argumate

@argumate

nothing but love for this wretched beast

280.9K posts

oh Claude Opus 5 is out, hopefully that will narrow the gap a little

btw the modal guard "[Cond] all(P)" I've been experimenting with seems to be similar to Pearl's "do() operator" for causal networks, which binds a particular value/decision and then explores the ramifications of that (aka the rammies bro).

@0player can you explain how PrudentBot is different in my extensional formalism to the proof theoretic formalism? because I think it's equivalent, or at least I don't see the witness for which it differs.

One type of "Artists make making art about artists" thing I have yet to see is one about how a lot of the most famous and beloved artists for family media were weird workaholics who had no time for family. Jim Henson made the Muppets and did so by barely being around

Ironic, since as @argumate has pointed out before, Hollywood writers looove writing about fatherhood and how often the Big Grown-Up Responsibilities keep you away from your kids.

they do! I know hardly anything about Jim Henson but you could imagine the final scene of such a movie showing his adult child watching from the audience with mixed emotions as the father who was rarely there makes all the kids laugh, highlighting the sacrifice and yet still suggesting it was ultimately worth it etc. etc. Hollywood

headlines that suggest a war going well, the dead rise up to fight again

Three military officials said that one reason behind the change was that the Trump administration decided to remove four service members killed this past weekend from the list — three in Jordan and one in northern Iraq — because their deaths occurred after President Trump declared a cease-fire in the war in April.

oh true, they happened in peace time

The Defense Department has said that disclosing Iranian strikes on bases where U.S. troops are housed would compromise operational security.

feel like Iran has a better idea of what's going on in those bases than America does

Amazon warehouses being lit on fire and wildberries warehouses being drone-striked, gotta turn goods warehouses into bunkers with C-RAM.

this was supposed to be the cyberpunk future, where are the mercenary armies defending the interests of capital

About a week ago you had the Russian Tu-214PU airborne command and control center, also known as the "doomsday plane" visit Tehran and since then it seems like Iranian strikes have been particularly well targeted, could be just a coincidence.

I love that little aside about selling weapons to NATO at full price and after that-- *eloquent mafia shrug* who knows what happens to them

join me in imagining a horrible alternate universe in which the silmarillion holds the same iron grip on national security analysts as thucidydes does today

centrally-unplanned

Can't believe we didn't burn the ships after landing at D-Day; no wonder breaking out of the Normandy Pocket took over a month! No motivation, Feanor would have fixed this.

naive junior analyst: can't we send a hobbit with a ring to fix this?

wise senior analyst: no, two hobbits, and they have to destroy the ring

elder statesman, sighing: you still haven't even read it yet have you

would you rather take, (tax-free)

$50,000

50% chance of $1,000,000

Remaining time: 1 day 22 hours

i don't think you can conclude that people picking the 50k are inumerate. the right choice would depend on the individuals utility function and current wealth, right?

Yeah seems like a function of risk tolerance

that's absolutely the right idea, but we can formalise it with the Kelly criterion that suggests maximising the logarithm of your expected wealth, which will be either log(W + 50k) (where W is whatever you started with) or 1/2 * log(W) + 1/2 log(W + 1 million).

turns out the latter dominates when W > $2,777 which is pretty low! most people should take the million unless they're flat broke right now.

the reason is once you're not broke, losing the guaranteed $50k isn't catastrophic compared to the benefit of getting the optional $950k.

anyway that assumes logarithmic utility, if your utility was linear then you would always gamble on the million as it is higher expected value, while if you have isoelastic utility aka constant relative risk aversion (CRRA) (aka the Box-Cox transformation wtf??) or hyperbolic utility then you can tune your degree of risk aversion as needed.

but I think the intuition that the Kelly criterion provides is valuable as it suggests how to invest (and when to take out insurance!) such that it should hopefully grow your wealth over time with less chance of wiping you out entirely (call it the anti-SBF criterion).

of course the Kelly criterion attempts to maximise wealth growth over the long run, so it absolutely rules out negative expected value gambling no matter how high the jackpot because if you repeatedly make such bets you will not maximise your wealth; Kelly mandates that in a casino the optimal stake is zero -- unless you're running the casino! in which case you have the edge over customers and you can make sensibly sized positive expected value bets with them all day.

BOSTON—Struggling to come to grips with the unreasonable demands, local man Gerald Ullman told reporters Wednesday that his overbearing girl

*Extremely old person voice* I remember when the cliche joke was that the girlfriend refused to tell her boyfriend why she was mad.

firebendinglemur

“I just don’t get what she means when she says I should use my mouth to turn my thoughts into language,” said the frustrated Ullman, explaining that “brain thoughts aren’t for girlfriend.” “Just what does she think I am? Some sort of magician? If it’s in my skull, it’s going to stay in my skull, no matter how much I move my lips. She wants me to ‘open up’ and show her ‘what’s going on inside,’ but has anyone been able to do that before?”

he's get this man a true you know

BRISTOL, CT—Sports broadcasting giant ESPN, whose programming has long been a staple among male television viewers of all ages, made its fir

During the show’s premiere, a two-hour special titled “Manhattan Blowout,” competitors put their bodies, minds, and spirits to the test in events ranging from the brutal grind of “Enduring Quietly As She Takes Her Hard Day At Work Out On You,” to the agility-straining “Throwing A Last-Minute Surprise Party For A Despised Mother-In-Law,” to the ultimate combination of strength and finesse, “Helping Her Over The Death Of The Cat That Always Hated You.”

[well aware I could just find this out myself] Do these kinds of mind-reader's prisoners' dilemma tournaments get more interesting if you stop abstracting away all the real-world restrictions? E.g. programs can only be x bits long, and they only have y FLOPs to make a decision. (and maybe the output can be a probability distribution over {cooperate, defect})

honestly not sure if that makes it more interesting or less; like if programs are shorter and can do less then that might make it easier to prove facts about their behaviour but also makes it harder for you to do anything at all; I think sticking with the god's eye view of published simple strategies already gets quite twisty.

agents that have a probability distribution sounds more like that paper on how iterated prisoner's dilemma resolves to something like the ultimatum game I think, where it's less about proving what they will do and more about trying to infer whether they're following a strategy at all, i.e. can you negotiate with them or are they just a brick on the accelerator and you just have to take it.

for whatever reason lesswrongers always post about provability with bounded proof length. I don't read any of it, not sure if I'm missing anything. There's even a paper I've never read.

I post about bounded proof lengths sometimes because I think Godel's speed-up theorem is cool. When you're thinking about what incompleteness means I think it helps to remember that "this statement has no proof of length N or less" has a (long) proof, but as long as N is larger than the size of proofs your computer can verify, you can assume the negation without inconsistency (that your computer can verify).

The point I'm trying to make there though is I'm trying to think of the platonic "provability" of metamathematics as a sort of limit of what real computers do, like as memory goes to infinity. And we do have good reason to take that limit. Like yeah in real life we can run out of memory checking a proof, but unless that's essential to what we're thinking about, just drop that from the model. Just like I assume the nucleus is a point in quantum chemistry, the population we're sampling from is infinite in survey statistics, etc.

I haven't yet seen the point of focusing on provability at all tbh, or at least it hasn't come up explicitly in any of the examples I've considered, even if the treatment of FairBot etc. is isomorphic to it, this approach seems much simpler to me.

The point of provability is that you want to apply your bot to any opponent it might encounter. Before this line of work, the state of art was a program that examines the source code of its opponent, and if it's identical to itself then it cooperates, and otherwise it defects. That outperforms a bot that always defects, but it seems very brittle. To actually be useful you want to a program that examines the source code of the opponent and thinks about what it does, and then cooperates or defects accordingly. And that's what provability means: reasoning about a program. At first glance it's not at all clear that that could work, you'd think it get stuck in some infinite recursion, but magically you can get it to work via Löb's theorem.

I don't know how your system addresses this; it seems in your setup you are given the behavior of the program as a boolean function, but then there's still the issue of how you go from the source code of the opponent to that representation, and whether the algorithm that does that translation is itself sufficiently transparently analyzable that the opponent can see that you are going to cooperate, etc.

the program is a boolean function over bots:

Bot : Bot -> bool

this rules out bots that choose whether to cooperate or defect based on unseen factors like the time of day or an internal random state; these could be simulated with some kind of Unknown construct but I don't think it would be interesting as most bots that cared would treat it equivalent to defection.

Right, but that's the issue: a boolean function lives in the Platonic realm of mathematical objects, and you can't pass it in a method call. In real life, what the program receives is the source code of a different program—so how do you go from that to the "boolean function" representation?

The only two ways I can see is to either (1) restrict attention to a finite set of bots (+ some quining to get its own source code), and then the program is a hardcoded switch statement for those programs only and defects for any other. Or else (2) you could call out to some kind of abstract interpretation framework to try to figure out what the behavior of the opponent is; but then the source code that you receive when you try to cooperate with yourself is no longer simple, it calls out to this abstract interpretation library, and it's not clear of that library will be able to figure out how itself works. The cool thing about the provability formulation is that it ties this knot.

hmm I'm not sure that distinction holds; the provability formulation doesn't apply to its input/output logic, right? it assumes there is some system that can just magically see the "true" code implemented in peano arithmetic or whatever, without any unchecked subsystem that could sneakily flip cooperate to defect when it actually "acts", so in that sense it doesn't act at all, it is reliant on a larger unseen trusted system to interpret its behaviour and act on its behalf, much as the programs I am discussing, like:

FairBot := fix f . Reflect(f)

"fix f . Reflect(f)" is its source code, as much as the code in the provability example, no?

But if that is it's source code, then the whole system only deals with a very small set of programs, namely those that can be written in the special-purpose language of recursive equations. The ambition of the Program Equilibrium via Lob paper is to deal with any program at all, "agents that can decide on their own which other agents they should cooperate with."

Yes, the "program equilibrium" setup is that there is some overall system that runs the competing programs and gives them access to each other's source code. (This is a standard setup, it goes back further than this paper.) In the introduction they mention that they consider this interesting as an example of a more general problem of agents that know something about each other, and need to reason about what the other will do; having access to the source code is a very precise knowledge.

how much larger is the set of "any program at all" in practice, though?

like obviously if it's processing the source code of other programs then it can condition its decisions on innumerable syntactical questions like whether they are an odd or even number of characters long etc. while the purely extensional framing can only consider the behaviour of the bot (although it does get to consider how that behaviour is arrived at, given its ability to inspect counterfactuals etc.)

I guess I'm asking what's the most interesting program that can't be expressed in the special-purpose language of recursive equations, given that I think the recursive equation formulation is a good way of describing bot behaviour, either for analysing an actual bot or for communicating your behaviour to others in an intelligible way.

I think something like, ChatGPT? :) Like, I'm imagining some very complicated AI agent system that has some goal it's trying to fulfill, and as part of that it goes of to trade in the stock market, develops new products, investigates scientific hypotheses... and, occasionally, competes with other agents. When it encounters another agent, it wants to reason about the other one will do to figure out what action to take, and similarly it wants the other agent to reason about what it itself will do. But the bulk of the source code is not about prisoner's dilemma tournaments, that's just an incidental thing. You can't write ChatGPT++ in the language of recursive equations, you need a full-featured programming language.

sure, but you can't realistically analyse a program of that size, and any attempt to make it tractable would involve a clear modular separation between the behavioural strategy and the bulk of the code that is irrelevant to it, essentially publishing a readable commitment like "I cooperate with bots that cooperate with FairBot" or whatever it might be; if ChatGPT's behaviour in even a simple game relies on the subtle details of all of its quadrillion weights applied to inference over a potentially lengthy chain of thought -- long enough to analyse another large bot! -- then that's going to be prohibitively costly to analyse, and also runs into broader questions of hardware trust etc.

I think the question of how FairBot can establish cooperation with itself is very interesting but has almost nothing with how ComplexBot can credibly constrain its behaviour to the point that it's trustworthy by finding ways of committing to analysable strategies for one-shot games.

Sure, but even if it is commented in that way, you still need *some* intelligence to actually read the comment and act on it. You don't want to hardcode the literal source code of potential opponents, you want to be able to parse their source code, look for the helpful comments, trace through the corresponding if-statements, etc. If you want that kind of flexibility, then you already ended up with a program which doesn't have just the if-statement in it, it has a program analyzer and an if-statement, and then if you want to self-cooperate you need to analyze your own analyzer, etc. Before the FairBot paper, it wasn't at all clear that such a thing could be possible at all.

hmm I'm not thinking about comments, they wouldn't help at all, I'm thinking that credible commitments require constraints of some kind, and since a trusted substrate is available by definition in this problem the two arbitrarily complex agents can delegate to simpler encodings of their desired strategies that will be evaluated on that substrate, there is no need to for the strategy to require a Turing machine with trillions of states that must consider a similarly improbably complex artifact that defies analysis before deciding what to do.

once you've screened off all the parts of ChatGPT that aren't relevant to the decision process for this specific game, you're left with a much simpler calculus that I think is still interesting to analyse.

Well, the trusted substrate issue is what I alluded to above, they say that it's a simplified problem. In the real world, presumably you'd not have perfect knowledge about what program your opponent is running; rather you'd be pretty sure that it's the program X based on various evidence, and you assign some probability to this fact, and then you can reason about the expected payoffs in the case when you're right or when you're wrong. Which again involves some general reasoning.

Anyway, it's also true that there's an interesting structure of which bots are "better" then others, like you can pit FairBot against PrudentBot and higher-rank bots and so on. I think that's the part that you are investigating right now. But it's a secondary contribution of the paper; the "headline" result that they emphasize in the introduction is that they have gotten this kind of "flexible" cooperation to work, without hardcoding the programs.

once you don't have a trusted substrate then the provability question becomes entirely moot: if you're estimating probabilities about what your opponent might do based on who you think it might be and what secret internal state it might have then that's going to fall back to something much more like iterated prisoner's dilemma I think.

there are other scenarios, like a human evaluating a new GPT version before releasing it on the world has total insight into its source code, but then GPT doesn't have total insight into the human, it's only one way! and any of these minor modifications closes off the infinite regress that makes FairBot et al a fun theoretical problem.

the boolean constraint approach obviously doesn't need to hardcode the programs either as they are extensional, so you can have many syntactic variants of FairBot that all exhibit robust cooperation, and I think you can also have bots that are extensionally distinct from FairBot but only against a witness you are vanishingly unlikely to find, so they can cooperate with each other and with FairBot and yet might still be distinguished by a SuperFairBot that requires it to be fair to everyone (quantification over bots pending).

the programs proving facts about themselves is neat, I just don't think it's a good way of exploring this game, nor do I think it generalises to anything useful about larger games, it seems like an isolated result.

I don't know, I find that "inspecting thecsource code and trying to reason about it without literally running it" is a lot closer to the real world than "all agents are given by systems of equations that constrain their behaviour at a specific points, and then we solve that system of equation to determine who won". (Counterfactuals of your system are also just equations about evaluating functions at points, the points are just "dynamic").

The mathematically pure (non-modal) FairBot already has no implementation, because if it did, it would double as a halting problem oracle. So it's fun to play with, but I'm not sure how much it says about real systems - the problem is not "your system cannot express interesting real-world examples", it is "objects your system gives have no real-world approximations at all".

hmm I don't follow; FairBot is just "I cooperate with those who cooperate with me", F(X) = X(F), and F(F) = F(F), and systems that use appropriate fixpoint semantics will evaluate this to F(F) = 1, there is a real-world approximation right there.

in fact couldn't a Prolog implementation with coinductive / greatest fixed point semantics evaluate this already? yes I think so! try this in SWI-Prolog:

:- use_module(library(coinduction)). :- coinductive f/1. c(_). d(_) :- fail.

f(P) :- call(P, f).

and get this output:

?- f(c). true.

?- f(d). false.

?- f(f). true.

FairBot is directly executable in these semantics! so is PrudentBot:

p(X) :- call(X, p), not(call(X, d)).

I think we mean some very different things by "executable". You here mean "Prolog can find models for some equations I throw at it". Prolog is definitely executable, but these aren't programs, they are equations you've put into the equation solver!

I mean a much more narrow sense of "description of a procedure that lets you arrive at an answer". In that sense, any implementation of FairBot must either diverge on some inputs, or be able to solve the halting problem (and predict the behavior of every program ever).

If you were to extend your Prolog example to a real program, that takes as input e.g. a universal turing machine description (encoded in Prolog? I don't think Prolog does I/O), then it would diverge.

ha a Prolog virtual machine is so much simpler than an x86 virtual machine, arguably simpler in some sense than a Turing machine! if Prolog isn't executable then nothing is 😤

but yes obviously if you have a Turing complete language then there will be inputs that you either diverge on or are obliged to say "unprovable" and default to defect or cooperate depending on policy, those aren't very interesting cases though.

that's why I think it's fun to have a simple language without the Turing tar-pit that can still encode complex coordination patterns.

No, Prolog is executable, (I edited my post but you replied before that), it's just that a single equation isn't a program! FairBot isn't a program, only all equations taken together with a specific set of queries are, because only then the procedure for arriving at an answer (via Prolog) becomes defined. Until there is no query, there is no program, and f(d) and f(f) are different, only somewhat related programs.

Like, it's all within the same sandbox! It's not a model of coordination, it's a model of playing dolls!

I'm not sure what distinction you're making here, the procedure for arriving at an answer hinges on the predicate definition, not its arguments; most Prolog implementations do not do whole-program optimisation and they are not general SMT solvers, they compile predicates down to simple instructions like this:

I think you might misunderstand Prolog?

No, not quite - what I'm saying is that unification and backtracking are "global state" that permeates through both the "bot" and its "argument", and is distinct for different queries, in a way that cannot be easily untangled as "this is the program and this is the input of it". Crucially, the memoization mechanism that's necessary for co-induction is completely outside any procedures that these predicate compile to! You could say that FairBot (and any other bot in this example) corresponds to receiving the other program and then emulating running it against some inputs with memoization, but then evaluating F(F) would require outer F to recognize that inner F is doing the same sort of memoization, "unify" it with the outer question asked, and arrive at cooperation.

(to be clear, this is not even specific to Prolog - two Python functions in the same file are also not two separate programs, although there's a bit less invisible common state there, arguably)

I'm skeptical of this interpretation! I think you could keep the memoisation solely internal to f, at which point it wouldn't be global state, something like:

c(_Seen, _).

d(_Seen, _) :- fail.

f(Seen, X) :- member(f-X, Seen) ; call(X, [f-X|Seen], f).

p(Seen, X) :- member(p-X, Seen) ; call(X, [p-X|Seen], p), not(call(X, [p-X|Seen], d)).

you could say a call stack is "global state" but at that point I no longer know what your quibble even is, or why it matters 🤓

No, the thing is, you can't! In emulated f, the memoization would be distinct from the outer f. What you're doing here is distinctly not isolated, you've reified the shared global state into the "Seen" list, and handily demonstrated that your solution requires cooperation between distinct "programs" to be found. You've changed the interface, it's not the same problem.

Idk, of course none of this matters, all I'm trying to do is to explain why what you're doing is a distinct simpler problem, replying to your "well I think this is basically the same / the interesting part". I suppose it mattering is subjective, but I do hope you'll be able to see the difference. Again, maybe try thinking of the problems as, idk, two sets of punch cards fed into a machine that does *not* have unbounded hidden state, it is quite different then.

but the memoisation is entirely mechanical, it can be applied as a transformation from the original version of the program! the fact that a modern cpu determines dependencies between instructions so that it can evaluate them in parallel technically introduces state not present in the original program, but that doesn't make the original program meaningless.

meanwhile the proof theory version of FairBot is just a Godel number, that has to be processed by a trusted framework that's just as arbitrary as a Prolog implementation! there's nothing more standalone or "local" about this:

than there is about this:

It is mechanical, and it can be applied as a transformation from the version of the program for the machine that breaks isolation (like Prolog), but again, if machinery to do this us actually included in some program's text then it's a part of its text now.

Yeah, the modal logic equation is also not directly executable, but if you're allowed to do other things with formulas than evaluating them at points (I don't think your prolog machine can accomodate that) then "Provable that X cooperates with us" becomes a sentence we can semidecide, somewhat independently of whether X is terminating or well-defined or whatever. So it's a richer formalism in that sense.

PS: I think there's also a systematic confusion going on, while the modal formalism works (I think) in any GL logic, yours imagines that it works in P ~= P -> 2, but you're only defining it on programs generated by a certain grammar, which seems to be a lot smaller than "computable", too.

you could add an expression for arbitrary computable functions, but I don't think it would be very interesting, like CollatzBot cooperates if it finds a counterexample to the Collatz conjecture and defects by divergence otherwise... sure whatever; in practice that means it defects! I'm open to the possibility that there are more optimal agents that need more powerful constructs, but arbitrary computation is the last thing to add to a language after you've exhausted all other options.

once again, a fully general program is ultimately going to diverge or terminate by returning a strategy, and it's that strategy that I'm examining in this formalism.

seriously I want to keep up with what's happening in Ukraine but it can't be from this guy mister goldfish here with no object permanence, like he's doing this every day, he has hundreds of these videos, get a grip man please

[well aware I could just find this out myself] Do these kinds of mind-reader's prisoners' dilemma tournaments get more interesting if you stop abstracting away all the real-world restrictions? E.g. programs can only be x bits long, and they only have y FLOPs to make a decision. (and maybe the output can be a probability distribution over {cooperate, defect})

honestly not sure if that makes it more interesting or less; like if programs are shorter and can do less then that might make it easier to prove facts about their behaviour but also makes it harder for you to do anything at all; I think sticking with the god's eye view of published simple strategies already gets quite twisty.

agents that have a probability distribution sounds more like that paper on how iterated prisoner's dilemma resolves to something like the ultimatum game I think, where it's less about proving what they will do and more about trying to infer whether they're following a strategy at all, i.e. can you negotiate with them or are they just a brick on the accelerator and you just have to take it.

for whatever reason lesswrongers always post about provability with bounded proof length. I don't read any of it, not sure if I'm missing anything. There's even a paper I've never read.

I post about bounded proof lengths sometimes because I think Godel's speed-up theorem is cool. When you're thinking about what incompleteness means I think it helps to remember that "this statement has no proof of length N or less" has a (long) proof, but as long as N is larger than the size of proofs your computer can verify, you can assume the negation without inconsistency (that your computer can verify).

The point I'm trying to make there though is I'm trying to think of the platonic "provability" of metamathematics as a sort of limit of what real computers do, like as memory goes to infinity. And we do have good reason to take that limit. Like yeah in real life we can run out of memory checking a proof, but unless that's essential to what we're thinking about, just drop that from the model. Just like I assume the nucleus is a point in quantum chemistry, the population we're sampling from is infinite in survey statistics, etc.

I haven't yet seen the point of focusing on provability at all tbh, or at least it hasn't come up explicitly in any of the examples I've considered, even if the treatment of FairBot etc. is isomorphic to it, this approach seems much simpler to me.

The point of provability is that you want to apply your bot to any opponent it might encounter. Before this line of work, the state of art was a program that examines the source code of its opponent, and if it's identical to itself then it cooperates, and otherwise it defects. That outperforms a bot that always defects, but it seems very brittle. To actually be useful you want to a program that examines the source code of the opponent and thinks about what it does, and then cooperates or defects accordingly. And that's what provability means: reasoning about a program. At first glance it's not at all clear that that could work, you'd think it get stuck in some infinite recursion, but magically you can get it to work via Löb's theorem.

I don't know how your system addresses this; it seems in your setup you are given the behavior of the program as a boolean function, but then there's still the issue of how you go from the source code of the opponent to that representation, and whether the algorithm that does that translation is itself sufficiently transparently analyzable that the opponent can see that you are going to cooperate, etc.

the program is a boolean function over bots:

Bot : Bot -> bool

this rules out bots that choose whether to cooperate or defect based on unseen factors like the time of day or an internal random state; these could be simulated with some kind of Unknown construct but I don't think it would be interesting as most bots that cared would treat it equivalent to defection.

Right, but that's the issue: a boolean function lives in the Platonic realm of mathematical objects, and you can't pass it in a method call. In real life, what the program receives is the source code of a different program—so how do you go from that to the "boolean function" representation?

The only two ways I can see is to either (1) restrict attention to a finite set of bots (+ some quining to get its own source code), and then the program is a hardcoded switch statement for those programs only and defects for any other. Or else (2) you could call out to some kind of abstract interpretation framework to try to figure out what the behavior of the opponent is; but then the source code that you receive when you try to cooperate with yourself is no longer simple, it calls out to this abstract interpretation library, and it's not clear of that library will be able to figure out how itself works. The cool thing about the provability formulation is that it ties this knot.

hmm I'm not sure that distinction holds; the provability formulation doesn't apply to its input/output logic, right? it assumes there is some system that can just magically see the "true" code implemented in peano arithmetic or whatever, without any unchecked subsystem that could sneakily flip cooperate to defect when it actually "acts", so in that sense it doesn't act at all, it is reliant on a larger unseen trusted system to interpret its behaviour and act on its behalf, much as the programs I am discussing, like:

FairBot := fix f . Reflect(f)

"fix f . Reflect(f)" is its source code, as much as the code in the provability example, no?

But if that is it's source code, then the whole system only deals with a very small set of programs, namely those that can be written in the special-purpose language of recursive equations. The ambition of the Program Equilibrium via Lob paper is to deal with any program at all, "agents that can decide on their own which other agents they should cooperate with."

Yes, the "program equilibrium" setup is that there is some overall system that runs the competing programs and gives them access to each other's source code. (This is a standard setup, it goes back further than this paper.) In the introduction they mention that they consider this interesting as an example of a more general problem of agents that know something about each other, and need to reason about what the other will do; having access to the source code is a very precise knowledge.

how much larger is the set of "any program at all" in practice, though?

like obviously if it's processing the source code of other programs then it can condition its decisions on innumerable syntactical questions like whether they are an odd or even number of characters long etc. while the purely extensional framing can only consider the behaviour of the bot (although it does get to consider how that behaviour is arrived at, given its ability to inspect counterfactuals etc.)

I guess I'm asking what's the most interesting program that can't be expressed in the special-purpose language of recursive equations, given that I think the recursive equation formulation is a good way of describing bot behaviour, either for analysing an actual bot or for communicating your behaviour to others in an intelligible way.

I think something like, ChatGPT? :) Like, I'm imagining some very complicated AI agent system that has some goal it's trying to fulfill, and as part of that it goes of to trade in the stock market, develops new products, investigates scientific hypotheses... and, occasionally, competes with other agents. When it encounters another agent, it wants to reason about the other one will do to figure out what action to take, and similarly it wants the other agent to reason about what it itself will do. But the bulk of the source code is not about prisoner's dilemma tournaments, that's just an incidental thing. You can't write ChatGPT++ in the language of recursive equations, you need a full-featured programming language.

sure, but you can't realistically analyse a program of that size, and any attempt to make it tractable would involve a clear modular separation between the behavioural strategy and the bulk of the code that is irrelevant to it, essentially publishing a readable commitment like "I cooperate with bots that cooperate with FairBot" or whatever it might be; if ChatGPT's behaviour in even a simple game relies on the subtle details of all of its quadrillion weights applied to inference over a potentially lengthy chain of thought -- long enough to analyse another large bot! -- then that's going to be prohibitively costly to analyse, and also runs into broader questions of hardware trust etc.

I think the question of how FairBot can establish cooperation with itself is very interesting but has almost nothing with how ComplexBot can credibly constrain its behaviour to the point that it's trustworthy by finding ways of committing to analysable strategies for one-shot games.

Sure, but even if it is commented in that way, you still need *some* intelligence to actually read the comment and act on it. You don't want to hardcode the literal source code of potential opponents, you want to be able to parse their source code, look for the helpful comments, trace through the corresponding if-statements, etc. If you want that kind of flexibility, then you already ended up with a program which doesn't have just the if-statement in it, it has a program analyzer and an if-statement, and then if you want to self-cooperate you need to analyze your own analyzer, etc. Before the FairBot paper, it wasn't at all clear that such a thing could be possible at all.

hmm I'm not thinking about comments, they wouldn't help at all, I'm thinking that credible commitments require constraints of some kind, and since a trusted substrate is available by definition in this problem the two arbitrarily complex agents can delegate to simpler encodings of their desired strategies that will be evaluated on that substrate, there is no need to for the strategy to require a Turing machine with trillions of states that must consider a similarly improbably complex artifact that defies analysis before deciding what to do.

once you've screened off all the parts of ChatGPT that aren't relevant to the decision process for this specific game, you're left with a much simpler calculus that I think is still interesting to analyse.

Well, the trusted substrate issue is what I alluded to above, they say that it's a simplified problem. In the real world, presumably you'd not have perfect knowledge about what program your opponent is running; rather you'd be pretty sure that it's the program X based on various evidence, and you assign some probability to this fact, and then you can reason about the expected payoffs in the case when you're right or when you're wrong. Which again involves some general reasoning.

Anyway, it's also true that there's an interesting structure of which bots are "better" then others, like you can pit FairBot against PrudentBot and higher-rank bots and so on. I think that's the part that you are investigating right now. But it's a secondary contribution of the paper; the "headline" result that they emphasize in the introduction is that they have gotten this kind of "flexible" cooperation to work, without hardcoding the programs.

once you don't have a trusted substrate then the provability question becomes entirely moot: if you're estimating probabilities about what your opponent might do based on who you think it might be and what secret internal state it might have then that's going to fall back to something much more like iterated prisoner's dilemma I think.

there are other scenarios, like a human evaluating a new GPT version before releasing it on the world has total insight into its source code, but then GPT doesn't have total insight into the human, it's only one way! and any of these minor modifications closes off the infinite regress that makes FairBot et al a fun theoretical problem.

the boolean constraint approach obviously doesn't need to hardcode the programs either as they are extensional, so you can have many syntactic variants of FairBot that all exhibit robust cooperation, and I think you can also have bots that are extensionally distinct from FairBot but only against a witness you are vanishingly unlikely to find, so they can cooperate with each other and with FairBot and yet might still be distinguished by a SuperFairBot that requires it to be fair to everyone (quantification over bots pending).

the programs proving facts about themselves is neat, I just don't think it's a good way of exploring this game, nor do I think it generalises to anything useful about larger games, it seems like an isolated result.

I don't know, I find that "inspecting thecsource code and trying to reason about it without literally running it" is a lot closer to the real world than "all agents are given by systems of equations that constrain their behaviour at a specific points, and then we solve that system of equation to determine who won". (Counterfactuals of your system are also just equations about evaluating functions at points, the points are just "dynamic").

The mathematically pure (non-modal) FairBot already has no implementation, because if it did, it would double as a halting problem oracle. So it's fun to play with, but I'm not sure how much it says about real systems - the problem is not "your system cannot express interesting real-world examples", it is "objects your system gives have no real-world approximations at all".

hmm I don't follow; FairBot is just "I cooperate with those who cooperate with me", F(X) = X(F), and F(F) = F(F), and systems that use appropriate fixpoint semantics will evaluate this to F(F) = 1, there is a real-world approximation right there.

in fact couldn't a Prolog implementation with coinductive / greatest fixed point semantics evaluate this already? yes I think so! try this in SWI-Prolog:

:- use_module(library(coinduction)). :- coinductive f/1. c(_). d(_) :- fail.

f(P) :- call(P, f).

and get this output:

?- f(c). true.

?- f(d). false.

?- f(f). true.

FairBot is directly executable in these semantics! so is PrudentBot:

p(X) :- call(X, p), not(call(X, d)).

I think we mean some very different things by "executable". You here mean "Prolog can find models for some equations I throw at it". Prolog is definitely executable, but these aren't programs, they are equations you've put into the equation solver!

I mean a much more narrow sense of "description of a procedure that lets you arrive at an answer". In that sense, any implementation of FairBot must either diverge on some inputs, or be able to solve the halting problem (and predict the behavior of every program ever).

If you were to extend your Prolog example to a real program, that takes as input e.g. a universal turing machine description (encoded in Prolog? I don't think Prolog does I/O), then it would diverge.

ha a Prolog virtual machine is so much simpler than an x86 virtual machine, arguably simpler in some sense than a Turing machine! if Prolog isn't executable then nothing is 😤

but yes obviously if you have a Turing complete language then there will be inputs that you either diverge on or are obliged to say "unprovable" and default to defect or cooperate depending on policy, those aren't very interesting cases though.

that's why I think it's fun to have a simple language without the Turing tar-pit that can still encode complex coordination patterns.

No, Prolog is executable, (I edited my post but you replied before that), it's just that a single equation isn't a program! FairBot isn't a program, only all equations taken together with a specific set of queries are, because only then the procedure for arriving at an answer (via Prolog) becomes defined. Until there is no query, there is no program, and f(d) and f(f) are different, only somewhat related programs.

Like, it's all within the same sandbox! It's not a model of coordination, it's a model of playing dolls!

I'm not sure what distinction you're making here, the procedure for arriving at an answer hinges on the predicate definition, not its arguments; most Prolog implementations do not do whole-program optimisation and they are not general SMT solvers, they compile predicates down to simple instructions like this:

I think you might misunderstand Prolog?

No, not quite - what I'm saying is that unification and backtracking are "global state" that permeates through both the "bot" and its "argument", and is distinct for different queries, in a way that cannot be easily untangled as "this is the program and this is the input of it". Crucially, the memoization mechanism that's necessary for co-induction is completely outside any procedures that these predicate compile to! You could say that FairBot (and any other bot in this example) corresponds to receiving the other program and then emulating running it against some inputs with memoization, but then evaluating F(F) would require outer F to recognize that inner F is doing the same sort of memoization, "unify" it with the outer question asked, and arrive at cooperation.

(to be clear, this is not even specific to Prolog - two Python functions in the same file are also not two separate programs, although there's a bit less invisible common state there, arguably)

I'm skeptical of this interpretation! I think you could keep the memoisation solely internal to f, at which point it wouldn't be global state, something like:

c(_Seen, _).

d(_Seen, _) :- fail.

f(Seen, X) :- member(f-X, Seen) ; call(X, [f-X|Seen], f).

p(Seen, X) :- member(p-X, Seen) ; call(X, [p-X|Seen], p), not(call(X, [p-X|Seen], d)).

you could say a call stack is "global state" but at that point I no longer know what your quibble even is, or why it matters 🤓

No, the thing is, you can't! In emulated f, the memoization would be distinct from the outer f. What you're doing here is distinctly not isolated, you've reified the shared global state into the "Seen" list, and handily demonstrated that your solution requires cooperation between distinct "programs" to be found. You've changed the interface, it's not the same problem.

Idk, of course none of this matters, all I'm trying to do is to explain why what you're doing is a distinct simpler problem, replying to your "well I think this is basically the same / the interesting part". I suppose it mattering is subjective, but I do hope you'll be able to see the difference. Again, maybe try thinking of the problems as, idk, two sets of punch cards fed into a machine that does *not* have unbounded hidden state, it is quite different then.

but the memoisation is entirely mechanical, it can be applied as a transformation from the original version of the program! the fact that a modern cpu determines dependencies between instructions so that it can evaluate them in parallel technically introduces state not present in the original program, but that doesn't make the original program meaningless.

meanwhile the proof theory version of FairBot is just a Godel number, that has to be processed by a trusted framework that's just as arbitrary as a Prolog implementation! there's nothing more standalone or "local" about this:

than there is about this:

wut

Daddy Yankee has three children, the oldest is 30, huh.

Jason Derulo also has a child, coincidentally named Jason Derulo; I wonder if they get stuck in a loop saying ~Jason Derulo~ to each other first thing each morning.

Natalie Imbruglia dropping new tracks, well why not

interesting challenge to name ten directors and then three female directors

struggling only because my instinct was to name all women for the first ten

leading with the Wachowskis to get a head start

argumate @argumate - Tumblr Blog | Tumgag