# Simulators  ## Metadata - Author: [[janus]] - Full Title: Simulators - Category: #articles - Summary: insert summary - My notes: - Document Tags: [[ai]] [[LLMs]] - Summary: GPT functions as a simulator that can mimic both agentic and non-agentic behaviors, drawing from a diverse range of training data. It doesn't behave like a traditional agent, as it doesn't have inherent goals but generates text based on patterns it has learned. This unique nature of GPT allows for new ways of thinking about AI capabilities and alignment. ## LLM Chats ## NotebookLM ## LLM Audio ## Highlights > Extrapolating the idea of “behavior cloning”, we might imagine GPT-N approaching a perfect mimic which serves up digital clones of the people and things captured in its training data. But that only tells a very small part of the story. GPT *is* behavior cloning. But it is the behavior of a universe that is cloned, not of a single demonstrator, and the result isn’t a static copy of the universe, but a *compression of the universe into a generative rule*. This resulting policy is capable of animating anything that evolves according to that rule: a far larger set than the sampled trajectories included in the training data, just as there are many more possible configurations that evolve according to our laws of physics than instantiated in our particular time and place and Everett branch. ([View Highlight](https://read.readwise.io/read/01jz2z82e2bkc5k77071reh3nv)) - Tags: [[ai]] [[ai aesthetics paper]] [[generating philosophy paper]] - Note: this is a really useful way of thinking about his theory > Extrapolating the idea of “behavior cloning”, we might imagine GPT-N approaching a perfect mimic which serves up digital clones of the people and things captured in its training data. But that only tells a very small part of the story. GPT *is* behavior cloning. But it is the behavior of a universe that is cloned, not of a single demonstrator, and the result isn’t a static copy of the universe, but a *compression of the universe into a generative rule*. This resulting policy is capable of animating anything that evolves according to that rule: a far larger set than the sampled trajectories included in the training data, just as there are many more possible configurations that evolve according to our laws of physics than instantiated in our particular time and place and Everett branch. ([View Highlight](https://read.readwise.io/read/01jz4t3qr6j0sa3wznpsty7s35)) - Tags: [[generating philosophy paper]] [[ai]] - Note: how janus sees llms. What LLMs in fact are > The idea of parameterized behavior cloning grows more unwieldy if we remember that GPT’s prompt continually changes during autoregressive generation. If GPT is a parameterized agent, then parameterization is not a fixed flag that chooses a process out of a set of possible processes. The parameterization *is* what is evolved – a successor “agent” selected by the old “agent” at each timestep, and neither of them need to have precedence in the training data. ([View Highlight](https://read.readwise.io/read/01jz7hjwz85v5t3p6mb1768jsv)) > Something which can predict everything all the time is more formidable than any demonstrator it predicts: the upper bound of what can be learned from a dataset is not the most capable trajectory, but the conditional structure of the universe implicated by their sum (though it may not be trivial to [extract that knowledge](https://www.lesswrong.com/tag/eliciting-latent-knowledge-elk)). ([View Highlight](https://read.readwise.io/read/01jz7hmnw2942d1d524hnffr5s)) > You can probably predict my proposed answer. The natural thing to do with a predictor that inputs a sequence and outputs a probability distribution over the next token is to sample a token from those likelihoods, then add it to the sequence and recurse, indefinitely yielding a *simulated* future. Predictive sequence models in the generative modality are **simulators** of a learned distribution. ([View Highlight](https://read.readwise.io/read/01jypfny6ha4n60hkerwt760r5)) - Tags: [[ai]] - Note: the simulation account of LLMs > I’ve ended several of the above sections with questions pointing to desiderata of a category that might satisfactorily classify GPT. > > What is the word for something that roleplays minus the implication that some*one* is behind the mask? > > What category, unlike the category of oracles, would make the importance of *process* specification obvious? > > What category would do justice to GPT’s ability to not only reproduce the behavior of its demonstrators but to *produce* the behavior of an inexhaustible number of counterfactual configurations? ([View Highlight](https://read.readwise.io/read/01jz0a3fyzw1hgx6akj5052je0)) > A fun way to test whether a name you’ve come up with is effective at evoking its intended signification is to see if GPT, a model of how humans are conditioned by words, infers its correct definition in context. ([View Highlight](https://read.readwise.io/read/01jz0a6r0fsqcjx8gkzy3vtcgm)) > If I wanted to be precise about what I mean by a simulator, I might say there are two aspects which delimit the category. GPT’s completion focuses on the teleological aspect, but in its talk of “generating” it also implies the structural aspect, which has to do with the notion of time evolution. The first sentence of the [Wikipedia article on “simulation”](https://en.wikipedia.org/wiki/Simulation) explicitly states both: > > A **simulation** is the imitation of the operation of a real-world process or system over time. ([View Highlight](https://read.readwise.io/read/01jypftcan97rcnkqps4pnvcdq)) > Tools: A tool is optimized to perform a specific task. A tool will not act without a command and will not optimize for any objective other than its specific task. Google Maps is an example of a tool. ([View Highlight](https://read.readwise.io/read/01jz0ab97n9p9hwymwhx4m3wzf)) - Tags: [[LLMs]] - Note: an llm was asked to give a definition of what a simulator was.. > Simulators: `A simulator is optimized to generate realistic models of a system. The simulator will not optimize for any objective other than realism,` although in the course of `doing so, it might generate instances of agents, oracles, and so on.` ([View Highlight](https://read.readwise.io/read/01jz0ab5xtp5r2hcn9bm68s4kd)) > To know the conditional structure of the universe[[27]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators/#fni3y95l8d8bo) is to know its laws of physics, which describe what is expected to happen under what conditions. The laws of physics are always fixed, but produce different distributions of outcomes when applied to different conditions. Given a sampling of trajectories – examples of situations and the outcomes that actually followed – we can try to infer a common law that generated them all. In expectation, the laws of physics are always implicated by trajectories, which (by definition) fairly sample the conditional distribution given by physics. Whatever humans know of the laws of physics governing the evolution of our world has been inferred from sampled trajectories. ([View Highlight](https://read.readwise.io/read/01k0h2z1te2z1d1emhqb7466rg)) > • The word “simulator” evokes a model of real processes which can be used to run virtual processes in virtual reality. > • It suggests an ontological distinction between the simulator and things that are simulated, and avoids the fallacy of attributing contingent properties of the latter to the former. > • It’s not confusing that multiple simulacra can be instantiated at once, or an agent embedded in a tragedy, etc. > • It does not imply that the AI’s behavior is well-described (globally or locally) as expected utility maximization. An arbitrarily powerful/accurate simulation can depict arbitrarily hapless sims. > • It does not imply that the AI is only capable of emulating things with direct precedent in the training data. A physics simulation, for instance, can simulate any phenomena that plays by its rules. > • It emphasizes the role of the model as a transition rule that evolves processes *over time*. The power of factored cognition / chain-of-thought reasoning is obvious. > • It emphasizes the role of the state in specifying and constructing the agent/process. The importance of prompt programming for capabilities is obvious if you think of the prompt as specifying a configuration that will be propagated forward in time. > • It emphasizes the interactive nature of the model’s predictions – even though they’re “just text”, you can converse with simulacra, explore virtual environments, etc. > • It’s clear that in order to actually *do* anything (intelligent, useful, dangerous, etc), the model must act through simulation *of something*. ([View Highlight](https://read.readwise.io/read/01jypg3y4yts0s1w9bcqrvr8kx)) > A simulator trained with machine learning is optimized to accurately model its training distribution – in contrast to, for instance, maximizing the output of a reward function or accomplishing objectives in an environment. ([View Highlight](https://read.readwise.io/read/01jypghk6bewj6r3kx4k4b9sbs)) > A strict version of the simulation objective, which excludes GANs, applies only to models whose output distribution is incentivized using a proper scoring rule[[24]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fnco7whsfoh2e) to minimize single-step predictive error. This means the model is directly incentivized to match its predictions to the probabilistic transition rule which implicitly governs the training distribution. As a model is made increasingly optimal with respect to this objective, the rollouts that it generates become increasingly statistically indistinguishable from training samples, because they come closer to being described by the same underlying law: closer to a perfect simulation. ([View Highlight](https://read.readwise.io/read/01jypgj9b5f82n7nwahs17cmzc)) > To know the conditional structure of the universe[[27]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fni3y95l8d8bo) is to know its laws of physics, which describe what is expected to happen under what conditions. The laws of physics are always fixed, but produce different distributions of outcomes when applied to different conditions. Given a sampling of trajectories – examples of situations and the outcomes that actually followed – we can try to infer a common law that generated them all. In expectation, the laws of physics are always implicated by trajectories, which (by definition) fairly sample the conditional distribution given by physics. Whatever humans know of the laws of physics governing the evolution of our world has been inferred from sampled trajectories. ([View Highlight](https://read.readwise.io/read/01jz7jabrvcs1j3hgbrcdd93f0)) > If we had access to an unlimited number of trajectories starting from every possible condition, we could converge to the true laws by simply counting the frequencies of outcomes for every initial state (an [n-gram](https://en.wikipedia.org/wiki/N-gram) with a sufficiently large n). In some sense, physics contains the same information as an infinite number of trajectories, but it’s possible to represent physics in a more compressed form than a huge lookup table of frequencies if there are regularities in the trajectories. ([View Highlight](https://read.readwise.io/read/01jz7jc15wqq6yyrk7xv5zggsx)) > **Models trained with the strict simulation objective are directly incentivized to reverse-engineer the (semantic) physics of the training distribution, and consequently, to propagate simulations whose dynamical evolution is indistinguishable from that of training samples.** I propose this as a description of the archetype targeted by self-supervised predictive learning, again in contrast to RL’s archetype of an agent optimized to maximize free parameters (such as action-trajectories) relative to a reward function. ([View Highlight](https://read.readwise.io/read/01jyr3bmsx0q3xr7b7fqepxcvd)) > But when grading tests in the real world, we do not say “the laws of physics got this problem wrong” and conclude that the laws of physics haven’t sufficiently mastered the course material. If someone argued this is a reasonable view since the test-taker was steered by none other than the laws of physics, we could point to a different test where the problem was answered correctly by the same laws of physics propagating a different configuration. The “knowledge of course material” implied by test performance is a property of *configurations*, not physics. ([View Highlight](https://read.readwise.io/read/01jz68nz8prg8q2k984zf9kmzk)) > The verdict that knowledge is purely a property of configurations cannot be naively generalized from real life to GPT simulations, because “physics” and “configurations” play different roles in the two (as I’ll address in the next post). The parable of the two tests, however, literally pertains to GPT. People have a tendency to draw [erroneous global conclusions](https://en.wikipedia.org/wiki/Fallacy_of_composition) about GPT from behaviors which are in fact prompt-contingent, and consequently there is a pattern of constant discoveries that GPT-3 exceeds previously measured capabilities given alternate conditions of generation[[29]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fnomelvf6lrng), which shows no signs of slowing 2 years after GPT-3’s release. ([View Highlight](https://read.readwise.io/read/01jz68pkpbyarnnx5qckz7peyn)) > Making the ontological distinction between GPT and instances of text which are propagated by it makes these discoveries unsurprising: obviously, different configurations will be differently capable and in general behave differently when animated by the laws of GPT physics. ([View Highlight](https://read.readwise.io/read/01jz68pynaax7ycs8khd49asqx)) > In the simulation ontology, I say that GPT and its output-instances correspond respectively to the **simulator** and **simulacra**. **GPT** is to a **piece of text output by GPT** as **quantum physics** is to a **person taking a test**, or as [**transition rules of Conway’s Game of Life**](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life#Rules) are to [**glider**](https://conwaylife.com/wiki/Glider). The simulator is a time-invariant law which unconditionally governs the evolution of all simulacra. ([View Highlight](https://read.readwise.io/read/01jyp6vkdppxwzdxah89dkdev0)) > I think that implicit type-confusion is common in discourse about GPT. “GPT”, the neural network, the policy that was optimized, is the easier object to point to and say definite things about. But when we talk about “GPT’s” capabilities, impacts, or alignment, we’re usually actually concerned about the behaviors of an algorithm which calls GPT in an autoregressive loop repeatedly writing to some prompt-state – that is, we’re concerned with simulacra. What we call GPT’s “downstream behavior” is the behavior of simulacra; it is primarily through simulacra that GPT has potential to perform meaningful work (for good or for ill). ([View Highlight](https://read.readwise.io/read/01jz68s09znj1k24xpyg22pm5s)) - Tags: [[LLMs]] [[ai aesthetics paper]] [[generating philosophy paper]] - Note: two levels in llms > GPT updated me on how simulation can be implemented with prosaic machine learning: > • **Self-supervised ML can create “behavioral” simulations of impressive semantic fidelity.** Whole brain emulation is not necessary to construct convincing and useful virtual humans; it is conceivable that observations of human behavioral traces (e.g. text) are sufficient to reconstruct functionally human-level virtual intelligence. > • **Learned simulations can be partially observed and lazily-rendered, and still work.** A couple of pages of text severely underdetermines the real-world process that generated text, so GPT simulations are likewise underdetermined. A “partially observed” simulation is more efficient to compute because the state can be much smaller, but can still have the effect of high fidelity as details can be rendered as needed. The tradeoff is that it requires the simulator to model semantics – human imagination does this, for instance – which turns out not to be an issue for big models. > • **Learned simulation generalizes impressively.** As I described in the section on [behavior cloning](http://localhost:1313/drafts/simulators-post/#heading=h.ugu71trvtemp), training a model to predict diverse trajectories seems to make it internalize general laws underlying the distribution, allowing it to simulate counterfactuals that can be constructed from the distributional semantics. > In my model, these updates dramatically alter the landscape of potential futures, and thus motivate [exploratory engineering](https://intelligence.org/files/ExploratoryEngineeringAI.pdf) of the class of learned simulators for which GPT-3 is a lower bound. That is the intention of this sequence. ([View Highlight](https://read.readwise.io/read/01jz6ajy6rs6bn8emrp1f4c5f2)) - Note: The power of simulation. > Everything can be trivially modeled as a utility maximizer, but for these reasons, a utility function is not a good explanation or compression of GPT’s training data, and its optimal predictor is not well-described as a utility maximizer. However, just because information isn’t compressed well by a utility function doesn’t mean it can’t be compressed another way. The [Mandelbrot set](https://en.wikipedia.org/wiki/Mandelbrot_set) is a complicated pattern compressed by a very simple generative algorithm which makes no reference to future consequences and doesn’t involve argmaxxing anything (except vacuously [being the way it is](https://www.lesswrong.com/posts/d2n74bwham8motxyX/optimization-at-a-distance#An_Agent_Optimizing_Its_Own_Actions)). Likewise the set of all possible rollouts of [Conway’s Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) – [some automata may be well-described as agents](https://www.lesswrong.com/posts/3SG4WbNPoP8fsuZgs/agency-in-conway-s-game-of-life), but they are a minority of possible patterns, and not all agentic automata will share a goal. Imagine trying to model Game of Life as an expected utility maximizer! ([View Highlight](https://read.readwise.io/read/01jynz6qk4sc5b5veggt5b5r88)) > Even though neither GPT’s behavior nor its training story fit with the traditional agent framing, there are still compatibilist views that characterize it as some kind of agent. For example, Gwern has said[[13]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fn4qv9vfo4ps7) that anyone who uses GPT for long enough begins to think of it as an agent who only cares about roleplaying a lot of roles. > That framing seems unnatural to me, comparable to thinking of physics as an agent who only cares about evolving the universe accurately according to the laws of physics. At best, the agent is an epicycle; but it is also compatible with interpretations that generate dubious predictions. > Say you’re told that an agent *values predicting text correctly*. Shouldn’t you expect that: ([View Highlight](https://read.readwise.io/read/01jynzgmzpmkpqdrhjhkvd376g)) > At first glance, GPT might resemble a generic “oracle AI”, because it is trained to make accurate predictions. But its log loss objective is myopic and only concerned with immediate, micro-scale correct prediction of the next token, not answering particular, global queries such as “what’s the best way to fix the climate in the next five years?” In fact, it is not specifically optimized to give *true* answers, which a classical oracle should strive for, but rather to minimize the divergence between predictions and training examples, independent of truth. Moreover, it isn’t specifically trained to give answers in the first place! It may give answers if the prompt asks questions, but it may also simply elaborate on the prompt without answering any question, or tell the rest of a story implied in the prompt. What it does is more like animation than divination, executing the dynamical laws of its rendering engine to recreate the flows of history found in its training data (and a large superset of them as well), mutatis mutandis. Given the same laws of physics, one can build a multitude of different backgrounds and props to create different storystages, including ones that don’t exist in training, but adhere to its general pattern. ([View Highlight](https://read.readwise.io/read/01jynzwds4jt4h3x7ttjby91ph)) > GPTs ability to arrive at true answers depends on not only the space to solve a problem in multiple steps (of the [right granularity](https://blog.eleuther.ai/factored-cognition/)), but also the direction of the flow of evidence in that *time*. If we’re ambitious about getting the truth from a finitely powerful GPT, we need to incite it to predict truth-seeking processes, not just ask it the right questions. Or, in other words, the more general problem we have to solve is not asking GPT the question[[20]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fnj2g0f81c6p) that makes it output the right answer, but asking GPT the question that makes it output the right question (…) that makes it output the right answer.[[21]](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators#fnn13gtuadzp) A question anywhere along the line that elicits a premature attempt at an answer could [neutralize the remainder of the process into rationalization](https://generative.ink/posts/methods-of-prompt-programming/#avoiding-rationalization). ([View Highlight](https://read.readwise.io/read/01jyp0777850nzhevk35rm9jp6)) > Natural language has the property of [*systematicity*](https://evjang.com/2021/12/17/lang-generalization.html): “blocks”, such as words, can be combined to form composite meanings. The number of meanings expressible is a combinatorial function of available blocks. A system which learns natural language is incentivized to learn systematicity; if it succeeds, it gains access to the combinatorial proliferation of meanings that can be expressed in natural language. What GPT lets us do is use natural language to specify any of a functional infinity of configurations, e.g. the mental contents of a person and the physical contents of the room around them, *and animate that*. That is the terrifying vision of the limit of prediction that struck me when I first saw GPT-3’s outputs. The words “behavior cloning” do not automatically evoke this in my mind. ([View Highlight](https://read.readwise.io/read/01jyp1arn61eckms6r5mneg1p0)) > This distinction is so obvious that it hardly ever merits mention. But import this distinction to the model of GPT as physics, and we generate a statement which has sometimes proven counterintuitive: *“GPT” is not the text which writes itself.* There is a categorical distinction between a thing which evolves according to GPT’s law and the law itself. ([View Highlight](https://read.readwise.io/read/01jz6fb7qckz54vpdthndvexsj)) > text automata ([View Highlight](https://read.readwise.io/read/01jz6g0wz3psa9936h66117hqw)) - Note: Q: What does this selection mean specifically? A: This selection explains that GPT and similar large language models function fundamentally as simulators—they generate text by modeling and predicting sequences based on learned statistical patterns rather than by pursuing explicit goals or utility maximization. The apparent agency or goal-directed behavior seen in their outputs arises from the simulated characters or scenarios they produce, not from coherent, underlying preferences or objectives within the model itself. Thus, GPT is best understood as a simulator of many possible agents and processes, rather than as a single agent with fixed goals or intentions. > By 2021, it was blatantly obvious that AGI was imminent. The elements of general intelligence were already known: access to information about the world, the process of predicting part of the data from the rest and then updating one’s model to bring it closer to the truth (…) and the fact that predictive models can be converted into generative models by reversing them: running a prediction model forwards predicts levels of X in a given scenario, but running it backwards predicts which scenarios have a given level of X. A sufficiently powerful system with relevant data, updating to improve prediction accuracy and the ability to be reversed to generate optimization of any parameter in the system is a system that can learn and operate strategically in any domain. ([View Highlight](https://read.readwise.io/read/01jz6fshrh85z1kra6rq7bc82d)) > Now I have updated to think that we will live, however briefly, alongside AI that is not yet foom’d but which has *inductively* learned a rich enough model of the world that it can simulate time evolution of open-ended rich states, e.g. coherently propagate human behavior embedded in the [real world](https://arbital.com/p/real_is_rich/). ([View Highlight](https://read.readwise.io/read/01jz6g77pfd70pjn5gjnsv9f7e)) - Note: Induction based simulation