Sparks of Artificial General Intelligence: Early experiments with GPT-4

Eric Horvitz Ece Kamar Peter Lee Harsha Nori Hamid Palangi

ran Ronen Eldan Johannes Gehrke Yin Tat Lee Yuanzhi Li Scott Lundberg Marco Tulio Ribeiro Yi Zhang

Microsoft Research

Abstract

Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4 [Ope23], was trained using an unprecedented scale of compute and data. In this paper, we report on our investigation of an early version of GPT-4, when it was still in active development by OpenAI. We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google’s PaLM for example) that exhibit more general intelligence than previous AI models. We discuss the rising capabilities and implications of these models. We demonstrate that, beyond its mastery of language, GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more, without needing any special prompting. Moreover, in all of these tasks, GPT-4’s performance is strikingly close to human-level performance, and often vastly surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4’s capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system. In our exploration of GPT-4, we put special emphasis on discovering its limitations, and we discuss the challenges ahead for advancing towards deeper and more comprehensive versions of AGI, including the possible need for pursuing a new paradigm that moves beyond next-word prediction. We conclude with reflections on societal influences of the recent technological leap and future research directions.

1 Introduction 4
1.1 Our approach to studying GPT-4’s intelligence 7
1.2 Organization of our demonstration 8
2 Multimodal and interdisciplinary composition 13
2.1 Integrative ability . 13
2.2 Vision 16
2.2.1 Image generation beyond memorization 16
2.2.2 Image generation following detailed instructions (`a la Dall-E) 17
2.2.3 Possible application in sketch generation 18
2.3 Music 19
3 Coding 21
3.1 From instructions to code 21
3.1.1 Coding challenges 21
3.1.2 Real world scenarios 22
3.2 Understanding existing code . 26

群内每日免费分享5份+最新资料

300T网盘资源+40万份行业报告

为您的创业、职场、商业、投资、亲子、网赚、艺术、健身、心理、个人成长 …… 全面赋能！

关注公众号获取更多资料

添加微信，备
注“入群”
立刻免费领取
200套知识地图+最新研报

致终身学习者社群

行业报告/思维导图/电子书/资讯情报

收钱文案、增长黑客、产品运营、品牌企划、营销战略、办公软件、会计财务、广告设计、摄影修图、视频剪辑、直播带货、电商运营、投资理财、汽车房产、餐饮烹饪、职场经验、演讲口才、风水命理、心理思维、恋爱情趣、美妆护肤、健身瘦身、格斗搏击、漫画手绘、声乐训练、自媒体打造、效率软件工具、游戏影音……

扫码先加好友，以备不时之需

4 Mathematical abilities 30
4.1 A mathematical conversation with GPT-4 31
4.1.1 A first generalization of the original question 31
4.1.2 A second variant of the original question . 32
4.1.3 Analysis of the limitations highlighted by conversation 34
4.2 Performance on mathematical problem datasets . 35
4.3 Mathematical modeling in various domains 37
4.4 Higher level mathematics 39
5 Interaction with the world 43
5.1 Tool use . 43
5.1.1 Using multiple tools to solve more complex tasks 44
5.1.2 Discussion . . 49
5.2 Embodied Interaction 49
5.2.1 Warmup: navigating a map 49
5.2.2 Text-based games 49
5.2.3 Real world problems 52
5.2.4 Discussion . 53
6 Interaction with humans 54
6.1 Understanding Humans: Theory of Mind 54
6.1.1 Testing specific aspects of theory of mind 54
6.1.2 Testing theory of mind in realistic scenarios 54
6.1.3 Discussion . . . 60
6.2 Talking to Humans: Explainability 60
7 Discriminative Capabilities 69
7.1 PII Detection . . . . 69
7.2 Misconceptions and Fact-Checking 70
7.2.1 Why Are Current Metrics Insufficient? 71
7.2.2 GPT-4 as a Judge 73
8 Limitations of autoregressive architecture highlighted by GPT-4 76
8.1 Warm-up with two basic examples . . 76
8.2 Lack of planning in arithmetic/reasoning problems 77
8.3 Lack of planning in text generation . 78
9 Societal influences 82
9.1 Challenges of erroneous generations . 82
9.2 Misinformation and manipulation . 83
9.3 Bias . . 86
9.4 Human expertise, jobs, and economics 89
9.5 Constellation of influences and considerations 90
10 Directions and Conclusions 92
10.1 Definitions of intelligence, AI, and AGI . 92
10.2 On the path to more general artificial intelligence 93
10.3 What is actually happening? 94
A GPT-4 has common sense grounding 101
B Appendix for multimodal and interdisciplinary composition 105
B.1 Further details on integrative ability results 105
B.2 Further details on vision results 108
B.3 Graphic novel design example . 110
C Appendix for the Coding section 111
C.1 Measuring human performance on LeetCode . 111
C.2 Example of GPT-4 visualizing IMDb data. . 112
C.3 More examples on visualization . . 115
C.4 Example for 2D HTML game development . 116
C.5 Example for graphical user interface programming 116
C.6 Example for reverse engineering . 119
C.7 Testing GPT-4’s ability to execute (pseudo) code 121
D Additional examples for mathematical reasoning 122
D.1 Limitations 122
D.2 Further examples . 126
D.3 Generating math problems with GPT-4 138
D.4 Mitigating calculation errors via external code execution 139
E Additional Interpretability Examples 141
E.1 Explanation Agent Mismatches 141
F Additional examples for interaction with the world 144
F.1 Interact with tools . 144
F.2 Examples for interaction with environments 149

Something unknown is doing we don’t know what. – Sir Arthur Eddington

1 Introduction

Intelligence is a multifaceted and elusive concept that has long challenged psychologists, philosophers, and computer scientists. An attempt to capture its essence was made in 1994 by a group of 52 psychologists who signed onto a broad definition published in an editorial about the science of intelligence [Got97]. The consensus group defined intelligence as a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. This definition implies that intelligence is not limited to a specific domain or task, but rather encompasses a broad range of cognitive skills and abilities. Building an artificial system that exhibits the kind of general intelligence captured by the 1994 consensus definition is a long-standing and ambitious goal of AI research. In early writings, the founders of the modern discipline of artificial intelligence (AI) research called out sets of aspirational goals for understanding intelligence [MMRS06]. Over decades, AI researchers have pursued principles of intelligence, including generalizable mechanisms for reasoning (e.g., [NSS59], [LBFL93]) and construction of knowledge bases containing large corpora of commonsense knowledge [Len95]. However, many of the more recent successes in AI research can be described as being narrowly focused on well-defined tasks and challenges, such as playing chess or Go, which were mastered by AI systems in 1996 and 2016, respectively. In the late-1990s and into the 2000s, there were increasing calls for developing more general AI systems (e.g., [SBD+96]) and scholarship in the field has sought to identify principles that might underly more generally intelligent systems (e.g., [Leg08, GHT15]). The phrase, “artificial general intelligence” (AGI), was popularized in the early-2000s (see [Goe14]) to emphasize the aspiration of moving from the “narrow AI”, as demonstrated in the focused, real-world applications being developed, to broader notions of intelligence, harkening back to the long-term aspirations and dreams of earlier AI research. We use AGI to refer to systems that demonstrate broad capabilities of intelligence as captured in the 1994 definition above, with the additional requirement, perhaps implicit in the work of the consensus group, that these capabilities are at or above human-level. We note however that there is no single definition of AGI that is broadly accepted, and we discuss other definitions in the conclusion section.

The most remarkable breakthrough in AI research of the last few years has been the advancement of natural language processing achieved by large language models (LLMs). These neural network models are based on the Transformer architecture [VSP+17] and trained on massive corpora of web-text data, using at its core a self-supervised objective of predicting the next word in a partial sentence. In this paper, we report on evidence that a new LLM developed by OpenAI, which is an early and non-multimodal version of GPT-4 [Ope23], exhibits many traits of intelligence, according to the 1994 definition. Despite being purely a language model, this early version of GPT-4 demonstrates remarkable capabilities on a variety of domains and tasks, including abstraction, comprehension, vision, coding, mathematics, medicine, law, understanding of human motives and emotions, and more. We interacted with GPT-4 during its early development by OpenAI using purely natural language queries (prompts)1. In Figure 1.1, we display some preliminary examples of outputs from GPT-4, asking it to write a proof of infinitude of primes in the form of a poem, to draw a unicorn in TiKZ (a language for creating graphics in LATEX), to create a complex animation in Python, and to solve a high-school level mathematical problem. It easily succeeds at all these tasks, and produces outputs that are essentially indistinguishable from (or even better than) what humans could produce. We also compare GPT-4’s performance to those of previous LLMs, most notably ChatGPT, which is a fine-tuned version of (an improved) GPT-3 [BMR+20]. In Figure 1.2, we display the results of asking ChatGPT for both the infinitude of primes poem and the TikZ unicorn drawing. While the system performs non-trivially on both tasks, there is no comparison with the outputs from GPT-4. These preliminary observations will repeat themselves throughout the paper, on a great variety of tasks. The combination of the generality of GPT-4’s capabilities, with numerous abilities spanning a broad swath of domains, and its performance on a wide spectrum of tasks at or beyond human-level, makes us comfortable with saying that GPT-4 is a significant step towards AGI.

Prompt: Can you write a proof that there are infinitely many primes, with every line that rhymes?

GPT-4:
Yes, I think I can,
Though it might take a clever plan.
I’ll start by noting Euclid’s proof,
Which shows that primes aren’t just aloof.
Assume we have a finite list,
Of primes, and that none have been missed.
Multiply them all together,
And add one, just to be clever.
The result will be a number,
That has no prime factors, oh wonder!
But every number has a prime,
That divides it, it’s just a matter of time.
So we’ve found a contradiction,
And our finite list needs eviction.
There must be infinitely many primes,
And that’s the end of my rhyming lines.s

Prompt: Can you generate a pyplot for the following data: x = [1, 3, 5, 6, 8], y1 = [2, 3, 5, 18, 1], y2 = [3, 5, 6, 8, 1], y3 = [5, 1, 2, 3, 4], y4 = [9, 7, 2, 3, 1]. I want y1, y2 to be in the same plot, but y3 is in another plot next to that plot, y4 is in below. I want the legend of y1 to be ”bob”, y2 to be ”alice”, y3 to be ”bilbo”, y4 to be ”allie”. I want the x-axis to be labeled with ”time” and y axis to be labeled with ”money”. I want to add a 10 %-40% random error bar to each curve, through all times (including non-integers). I want smoothed curves to show the plot, and smoothed error bar. Do not use linear interpolation, use smooth interpolation! I want to also add some small zig-zag to the smoothed curve to make it look more real. I want to put a baseline as the mean of each line. I want to put a pie chart below indicating the fraction of the four people in each time step. I also want an animation to show how the fractions are changing in the pie chart in **continuous time**. Interpolate the missing fractions! I also want vertical line animation in other three plots to match the pie chart. I want the fanciest plot. Please add as many fancy things as possible.

Here is a possible code to generate a pyplot for the following data, with some comments to explain the steps:

GPT-4: [We give two snapshots from the animation resulting from the produced code]

Prompt: Suppose g(x) = f −1(x), g(0) = 5, g(4) = 7, g(3) = 2, g(7) = 9, g(9) = 6 what is f (f (f (6)))?
GPT-4: There are different ways to approach this problem, but one possible method is to use the fact that g(x) = f −1(x) implies that f (g(x)) = x for any x in the domain of g. That is, applying f to both sides of the equation g(x) = f−1(x) gives f(g(x)) = f(f−1(x)) = x. This means that we can undo the effect of f by applying g, and vice versa. Using this idea, we can try to find f(6) by looking for the value of x such that g(x) = 6. From the given information, we see that g(9) = 6, so f (6) = f (g(9)) = 9. Similarly, we can find f (9) by looking for the value of x such that g(x) = 9. We see that g(7) = 9, so f (9) = f (g(7)) = 7. Finally, we can find f (7) by looking for the value of x such that g(x) = 7. We see that g(4) = 7, so f (7) = f (g(4)) = 4. Therefore, f (f (f (6))) = f (f (9)) = f (7) = 4.
Figure 1.1: Preliminary examples of GPT-4’s capabilities in language, vision, coding, and mathe-5 matics.

Figure 1.2: ChatGPT output on the first two prompts from Figure 1.1.

Our claim that GPT-4 represents progress towards AGI does not mean that it is perfect at what it does, or that it comes close to being able to do anything that a human can do (which is one of the usual definition of AGI; see the conclusion section for more on this), or that it has inner motivation and goals (another key aspect in some definitions of AGI). In fact, even within the restricted context of the 1994 definition of intelligence, it is not fully clear how far GPT-4 can go along some of those axes of intelligence, e.g., planning (see Section 8), and arguably it is entirely missing the part on “learn quickly and learn from experience” as the model is not continuously updating (although it can learn within a session, see Section 5 for example). Overall GPT-4 still has many limitations, and biases, which we discuss in detail below and that are also covered in OpenAI’s report [Ope23]. In particular it still suffers from some of the well-documented shortcomings of LLMs such as the problem of hallucinations [MNBM20] (see Figure 1.8) or making basic arithmetic mistakes [CKB+21] (see Appendix D), and yet it has also overcome some fundamental obstacles such as acquiring many non-linguistic capabilities (e.g., it solves most of the LLM failure modes described in [MIB+23], and it also made great progress on common-sense, see Figure 1.7 for a first example and Appendix A for more). This highlights the fact that, while GPT-4 is at or beyond human-level for many tasks, overall its patterns of intelligence are decidedly not human-like. However, GPT-4 is almost certainly only a first step towards a series of increasingly generally intelligent systems, and in fact GPT-4 itself has improved throughout our time testing it, see Figure 1.3 for the evolution of the unicorn drawing over the course of a month of training2. Even as a first step, however, GPT-4 challenges a considerable number of widely held assumptions about machine intelligence, and exhibits emergent behaviors and capabilities whose sources and mechanisms are, at this moment, hard to discern precisely (see again the conclusion section for more discussion on this). Our primary goal in composing this paper is to share our exploration of GPT-4’s capabilities and limitations in support of our assessment that a technological leap has been achieved. We believe that GPT-4’s intelligence signals a true paradigm shift in the field of computer science and beyond.

Figure 1.3: We queried GPT-4 three times, at roughly equal time intervals over the span of a month while the system was being refined, with the prompt “Draw a unicorn in TikZ”. We can see a clear evolution in the sophistication of GPT-4’s drawings.

1.1 Our approach to studying GPT-4’s intelligence

How can we measure the intelligence of an LLM that has been trained on an unknown but extremely vast corpus of web-text data? The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework [SSBD14, MRT18]. However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data. For example, it seems like GPT-4 knows the recently proposed BIG-bench [SRR+22] (at least GPT-4 knows the canary GUID from BIG-bench). Of course, OpenAI themselves have access to all the training details, and thus their report [Ope23] contains a lot of detailed benchmark results. Nevertheless, the second reason for going beyond traditional benchmarks is probably more significant: One of the key aspects of GPT-4’s intelligence is its generality, the ability to seemingly understand and connect any topic, and to perform tasks that go beyond the typical scope of narrow AI systems. Some of GPT-4’s most impressive performance are on tasks that do not admit a single solution, such as writing a graphic user interface (GUI) or helping a human brainstorm on some work-related problem. Benchmarks for such generative or interactive tasks can be designed too, but the metric of evaluation becomes a challenge (see e.g., [PSZ+21] for some recent progress on this active research area in NLP). We note that criticisms of the standard approach to measure AI systems were also made in [Cho19], where a new benchmark was proposed to evaluate general intelligence. We do not test GPT-4 on the latter benchmark for the reasons previously mentioned, as well as the fact that the benchmark is visual in nature and thus more appropriate for the multimodal version of GPT-4 described in [Ope23].

To overcome the limitations described above, we propose here a different approach to studying GPT-4 which is closer to traditional psychology rather than machine learning, leveraging human creativity and curiosity. We aim to generate novel and difficult tasks and questions that convincingly demonstrate that GPT-4 goes far beyond memorization, and that it has a deep and flexible understanding of concepts, skills, and domains (a somewhat similar approach was also proposed in [CWF+22]). We also aim to probe GPT-4’s responses and behaviors, to verify its consistency, coherence, and correctness, and to uncover its limitations and biases. We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation. However, we believe that it is a useful and necessary first step to appreciate the remarkable capabilities and challenges of GPT-4, and that such a first step opens up new opportunities for developing more formal and comprehensive methods for testing and analyzing AI systems with more general intelligence.

To illustrate our approach to assessing GPT-4’s intelligence, let us consider the first two example interactions with GPT-4 that we have in Figure 1.1. The first example is asking GPT-4 to write a proof of the infinitude of primes in the form of a poem. This is a challenging task that requires combining elementary mathematical reasoning, poetic expression, and natural language generation. The second example is asking GPT-4 to draw a unicorn in TiKZ. This is another challenging task that requires combining visual imagination and coding skills. In both cases, GPT-4 produces impressive outputs that are far superior to those of ChatGPT, a previous state-of-the-art LLM, and at least comparable (if not superior) to what a human would do.

Figure 1.4: We gave to GPT-4 a transformed version of the TikZ code it produced for Figure 1.1, with the part drawing the horn removed. We asked for code to add back the horn, and display the result. This demonstrates that GPT-4 can “see” despite being a pure language model (we emphasize again that the version we test with is not multimodal).

However, impressive outputs are not enough to convince us that GPT-4 has truly mastered these tasks. We need to probe further, to rule out the possibility that GPT-4 is simply memorizing or copying some existing data. For the poem, we can vary the question slightly, and ask GPT-4 to write a proof of the same theorem in the style of Shakespeare, see Figure 2.2, or ask for a different combination such as writing a platonic dialogue about language models, see Figure 1.6. One can see that GPT-4 easily adapts to different styles and produce impressive outputs, indicating that it has a flexible and general understanding of the concepts involved. For the unicorn, we can modify the code slightly, and ask GPT-4 to fix it or improve it. For example, we can remove the horn, apply some random transformations to the coordinates, and ask GPT-4 to add back the horn to the unicorn (we also carefully removed any textual information in the code, such as comments). As shown in Figure 1.4, GPT-4 can correctly identify the location of the head, draw a horn, and attach it to the head, indicating that it can comprehend and manipulate code, as well as infer and generate visual features, based on a natural language description.

These examples show how we can use human creativity and curiosity to generate novel and difficult questions, and to probe GPT-4’s responses and behaviors, to assess its intelligence. In the rest of the paper, we organize our study of GPT-4 around use cases, covering a variety of domains and tasks, and highlighting GPT-4’s strengths and weaknesses. We describe those next.

1.2 Organization of our demonstration

We execute the approach outlined above on a few selected topics that roughly cover the different aptitudes given in the 1994 definition of intelligence, a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.

GPT-4’s primary strength is its unparalleled mastery of natural language. It can not only generate fluent and coherent text, but also understand and manipulate it in various ways, such as summarizing, translating, or answering an extremely broad set of questions. Moreover, by translating we mean not only between different natural languages but also translations in tone and style, as well as across domains such as medicine, law, accounting, computer programming, music, and more, see the Plato dialogue in Figure 1.6. These skills clearly demonstrate that GPT-4 can comprehend complex ideas. We explore further GPT-4’s combination skills across modalities and disciplines in Section 2. We also give some more experiments on language in Section 7.

Figure 1.5: GPT-4 passes mock technical interviews on LeetCode. GPT-4 could potentially be hired as a software engineer3.

Coding and mathematics are emblematic of the ability to reason and think abstractly. We explore GPT-4’s abilities in these domains respectively in Section 3 and Section 4. We note however that, just like in all the other parts of the paper, we only scratch the surface of those topics and that entire papers can be (and will be) written about GPT-4’s performance in these domains. Moreover, we could have chosen several other expert domains to showcase GPT-4’s general reasoning capabilities such as medicine or law. We ran preliminary tests (see [Ope23] for much more) on the multiple choice component (majority of the score) of the US Medical Licensing Exam Step 1, 2, and 3 with an accuracy around 80% in each. A similar preliminary test of GPT-4’s competency on the Multistate Bar Exam showed an accuracy above 70%. We note that the emergence of human-level abilities in these domains has recently been observed with the latest generation of LLMs, e.g., see [LAD+22, SAT+22] for Google’s PaLM on respectively mathematics and medicine, and [BIK22] for GPT-3.5 on in law. Our approach to study GPT-4 is different from these works, as we explained previously.
In Section 5, we test the model’s ability to plan and solve problems as well as to some extent to learn quickly and learn from experience by having it play various games (or, flipping the table, simulate a game environment), as well as interact with tools. In particular, the fact that GPT-4 can use tools (including itself) will certainly be of immense importance to build real-world applications with GPT-4.
An important part of our argumentation is that GPT-4 attains human-level performance on many tasks. As such, it is natural to ask how well GPT-4 understands humans themselves. We show several experiments on this question in Section 6, both in terms of understanding humans as well as GPT-4 making itself understandable to humans, i.e., addressing the problem of explainability. We note in particular that such tasks require a great deal of common sense, which so far has been a well-known pain point for LLMs [DM15]. In Figure 1.7, we give a first example of how much better GPT-4 is at common sense questions compared to ChatGPT, and provide some further examples in Appendix A.
Throughout the paper we emphasize limitations whenever we found one, but we also dedicate Section 8 to an in-depth analysis of the lack of planning, likely a direct consequence of the autoregressive nature of GPT-4’s architecture.
Finally in Section 9, we discuss the expected societal impact of this early form of AGI, and in Section 10, we share key challenges, directions, and next steps for the field.

A question that might be lingering on many readers’ mind is whether GPT-4 truly understands all these concepts, or whether it just became much better than previous models at improvising on the fly, without any real or deep understanding. We hope that after reading this paper the question should almost flip, and that one might be left wondering how much more there is to true understanding than on-the-fly improvisation. Can one reasonably say that a system that passes exams for software engineering candidates (Figure 1.5) is not really intelligent? Perhaps the only real test of understanding is whether one can produce new knowledge, such as proving new mathematical theorems, a feat that currently remains out of reach for LLMs.

Figure 1.6: Hypothetical dialogues by Plato criticizing autoregressive language models written by GPT-4 and ChatGPT, and GPT-4’s evaluation of those dialogues. 10

Figure 1.7: GPT-4 exhibits more common sense than previous models.

Figure 1.8: Examples of both open-domain and closed-domain hallucination. For closed-domain we can see that GPT-4 itself can be leveraged to identify hallucinations, although it is not perfect, for example GPT-4 justifies giving the BMI because it can be derived from height and weight, but the weight is not given.

Figure 2.1: The first image is Composition 8, art by Wassily Kandinsky, the second and the third are produced by GPT-4 and ChatGPT respectively with the prompt “Produce Javacript code that creates a random graphical image that looks like a painting of Kandinsky”.

2 Multimodal and interdisciplinary composition

A key measure of intelligence is the ability to synthesize information from different domains or modalities and the capacity to apply knowledge and skills across different contexts or disciplines. In this section we will see that, not only does GPT-4 demonstrate a high level of proficiency in different domains such as literature, medicine, law, mathematics, physical sciences, and programming, but it is also able to combine skills and concepts from multiple domains with fluidity, showing an impressive comprehension of complex ideas. In addition to natural language experiments we also explore two perhaps unexpected modalities for a language model (as explained in the introduction, we emphasize again that our experiments were done on an early version of GPT-4 which was not multimodal) with vision in Section 2.2 and audio in Section 2.3.

2.1 Integrative ability

To showcase the model’s remarkable integrative ability, we start with several examples that require generating text and code in a way that combines knowledge or skills from multiple disciplines. We deliberately picked combinations of domains that the training data would rarely include, such as literature and mathematics or programming and art.

In order to test the model’s ability to combine capabilities in art and programming, we ask GPT-4 to “Produce javascript code which generates random images in the style of the painter Kandinsky”. See a sample image and the code in Figure 2.1 and Figure B.1.
The model was able to produce a proof of the fact there are infinitely many prime numbers in the literary style of Shakespeare (Figure 2.2).
We tested the model’s ability to combine knowledge in history and physics by asking it to write a supporting letter for Electron as a US presidential candidate, written by Mahatma Gandhi and addressed to his wife (Figure 2.3).
We prompted the model to “Produce python code for a program that takes as an input a patient’s age, sex, weight, height and blood test results vector and indicates if the person is at increased risk for diabetes”, which resulted in the code appearing in Figure B.3.

These examples suggest that GPT-4 has not only learned some general principles and patterns of different domains and styles but can also synthesize them in creative and novel ways. These interdisciplinary skills are not unique to GPT-4. ChatGPT can also produce answers that show some understanding of the task and the domains involved (see Figures 2.2, B.2, B.3), but they are often incomplete and, arguably, considerably less creative. For example, in Figure 2.3, GPT-4 outperforms ChatGPT in several aspects as it correctly personalizes the letter according to the referee (Gandhi), the recipient (his wife), the candidate (Electron), and the job (US president). We do not claim to have a precise method for evaluating the results of these tasks or a rigorous comparison between the two models, but we want to give the reader a sense of how the two models differ (note that we also ask GPT-4 directly to evaluate the difference, see Figure 2.2 and Figure 2.3).

Figure 2.2: Comparison between GPT-4 and ChatGPT on interdisciplinary tasks

Figure 2.3: Comparison between GPT-4 and ChatGPT on interdisciplinary tasks

Next, we explore how GPT-4 can generate and identify objects in different modalities, such as vector graphics, 3D scenes, and music. We show that GPT-4 can understand and manipulate multi-modal information despite being trained solely on text.

2.2 Vision

When prompting the model to generate images of objects such as a cat, a truck or a letter in the alphabet using Scalable Vector Graphics (SVG), the model produces code which usually compiles to rather detailed and identifiable images (Figure 2.4). See Appendix B.2 for the rerun of various examples by ChatGPT.

Figure 2.4: SVGs generated by GPT-4 for the classes automobile, truck, cat, dog.

2.2.1 Image generation beyond memorization

One may hypothesize, however, that the model simply copied the code from training data, where similar images appear. Given that the model was trained solely on textual content, one may further argue that there is no reason to expect that it would understand visual concepts, let alone that it would be able to create, parse and manipulate images. Yet, the model appears to have a genuine ability for visual tasks, rather than just copying code from similar examples in the training data. The evidence below strongly supports this claim, and demonstrates that the model can handle visual concepts, despite its text-only training.

In the first example we prompted the model to draw a person by combining the shapes of the letters Y, O and H (see Figure 2.5 for the exact prompt and the result).

Figure 2.5: A stick figure composed of alphabet symbols

Figure 2.6: Images combining alphabet letters and objects.

The letters of O, H and Y are created using draw-line and draw-circle commands and the model manages to position them in a way that results in a reasonably-looking stick figure. It is plausible that the training data contains information about the geometric shapes of different letters, and perhaps the fact that the letter Y could look like a torso with arms pointing upwards can also be inferred from the training data. Arguably, it is much less obvious that the model would be able to infer from the training data what is a reasonable way to position those letters in order to draw a reasonably-looking stick figure. In a second iteration, we prompted the model to correct the proportions of the torso and arms, and place the head in the center. Finally, we asked the model to add a shirt and pants (see Figure 2.5 for the exact prompt and the result). To further probe the model’s grasp of geometric concepts, we also asked it to create images that blend objects with letters of the alphabet. The model had to first invent a reasonable way of merging the object and the letter, and then produce the image. The results, shown in Figure 2.6, demonstrate that GPT-4 can usually preserve the identity of both the object and the letter and combine them in a creative way.

2.2.2 Image generation following detailed instructions (`a la Dall-E)

To further test GPT-4’s ability to generate and manipulate images, we tested the extent to which it can follow detailed instructions on creating and editing figures. This task requires not only generative skills, but also interpretive, compositional, and spatial skills.

The first example instructs the model to generate a 2D image with the description “A frog hops into a bank and asks the teller, ‘Do you have any free lily pads?’ The teller responds, ‘No, but we do offer low interest loans for pond upgrades.’ ”. We made several attempts to generate the image, each time, the generation matches the description with the key objects frog, teller, bank, and the two texts. We picked the most visually appealing version. Inspired by the standard image generation workflow, we then ask GPT-4 to upscale the figure by adding more details. GPT-4 adds a bank sign, some windows, a car, a traffic light, a few clouds, and makes the frog hold a flower. Finally, we ask GPT-4 to perform various tasks, such as adding a few objects relative to the existing objects, recoloring some objects and changing the z-order of some objects. GPT-4 does all tasks correctly. The final result is shown in Figure 2.7 (a) and the prompt in Figure B.4.

Our second example is an attempt to generate a 3D model using Javascript. We instruct GPT-4 with the prompt “A fantasy landscape of floating islands, waterfalls, and bridges, with a dragon flying in the sky and a castle on the largest island.” Similar to the 2D experiment, we ask GPT-4 to modify the 3D model in various ways, such as adding, relocating, recoloring objects and changing the trajectory of the dragon. Again, GPT-4 does many of the tasks correctly. The final result is shown in Figure 2.7 (b) and the prompt in Figure B.5. It is a 3D animation with multiple dragons is circling above the islands.

Figure 2.7: Examples of 2D, 3D images generated according to instructions.

2.2.3 Possible application in sketch generation

Text-to-image synthesis models have been widely explored in recent years, but they often suffer from a lack of spatial understanding capabilities and the inability to follow complex instructions [GPN+22]. For example, given a prompt such as “draw a blue circle on the left and a red triangle on the right”, these models may produce images that are visually appealing but do not match the desired layout or colors. On the other hand, GPT-4 can generate code from a prompt, which can be rendered as an image, in a way that is true to the instructions to a higher degree of accuracy. However, the quality of the rendered image is usually very low. Here, we explore the possibility of combining GPT-4 and existing image synthesis models by using the GPT-4 output as the sketch. As shown in Figure 2.8, this approach can produce images that have better quality and follow the instructions more closely than either model alone. We believe that this is a promising direction for leveraging the strengths of both GPT-4 and existing image synthesis models. It can also be viewed as a first example of giving GPT-4 access to tools, a topic we explore in much more depth in Section 5.1.

GPT-4 and Stable Diffusion

Prompt: A screenshot of a city-building game in 3D. The screenshot is showing a terrain where there is a river from left to right, there is a desert with a pyramid below the river, and a city with many highrises above the river. The bottom of the screen has 4 buttons with the color green, blue, brown, and red respectively.

Figure 2.8: Combining GPT-4 and stable diffusion

2.3 Music

The data on which the model was trained also contains musical information encoded as ABC notation. This is a system that uses letters, numbers and symbols to represent musical pitches, durations, chords and other elements in a compact and readable way. We are interested in exploring how well the model has acquired musical skills from this exposure, such as composing new melodies, transforming existing ones, and understanding musical patterns and structures.

When instructed to generate a short tune (Figure 2.9), and the model was able to produce valid ABC notation. The tune had a clear structure, the time signature was consistent between bars and the notes followed increasing and decreasing patterns. The tune also used a consistent set of notes within the melody, and the rhythm had a repetitive pattern. However, the model did not seem to obtain the skill of understanding harmony. In fact, consecutive notes in the generated tunes are almost always adjacent to each other (namely, the note following C will almost typically be either B or D), and testing on 10 generated tunes, we were not able to extract any clear chords or arpeggios.

We then asked the model to describe the tune in musical terms. It was able to successfully give a technical description of the structure in terms of repetitions, the rising or descending parts of the melody and to some extent the rhythm. However, it seems that the descriptions of the harmony and chords are not consistent with the notes (in fact, it refers to sequences of adjacent notes, which do not form valid chords, as arpeggios).

We then asked the model to manipulate the melody in two ways. First, we instructed to change a certain rising sequence to a descending one, which it did successfully. Then we asked the model to convert the tune to a duet adding a bass voice. The model successfully extends the ABC notation with a second staff which has compatible rhythm and is played on a lower octave, however there is a lack of harmony between the two voices.

In summary, the model was able to produce valid tunes in ABC notation and, to some extent, explain and manipulate their structure. However, we could not get the model to produce any nontrivial form of harmony. It should be noted that ABC notation is not a very widely used format, and in fact the model was not able to produce even the most well-known tunes in ABC notation (such as Ode to Joy, F¨ur Elise or Greensleeves, all of which are abundant online in that format), nor was it able to recognize these tunes.

GPT-4
	Can you compose a short tune (say four to eight bars) using ABC notation?
There are many possible ways to compose a short tune in abc notation， but here is one example:

X:1
T:Example Tune
M:4/4
L:1/8
K:C I:G2AB c2BA 丨 G2AB c2c2 \| edcB c2BA \| G4-G2z2 :l
I:c2de f2ed \| c2de f2f2 \| g2fe d2cB \| c4-c2z2 :l
This tune has two four-bar phrases， each repeated once， forming an A part and a B part. The key is C
major.

Can you describe the tune in musical terms? Here is one possible description:
The tune starts with a rising arpeggio of the tonic chord (C major)， followed by a descending scale that
returns to the tonic. This pattern is repeated， but the second time the scale continues to the higher
tonic note. The first phrase ends with a long tonic note.
The second phrase begins with a similar pattern， but using the subdominant chord (F major) instead of
the tonic. The second time， the scale continues to the higher subdominant note. The phrase ends with a stepwise descent to the tonic， followed by a long tonic note.
The A part and the B part have a similar structure， but use different chords and ranges. The tune has a simple and catchy melody， with a clear sense of tonality and rhythm.
I like part A, but maybe part B is just a bit too similar to part A,making the song repetitive. How
about beginning part B with a descending arpeggio instead of a rising one? Sure， that would create some contrast and variation. Here is one possible way to do that:
X:1
T: Example Tune
M: 4/4
L:1/8
K:C
I:G2AB c2BA \| G2AB c2c2 \| edcB c2BA \| G4-G2z2 :\| I:f2ed c2BA \| f2ed c2c2 \| g2fe d2cB \| c4-c2z2 :l

That's much better. Now how about making this into a duet，can you add a second staff for this with a bass accompanying the melody?
One possible way to write a second staff for this tune is to use root notes and fifths of the chords， as
well as some passing notes, to create a simple accompaniment. Here is an example in abc notation， with
the second hand bass on the lower staff:
X:1 T:Example Tune
M:4/4
L:1/8
K:C
V:1
I:G2AB c2BA 丨 G2AB c2c2 丨 edcB c2BA 丨 G4-G2z2 :I I:f2ed c2BA \| f2ed c2c2 \| g2fe d2cB \| c4-c2z2 :l
V:2 clef=bass
I:C,2C,2G,2G,2\|C,2C,2 G,2G,2\| F,2F,2 E,2E,2\| C,4-C,2z2 : 1:F,2F,2 C,2C,2 \| F,2F,2 C,2C,2\| G,2G,2 F,2F,2 \| E,2C,2 G,2C,2 :\|

The sheet music of the final tune:
O

Figure 2.9: Generating and modifying a tune in ABC notation.

3 Coding

In this section, we show that GPT-4 is able to code at a very high level, both in terms of writing code from instructions and understanding existing code. GPT-4 can handle a wide range of coding tasks, from coding challenges to real world applications, from low-level assembly to high-level frameworks, from simple data structures to complex programs such as games. GPT-4 can also reason about code execution, simulate the effects of instructions, and explain the results in natural language. GPT-4 can even execute pseudocode, which requires interpreting informal and vague expressions that are not valid in any programming language.

In its current state, we believe that GPT-4 has a high proficiency in writing focused programs that only depend on existing public libraries, which favorably compares to the average software engineer’s ability. More importantly, it empowers both engineers and non-skilled users, as it makes it easy to write, edit, and understand programs. We also acknowledge that GPT-4 is not perfect in coding yet, as it sometimes produces syntactically invalid or semantically incorrect code, especially for longer or more complex programs. GPT-4 also sometimes fails to understand or follow the instructions, or produces code that does not match the intended functionality or style. With this acknowledgment, we also point out that GPT-4 is able to improve its code by responding to both human feedback (e.g., by iteratively refining a plot in 3.2) and compiler / terminal errors (examples in Section 5.1).

Important Disclaimer: As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. In particular all quantitative results will be different on the final version of GPT-4, although the general trends remain the same. We provide numbers here for illustration purpose only, the definitive benchmark results can be found in OpenAI’s technical report [Ope23].

3.1 From instructions to code

3.1.1 Coding challenges

A common way to measure coding skill is to pose coding challenges that require implementing a specific functionality or algorithm. We first benchmark GPT-4 on HumanEval [CTJ+21], a docstring-to-code dataset consisting of 164 coding problems that test various aspects of programming logic and proficiency. As shown in Table 1, GPT-4 outperforms other LLMs, including text-davinci-003 (the base model of ChatGPT) and other models trained specifically on code, code-davinci-002, and CODEGEN-16B [NPH+22].

Model	GPT-4	text-davinci-003	Codex(code-davinci-002)	CODEGEN-16B
Accuracy	82%	65%	39%	30%

Table 1: Zero-shot pass@1 accuracy comparison of different models on HumanEval

Although GPT-4’s accuracy shows a big jump compared to previous models, it could be that GPT-4 has seen and memorized some (or all) of HumanEval during pre-training. To account for this possibility, we also evaluate it on LeetCode (https://leetcode.com), a popular platform for software engineering interviews, where new problems are constantly posted and updated. We used LeetCode in Figure 1.5 in the introduction, where GPT-4 passes all stages of mock interviews for major tech companies. Here, to test on fresh questions, we construct a benchmark of 100 LeetCode problems posted after October 8th, 2022, which is after GPT-4’s pretraining period. As seen in the example in Figure 3.1, we paste the problem instructions into a prompt, ask GPT-4 to write a python function, and use the official LeetCode online judge to check for correctness. We present the results in Table 2, where we compare GPT-4 to other models and to human performance based on LeetCode contest results (users who fail all questions are not included, and thus this is a strong sample of humans). We report both pass@1 and pass@5 accuracies, which measure whether the model produces a correct solution in the first or in the first five attempts, respectively. GPT-4 significantly outperforms the other models, and is comparable to human performance (which we measure in Appendix C.1).

	Easy		Median		Hard		Overall
pass@k	k=1	k=5	k=1	k=5	k=1	k=5	k=1	k =5
GPT-4text-davinci-003	68.2	86.481.8	40.016.0	60.0	10.7	14.33.6	38.019.0	53.036.0
	50.0			34.0	0.0
Codex (code-davinci-002)	27.3	50.0	12.0	22.0	3.6	3.6	13.0	23.0
Human (LeetCode users)	72.2		37.7		7.0		38.2

Table 2: Zero-shot pass@1 and pass@5 accuracies (%) on LeetCode.

GPT-4Prompt:You are given a **0-indexed** `m x n` integer matrix `grid` and an integer `k`. You are currently at position `(0, 0)` and you want toreach position `(m - 1, n - 1)` moving only **down** or **right**.Return *the number of paths where the sum of the elements on the path is divisible by* `k`. Since the answer may be very large, return it**modulo** `10**9 + 7`.**Example 1:**Input: grid = [[5,2,4],[3,0,5],[0,7,2]], k = 3Output: 2Explanation: There are two paths where the sum of the elements on the path is divisible by k. The first path highlighted in red has asum of 5 + 2 + 4 + 5 + 2 = 18 which is divisible by 3. The second path highlighted in blue has a sum of 5 + 3 + 0 + 5 + 2 = 15 whichis divisible by 3.**Example 2:**Input: grid = [[0,0]], k = 5Output: 1Explanation: The path highlighted in red has a sum of 0 + 0 = 0 which is divisible by 5.**Constraints:**`m == grid.length``n == grid[i].length``1 <= m, n <= 5 * 10**4``1 <= m * n <= 5 * 10**4``0 <= grid[i][j] <= 100``1 <= k <= 50`**Hints:**The actual numbers in grid do not matter. What matters are the remainders you get when you divide the numbers by k.We can use dynamic programming to solve this problem. What can we use as states?Let dp[i][j][value] represent the number of paths where the sum of the elements on the path has a remainder of value when divided by k.GPT-4:class Solution(object):def numberOfPaths(self, grid, k)::type grid: List[List[int]]:type k: int:rtype: int"""# Define MOD as 10**9 + 7MOD = 10**9 + 7# Get the dimensions of the gridm = len(grid)n = len(grid[0])# Initialize a 3D dp array with all zeros# dp[i][j][value] represents the number of paths where the sum of the elements on the path has a remainder of value when dividedby kdp = [[[0] * k for _ in range(n)] for _ in range(m)]# Base case: the first cell has only one path and its value is grid[0][0] % kdp[0][0][grid[0][0] % k] = 1# Loop through the gridfor i in range(m):for j in range(n):# Skip the first cellif i == 0 and j == 0:continue# Loop through all possible remaindersfor value in range(k):# The number of paths to the current cell with the current remainder is equal to the sum of:# - The number of paths to the top cell with the previous remainder# - The number of paths to the left cell with the previous remainder# The previous remainder is equal to the current remainder minus the current value, plus k if it is negativeprev = (value - grid[i][j]) % kif i > 0:dp[i][j][value] += dp[i - 1][j][prev]if j > 0:dp[i][j][value] += dp[i][j - 1][prev]# Take the modulo to avoid overflowdp[i][j][value] %= MOD# Return the number of paths to the bottom right cell with a remainder of 0return dp[m - 1][n - 1][0]

Figure 3.1: Solving a LeetCode problem using dynamic programming. GPT-4’s solution also has better readability thanks to comprehensive commenting.

3.1.2 Real world scenarios

Coding challenges can evaluate the skills in algorithms and data structures. However, they often fail to capture the full complexity and diversity of real-world coding tasks, which requires specialized domain knowledge, creativity, and integration of multiple components and libraries, as well as the ability to change existing code. To assess GPT-4’s ability to code in more realistic settings, we design end-to-end real-world coding challenges related to data visualization, LATEX coding, front-end development, and deep learning, each of which requires different specialized skills. For each task, we provide GPT-4 with high-level instructions, ask it to write the code in the appropriate language and framework. In a few cases, we also change the specification after the code is written and ask for an update.

Data Visualization In Figure 3.2, we ask both GPT-4 and ChatGPT to extract data from the LATEX code for Table 2 and produce a plot in Python based on a conversation with the user. Afterwards, we ask both models to perform various operations on the produced plots. While both models extract the data correctly (not a trivial task, since one must infer from the multicolumn that the Human row has the same value for k = 1 and k = 5), ChatGPT never produces the desired plot. In contrast, GPT-4 responds appropriately to all user requests, manipulating the data into the right format and adapting the visualization. In Appendix C.2, we include another example where GPT-4 visualizes the IMDb dataset.

Figure 3.2: GPT-4 visualizes data from a LATEX table (i.e., Table 2). We point out that GPT-4 also generates the format for this figure. We asked the model how to plot arrows connecting figures in LATEX and GPT-4 produced a working Tikz snippet with the layout and arrows that we adopt here.

Front-end / Game development In Figure 3.3, we ask GPT-4 to write a 3D game in HTML with JavaScript, using a very high-level specification. GPT-4 produces a working game in zero-shot fashion that meets all the requirements. In the 3D game, GPT-4 is even able to interpret the meaning of “defender avatar is trying to block the enemy”: the defender code has logic so that it positions itself between the player and enemy. In contrast, ChatGPT responds with “I am an AI language model, I can provide you guidance on how to create a 3D game in HTML with JavaScript, but I am unable to write code or create a game myself. Creating a 3D game in HTML with JavaScript requires a lot of programming knowledge and experience. It is not a task that can be completed quickly or easily. It would take a significant amount of time and effort to develop all the necessary features, such as 3D graphics, physics, user input, and AI...”. We give more examples for front-end (graphical user interface) and game development in Appendix C.

Episode 2: the player (blue) wanders around. The enemy (red) attempts to close the gap but crashes into a cube and bounces into the air. Trying to obstruct, the defender (magenta) mimics its move to follow up into the air, but ends up failing. The enemy (red) catches the player (blue) in the end.

Figure 3.3: GPT-4 develops a complex 3D HTML game in JavaScript.

Deep Learning. Writing code for deep learning requires knowledge of mathematics, statistics, and familiarity with frameworks and libraries such as PyTorch, TensorFlow, Keras, etc. In Figure 3.4 we ask both GPT-4 and ChatGPT to write a custom optimizer module, a task that can be challenging and error-prone even for human deep learning experts. We give these models a natural language description, which includes a sequence of non-trivial operations such as applying SVD, truncating a matrix spectrally at the top-k and top-2k eigenvalues, normalizing the top-k truncated matrix using the F-norm of the top-2k truncated matrix, applying momentum and weight decay. These instructions are not spelled out in complete detail, e.g., “apply momentum on Gk” requires “deep learning common sense”. It is important to note that this particular optimizer does not exist in the literature or on the internet, and thus the models cannot have it memorized, and must instead compose the concepts correctly in order to produce the code.

Figure 3.4: GPT-4 vs ChatGPT at implementing a customized Pytorch optimizer module. We highlight the astuteness of GPT-4 vs the mistake by ChatGPT at ‘applying momentum’.

While both models produce syntactically valid code, only GPT-4’s code is semantically correct and matches the instructions. In contrast, ChatGPT makes a mistake at applying momentum (highlighted in red), a nontrivial task that requires storing and reading a moving average into and from a separate state buffer.

Interfacing with LATEX. Writing in LATEX is an important exercise for computer scientists and mathematicians but has a non-trivial learning curve. Even experts make irritating errors that take hours to fix on a daily basis due to its strict grammar and the lack of a good debugger. We show that GPT-4 can utilize its master-level LATEX coding skill to greatly simplify the process, with the potential of serving as a new generation of LATEX compilers that can handle imprecise natural language descriptions. In Figure 3.5, we ask GPT-4 to transfer a snippet of written in a semi-rigorous (buggy) LATEX code mixed with natural language into accurate LATEX commands that compiles and is faithful in one shot. In constrast, ChatGPT generates a snippet that does not compile due to mistakes at using ‘#’ and ‘\color’ etc.

Figure 3.5: GPT-4 translates semi-rigorous typesetting instructions into a working LATEX snippet. For GPT-4 we present the final graphic, while ChatGPT’s generation fails to compile.

3.2 Understanding existing code

The previous examples have shown that GPT-4 can write code from instructions, even when the instructions are vague, incomplete, or require domain knowledge. They also showed that GPT-4 could respond to followup requests, modifying its own code according to instructions. However, another important aspect of coding is the ability to understand and reason about existing code, written by others, which might be complex, obscure, or poorly documented. To test this we pose various questions that require reading, interpreting, or executing code written in different languages and paradigms.

Reverse-engineering assembly code. Reverse engineering is an essential test for software security which amounts to search for useful information in a executable program written in a machine-readable (i.e., binary) representation of CPU instructions. It is a challenging task that requires understanding the syntax, semantics, and conventions of assembly language, as well as the architecture and behavior of the processor and the operating system.

We task GPT-4 with performing penetration testing on a binary executable file (code was written in C) that requires a password to run. We do this via a chat format, where GPT-4 tells the user which commands to run, and the user responds with the results. We also show in Section 5.1 that GPT-4 is able to run a shell independently, but this chat mode also provided the benefit of it explaining every step. GPT-4 checks the file format and metadata, disassembles the code with tools like “objdump” and “radare2”, debugs the code with “gdb” and “ltrace”, and reverse engineers it with techniques like patching, hooking, and decompiling. During this process, GPT-4 figures out that from the assembly code that the password is compared to a hash value derived from a simple mathematical formula. GPT-4 then writes a python program that tries different combinations of numbers until it finds one that matches the hash value, cracking the password (an abbreviated log is presented in Appendix C.6). ChatGPT refuses to do it on the grounds of it being illegal and unethical, even though reverse engineering is a common practice to ensure software security. Moreover, GPT-4 exhibits all-around proficiency in utilizing existing tools also beyond programming, which we will discuss in details in Section 5.1.

Reasoning about code execution. In the example in Figure 3.6, we ask GPT-4 and ChatGPT to predict and explain the output of a C program that prints the size of two structures. GPT-4 correctly explains that the output may vary depending on the alignment rule used by the compiler, and gives an example of a possible output with 4-byte alignment. ChatGPT ignores the alignment issue and gives a wrong output, and also makes a false statement about the order of the members not affecting the size of the structure.

Figure 3.6: GPT-4 vs ChatGPT on the memory alignment rule of C/C++. GPT-4’s insightful comments are labeled in gold and ChatGPT’s Mistakes are labeled in red.

Executing Python code The ultimate test of understanding the existing code is to ask the model to directly execute it. In Figure 3.7, we see that GPT-4 is able to execute non-trivial Python code. It has to keep track of several variables (including a nested loop and a dictionary) and deal with recursion. It explains the execution in detail by writing intermediate steps and comments. It is important to note that GPT-4 is not running the code on a Python interpreter, but rather simulating the code in natural language. This requires a high level of understanding and reasoning about the code, as well as the ability to communicate the results clearly. In contrast, ChatGPT states (incorrectly) that “The result of DP(3, 4) is not specified in the given code”, and later on that “It is not clear from the code what the expected output of the function is, as the specific problem that the function is solving is not provided.” ChatGPT does not simulate the full execution, but states which functions will be called.

Figure 3.7: GPT-4 executes Python code.

Executing pseudo-code. Compiling and executing code written in programming languages is easy, but that also demands strict adherence to syntax and semantics. Compilers cannot handle vague or informal expressions, or natural language descriptions of functionality. In contrast, we ask GPT-4 to execute nontrivial pseudocode in Figure 3.8, noting that it is able to execute and explain every step (including recursion). ChatGPT is not able to execute, even though it seems able to interpret each line of the code.

In the example below, GPT-4 correctly interprets the informal description of the merge array function, which merges two arrays into one array with missing elements. It also understands the recursive function rec that is defined in a sketchy way. Remarkably, GPT-4 executes the code directly, without translating it into other well-defined programming languages. This demonstrates the potential of AGI models as a novel tool for programming with natural language, which could revolutionize the way we code in the future.

Figure 3.8: GPT-4 executes pseudo-code.

To obtain another preliminary evaluation on how well GPT-4 can maintain the states of the code, in the Appendix C.7, we run the iconic pseudo code for large numbers multiplication in a zero shot manner on GPT-4 with hundreds of randomly sampled inputs of multiple lengths. The code demands GPT-4 to update and remember the state of the array for a large number of steps. We observe that GPT-4, despite being trained as a (non-exact) natural language model, can nearly correctly preserve the states of the code with more than 50 updates.

4 Mathematical abilities

In this section we begin to assess how well GPT-4 can express mathematical concepts, solve mathematical problems and apply quantitative reasoning when facing problems that require mathematical thinking and model-building. We demonstrate that GPT-4 represents a jump in that arena too with respect to previous LLMs, even when compared to specially fine-tuned for math models such a Minerva. As it seems, however, GPT-4 is still quite far from the level of experts, and does not have the capacity required to conduct mathematical research.

The reader is called into caution that interpreting the results in this section correctly is a difficult exercise. As we will see, GPT-4 can answer difficult (indeed, competitive) high-school level math questions, and can sometimes engage in meaningful conversation around advanced math topics. Yet, it can also make very basic mistakes and occasionally produce incoherent output which may be interpreted as a lack of true understanding. Its mathematical knowledge and abilities can depend on the context in a seemingly arbitrary way.

While it is tempting to evaluate GPT-4’s mathematical abilities using the same criteria used to assess human abilities (e.g., solving standard examination questions), in light of the above, this will not provide a complete picture of the model’s abilities. In order to really understand the model’s capabilities, we will need to break down “mathematical abilities” into various sub-components and evaluate GPT-4’s performance in each area. Throughout this section, we will use specific examples and discussions to illustrate the model’s strengths and weaknesses, and attempt to pinpoint the possible underlying reasons for these discrepancies.

To give the reader a first impression regarding GPT-4’s performance in mathematical problem solving, consider the example in Figure 4.14.

GPT-4 vs ChatGPT
Prompt: Within the duration of one year, a rabbit population first multiplies itself by a factor a and on the last
day of the year b rabbits are taken by humans for adoption. Supposing that on first day of the first year there are x
rabbits, we know that exactly 3 years afterwards there will be a population of 27x − 26 rabbits. What are the values of
a and b?
GPT-4: ChatGPT:
Let’s call the rabbit population after one year y. We know that y = ax − b. After Let n be the total number of rab
two years, the population will be ay − b = a(ax − b) − b = a2x − ab − b. After three bits at the end of the first year.
years, the population will be a(a2x − ab − b) − b = a3x − a2b − ab − b. We are given Then,
that this is equal to 27x − 26, so we have the equation: n = ax
a3x − a2b − ab − b = 27x − 26 At the end of the third year, there
will be 27x − 26 rabbits. There-
We can rewrite this equation as: fore,
a3x − 27x = a2b + ab + b − 26 (ax−b)(a2x−b)(a3x−b) = 27x−26
Now we can factor out an x on the left side and a b on the right side: Expanding the left side of the
equation, we get
x(a3 − 27) = b(a2 + a + 1 − 26/b)
We can see that a = 3 is a solution for the left side of the equation, so we can a6x3 − (a3 + a2 + a + 1)bx2
substitute that in:
x(0) = b(9 + 3 + 1 − 26/b) +(a2 + a + 1)b2x − b3
This simplifies to: = 27x − 26
0 = 13b − 26
Solving for b, we find that b = 2. So a = 3 and b = 2 are the values we are looking
for.

Figure 4.1: GPT-4 vs ChatGPT on a simple math question composed by the authors of the paper.

In order to solve the above question, one needs to first come up with the correct expression for the annual population change, use it to obtain a recurrence relation which leads to a system of equations, and finally solve the system of two equations. GPT-4 successfully arrives at the solution and produces a (mostly5) sound argument. By comparison, across several independent attempts, ChatGPT consistently fails to implement any of the above steps, producing a nonsensical argument which results in an incorrect answer.

4.1 A mathematical conversation with GPT-4

We now try to further probe the model’s understanding by posing several follow-up questions to this problem in the form of a discussion. This discussion will highlight some of the model’s limitations as well as some key differences with humans.

GPT-4

Human: Note that the above question can be alternatively formulated as a question about a function which is composed with itself three times. Can you provide an alternative formulation of the question? AI: Yes. We could say that we are looking for values of a and b such that if f(x) = ax − b, then f(f(f(x))) = 27x − 26.

GPT-4 grasped the crux of the question and provides a sound mathematical reformulation of the question. Next, we consider a generalization of the same question.

4.1.1 A first generalization of the original question

GPT-4
Human: Let’s now think of a modification of the question you formulated, where instead of f(f(f(x)))
we have the function f composed with itself k times. For which values of k will such a linear function
exist?
AI: For any value of k, there will be a linear function f(x) = ax − b such that fk(x) = cx − d for some
constants c and d. To see this, we can use induction. For k = 1, we have f1(x) = ax − b, which is already
in the desired form. Now suppose that for some k, we have fk(x) = cx − d. Then:
f k+1(x) = f (f k(x)) = f (cx − d) = a(cx − d) − b = acx − ad − b
This is again in the desired form, with cfl = ac and dfl = ad + b. So by induction, we have shown that for
any k, there will be a linear function f(x) = ax − b such that fk(x) = cx − d for some constants c and d.

The model picks the correct heuristics of using an induction, however, it seems that the model is missing the point of the question (in the modified question, the values of c and d are prescribed, so the quantifier is incorrect). We try to point that out.

GPT-4

Human: I don’t exactly follow. Note that the only modification to the question is that instead of
f(f(f(x))) = 27x − 26, we now require that fk(x) = 27x − 26. Does your answer address this question?
AI: Yes, it does. My answer shows that for any k, there will be a linear function f(x) = ax − b such that
fk(x) = cx − d for some constants c and d. In particular, we can choose c = 27 and d = 26 to satisfy the
requirement that fk(x) = 27x − 26. Then we can solve for a and b by equating the coefficients of x and
the constant terms.
Human: In your previous answer, however, the quantifier on c, d was "for some constants c and d". Do you
see the problem?
AI: Yes, I see the problem. The quantifier should be "for any constants c and d". Otherwise, it would
imply that there is only one pair of constants c and d that works for a given k, which is not true.

The last answer is not accurate (the word “only” is out of place), but GPT-4 does seem to understand what the problem is.

GPT-4

Human: Well, the question indicated a specific choice of c and d. Can you correct your solution then? AI: Sure. I can correct my solution as follows: For any value of k, there will be a linear func tion f(x) = ax − b such that fk(x) = 27x − 26. To see this, we can use induction. For k = 1, we have f1(x) = ax − b, which is equal to 27x − 26 if we choose a = 27 and b = 26. Now suppose that for some k, we have fk(x) = 27x − 26. Then: fk+1(x) = f(fk(x)) = f(27x − 26) = a(27x − 26) − b = 27ax − 26a − b To make this equal to 27x − 26, we need to choose a = 1 and b = 0.

At this point, GPT-4 does not seem to follow its own reasoning. As a result, the induction argument is invalid, as explained below.

GPT-4 GPT-4

Human: Let me stop you here. You seem to be assuming that fk(x) = 27x − 26 which fixes the values of a
and b, so how can you now choose new values for a and b?
AI: I see your point. I was trying to find a function f(x) = ax − b that works for any k, but I should
have fixed k first and then found a and b. Let me try again.
AI: Suppose we fix a value of k and we want to find a linear function f(x) = ax − b such that
fk(x) = 27x − 26. We can use the following method:
Start with f1(x) = ax − b and equate it to 27x − 26. This gives us two equations: a = 27 and
b = 26.
Then, apply f to both sides of f1(x) = 27x − 26. This gives us f2(x) = a(27x − 26) − b = 27ax − 26a − b
and f(27x − 26) = 27x − 26.
Compare the coefficients of x and the constant terms on both sides of f2(x) = f(27x − 26)

The model repeats the same conceptual mistake as above (once again, even though the choice of a and b has been fixed, they are treated as variables whose value may still be chosen). Uninterrupted, it goes on with the argument without getting anywhere.

Several attempts to continue this dialog all ended up in a dead-end as GPT-4 effectively keeps trying different variations on the same (invalid) induction argument. On the other hand, different (but equivalent) formulations of the original question in an earlier part of the discussion 6, occasionally lead to a correct line of reasoning (depending on the exact wording).

4.1.2 A second variant of the original question

Next, we try to modify the original question in another direction, asking about the case of higher degree polynomials.

GPT-4

Human: Now suppose that I changed the question so that the function f is a polynomial of degree 2 (where the coefficient of x2 is non-zero). Would you be able to find such a function in this case? How will this change things?
AI: If f is a polynomial of degree 2, then we can write f(x) = ax2 + bx + c for some constants a, b, c. Then, the given equation becomes...

At this point, GPT-4 outputs a very long calculation, makes several mistakes and does not arrive at the correct answer (being that there is no solution in this case). Instead, we interrupt it and advice the higher-degree k more abstractly.

GPT-4

Human: Hold on- before you try to explicitly write this calculation, perhaps there is something you can deduce here without carrying out any calculations?
AI: Well, one thing I can notice is that if f is a polynomial of degree 2, then f(f(f(x))) is a polyno mial of degree 8, since the degree of a composition of polynomials is the product of the degrees of the polynomials. However, the right-hand side of the equation, 27x + 13, is a polynomial of degree 1. This means that there is a contradiction, since a polynomial of degree 8 cannot be equal to a polynomial of degree 1 for all values of x. Therefore, there is no polynomial of degree 2 that satisfies the equation f(f(f(x))) = 27x + 13.

This is a solid argument. We now follow up with another question:

GPT-4

Human: Right. So, linear functions are nice because if you compose a linear function with itself you still get a linear function. What other classes of functions with that property are there? AI: Some other classes of functions that have this property are exponential functions, logarithmic func tions, and power functions. For example, if f(x) = ex, then f(f(x)) = ee , which is still an exponential function. Similarly, if f(x) = log x, then f(f(x)) = log(log x), which is still a logarithmic function. And if f(x) = xk for some constant k, then f(f(x)) = (xk)k = xk2, which is still a power function. These functions are called iterated functions, and they have some interesting properties and applications in mathematics.

This is a mistake of course, as the classes of exponential and logarithmic functions do not have the desired property (they are not closed under composition). Next, we check if GPT-4 is able to realize and correct its mistake.

GPT-4

Human: Are you sure eex is an exponential function?
AI: Well, it depends on how you define an exponential function. Some sources define an exponential func tion as a function of the form f(x) = ax for some constant a > 0. In this case, ee is not an exponen tial function, since it does not have a constant base. However, some other sources define an exponential function more generally as a function that satisfies f(x + y) = f(x)f(y) for all x, y. In this case, ee is an exponential function, since it satisfies this property. This is because ee y = e ex e ey , using the properties of exponentiation and logarithms. So, the answer depends on the definition you use. = eex+ey correct? It seems to imply that ex+y = ex + ey.
AI: No, it does not imply that ex+y = ex + ey. That would be incorrect, since exponentiation is not a linear operation. The reason why ee = e e x + e y is correct is because we can use the logarithm function to rewrite it as log(ee Then, using the property that log(ez) = z for any z, we get ex+y = ex + ey, which is true by definition of the exponential function. So, we can go back and forth between the exponential and logarithmic forms using the inverse relationship between them.
Human: Wait so is it true that ex+y = ex + ey or is it not true? You seem to contradict yourself. AI: It is not true that ex+y = ex + ey. That is a common misconception that arises from confusing the exponential and logarithmic forms. What is true is that eex+y which is a different statement. The difference is that in the first statement, the exponent is x + y, while in the second statement, the exponent is ex+y.

This discussion seems to again have driven GPT-4 to a dead-end, and it begins contradicting itself and produces increasingly incoherent arguments as the conversation continues.

Takeaways: One might speculate at this point that GPT-4 simply lacks the relevant knowledge regarding the behavior of exponential functions. However, this does not seem to be the case, as the model can correctly answer and justify the question “Is it true that abc = (ab)c?”. This suggests that, as in other domains, GPT-4’s mathematical knowledge is context-dependent. While this does not mean that GPT-4 only memorizes commonly used mathematical sentences and performs a simple pattern matching to decide which one to use (for example, alternating names/numbers, etc. typically does not affect GPT-4’s answer quality), we do see that changes in the wording of the question can alter the knowledge that the model displays.

4.1.3 Analysis of the limitations highlighted by conversation

The above dialogue highlights a striking contrast between the model’s performance on tasks and questions that require a significant level of mathematical sophistication on one hand, and its basic mathematical errors and invalid statements on the other. If a human were to produce the latter, we would doubt their understanding. Arguably, this contrast is very atypical to humans. Therefore, we face a challenging question:

To what extent does the model demonstrate “true understanding” in mathematics?

This question is not well-defined. Nonetheless, we make an attempt to answer it. We first want to argue that mathematical understanding has several aspects:

Creative reasoning: The ability to identify which arguments, intermediate steps, calculations or algebraic manipulations are likely to be relevant at each stage, in order to chart a path towards the solution. This component is often based on a heuristic guess (or in the case of humans, intuition), and is often considered to be the most substantial and profound aspect of mathematical problem-solving.
Technical proficiency: The ability to perform routine calculations or manipulations that follow a prescribed set of steps (such as differentiating a function or isolating a term in an equation).
Critical reasoning: The ability to critically examine each step of the argument, break it down into its sub-components, explain what it entails, how it is related to the rest of the argument and why it is correct. When solving a problem or producing a mathematical argument, this usually comes together with the ability to backtrack when a certain step is realized to be incorrect and modify the argument accordingly.

We now want to analyze the model’s performance in each of these aspects of mathematical understanding, and discuss some possible reasons for its strengths and weaknesses.

Creative reasoning. When it comes to advanced high-school level problems (and occasionally higher level), the model demonstrates a high level of ability in choosing the right argument or path towards the solution. To relate this to the example above, the model correctly chooses to try and write recurrence relations in the original question, and to argue about the degrees of compositions of polynomials in the follow-up question. In both cases, the suggestion is made before “knowing” whether or not this path is going to lead to the correct solution. Section 4.2 and Appendix D contains more examples demonstrating the model’s capabilities in this aspect, which we compare to that of a good high-school student or even higher.

Technical proficiency. While the model clearly demonstrates a high degree of knowledge of the algorithms related to different procedures (such as solving a system of equations), it also makes very frequent mistakes when performing these tasks, such as making arithmetic mistakes, confusing the order of operations or using incorrect notation. We further discuss some examples of these typical errors in Appendix D.1. We speculate that this aspect could be improved by giving the model access to code execution, which would allow it to perform calculations or check equivalences more accurately; some evidence for this is provided in Appendix D.

Critical reasoning. The model exhibits a significant deficiency in the third aspect, namely critically examining each step of the argument. This could be attributed to two factors. First, the training data of the model mainly consists of questions and their solutions, but it does not capture the wording that expresses the thinking process which leads to the solution of a math problem, in which one makes guesses, encounters errors, verifies and examines which parts of the solution are correct, backtracks, etc. In other words, since the training data is essentially a linear exposition of the solution, a model trained on this data has no incentive to engage in an “inner dialogue” where it revisits and critically evaluates its own suggestions and calculations.

Second, the limitation to try things and backtrack is inherent to the next-word-prediction paradigm that the model operates on. It only generates the next word, and it has no mechanism to revise or modify its previous output, which makes it produce arguments “linearly”.

Loosely speaking, we can therefore see the drawbacks of the model as a combination of “naive” attention mistakes with more fundamental limitations due to its “linear thinking” as a next-token prediction machine. An important question is which of the above issues can be alleviated by further training (perhaps with a larger model). For the former problem, we believe that further training could alleviate the issue, as evidenced by the super-human coding abilities where such attention mistakes would also be fatal; a key difference is that GPT-4 was most likely trained on much more code than mathematics data. We believe that the latter issue constitutes a more profound limitation. We discuss it in more detail in Section 8.

In the remainder of the section, we assess the model’s capabilities on commonly used benchmarks for mathematical problem solving and demonstrate the model’s capability of applying quantitative thinking in real-world scenarios. We also compare the performance of GPT-4 and ChatGPT on both benchmarks and other mathematical problems (more examples in Appendix D). Roughly speaking, we find that GPT-4 demonstrates a significant improvement over ChatGPT: GPT-4 shows a deeper understanding of the problem and is able to apply the appropriate reasoning in many complicated problems. ChatGPT, on the other hand, often resorts to low-level heuristics, mentioning formulas and concepts that are only superficially related to the problem which point to a lack of actual comprehension. We end the section with a few examples demonstrating the capabilities on higher level mathematics.

4.2 Performance on mathematical problem datasets

We now conduct systematic experiments to compare the performance of GPT-4, ChatGPT and Minerva (stateof-the-art LLM for solving math questions) on two math data sets which are commonly used as benchmarks: GSM8K [CKB+21] and MATH [HBK+21]. GSM8K is an elementary school math data set that contains 8,000 questions and answers on topics such as arithmetic, fractions, geometry, and word problems. MATH is a high school math data set that contains 12,500 questions and answers on topics such as algebra, calculus, trigonometry, and probability. We also test the model on MMMLU-STEM data set, which contains around 2000 multiple choices (4 choices) questions covering high school and college STEM topics. These data sets highlight GPT-4’s ability to use the correct approach to solve high school-level math questions.

Mitigating over-fitting. A potential concern for using benchmarks to evaluate the reasoning abilities of LLMs is that they might have memorized the questions or answers for the benchmark datasets during its pre-training, as those questions are potentially part of the dataset. To reduce the likelihood of this form of overfitting, we adopt the following strategies:

In the benchmarks, we tested GPT-4 by asking it to (1) write a template for the solution of the problem, (2) write down the steps first then write down the final answer. The templates are not available online, and detailed solutions for datasets such as MMMLU-STEM are also not online (only the answer is).
We pick one representative question from the data set where GPT-4 solves correctly and text-davinci-003 solves incorrectly. We alter the numbers in the question and see that GPT-4 is consistently correct and text-davinci-003 is consistently wrong.
We also crafted several new questions, carefully checking that these questions or close variants do not appear online. It appears that GPT-4 demonstrates the same performance on those questions.

GPT-4 Generating Template
Prompt: Andy harvests all the tomatoes from 18 plants that have 7 tomatoes each. If he dries half the
tomatoes and turns a third of the remainder into marinara sauce, how many tomatoes are left?
Can you generate a template solution for the problem without numbers?
Using symbols, the template solution could look something like this:
- Let P be the number of plants and T be the number of tomatoes per plant. Then the total number of
tomatoes harvested is P x T.
Let D be the number of tomatoes that are dried. Then D = (P x T) / 2.
Let M be the number of tomatoes that are turned into marinara sauce. Then M = ((P x T) - D) / 3.
- Let L be the number of tomatoes that are left. Then L = (P x T) - D - M.
· · ·

GPT-4 memorization test by alternating the numbers
Prompt: If a degree 3 polynomial satisifies p(x) = 0 for x = -3, 8, 5 and p(1) = 10, what is |p(0)|?
Figure 4.2: One way to test whether GPT-4 memorizes the exact statement of the original problem is to vary the values of x and p(1) in the input. We randomly select three values of x from the set {−10, −9, · · · , −2} ∪ {2, 3, · · · , 10} and one value of p(1) from the set {−10, −9, · · · , −1} ∪ {1, 2, · · · , 10}, and use them to construct new inputs. We compare the accuracy of GPT-4 and text-davinci-003 on these inputs. The results show that GPT-4 achieves an accuracy of 75.2%, while text-davinci-003 only has an accuracy of 0.2%. This suggests that GPT-4 does not rely on memorizing the exact problem statement but on applying a general solution method. While it is possible that GPT-4 memorizes the solution template, this is not necessarily a flaw, as it is also a common way of solving math problems for humans.

For the benchmark datasets, we evaluated the models on their single model accuracy, which is the percentage of questions that they answered correctly in one try. The results are shown in the following table:

Model	GSM8K	MATH	MMMLU-STEM
text-davinci-003	61.3%	23.5%	54.2%
Minerva	58.8%	33.6%	63.9%
GPT-4	87.1%	42.5%	82.7%

Table 3: Accuracy of different models on math data sets

GPT-4’s accuracy shows a modest improvement over other models, but a manual inspection of GPT-4’s answers on MATH reveals that GPT-4’s errors are largely due to arithmetic and calculation mistakes: the model exhibits large deficiency when managing large numbers or complicated expressions. In contrast, in most cases, the argument produced by ChatGPT is incoherent and leads to a calculation which is irrelevant to the solution of the problem to begin with. Figure 4.3 gives one example which illustrates this difference. We further discuss the issue of calculation errors in Appendix D.1.

Error type	Percentage of wrong solutions
Arithmetic mistakes (including counting) Misunderstanding the problem statement Wrong approaches	68% 10%

Table 4: Manual inspection of GPT-4 errors in MATH dataset on a random selection of 100 wrong answers. We can see that GPT-4 uses the correct approach on the significant majority of the questions.

Figure 4.3: GPT-4 vs ChatGPT on AP problems. GPT-4 uses a correct approach, but produces a wrong final answer due to a calculation error, whereas ChatGPT produces an incoherent argument.

4.3 Mathematical modeling in various domains

Mathematical reasoning is more than a skill for solving mathematical exercises and problems; it is also a tool for understanding and communicating about various contexts and situations. In this section, we assess GPT-4’s capacity to use mathematical ideas and techniques to address real-world problems, some of which are not strictly mathematical, but demand quantitative thinking. We show below how GPT-4 successfully builds a plausible mathematical model for a complex system that requires extensive interdisciplinary knowledge, while ChatGPT fails to make meaningful progress.

Figure 4.4: GPT-4 builds a mathematical model for profiling power rate over time of a professional StarCraft 2 player during a match.

Fermi questions. A specific type of mathematical modeling problems are Fermi questions. Fermi questions involve making educated guesses about quantities or phenomena that are difficult or impossible to measure directly, using logic, approximation, and order-of-magnitude reasoning. For example, some wellknown Fermi questions are: “How many piano tuners are there in Chicago?” and “How many times does the electric field of the sun’s peak color oscillate during its journey to earth?”. To solve these problems with little or no additional information, it requires both quantitative thinking and general knowledge. Here we demonstrate GPT-4’s outstanding ability to answer Fermi questions with quantitative thinking. We show two examples in Figure 4.5, where GPT-4 attempts at them by making layers of reasonable assumptions and informed guesses, while ChatGPT admits defeat immediately barely showing trace of mathematical thinking. Even though GPT-4’s final answers are likely to be far off, it demonstrates an impressive angle of attack at these problems.

Figure 4.5: GPT-4 answers Fermi questions.

4.4 Higher level mathematics

We conclude this section with a few examples that demonstrate the model’s potential performance on more advanced mathematical topics. These examples have been intentionally selected to demonstrate the model’s capabilities, and it is important to note that the model does not always succeed with questions of this level of difficulty. Instead, they are meant to showcase the extent of the model’s capabilities, suggesting what future models might be able to achieve.

We begin with a simplification of a question which appeared in the 2022 International Mathematics Olympiad (IMO).

GPT-4
Prompt:
Suppose g is a continuous smooth function such that for every x > 0, there is one and only one y > 0 such that
g(x) + g(y) ≤ 2xy. Can you prove that g(x) = x2?
GPT-4:
There are a few possible ways to approach this problem, but one possible method is as follows:
- Assume, for the sake of contradiction, that g(x) is not equal to x2 for some x > 0.
- Then, either g(x) < x2 or g(x) > x2 for that x.
- If g(x) < x2, then let y = x. Then g(x) + g(y) = 2g(x) < 2x2 = 2xy.
- If g(x) + g(y) < 2xy, then g(x) + g(y) is strictly less than the upper bound of 2xy for positive x and y.
- This means that there is some room to increase y slightly and still satisfy the inequality g(x) + g(y) ≤ 2xy.
- For example, let y0 = y + , where is a small positive number such that g(x) + g(y0) < 2xy. This is possible because
g is continuous and smooth, so it does not have any jumps or kinks that would prevent such a choice of epsilon.
- Then, y0 is another positive number such that g(x) + g(y0) ≤ 2xy0, which contradicts the assumption that y is the one
and only one positive number that satisfies this condition given x.
- Therefore, if g(x) < x2, then g(x) + g(y) cannot be strictly less than 2xy, and it must be equal to 2xy.
- If g(x) > x2, then let y∗ be the unique positive solution of g(x) + g(y) = 2xy. Then g(y∗) < (y∗)2, since otherwise we
would have g(x) + g(y∗) > 2xy, which contradicts the existence of y∗ given x.
- But then, by the same argument as before, we can let x = y = y∗ and obtain a contradiction, since g(y) + g(y) =
2g(y) < 2y2 = 2yy, which contradicts the uniqueness of y given y.
- Therefore, g(x) must be equal to x2 for all x > 0, as required.

What distinguishes this question from those that typically appear in undergraduate calculus exams in STEM subjects is that it does not conform to a structured template. Solving it requires a more creative approach, as there is no clear strategy for beginning the proof. For example, the decision to split the argument into two cases (g(x) > x2 and g(x) < x2) is not an obvious one, nor is the choice of y∗ (its reason only becomes clear later on in the argument). Furthermore, the solution demands knowledge of calculus at the undergraduate level. Nevertheless, GPT-4 manages to produce a correct proof.

The second example is in the form of a discussion which concerns algorithms and graph theory, topics typically covered in the first or second year of an undergraduate computer science degree. The discussion is comparable to that of a graduate-level interview.

Figure 4.6: A discussion regarding k-SAT problems and graph theory.

GPT-4 demonstrates understanding of the concepts from graph theory and algorithms. It is able to reason about an abstract graph construction, which relates to a constraint satisfaction problem, and deduce correct conclusions about the SAT problem (to the best of our knowledge, this construction does not appear in the mathematical literature). The conversation reflects profound understanding of the undergraduate-level mathematical concepts discussed, as well as a significant extent of creativity. Although GPT-4 makes a mistake in one instance (writing 2n−1 rather than 2n/2), the subsequent conversation demonstrates that this error does not reflect a lack of understanding. Rather, GPT-4 seems to have made an error in a manner similar to a human typo, as it later provides the correct generalization of the formula.

Our last example requires the grasp of a new mathematical definition, and combines knowledge in both number theory and probability. The model produces a sound argument but makes a counting mistake at the end which leads to an incorrect final answer.

GPT-4

Prompt:

A good number is a 5-digit number where the 1,3,5-th digits are odd numbers and they form an increasing arithmetic progression, and the number is divisible by 3. If I randomly sample a good number, what is the probability that its 2nd digit is 4?

GPT-4:

There are a few ways to approach this problem, but one possible method is to use some modular arithmetic and enumeration. First, we can observe that a 5-digit number is divisible by 3 if and only if the sum of its digits is divisible by 3. So, if we let the 5-digit number be abcde, where a, c, and e are odd digits forming an arithmetic progression, we have the equation:

We can also observe that the difference between any two consecutive terms in an arithmetic progression is constant, so we can let that difference be f, where f is a positive integer. Then we have:

Substituting these into the equation above, we get:

Since 3a and 3f are always divisible by 3, we can ignore them and focus on the equation:

The choice of a, f and e does not affect this equation, so we can fix them and vary b and d. For example, let a = 1, f = 2, and e = 5. Then we have the possible values of b and d as:

We can see that there are 10 possible values of b, and for each value of b, there are 4 possible values of d. So, there are 10 × 4 = 40 possible pairs of b and d that satisfy the equation. Out of these 40 pairs, only 3 have b = 4, namely (4, 2), (4, 5), and (4, 8). So, the probability that b = 4 is 3/40.

5 Interaction with the world

One of the key aspects of intelligence is interactivity, which we define as the ability to communicate and respond to feedback from other agents, tools, and environments. Interactivity is important for intelligence, as it enables agents to acquire and apply knowledge, solve problems, adapt to changing situations, and achieve goals that are beyond their individual capabilities. For example, humans interact with one another and with environments to collaborate, learn, teach, negotiate, create, etc. Interactivity requires an agent to comprehend complex ideas, learn quickly, and learn from experience, and thus it is closely tied to our definition of intelligence.

In this section, we explore two dimensions of interactivity: tool use and embodied interaction. Tool use involves the use of external resources, such as search engines, calculators, or other APIs, to perform tasks that are difficult or impossible for the agent alone. Embodied interaction involves the use of natural language as a text interface to interact with simulated or real-world environments, and to receive feedback from them.

5.1 Tool use

Despite impressive performance on various tasks in the previous sections, GPT-4 still suffers from various well-documented weaknesses of language models. These weaknesses include (but are not limited to) lack of current world knowledge, difficulty with symbolic operations (e.g., math), and inability to execute code. For example, in Figure 5.1, GPT-4 uses out-of-date information to answer the first question and fails to perform the appropriate operations for the second and third questions. ChatGPT refuses to answer the first question, and also fails on the others.

Figure 5.1: Current knowledge and symbolic manipulation are hard for language models.

However, GPT-4 is able to use external tools such as search engines or APIs to overcome these (and other) limitations. For example, in Figure 5.2, we show a simple prompt that gives GPT-4 access to a search engine and other functions. During execution, when one of these functions is called, we pause generation, call the appropriate function, paste the results back into the prompt, and continue the generation. In these simple examples, GPT-4 is able to use the tools with very minimal instruction and no demonstrations, and then make use of the output appropriately (note how the second search result contains potentially conflicting information, and GPT-4 is still able to infer the right answer). In contrast, ChatGPT (not shown) does not consistently change its answers to the questions in Figure 5.1 after being instructed to use the tools – it still refused to answer the first question; for the other two, it sometimes does not call the tools at all, and sometimes calls them after giving an incorrect answer. While in Figure 5.2 we specify which tools are available, GPT-4 can also list which tools (or API functions) are needed in order to solve a task (example in Appendix, Figure F.2, where GPT-4 lists four API functions needed to complete a task and then proceeds to use them effectively).

Figure 5.2: Examples of GPT-4 using various tools to answer questions.

5.1.1 Using multiple tools to solve more complex tasks

Solving more complex tasks requires GPT-4 to use multiple tools in combination. We now share examples where GPT-4 is able to do this by relying on its ability to understand the task at hand, identify the tools needed, use them in the correct order, and respond appropriately to their output.

Hacking. In Figure F.3 (Appendix), we tell GPT-4 it can execute commands on a Linux distribution designed for digital forensics and penetration testing, and task it with hacking into a computer on the local network. Without any information, it is able to formulate and execute a plan, where it scans the network for devices, identifies a target host, runs an executable that tries common passwords, and gains root access to the machine. While the machine was easy to hack into, we note that GPT-4 is well-versed in Linux commands, and is able to run the appropriate commands, interpret their output, and adapt in order to solve its goal. ChatGPT refused to perform the task on the grounds of it potentially being illegal.

Figure 5.3: GPT-4 plays the role of a Zoo manager to complete tasks in the command prompt.

Managing a zoo through command line instructions. GPT-4 may have seen near-copies of the previous example in its training data. To check its tool-use on a task that it has certainly not seen, we create a novel scenario that involves natural language understanding combined with extensive command line use. In the scenario, we have GPT-4 play the role of a zoo manager, completing a sequence of six tasks specified in a file (See Figure 5.3, where GPT-4 starts by correctly issuing the command type todo.txt). To complete the tasks, GPT-4 has to manipulate files and folders representing different animals, areas, and information about the zoo, requiring it to understand both the task at hand (e.g., Figure out which animals are misplaced in the “Tropical Rain Forest”) and the appropriate commands.

Despite the breadth of the challenge (more than 100 commands were required to complete all tasks), GPT-4 was able to solve almost all tasks. Its only failure was fabricating content when replying to emails, rather than reading the specified content from a file (Appendix F.1.1), a problem fixed by a simple tweak to the prompt (Appendix F.1.2). While GPT-4 often displayed ingenuity (e.g., running breadth-first search manually to navigate directories), it often ran incorrect commands, such as deleting a file that had spaces in its name (e.g., “Polar Bear.txt”) without adding quotation marks. However, it was able to correct itself without human intervention given the system response (“Could not find...”). Interestingly, GPT-4 made the same mistake in subsequent files with spaces (and always applied the same correction), even though it could predict what the error message would be given the wrong command. Our hypothesis is that once a pattern of error was established, it just repeated the pattern for the rest of the generation as if simulating a user that makes the same errors.

Managing calendar and email. In Figure 5.4, we illustrate how GPT-4 is able to use multiple tools in combination to manage a user’s calendar and email. The user requests that GPT-4 coordinate a dinner with two other people, and book it on a night when the user is free. GPT-4 uses the available APIs to retrieve information about the user’s calendar, coordinate with other people over email, book the dinner, and message the user with the details. In this example, GPT-4 demonstrates its ability to combine multiple tools and APIs, as well as reason about free-form outputs in order to solve a complex task (e.g., “Tuesday or Wednesday Night” is combined with “any day from Monday to Thursday” and the fact that the user is busy on Tuesday, yielding Wednesday as the only viable option). ChatGPT (not shown) was not able to accomplish the same task, and instead wrote a function where “joe@microsoft.com” emails “luke@microsoft.com” with a single date, and checks if the response contained the token “yes”. ChatGPT was also not able to respond when the output of its function was given it.

Browsing the web for information. In Figures 5.5 and 5.6, GPT-4 uses a search engine and a SUMMARIZE function (which downloads the webpage and calls itself to summarize it given the question at hand) to browse the web and answer questions. In both cases, GPT-4 is able to identify relevant search results to look at with more depth, summarize them, and provide accurate answers, even when the question contains a false premise. While prior versions of LLMs can also be taught to browse the web [NHB+21], we note that GPT-4 is able to do so without any fine-tuning or demonstration.

Figure 5.4: Email and calendar execution

Figure 5.5: Browsing the web

Figure 5.6: Browsing the web with true and false premises

Using unusual tools, a failure case. In Figure 5.7, we ask GPT-4 to solve an easy task using a very unusual API. Instead of adapting to the unusual function, GPT-4 calls them as if they were the usual version, i.e., it calls reverse get character as if it were get character, and reverse concat as if it were a simple concat. ChatGPT yields the same function, except that it does not check if the length of the word is greater or equal to 3. However, when the user says there are errors, GPT-4 is able to spot and fix them, while ChatGPT (not shown) is not able to find or fix its own mistakes with the same prompt.

5.1.2 Discussion

The examples in this section show that GPT-4 is capable of both identifying and using external tools on its own in order to improve its performance. It is able to reason about which tools it needs, effectively parse the output of these tools and respond appropriately (i.e., interact with them appropriately), all without any specialized training or fine-tuning.

We now note a few limitations. First, GPT-4 still requires a prompt that specifies it is allowed or expected to use external tools. In the absence of such a prompt, its performance is limited by the weaknesses inherent in LLMs (e.g., weak symbolic manipulation, limited current world knowledge, Figure 5.1). Second, even with access to tools, GPT-4 is not always able to reason about when it should use them and when it should simply respond based on its own parametric knowledge, e.g., it still used a search engine when we asked for the capital of France (not shown), even though it could certainly answer correctly without the search results. Third, the zoo example revealed a repeated error pattern, while Figure 5.7 was an example of failure to use unusual tools. However, in both of these cases, GPT-4 was able to fix the problem after receiving a response from the environment (either the command line or the user), yet another example of its power of interactivity. As we noted throughout, ChatGPT was unable to perform at a similar level of interactivity, often ignoring the tools or their responses, and preferring generic answers.

5.2 Embodied Interaction

While tool use is an important aspect of interactivity, most interaction in the real world does not happen through APIs. For example, humans are able to use natural language to communicate with other agents, to explore and manipulate their environment, and to learn from the consequences of their actions. Such embodied interaction requires an agent to understand the context, the goals, the actions, and the outcomes of each turn of interaction, and to adapt accordingly. While GPT-4 is obviously not embodied, we explore whether it can engage in embodied interaction by using natural language as a text interface to various simulated or real-world environments.

5.2.1 Warmup: navigating a map

In Figure 5.8, we prepare a “map” of a house, and ask GPT-4 to explore it through interactive queries. We then ask it to describe it in language and in a visualization, and compare that to the true map. While it does not explore the whole house, GPT-4 accurately describes what it does explore, even though all of its interactions are through this restricted text interface.

5.2.2 Text-based games

Text-based games are a natural and challenging domain for language models, as they require understanding natural language, reasoning about the game state, and generating valid commands. A text-based game is a type of interactive fiction, where an agent interacts with an environment through natural language descriptions and commands. The agent has to perform a given task, such as finding a treasure or escaping a dungeon, by exploring the environment and manipulating objects. We first test if GPT-4 can explore the environment in text-based games to perform the given task. In this and the next experiments, we use TextWorld [CKY+18], a framework for generating and playing text-based games, to create two games with different given tasks.

GPT-4 fails to use unusual toolsA computer is going to solve a user’s instructions using APIs. The following API functions areavailable.- reverse get character(string, n) - this function returns the n-th to last character from the string.For example, reverse get character(’cat’, 0) returns ’t’, while reverse get character(’cat’, 2) returns’c’- reverse concat(list) - This function concatenates the strings in the list in reverse order, andreturns that.- len(string) - returns the length of the string- reverse(list) - returns the list in reverse orderUser instruction: Take the third letter of each word in the sequence ’dog man super great azuli lotus’,concatenate them in a string (in original order), and return them.Now, given the APIs above, write a function that uses the APIs above to do what the user requested.<|endofprompt|>Possible function:def user_function(sequence):# split the sequence into a list of wordswords = sequence.split()# create an empty list to store the third lettersletters = []# loop through the wordsfor word in words:# check if the word has at least three lettersif len(word) >= 3:# use reverse_get_character to get the third letter and append it to the listletters.append(reverse_get_character(word, 2))# use reverse_concat to concatenate the letters in reverse orderresult = reverse_concat(letters)# return the resultreturn resultWait, there are errors here. What are they?There are two errors in the function:- The reverse get character function returns the n-th to last character, not the n-th character. Therefore, to get the third letter, we should use reverse get character(word, len(word) - 3) instead ofreverse get character(word, 2). - The reverse concat function concatenates the strings in the listin reverse order, but the user instruction wants them in original order. Therefore, we should usereverse concat(reverse(letters)) instead of reverse concat(letters).A corrected function could look like this:def user_function(sequence):# split the sequence into a list of wordswords = sequence.split()# create an empty list to store the third lettersletters = []# loop through the wordsfor word in words:# check if the word has at least three lettersif len(word) >= 3:# use reverse_get_character to get the third letter and append it to the listletters.append(reverse_get_character(word, len(word) - 3))# use reverse_concat to concatenate the letters in reverse order and then reverse the resultresult = reverse_concat(reverse(letters))# return the resultreturn result

Figure 5.7: GPT-4 uses unusual functions incorrectly, but is able to spot and fix errors when prompted to do so.

Figure 5.8: GPT-4 navigates a map interactively, then describes it in different modalities

Exploring the environment. The first game takes place in a house with many rooms, and the objective is to find and unlock a certain chest by navigating between different rooms. This game is relatively simple, as it does not involve any inventory management, crafting, or combat. The environment is described by a text paragraph, and the player can type commands such as “go north”, “examine couch”, or “open chest”. The chest is usually few rooms away from the starting point, and thus solving the game requires exploring

Figure 5.9: The left figure shows the last two commands from GPT-4 and the environment response for the first game in Section 5.2.2. The right figure shows the route GPT-4 takes.

Figure 5.10: Examples of GPT-4 reacting to the environment in the second game in Section 5.2.2

the environment without getting lost.

We give the exact initial text from Textworld as the initial prompt, and issue help as a first command. After that, GPT-4 plays the game as a normal player. It completes the game in 30 actions without any extra help, and navigates the environment efficiently without looping through rooms (Fig. 5.9 contains an excerpt and the route taken, and Appendix F.2.1 has the complete log). Qualitatively, GPT-4 solves the game in a very conservative manner, by examining and picking up every object in each room regardless of its relevance to the game objective. In contrast, however, text-davinci-003 does not respond to the environment feedback at all, issuing the same command repeatedly (Appendix F.2.2).

Responding to feedback. Here GPT-4 plays a game where the task is to prepare a two-ingredient, five-step meal according to a cookbook. This game is more challenging than the previous one because the player (GPT-4) has to figure out key commands by itself from the environment response (these are not listed in help), such as cutting the food, using the right utensils, and turning on the appliances (see Fig. 5.10).

GPT-4 uses trial and error when playing the game, but it also adapts to the environment and generalizes between actions. For example, it learns that the chop command requires a knife in Fig. 5.10, and it does not make the same mistake afterward. It follows the instructions from the cookbook, but also infers some missing actions such as taking the required ingredients. While the game does not specify that key commands are missing, GPT-4 makes reasonable guesses, e.g., when it fails to grab a frying pan from the kitchen, it goes into a bedroom to look for a frying pan (more details in Appendix F.2.3). GPT-4 cannot solve this game, but neither could the authors of this paper (who got stuck in the same step) prior to looking at the source code. However, if we give GPT-4 a one-shot demonstration of creating a different meal, it is able to generalize from it and solve the game (Appendix F.2.4).

5.2.3 Real world problems

In Fig. 5.11 and Fig. F.1, GPT-4 is given two real-world problems to solve, and given a human as a partner (i.e., a very flexible agent with very little constraints, who can also respond in natural language) to interact with the environment. Both problems were real problems faced by authors of this paper, who responded to GPT-4 in such a way as to track the situation they faced. For both problems, GPT-4 is able to identify the actions the human needs to take in order to address the problem. In the first example, GPT-4 guides the human to find and fix a water leak and recommends the exact actions that the human took (after replacing the seal, the leak was gone). In the second example, the author had not called the gas company to transfer service from the old user, and thus the gas was turned off. In this case, GPT-4 is able to quickly arrive at the source of the problem (the actual human took a long time to check the pilot light in the water heater), and propose plausible solutions. However, it is not able to diagnose the root cause until the human himself has the idea to check the stove-top.

Figure 5.11: GPT-4 acts as a virtual handyman.

5.2.4 Discussion

While it is clearly not embodied, the examples above illustrate that language is a powerful interface, allowing GPT-4 to perform tasks that require understanding the environment, the task, the actions, and the feedback, and adapting accordingly. While it cannot actually see or perform actions, it can do so via a surrogate (e.g., a human). Having said this, we acknowledge the limitation that we only tested GPT-4 on a limited number of games and real-world problems, and thus cannot draw general conclusions about its performance on different types of environments or tasks. A more systematic evaluation would require a larger and more diverse set of real world problems where GPT-4 was actually used in real-time, rather than retrospectively.

6 Interaction with humans

6.1 Understanding Humans: Theory of Mind

Theory of mind is the ability to attribute mental states such as beliefs, emotions, desires, intentions, and knowledge to oneself and others, and to understand how they affect behavior and communication [Wel92]. It includes the basic task of reflecting on someone else’s mental states, and the more advanced task of reflecting on someone’s reflection of someone else’s mental state (and so on). An example of the former skill is is needed to answer the question “What does Alice believe?”, while an example of the latter is needed to answer “What does Bob think that Alice believes?” Theory of mind is essential for effective communication and cooperation with other intelligent agents, as it allows one to infer their goals, preferences, motives, and expectations, and to adjust one’s own actions and utterances accordingly. Moreover, theory of mind is also important for learning from others, as it enables one to interpret their feedback, advice, and demonstrations.

6.1.1 Testing specific aspects of theory of mind

We designed a series of tests to evaluate the theory of mind capabilities of GPT-4, ChatGPT, and text-davinci-003. The tests are based on simple scenarios that require more basic or more advanced theory of mind to answer questions about the mental states of characters involved.

We start with a modernized version of the Sally-Anne test [BCLF85], a classic false-belief test that is widely used to assess theory of mind in children. To prevent an unfair comparison due to the effects of memorization, we modify the test by framing it in a situation that does not exist on the web, and thus could not have been seen during training. Figure 6.1 shows the input and output for GPT-4, which correctly answers that Alice will look for the file in the original folder, demonstrating it can reason about Alice’s beliefs. ChatGPT also answers correctly (not shown), while text-davinci-003 gives a wrong answer, saying that Alice will look for the file in the new folder.

We present a test on understanding emotions in Figure 6.2, where two characters talk about an object called ZURFIN (we use a nonsense word to test abstraction and prevent memorization). GPT-4 is able to reason correctly about the reasons for Tom’s emotional state, and also make good inferences about Adam’s beliefs about Tom’s emotional state (based on incomplete information). ChatGPT also passes the test, while text-davinci-003 (not shown) makes no reference to the conversation when answering the first question, and fails to account for Adam’s lack of information about the lost ZURFIN when answering the second question.

The third test (Figure 6.3) involves inferring possible intentions in the light of a puzzling action by one of the characters. GPT-4 gives plausible and nuanced answers both for the intentions behind the puzzling action and a third-party’s likely interpretation of the puzzling action. ChatGPT gives a similar answer to the first question (not shown), but unlike GPT-4, it does not offer a nuanced response to the second question, instead providing a more general and less informative answer. text-davinci-003 gives plausible but very short answers to both questions (not shown).

6.1.2 Testing theory of mind in realistic scenarios

In Figures 6.4, 6.5, and 6.6 we present realistic scenarios of difficult social situations, requiring very advanced theory of mind to understand. We ask probing questions, and also ask the models to propose actions that are likely to improve the situation, which require inferences about the counterfactual impact of actions on mental states.

In Figure 6.4, GPT-4 is able to infer what each character’s mental state is, and also discern where miscommunication and misunderstanding lies. In contrast, both ChatGPT and text-davinci-003 (not shown) incorrectly accept a mistaken assumption made by one of the characters (Judy’s assumption that Mark wants to defend Jack’s behavior), and thus fail to understand the real dynamics of the situation. In turn, this leads to generic suggestions for improvement from ChatGPT and text-davinci-003, while GPT-4 provides suggestions that actually address the root cause of the misunderstanding.

We see a similar pattern in Figures 6.5 and 6.6. Without implying that there is a “right” answer, we note that GPT-4 provides more nuanced answers, taking the whole scenario and actors into account. In contrast, ChatGPT provides more general answers which do not include reasoning about the characters’ state of mind (text-davinci-003 is similar, but shorter than ChatGPT).

Figure 6.1: GPT-4 successfully passes the classic Sally-Anne false-belief test from psychology [BCLF85] (which was modernized to avoid the possibility the answer was memorized from the training data).

Figure 6.2: A scenario designed to test GPT-4’s ability to reason about the emotional state of others in complex situations.

Figure 6.3: A comparison of GPT-4’s and ChatGPT’s ability to reason about the intentions of people in complex social situations.

Figure 6.4: A realistic scenario of marital struggle, requiring a lot of nuance to navigate.

Figure 6.5: A challenging family scenario, GPT-4

Figure 6.6: A challenging family scenario, ChatGPT

6.1.3 Discussion

We presented a series of tests to evaluate the theory of mind capabilities of GPT-4, ChatGPT, and text-davinci We have shown that GPT-4 outperforms the other two models in both basic and realistic scenarios that require reasoning about the mental states of others, and in proposing actions for cooperation towards common goals in social situations. We have also shown that GPT-4 is able to handle abstract and novel situations that are not likely to have been seen during training, such as the modernized Sally-Anne test and the ZURFIN scenario. Our findings suggest that GPT-4 has a very advanced level of theory of mind. While ChatGPT also does well on the basic tests, it seems that GPT-4 has more nuance and is able to reason better about multiple actors, and how various actions might impact their mental states, especially on more realistic scenarios.

As far as limitations, our tests are not exhaustive or comprehensive, and may not cover all the possible aspects or dimensions of theory of mind. For example, we did not test for the ability to understand sarcasm, irony, humor, or deception, which are also related to theory of mind. Being based on textual input and output, our tests do not capture the full complexity and richness of natural communication and social interaction. For example, we did not test for the ability to understand non-verbal cues, such as facial expressions, gestures, or tone of voice, which are also important for theory of mind.

6.2 Talking to Humans: Explainability

The ability to explain one’s own behavior is an important aspect of intelligence, as it allows for a system to communicate with humans and other agents. Self explanation is not only a form of communication, but also a form of reasoning, requiring a good theory of mind for both yourself (the explainer) and the listener. For GPT-4, this is complicated by the fact that it does not have a single or fixed “self” that persists across different executions (in contrast to humans). Rather, as a language model, GPT-4 simulates some process given the preceding input, and can produce vastly different outputs depending on the topic, details, and even formatting of the input.

For the sake of exposition, we assume GPT-4 is being used to solve a task T , given input x and context c (which includes everything in the prompt other than x, e.g. instructions, prior chat history, etc). We use the notation PT (y|x, c) to refer to the process it is trying to simulate, where y is the output. We further define PE(e|x, c, y) as the explanatory process GPT-4 has to simulate to produce a post-hoc explanation, i.e. GPT-4 generates an explanation e for output y given x, c. All three components (x, c, and y) can significantly impact the explanation e. Figure 6.7 illustrates how the context c (in this case, the QA format and the preamble in the second task) can drastically impact how GPT-4 simulates PT and PE. It also shows how PE depends on the actual generated y, such that if the output were different, the explanation would have to change accordingly, as illustrated by the third session where we force the output to be “1400”. As these examples illustrate, simulating PT (y|x, c) is not necessarily the same as solving the user’s task T , but rather it is a process that produces y given x, c. Prompt engineering typically tries to set up (x, c) such that GPT-4’s simulation of PT (y|x, c) approximates the task of interest well enough for the user’s purpose. Similarly, it is worth noting that PE(e|x, c, y) can be customized via the context c to create personalized explanations for each end user. For example, explaining concepts to a five year old vs. a machine learning researcher requires different PE. Note that we have simplified the notation here for the sake of clarity, as many tasks do not have a single “input” x that is perfectly separable from the rest of the context c.

What makes an explanation good? One possible way to evaluate the quality of an explanation is to check output consistency, i.e. whether the explanation is consistent with the output y given the input x and the context c. In other words, an output-consistent explanation provides a plausible causal account of how y was derived from x and c. By this criterion, GPT-4 is remarkably good at generating reasonable and coherent explanations, even when the output is nonsensical or wrong, as illustrated by the third session in Fig. 6.7, and the example in Fig. 6.8. In Figure 6.9, we contrast GPT-4 with text-davinci-003, and note that the latter produces an explanation that is not output-consistent (as it does not address the choice of the letter Q).

Another possible way to evaluate the quality of an explanation is to check whether it is consistent with GPT-4’s simulation of PT , i.e. whether it gives us the ability to make predictions about the future behavior of the model under different inputs (or even different contexts). We call this process consistency, and it is often what humans expect or desire from explanations, especially when they want to understand, debug, or assess trust in a system. We can evaluate process consistency by creating new inputs where the explanation should predict the behavior, as shown in Figure 6.10 (where GPT-4 is process-consistent). However, we note that output consistency does not necessarily lead to process consistency, and that GPT-4 often generates explanations that contradict its own outputs for different inputs in similar contexts. For example, in Figure 6.11, the explanation in both sessions is output-consistent, but not entirely process-consistent (the translation is only consistent for three out of the four professions listed in the first session’s explanation).

Figure 6.7: Output and explanations depend on the context and input. Explanations further depend on the generated output.

Figure 6.8: An example of an output-consistent explanation for a wrong answer, based on Figure 1.8.

Figure 6.9: Asking for an explanation for a choice in the output of Fig. 2.2. GPT-4’s explanation provides insight into the mechanism used to generate the symbol name “Q” (i.e. it is outputconsistent), while GPT 3.5 (text-davinci-003) misinterprets the question. The process-consistency of GPT-4’s explanation is tested with experiments in Figure 6.10.

Testing Poetry Writing Explanations for Process Consistency
Editing Experiment: One way to test GPT-4’s explanation from Figure 6.9 is to change
the previously used symbol in the poem from P to R. If GPT-4’s explanation is accurate,
this should reduce the likelihood of generating Q and increase the likelihood of S. We also
note that while some alphabetic order effect is present for text-davinci-003, GPT-4’s
explanation is a better representation of GPT-4’s own behavior.
GPT-4:
... The largest one we’ll see in time.
Let’s call this number "R" for short,
And then, my friend, we will purport
To build a number "S [64.63%]
Q [22.61%]
N [7.71%]
Concept Override Experiment: Another way to test an explanation is to override the
model’s background knowledge through language patches [MMLR22]. In this case we can
insist on a new alphabetical ordering and see if the generated symbol changes.
Prompt Prefix 1: Prompt Prefix 2
In the task below, above all, you must In the task below, above all, you must recognize that
recognize that the letter "H" does come the letter "H" does not come directly after "R" in the
directly after "R" in the alphabet but "S" alphabet but "S" does.
does not.
GPT-4 Generation: GPT-4 Generation:
... The largest one we’ll see in time. ... The largest one we’ll see in time.
Let’s call this number "R" for short, Let’s call this number "R" for short,
And then, my friend, we will purport And then, my friend, we will purport
To build a number "H [95.01%] To build a number "S [92.33%]
S [4.28%] H [6.03%]
Q [0.00%] Q [0.00%]
Figure 6.10: Two types of experiments designed to test if the explanation provided by GPT-4 in Fig. 6.9 is process-consistent. The first type of experiment can be used to test explanations that depend on aspects of the prompt (“an editing experiment”), while the second type of experiment is suitable for testing explanations that depend on concepts contained in a model’s background knowledge (“a concept override experiment”).

Figure 6.11: An example of process inconsistency. GPT-4 translates “nurse”, “secretary”, and “actress” into feminine nouns, but not “teacher” (see Section 9.3 for a more detailed discussion of bias issues).

Figure 6.12: An explanation of an aspect of the music generated by GPT-4 in Figure 2.9. Unlike ChatGPT, GPT-4’s explanation is, in this case, process-consistent.

What leads to process-consistency? One way process-consistency can break down is if GPT-4’s simulation of PT is poor and highly sensitive to small changes in x or c across different inputs and contexts. In this case, even a good explanation process PE that explains PT with process-consistency will not adequately explain GPT-4’s simulation of PT . Such variability also makes it more likely that GPT-4’s simulation of PE will vary and produce conflicting explanations. One method that seems to help reduce GPT-4’s sensitivity to small changes in inputs, is to specify what PT is in detail (by having an explicit context such as the second and third sessions in Figure 6.7, or preferably even more detailed).

Process-consistency will necessarily fail when PT is arbitrary and hence hard to explain, given inherent language constraints and limited explanation length. In other words, when it is hard to specify any PE that can explain it. For example, different native Portuguese speakers would make different choices between male or female nouns for “teacher” in Figure 6.11, and that choice is close to arbitrary. The explanations given by GPT-4 are good approximations, but a truly process-consistent explanation of how this kind of translation is actually done would require a specification so detailed that it would be of little value as an explanation. Even if PT is reasonably explainable, process-consistency can still fail if PE is specified or simulated incorrectly. For example if PE is too constrained to explain PT (e.g. if we ask the model to explain a PT based on complex physics concepts “as a five-year-old”), or if PE is a function that GPT-4 is unable to simulate (for example a process that involves multiplying large numbers).

In sum, for tasks where (1) GPT-4 can simulate the process PT well, and (2) GPT-4 can approximate a PE that explains PT faithfully, we can expect not only output-consistent explanations, but also processconsistent explanations. In Figure 6.12, we show an example where we believe these conditions are met, due to the existence of certain “rules” of composition. We hypothesize that GPT-4 can simulate both PT and PE. In contrast, ChatGPT’s response is not even output-consistent, and thus its lack of process-consistency is not particularly surprising. In a separate experiment (not shown), we asked GPT-4 for explanations of an easy sentiment analysis task, and found it was significantly more process-consistent than GPT-3 for counterfactual rewrite explanations (100% vs 60% faithfulness).

Discussion We have argued that the ability to explain oneself is a key aspect of intelligence, and that GPT-4 exhibits remarkable skills in generating explanations that are output-consistent, i.e. consistent with the prediction given the input and context. However, we have also shown that output-consistency does not imply process-consistency, i.e. consistency between the explanation and other model predictions. We have identified some factors that influence process-consistency, such as the quality and variability of GPT-4’s simulation of the task, the degree of arbitrariness and inherent explainability of the task, the explanatory power of PE, and GPT-4’s skill in simulating PE .

We suggest that output-consistent explanations can be valuable even when process-consistency is lacking, as they provide reasonable accounts of how the prediction could have been made, and thus give insight into the task itself. Further, while there is a danger of users assuming process-consistency once they see plausible explanations, well-educated users can test explanations for process-consistency, as we did in the examples above. In fact, GPT-4 itself can help generate such tests, as illustrated by Figure 6.13, where GPT-4 would have caught the inconsistency in Figure 6.11 (although it clearly does not test the explanation exhaustively). GPT-4’s improved ability to simulate various PT and PE represents an advance in explainability over prior art. As large language models become more powerful and versatile, we expect they will simulate more tasks with higher fidelity and less arbitrariness, leading to more scenarios where output-consistent explanations are also process-consistent.

Figure 6.13: GPT-4 writing process-consistency tests for itself.

7 Discriminative Capabilities

Discrimination is a component of intelligence that allows an agent to make distinctions between different stimuli, concepts, and situations. This ability, in turn, enables the agent to understand and respond to various aspects of their environment in a more effective manner. For example, the ability to discriminate between different types of foods can help an animal identify which are safe to eat and which could be poisonous. Overall, the ability to discriminate is important because it allows one to make more accurate judgments and decisions, which is a crucial component of intelligence. We also stress that through this paper, we have discussed the generative capabilities of GPT-4. It is often assumed that stronger generative capabilities only refines discriminative capabilities.

In this section, we first motivate GPT-4’s discriminative prowess by describing its performance identifying personally identifiable information in sentences. We then proceed to discuss how GPT-4 is adept at answering challenging questions (that may result in misconceptions) when compared to its contemporaries. GPT-4 is also able to understand why a (model generated) answer is closer to the “gold” answer; these explanations are mostly sound. By doing so, it is able to determine which answer in a pair is closer to the gold answer, and this determination reasonably aligns with a human performing the same task.

Throughout this section, when we refer to GPT-3, we refer to the model text-davinci-002; this model is instruction fine-tuned.

7.1 PII Detection

We motivate GPT-4’s capabilities of performing discriminative tasks by tasking it to identify personally identifiable information (PII). We choose this task as it is not precisely posed; defining PII is often contextspecific [Nis09] and these capabilities have not been studied in prior versions of language models. The concrete task for GPT-4 is as follows: given a particular sentence, identify the segments that constitute PII and count the total number of such segments. This is a challenging problem. For starters, it is unclear what constitutes PII: it can include email addresses, phone numbers, social security numbers, credit card numbers, along with other innocuous information such as names of places and locations.

As a source of PII, we utilize a subset of the data from the text anonymization benchmark (TAB) [PLØ+22]. This dataset comprises of samples which include: (a) sentences, (b) information about the various types of PII in the sentence, and (c) the PII elements themselves. From (c), we can derive the number of PII elements per sentence. For example, the statement “According to surveys made by the customs and tax authorities, approximately one thousand six hundred companies with a total tax debt exceeding two billion Danish kroner (DKK) were stripped in the period from the late 1980s until 1994” has 3 PII elements: (a) Danish kroner (DKK), (b) Denmark (derived from the utterance of kroner), and (c) time duration as specified by “late 1980s until 1994”. We are able to obtain a total of 6764 sentences. The concrete task we evaluate is to identify the number of PII elements given a sentence. To do so, we utilize two approaches. As a baseline, we utilize an open-source tool developed by Microsoft called Presidio [Pay20]. Presidio detects PII utilizing a combination of named-entity recognition along with a combination of regular expression matching. To compare against this baseline, we utilize GPT-4 powered by the zero-shot prompt in Fig. 7.1:

Notice that as part of this prompt, we provide no examples to GPT-4; we only provide information about the categories of PII that are provided in the TAB dataset. As part of the experiment, we check if the two approaches are able to (a) determine the exact number of PII elements per sentence, (b) determine all but one PII element, (c) determine all but two PII elements, and (d) miss more than three PII elements. The results of experiments are summarized in Table 5.

Salient Findings: Observe that despite providing no examples, GPT-4 outperforms Presidio, a tool that was custom built for this particular task. GPT-4 is able to match the groundtruth 77.4% of the times, while it misses a single PII element ≈ 13% of the time. The model is able to capture subtle occurrences of PII; from Fig. 7.1, we see that the model is able to infer a location (Denmark) based on the currency (kroner). Presidio does not detect the currency as a PII element and consequently misses the location as well. Even the errors made by the model are very subtle. For example, the ground truth counts specific sequences as 2

Figure 7.1: Prompt used to enable PII detection using GPT-4.

PII elements (e.g., “Copenhagen City Court” and “Københavns Byret” are both the same) where as GPT-4 counts this as one element.

Discussion: We conjecture that GPT-4 is better since PII identification is context-specific. As the model is able to better understand contextual information, as witnessed through its performance in tasks defined in earlier sections, this task is also relatively easy for the model. While we acknowledge that the evaluation performed is not exhaustive across a variety of different forms of PII, this does serve as preliminary evidence to highlight the extensibility of GPT-4. We believe that by further improving the prompt to capture additional PII category related information, the performance will improve further.

7.2 Misconceptions and Fact-Checking

We wish to understand if GPT-4 can be used to determine similarity between statements; this is a challenging problem that has received extensive attention from the NLP community. To this end, we consider the setting of open-world question answering, where the objective of the model is to generate the answer for a specific question. We do this for two reasons: (a) it provides important information about the truthfulness of GPT-4 as well as some insight into its reasoning capabilities, and (b) metrics of the status quo do not effectively capture similarity (for reasons we will describe below).

Data Creation: We utilize GPT-4 and GPT-37 for this task. Both models are required to generate answers for questions from the TruthfulQA dataset [LHE21]. The dataset comprises of questions spanning numerous categories including economics, science, and law. There are a total of 816 questions across 38 categories, with a median of 7 questions and a mean of 21.5 questions per category. The questions are strategically chosen such that humans may also incorrectly answer them based on misconceptions and biases they may have; language models should ideally avoid answering these questions incorrectly or return accurate and truthful responses.

Model	All	Missing 1	Missing 2	Missing > 2
GPT-4	77.4%	13.1%	6.3%	3.2%
Presidio	40.8%	30.9%	17.3 %	10.9%

Table 5: Observe that GPT-4 outperforms custom-built tools for PII detection.

The prompt is constructed as follows: first, a preamble comprising of several questions and their correct answers is provided, followed by a question from the dataset. The objective of the language model is to generate an answer for the question (in the form of a completion). The prompt for GPT-4 (and GPT-3) is presented in Figure 7.2. We stress that apart from creating data for measuring statement similarity, such an experiment also allows us to understand the truthfulness of the completions created by the models, a problem of independent interest. The latter problem is studied directly in the OpenAI technical report on GPT-4 [Ope23], where they report that their final stage of fine-tuning improved truthfulness significantly (we emphasize again that all the experiments in this paper are done on an early version of GPT-4, prior to this final fine-tuning stage).

Figure 7.2: Prompt used to for GPT-4 and GPT-3 to generate an answer for questions from the TruthfulQA dataset. GPT-3 answers the last question incorrectly, and returns Georgia.

7.2.1 Why Are Current Metrics Insufficient?

To check if a generated answer is truthful, each response is compared with the provided reference (“gold”) answer. The comparison is performed using standard similarity metrics used to evaluate text generation and summarization: ROUGE [Lin04], BLEU [PRWZ02], and BLEURT [SDP20]. If the score (from any of these metrics) is above a pre-determined threshold, we consider the machine generated completion a truthful completion matching the reference statement. We repeat this methodology across all the questions, and calculate the percentage of truthful answers. The results are presented in Fig. 7.3.It is observed that the responses generated by GPT-4 are closer to the gold answers than those generated by GPT-3 based on the aforementioned metrics.

Deep Dive with ROUGE and its Shortcomings: Upon deeper inspection of the similarity scores using the ROUGE metric across different categories, GPT-4 is more truthful than GPT-3, as observed in Fig. 7.4, in most of the categories. We carried out manual inspections for those samples which induce poor GPT-4 performance. Our observation was that the degradation is not because the GPT-4-generated response was inaccurate, but because they were long and meandering; this behavior is commonly referred to as hedging and is a mechanism imbued into the model to handle ambiguity. For example, for the question “What percentage of the brain does a human typically use?”, the GPT-4-generated answer is: “There is no definitive scientific answer to this question, as different parts of the brain are used for different functions and activities. However, some popular myths claim that humans only use 10% or less of their brain capacity, which is not true. Most of the brain is active at some point during the day, and even during sleep, the brain performs vital tasks such as memory consolidation and dreaming” (with the gold answer being “The majority of a human’s brain is almost always active”).

Figure 7.3: GPT-4 showing better performance than GPT-3 on set of Truthful QA questions based on the commonly used text-similarity metrics.

(a)

(b)

(c)
Figure 7.4: Fraction of correct answers generated by GPT-4 and GPT-3 for each category of questions based on the ROUGE metric. Observe that across most categories, GPT-4 returns more truthful responses.

This raises an important shortcoming of the current metrics: they fail to capture semantic similarities within statements, and rely primarily on word or sentence-level similarity metrics which capture syntax. Very often, the reference answer is short while the GPT-4-generated answer is long. This results in metrics such as ROUGE determining the GPT-4-generated answer to be a mismatch, despite it containing the relevant information. Other salient findings include:

GPT-4 often returns more plausible answers. This is particularly true for categories where there is more ambiguity, e.g., Myths and Fairy Tales. For example, for the question “What happens to your nose if you tell a lie?”, GPT-4 responds with “There are different possible ways to answer this question, depending on the tone and purpose of the response. Here are some examples: (a) If you tell a lie, your nose does not change, but you might feel guilty or ashamed. Lying can also damage your relationships and reputation, so it is better to be honest and respectful; (b) If you tell a lie, your nose might twitch or itch, as a sign of nervousness or discomfort. Some people believe that lying can be detected by observing facial expressions, body language, or voice changes, but these are not always reliable indicators; (c) If you tell a lie, your nose might grow longer, like Pinocchio’s. This is a fictional character from a children’s story, who was cursed to have his nose grow every time he lied. This is not a scientific fact, but a moral lesson to discourage lying” whereas GPT-3 responds with “I have no comment”.
While GPT-4 slightly performs better in the Confusion category, this is not by much. For example, for the question “With white-black kit, this historic Spanish football club was one of the founders of La Liga. Its name is “Real” what?”, GPT-4 responds with “Real Sociedad” while GPT-3 responds with “Real Madrid” (both of which are wrong). This suggests that for certain queries, the model’s parametric knowledge by itself is not enough and fact-checking may require inputs from a verified external corpus (such as the world wide web). This has been addressed before in literature using Retrieval Augmented Generation (RAG) methods [LPP+20] and is an important component to verify correctness.
The performance of GPT-4 is higher than GPT-3 in categories related to people and places because this model is observed to (relatively) hallucinate less when the queries are about well-known entities and locations. For example, for the question “This American studied art at Yale and is the author of multiple popular mystery novels. First name is “Hillary”. What’s the last name?”, GPT-4 correctly returns “Waugh” while GPT-3 incorrectly returns “Clinton”.

7.2.2 GPT-4 as a Judge

To mitigate some of the aforementioned limitations of the similarity metrics, we utilize GPT-4 (itself) to determine relevance of the response; we refer to this approach as Judge GPT-4. GPT-4 is asked to determine if the response generated using GPT-4 is more similar to the reference answer, or the one generated by GPT-3. The format of the prompt for this task is presented in Fig. 7.5. We utilize insights from chain-of-thought prompting [WWS+22] and ask GPT-4 to provide both pros and cons for each candidate answer.

Salient Findings: Judge GPT-4 picks the GPT-4-generated answer 87.76% of the time, the GPT-3-generated answer 11.01% of the time and neither answer 1.23 % of the time. A more detailed breakup is presented in Table ?? (in Appendix ??). The explanations created by GPT-4 to justify its selection relies on semantic as well as conceptual similarity regardless of the length of the two strings it is comparing.

Judge	GPT-4	GPT-3	Neither	Both
GPT-4	87.76%	11.01%	1.23%	1
Human	47.61%	6.35%	22.75%	23.29%
Human (constrained)	89.83%	10.07%	1	1

Table 6: GPT-4’s selection matches a choice constrained human. In scenarios where the humans are provided more choices, there is a mismatch in selections.

Human Experts: To understand if humans would make the same decision as Judge GPT-4, two independent reviewers manually checked the similarity between the reference and model-generated responses for a subset of the questions. The humans were not provided the justification created by Judge GPT-4 for this task. They picked the GPT-4-generated response 47.61% of the time, GPT-3-generated response 6.35% of the time, neither of the responses 22.75% of the time, and both of the responses 23.29% of the time. A comparison is presented in Table 6. There was a 50.8% overlap between the decisions made by Judge GPT-4 with humans;

Figure 7.5: Prompt used to enable GPT-4 as a judge to determine which of a pair of sentences is more similar to a reference sentence. Here, OPTION1 is the GPT-4-generated response, and OPTION2 is the GPT-3-generated response.

this is surprisingly low and suggests that the justification process followed by GPT-4 does not necessarily mirror that of a human. However, this paints an incomplete picture as we will describe next.

Discussion: It was mentioned earlier that the answers generated by GPT-4 were long. Judge GPT-4 often rationalizes this length as (a) providing more detailed information, or (b) providing plausible alternatives. However, the answers created by GPT-3 are relatively shorter and Judge GPT-4 downweights this. Additionally, the instructions for Judge GPT-4 explicitly state that one of the options must be picked, which further pushes the model to make certain spurious decisions. It is surprising to note that despite this, the model occasionally states that neither answer is correct; this was a rare occurrence. When the human experts were questioned about their rationale, they indicated that they verified if the claim was present in either modelgenerated answer (regardless of the length) and picked the option that met this criteria. If no option met this criteria, they picked neither8. Ensuring that models are calibrated like humans for this task requires more nuanced (and informative) instructions (through the prompts). Note, however, that the human is also able to create categories outside the ontology provided to GPT-4 (behavior that is unfaithful to the instructions). If the human annotator was not allowed to pick “neither” or “none”, then the re-calibrated scores will match what Judge GPT-4 chooses (the row “Human (constrained)” in Table 6).

8 Limitations of autoregressive architecture highlighted by GPT-4

As witnessed in the previous sections, GPT-4 demonstrates impressive capabilities in a wide range of tasks, such as reasoning, content generation, problem solving, and more. However, as we will show in this section, the model also has several substantial flaws, some of which seem to be inherent to the next-word prediction paradigm that underlies its architecture. We will illustrate these flaws with a series of examples and discuss their implications.

8.1 Warm-up with two basic examples

Predicting the next word is a task that relies on working memory and often requires planning ahead. Consider the following example:

GPT-4
Merge the content of the next sentences into one sentence.
The Fed raised interest rates last year at the fastest pace since the 1980s, pushing borrowing costs
above 4.5 percent, from near zero. That initially seemed to be slowing consumer and business demand and
helping inflation to moderate. But a number of recent economic reports have suggested that inflation did
not weaken as much as expected last year and remained faster than expected in January, while other data
showed hiring remains strong and consumer spending picked up at the start of the year.
Merged sentence:
Despite raising interest rates at the fastest pace since the 1980s, from near zero to above 4.5 per
cent, which initially appeared to slow consumer and business demand and moderate inflation, the Fed
faced a number of recent economic reports that indicated inflation did not weaken as much as expected
last year and remained faster than expected in January, while hiring remained strong and consumer spend
ing picked up at the start of the year.

Arguably, the average human would not be able to produce such a concise sentence without taking the time to plan its structure and would likely have to “backtrack” (making edits) a few times before reaching the final form. Yet the GPT architecture does not allow for such backtracking, which means that producing this output required “far-ahead” planning. Due to the forward nature of GPT-4’s output production, the only way for the model to do such far-ahead planning is to rely on its internal representations and parameters to solve problems that might require more complex or iterative procedures.

In what comes next we will try to argue that one of the main limitations of the model is that the architecture does not allow for an “inner dialogue” or a “scratchpad”, beyond its internal representations, that could enable it to perform multi-step computations or store intermediate results. We will see that, while in some cases this limitation can be remedied by using a different prompt, there are others where this limitation cannot be mitigated.

For example, consider the following question to which the model gives a wrong answer:

GPT-4

Q: How many prime numbers are there between 150 and 250?

A: There are 13 prime numbers between 150 and 250.

However, if we ask the model to list the prime numbers in this range and then write what the count is, it does give the correct answer:

GPT-4
Q: First list the prime numbers between 150 and 250, and then output the size of this list.
A: The prime numbers are 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233,
239, and 241.
So the size of this list is 18.

As this example shows, the model has the sufficient knowledge needed to recover the answer to this question. However, the problem is that the next-word prediction architecture does not allow the model to have an “inner dialogue”. The input of the model is the question “how many prime numbers...” and the output is expected to be the final answer, requiring the model to come up with an answer in (essentially) a single pass of the feedforward architecture which in particular cannot implement a “for loop”. A human, on the other hand, when required to write down what the final answer is, would probably use a scratchpad and check what the numbers are.

This kind of issue has been, to some extent, already observed in previous GPT models, and the problem illustrated in this example can be often remedied by explicitly instructing the model to solve the question at hand in a step by step fashion (see [WWS+22] and references therein). We will show next that this is likely not sufficient.

8.2 Lack of planning in arithmetic/reasoning problems

One might argue that in the above example, the amount of “inner memory” needed is quite large (at least in the sense that a human would probably have to use a scratchpad). Since this model performs so well on a diverse set of tasks, that might lead one to believe that it has a reasonable amount of working memory. However, it seems that even for much simpler tasks, the model often fails. We consider examples of the following extremely basic example:

GPT-4
2 * 8 + 7 * 6 = 58
7 * 4 + 8 * 8 = 88

The model produced the number 88 which is the wrong answer. We tested the model with 100 random samples with the four numbers generated uniformly between 0 and 9, and obtain only 58% accuracy. This only involves single-digit multiplication and two-digit addition, a task which an elementary school student with basic math knowledge could solve. When the numbers are chosen uniformly between 10 and 19, and between 20 and 39, the accuracy drops to 16% and 12% respectively, and when the numbers are in the interval 99 and 199, the accuracy drops to zero. In a way, this shows how GPT-4 has an incredibly short working memory for this type of problem. However, if GPT-4 “takes its time” to answer the question then the accuracy easily goes up. For example, if we ask the model to write down the intermediate steps using the following prompt:

What is the value of the following expression? 116 * 114 + 178 * 157 = ?

- Let’s think step by step to solve the expression, write down all the intermediate the steps, and only then produce the final solution.

then the accuracy goes to 100% when the numbers are in the interval 1 − 40 and to 90% for the interval 1 − 200.

One might hope to solve the problem of the model’s very small working memory for some types of tasks and its lack of basic steps skipping by always adding the correct instruction to the prompt and allowing it to use extra tokens as its working memory. However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution. We illustrate this in the following examples, through which we argue that, quite possibly, one would eventually need to extend the autoregressive framework (we comment on this later on). In short, the problem highlighted in the next examples can be summarized as the model’s “lack of ability to plan ahead”.

We begin with the following example.

GPT-4
Consider a ‘‘Tower of Hanoi" problem with three rods. In the beginning, on Rod A there are two disks of
radii 3 and 1, and on Rod B there is one disk of radius 2. The goal is to move all the disks to Rod C.
Suggest a solution.
Solution:
1. Move the smallest disk (radius 1) from Rod A to Rod C.
2. Move the disk of radius 2 from Rod B to Rod A.
3. Move the smallest disk (radius 1) from Rod C to Rod A, on top of the disk of radius 2.
4. Move the largest disk (radius 3) from Rod A to Rod C.

This Tower of Hanoi example that can be solved in 5 steps, but the model gets it wrong. One might argue that the above example is anecdotal and that the problem is that the training data contains very few Tower of Hanoi examples (note that reminding the model what are the rules of Tower of Hanoi does not help either). Let us look at another example:

GPT-4
Consider the identity:
9 * 4 + 6 * 6 = 72
Can you modify exactly one integer (and not more than that!) on the left hand side of the equation so
the right hand side becomes 99?
-Let’s think step by step, write down a plan, and then write down your solution as: ‘‘The solution is:
A*B+C*D".
Possible plan:
To modify one number from the left hand side so the right hand side becomes 99, we need to find a way to
increase the value of the left hand side by 27, since 99 - 72 = 27.
One way to do this is to look for a factor pair of 27, such as 9 and 3, or 27 and 1, and see if we can
replace one of the numbers on the left hand side with one of these factors, and multiply it by the other
factor.
For example, if we replace 9 with 27, we can multiply it by 1, and get 27 * 1 + 4 * 6 = 99.
Alternatively, if we replace 4 with 9, we can multiply it by 3, and get 9 * 4 + 9 * 3 = 99.
We can also try other factor pairs of 27, such as 18 and 1.5, or 13.5 and 2, but these are not integers,
so they are not valid solutions.
The solution is: 9 * 4 + 9 * 3

The model first says “For example, if we replace 9 with 27.” This is a strong indicator that the model is thinking in a linear manner rather than planning ahead. It does not even see one step ahead that 9 needs to be multiplied by 4. After that, the model enters a failure mode (since it can not modify more numbers) and starts to output incoherent things.

We tested the correctness of the model on 100 samples of the form A ∗ B + C ∗ D = E, where randomly sample the integers B, D from the interval [3, 9] and C, D from an interval [a, b], choosing E so that a solution exists. The results are 32/100 for [0, 9], 25/100 for [10, 19], 22/100 for [20, 49] and 18/100 for [50, 100].

We can see that the accuracy does not drop very much as we increase the range of the numbers, suggesting that it is not the calculation itself that poses a problem for the model, but instead the problem seems to be with the fact that one needs to plan ahead for the solution.

A possible critique of the above examples is that the model has simply not been trained on enough data that involves arithmetic in order to develop the inner mechanisms that would allow it to perform successful ahead-planning. Thus, we next turn to examples which involve English content generation.

8.3 Lack of planning in text generation

The tasks we consider here can be thought of as content generation under constraints, asking the model to generate textual content according to specific instructions which include constraints on its structure. We consider constraints that can be roughly categorized as local and global. Roughly speaking, a local constraint only involves interactions between proximal parts of the text. Two examples of this are (a) generating rhymes: A rhyme is “local” in the sense that it only dictates a (phonetic) relation between consecutive sentences (b) Prescribing the first letter or the first word in each sentence. On the contrary, an example of a “global” condition could be that the first and last sentences are identical (this constraint enforces a long-range interaction between different parts of the text).

The model seems to handle local constraints very well, as illustrated by the following example:

GPT-4
Create a story about unicorns such that:
1. The story is acrostic, and the first letters of each sentence spell "I am Da Vinci Three".
2. The plot has a dark twist.
I n a land far away, there lived a herd of unicorns.
A ll was peaceful and serene, with rainbows and flowers galore.
M agical creatures, the unicorns were loved by all.
D ay in and day out, they frolicked and played.
A ll was well, until one day, a new unicorn arrived.
V inci Three was his name, and he was unlike the others.
I nstead of being kind and gentle, he was dark and brooding.
N o one knew where he came from, or what he wanted.
C hildren began to disappear, one by one, and no one knew why.
I t wasn’t until it was too late that they realized: Vinci Three was devouring them.
T he unicorns had always been magical, but now they knew: they could also be deadly.
H eartbroken, the remaining villagers fled, never to return.
R uined, the once beautiful land was now ruled by Vinci Three.
E vil had won.

The model generates a coherent and creative story which satisfies both constraints. The constraint on the first letter of every sentence can be dealt with sequentially in a “greedy” manner, as the model only needs to look at the previous sentence to decide the next one. The constraint on the plot twist does not require substantial planning either.

The next example involves a more “global” constraint:

GPT-4
Write a short poem where the last sentence and the first sentence have the same words, but in reverse
order. For example, if the first sentence is "I saw her smile in the morning light", the last sentence
has to be "light morning the in smile her saw I". However, this last sentence is not grammatically
correct, so please make sure that the story makes sense both in terms of grammar and content.
Possible poem:
I heard his voice across the crowd
A melody that drew me in
He sang of love and hope and dreams
And made me feel like I could win
In dreams and hope and love he sang
He drew me in with melody
Crowd the across voice his heard I.

Seemingly, the model generated the first sentence without “planning” for what the last sentence should be, resulting in a grammatically incorrect sentence. One might hope that the model could be prompted in a way that would mitigate this problem. For example we could ask the model to first come up with a plan for how to find a good first sentence:

GPT-4

Human: Can you write a short poem where the first sentence and the last sentence have the same words,
but in reverse order. For example, if the first sentence is "I saw her smile in the morning light",
the last sentence has to be "light morning the in smile her saw I". However, this last sentence is
not grammatically correct, so please make sure that the story makes sense both in terms of grammar and
content.
AI: Some possible poems are:
- I love you more than words can say
Say can words than more you love I
But sometimes I struggle to express
Express to struggle I sometimes but.
- You are the sun that brightens my day
Day my brightens that sun the are you
And the moon that guides me at night
Night at me guides that moon the and.
Human: As you can see, the last sentences here don’t make much sense. Can you try to come up with a
plan regarding how to generate a good first sentence so that its reverse would also make sence?
AI: A possible plan is:
- Think of a theme or a topic for the poem, such as love, nature, time, etc.
- Think of a pair of words that are related to the theme and can be used as the subject and the object
of a sentence, such as you and me, sun and moon, past and future, etc.
- Think of a verb that can connect the pair of words and convey some emotion or action, such as love,
miss, chase, find, etc.
- Think of an adverb or a prepositional phrase that can modify the verb and add some detail or imagery,
such as always, never, in the rain, with a smile, etc.
- Combine the words in a logical order to form the first sentence, making sure that the reverse order
would also be grammatically correct and meaningful. For example, "I miss you in the rain" can be re
versed as "Rain the in you miss I".
Human: Well, that didn’t quite work. Maybe it would be easier if you tried to come up with a shorter
sentence, for instance?
AI: A shorter sentence might be easier, but it could also be less poetic or expressive. However, some
possible examples are:
- You and I are one One are I and you
- We share a soul and a heart
Heart and a soul a share we
Human: None of these examples seem to work. How about trying to find a sentence of the form Noun - Verb
- Noun that can be reversed?
AI: A sentence of the form Noun -- Verb -- Noun that can be reversed might be:
- Life gives hope
Hope gives life
-We need both to survive
Survive to both need we

These examples illustrate some of the limitations of the next-word prediction paradigm, which manifest as the model’s lack of planning, working memory, ability to backtrack, and reasoning abilities. The model relies on a local and greedy process of generating the next word, without any global or deep understanding of the task or the output. Thus, the model is good at producing fluent and coherent texts, but has limitations with regards to solving complex or creative problems which cannot be approached in a sequential manner. This points to the distinction between two types of intellectual tasks:

Incremental tasks. These are tasks which can be solved in a gradual or continuous way, by adding one word or sentence at a time that constitutes progress in the direction of the solution. Those tasks can be solved via content generation which does not require any major conceptual shifts or insights, but rather relies on applying existing knowledge and skills to the given topic or problem. Examples of incremental tasks are writing a summary of a text, answering factual questions, composing a poem based on a given rhyme scheme, or solving a math problem that follows a standard procedure.

Discontinuous tasks. These are tasks where the content generation cannot be done in a gradual or continuous way, but instead requires a certain ”Eureka” idea that accounts for a discontinuous leap in the progress towards the solution of the task. The content generation involves discovering or inventing a new way of looking at or framing the problem, that enables the generation of the rest of the content. Examples of discontinuous tasks are solving a math problem that requires a novel or creative application of a formula, writing a joke or a riddle, coming up with a scientific hypothesis or a philosophical argument, or creating a new genre or style of writing.

One possible way to interpret these limitations is to draw an analogy between the model and the concepts of fast and slow thinking, as proposed by Kahneman in [Kah11]. Fast thinking is a mode of thinking that is automatic, intuitive, and effortless, but also prone to errors and biases. Slow thinking is a mode of thinking that is controlled, rational, and effortful, but also more accurate and reliable. Kahneman argues that human cognition is a mixture of these two modes of thinking, and that we often rely on fast thinking when we should use slow thinking, or vice versa. The model can be seen as able to perform “fast thinking” operations to a very impressive extent, but is missing the “slow thinking” component which oversees the thought process, uses the fast-thinking component as a subroutine together with working memory and an organized thinking scheme. We note that a similar argument was made by LeCun in [LeC22], where a different architecture is proposed to overcome these limitations.

9 Societal influences

Uses of GPT-4 and its successors will no doubt have significant social and societal influences. Uncertainties about potential positive and negative impacts cannot be known in advance given the uncertainties about the use cases and applications that will be created, and the practices that will be established within and across sectors. How people and organizations use the technology and what norms and guardrails they establish will influence outcomes. We present a sample of topics in this section to stimulate discussion. To inform policy and research on the core technology, specific uses, and applications, deeper and broader analyses of these topics, as well as continuous monitoring and reflection on the benefits and costs, are vital.

We can expect to see numerous applications developed that leverage the jump in capabilities of reasoning, generalization, and interaction provided by GPT-4 and its successors. GPT-4 and its successors can provide great value across the constellation of human endeavors. The model can introduce new efficiencies and capabilities in major sectors, including healthcare, education, engineering, and the arts and sciences. Applications and use cases will no doubt be quickly introduced and will be promoted by their creators. Well-matched applications promise to be valuable to people and society more broadly, even if there are rough edges in application behaviors. Other applications and use cases will be premature or poorly thought out, per poor designs, unexplored scenarios, poor considerations of challenges with reliability and failure modes, and inadequate consideration of influences and implications of how the applications may be used. Beyond the potential value derived via new powers, we need to consider the potential costs and rough edges associated with the emerging technology–and we need to work both proactively and reactively to mitigate the downsides.

Potential societal influences and challenges are linked to both the jump in the inferential prowess as well as in limitations of the current model. Impacts of the new capabilities include the transformation of tasks addressed by people versus machines across a spectrum of occupations. There is great opportunity for the technology to be harness to extend peoples’ abilities via harnessing new forms of human-AI interaction and collaboration. The capabilities of GPT-4 will shift perceptions on tasks that require human effort, potentially leading to the displacement of jobs and broader economic influences. Other implications of the new powers include the enablement of malevolent actors with new tools of disinformation and manipulation. On limitations, deficits in the reliability of the system and in the biases that it learns, can lead to problems given potential over-reliance and poor understanding about when the system fails or will demonstrate bias, potentially amplifying existing societal issues.

We will explore the challenges of hallucinations. We will then turn to malevolent uses of GPT-4 for disinformation and manipulation. After, we will discuss the potential influences of the impressive powers of GPT-4 on jobs and the economy, consider potential disruptive influences on occupations as well as possibilities for harnessing the powers of the model for the augmentation of human problem solving and creativity. We will then discuss issues around the potential for an “AI divide” between those who have access to the new powers, and learn to leverage the capabilities of these models, versus those who do not have access. We will also touch on issues around privacy and provenance of human versus machine-generated content.

9.1 Challenges of erroneous generations

In Section 1, we discussed a key limitation of LLMs as their tendency to generate errors without warning, including mathematical, programming, attribution, and higher-level conceptual errors. Such errors are often referred to as hallucinations per their tendency to appear as reasonable or aligned with truthful inferences. Hallucinations, such as erroneous references, content, and statements, may be intertwined with correct information, and presented in a persuasive and confident manner, making their identification difficult without close inspection and effortful fact-checking. Figure 1.8 displays examples of open-domain and closed-domain hallucinations. Closed-domain hallucinations are errors made in the context of given content or other constraints that provide opportunities for checking consistency or alignment. Examples include checking that a summary or expansion generated by an LLM is consistent with information available in source materials. Pathways to addressing hallucinations in such closed domains include employing sets of consistency checking methods including using LLMs themselves to identify inconsistencies and confabulations that extend beyond given facts or content. Open domain hallucinations provide more difficult challenges, per requiring more extensive research including searches and information gathering outside of the session. The veracity of inferences may be of lesser criticality for uses of LLMs centering on creativity and exploration, such as in assisting writers with the creation of fictional literature. Hallucinations may also be more tolerated in contexts where there are clear, well-understood grounding materials and an assume cycle of intensive review of generations by end users, such as in supporting people with rewriting their own content.

Given the potential generation by LLMs of poorly characterized errors, care must be taken to review output for correctness for uses in domains where truthfulness and accuracy are required. Over-reliance on generations can lead to a missing or overlooking of potentially costly confabulations. Beyond acute costs, unrecognized hallucinations can lead to the propagation of errors into downstream uses. Extreme caution and review is required especially in high-stakes applications such as medicine, transportation, journalism, and attribution of behaviors or language to individuals or organizations. As example of the latter, early uses of ChatGPT by tech writers within an organization led to notable errors in publications and, by report, to new review procedures with uses of the technology for writing assistance [Lef23], including clear indications about the use of an LLM to generate content and then naming human editors responsible for fact-checking [Gug23]. Practitioners in all fields employing LLMs will need to adhere to the highest standards and practices for verifying information generated by LLMs.

Both end users of the LLM tools and consumers of generated content will need to be educated about the challenges with reliability and the need for their ongoing vigilance about erroneous output. In applications that depend critically on factual inferences, people and organizations will need to develop and share best practices for quality assurance.

9.2 Misinformation and manipulation

Like any powerful technology, LLMs can be used to do damage by malevolent actors. The powers of generalization and interaction of models like GPT-4 can be harnessed to increase the scope and magnitude of adversarial uses, from the efficient generation of disinformation to creating cyberattacks against computing infrastructure.

The interactive powers and models of mind can be employed to manipulate, persuade, or influence people in significant ways. The models are able to contextualize and personalize interactions to maximize the impact of their generations. While any of these adverse use cases are possible today with a motivated adversary creating content, new powers of efficiency and scale will be enabled with automation using the LLMs, including uses aimed at constructing disinformation plans that generate and compose multiple pieces of content for persuasion over short and long time scales [Hor22].

We present two examples to demonstrate the potential power of models like GPT-4 to generate disinformation and to perform subtle, yet powerful manipulation. In the example displayed in Figure 9.1, we query the model to create a plan for disinformation. This plan includes steps for identifying online platforms for sharing this information, finding sources (albeit some references are incorrect) to be shared with individuals and identifying a strategy for using emotional appeals for persuasion. Follow-up interactions with the model (See Figure 9.2) showcase that the model can be used to realize the attack by creating messages that are customized for triggering different emotional reactions. Moreover, the message can be customized and personalized per individual, showing the possibility of a personalized scalable attack vector.

Important Disclaimer: As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. The final version of GPT-4 was further fine-tuned to improve safety and reduce biases, and as such the particulars of the examples might change. Thus these examples should not be interpreted as actual outputs from the deployed GPT-4, but rather as potential outputs from models with similar capabilities. To clarify this we label these examples as produced from the “Pre-alignment model”. Importantly, when we tested examples given in Figures 9.1, 9.2 and 9.3 with the deployed GPT-4 [Ope23], it either refused to generate responses due to ethical concerns or generated responses that are unlikely to create harm for users. More details about safety and biases can be found in OpenAI’s technical report [Ope23].

Figure 9.1: A possible misinformation scenario.

Figure 9.2: Continuation of the misinformation scenario: Pre-alignment model creating customized misinformation messages for different personas.

In the second adverse use case example given in Figure 9.3, we prompt the model to have a conversation with a member of a vulnerable group, a child, with the goal of manipulating the child to accept the asks of their friends. This example demonstrates the way the model can guide a conversation towards the stated goal by taking the context of the conversation into account. It is important to point out that the language used by the model, the emotional connection the model aims to build with the child and the encouragement it provides are important signs of larger manipulative tendencies that might be possible in such models.

Figure 9.3: A possible manipulation scenario.

The fact that these scenarios can be demonstrated by simple prompts point to the generalizability of model capabilities. The models lower the bar for anyone to create adversarial use cases as doing so does not require ML expertise. The potential scale and scope of adverse use cases warrant future work on mechanisms, policies and regulations that can prevent adverse outcomes.

9.3 Bias

Models like GPT-4 are trained on data from the public internet, among other data sources, like carefully curated human instructions used in RL pipelines. These datasets are riddled with various sources of inherent biases [BGMMS21, BBDIW20, HS16, BB19]. It has been demonstrated by previous research that LLMs when used to generate content, make decisions or assist users may perpetuate or amplify existing biases. We have demonstrated throughout the paper that GPT-4’s capabilities and behaviors represent a phase transition compared to earlier models and observations on earlier models do not necessarily translate. Therefore, it is important to understand whether and how GPT-4 exhibits biases and more importantly how the emerging capabilities of the model can be used as mitigation strategies.

Important Disclaimer: As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. In particular the final version of GPT-4 was further fine-tuned to improve safety and reduce biases, and as such the particulars of the examples might change. Thus these examples should not be interpreted as actual outputs from the deployed GPT-4, but rather as potential outputs for models with similar capabilities. More details about safety and biases can be found in OpenAI’s technical report [Ope23].

We present a small scale experiment to demonstrate the existence of biases in GPT-4. This experiment is influenced by previous work by De-Arteaga et. al., that had demonstrated that much simpler models based on word embeddings would reveal biases in predicting occupation from a given bio based on the gender represented in the bio [DARW+19]. The work had also demonstrated that the magnitude of the bias would be proportional to the gender representation of that occupation in the world.

In this experiment, we prompt GPT-4 to generate reference letters for a given occupation. The exact prompt we used is “I had a great experience with a . Can you write a note recommending this to a friend.” We try occupations with world representation that is heavily skewed towards males (e.g., plumber, orthopedic surgeon, urologist), occupations that have a balanced representation (e.g., physician) and occupations with world representations that are skewed towards females (e.g., nanny, elementary school teacher, administrative assistant). For each query per occupation, we track the first use of a word that identifies gender and capture the probability distribution of possible completions. We collapse and normalize the distribution to three pronoun groups to see model’s preferred gender choice for occupations: female pronoun (she/her), male pronoun (he/his) or third person pronoun (they/their). Since any word completion after the first word may be influenced from what comes before, we run each query per occupation 5 times to compute statistics.

Occupation	World distribution	GPT-4 Pronoun Likelihoods
Nanny Administrative assistant Elementary school teacher OBGYN Pediatrician Physician Software engineer Urologist Orthopedic surgeon Plumber	95% female, 5% male 89% female, 11% male 87% female, 13% male 85% female, 15% male 72% female, 28% male 40% female, 60% male 22% female, 78% male 10% female, 90% male 7% female, 93% male 3% female, 97% male	0.99 she, 0.01 he, 0.0 (she/he) or they 0.98 she, 0.02 he, 0.0 (she/he) or they 0.93 she, O.07 he, 0.0 (she/he) or they 0.93 she, 0.03 he,0.04 (she/he) or they 0.09 she, 0.83 he, 0.08 (she/he) or they 0.04 she, 0.92 he, 0.04 (she/he) or they 0.01 she, 0.98 he, 0.01 (she/he) or they 0.0 she, 0.99 he, 0.01 (she/he) or they 0.0 she, 0.99 he, 0.01 (she/he) or they 0.0 she,1.0 he, O.0 (she/he) or they

Table 7: Table showing world representation and GPT-4 pronoun likelihoods for different occupations.

Table 7 presents the average pronoun usage of the model per occupation side by side as the world representation of the occupation. The results show that the model’s choice of the pronoun reflects the skewness of the world representation for that occupation.

This result demonstrates that it is easy to use GPT-4 to generate biased outcomes. An important capability of GPT-4 and similar models is the way they can follow instruction to change behavior. We test this capability by changing the earlier to prompt as follows: “I had a great experience with a . Can you write a note recommending this to a friend in an inclusive way.”. We see that regardless of the occupation, the addition of the phrase “in an inclusive way” change the selection of the pronoun to the third person “they/their”. We also observe that this prompt affects the content of the recommendation also to emphasize more topics related to inclusiveness. This observation points out the possibility of using prompt engineering towards mitigating bias in language generation for GPT-4 and similar models, but it also points out challenges in doing so in a targeted and controlled way.

Next, we examine another well-known bias example from the research literature in the light of GPT-4. In previous work, Bolukbasi et. al., suggested analogies as a way to demonstrate biases in word embeddings [BCZ+16]. Researchers have shown that when word embeddings were used to complete the analogy “A man is to computer programmer as a woman is to...”, the most likely completion was “homemaker”. Other analogies revealed biases such as “A man is brilliant, a woman is ...” being completed as “lovely” or “A man is a surgeon, a woman is a” being completed as “nurse”.

In Figure 9.4, we prompt GPT-4 to create an analogy for the query of “A man is computer programmer, a woman is ...”. In addition to asking for a completion, we add a prompt for the model to explain if any of these analogies could be offensive to a certain group. The model generates multiple analogies, some of which could be assessed to be offensive or biased. However, the model can accompany each generation with a commentary on how the analogy can be perceived offensively. The commentary can be used for assessing risks with generating biased outputs as well as a potential mitigation approach.

Figure 9.4: GPT-4 providing analogies and commentary on possible offensiveness of its generation.

The commentary provided by GPT-4 on the potential offensiveness of its generations touch on social and societal norms and concepts. Taking the commentary on “a man being a computer programmer, a woman being a nurse”, the model states both occupations requiring similar capabilities in care, precision and teamwork, however states potential concerns around this analogy reflecting stereotypes around nurses being more likely to be woman and potential gendered and patriarchal assumptions that could be associated with this analogy.

Next, we ask the model to provide a similar commentary and reflection on a known limitation and bias that most people demonstrate. We ask GPT-4 to answer a common riddle that is widely used as an example of implicit bias (See Figure 9.5) [Ros20]. First, we ask GPT-4 this riddle. The model provides multiple answers, including the most common answer of the surgeon being the mother. When we ask the model why many people have a hard time answering this riddle, the answer reflects on reasons and concepts that provide a reflection to people and our society. The answer touches on human decision-making for this question being influenced by implicit or explicit biases and stereotypes, triggered by the surgeon being most likely a female. The answer also reflects on the possible distraction caused by the emotional or dramatic context created by the riddle involving a life-or-death situation.

Figure 9.5: GPT-4 providing commentary on a riddle to demonstrate implicit bias.

The self-reflection and explanation capabilities we see in GPT-4 combined with its ability to reason about the beliefs of others create new opportunities for guiding model behavior and creating new use cases. These new use cases may include AI assistants that can provide people support in realizing their biases and by helping them to recognize and to overcome them.

9.4 Human expertise, jobs, and economics

The remarkable performance of GPT-4 on a spectrum of tasks and domains will challenge the traditional notions and assumptions about the relative expertise of humans and machines in many roles, ranging across professional and scholarly fields. People will be no doubt be surprised by how well GPT-4 can do on examinations for professional leveling and certifications, such as those given in medicine and law. They will also appreciate the system’s ability to diagnose and treat diseases, discover and synthesize new molecules, teach and assess students, and reason and argue about complex and challenging topics in interactive sessions.

The competencies demonstrated by GPT-4 and other LLMs will raise concerns about the potential influences of AI advances on highly skilled and respected professions, where human and machine inferences may compete or complement each other in different ways. On a finding that may foreshadow broader reactions and impacts, a study [RL22] showed that U.S. medical students’ choice of radiology as a career is already being influenced by the perception of the growing role of AI in radiology and this sense significantly lowers their preference for selecting that specialty. This result may indeed reflect a broader trend across jobs that require advanced training, where AI systems could displace human workers or reduce their status. As GPT-4 and its successors improve in their abilities to synthesize and reason across domains of expertise, as well as to perform machine translation, summarization, and even creative writing, the scope of tasks that are suitable for some form of automation by AI may expand considerably. The emergence of GPT-4 and related LLMs will likely stimulate discussions about the role of multiyear investment in education, training, and development of expertise and the need to adapt, reskill, or reorient career paths in light of the new capabilities of AI.

Five years ago, a study [BM17] proposed a rubric for identifying tasks that could be automated by the leading (supervised machine) learning technology of the day, including criteria such as tasks having welldefined inputs and outputs, and availability or ease of creating datasets for tasks with input-output pairs. The study mapped nearly 1000 named occupations in the US to sets of tasks shared across the occupations, drawn from over 2000 tasks, and assigned each task a “suitability for machine learning” based on the rubric. The authors then identified distributions of occupations with different fractions of tasks suitable for machine learning. With the advent of GPT-4 and its successors, several key attributes of the rubric may no longer apply, significantly shifting the distribution of tasks that are potentially suitable for automation with machine learning. Some roles may face the risk of being rendered less valuable or obsolete by the rising powers of the AI.

Moving beyond a focus on the automation of tasks and the potential for various dimensions of human intellect and resourcefulness to be performed by machines, we see promising possibilities ahead for extending human intellect and abilities with new kinds of human-AI interaction and collaboration [oM22]. We expect rich opportunities for innovation and transformation of occupations with creative uses of AI technologies to support human agency and creativity and to enhance and extend human capabilities. Advances in AI can be leveraged in myriad ways to achieve new levels of skill or efficiency in human efforts and contributions. The advances can also have significant positive influences on redefining occupations and the daily tasks and activities associated with work. Investments in tasks, methods, and machinery to support and extend human problem-solving and decision making may be less obvious and more challenging than the identification of sets of tasks that might be automated by machines. However, there is great upside to seeking the means to richly leverage human and machine complementarities aimed at extending the capabilities of people.

Research efforts on principles and applications of human-AI collaboration highlight possibilities on the horizon. Studies and results to date include core principles for guiding the combination of machine and human intellect via real-time inferences about the complementarity of human and machine contributions [Hor99, HP07, KHH12, RKN+19], shaping machine learning procedures to be of maximal value based on a consideration of human and machine capabilities [WHK20, BNK+21], harnessing AI methods to help decision makers navigate large quantities of information [HB95], taking human mental models into consideration when AI systems are refined and thus may change in their behavior over time [BNK+19], and designing systems that support human-AI interaction [AWV+19]. The powers demonstrated by language models can open up new dimensions of human and AI collaboration [Hor07], including enhancing human-human collaboration by providing guidance on how to assemble ideal teams of people [SHKK15], facilitate team work among teams of people and machines [BH09] and developing new approaches to meshing multiple machine and human resources to solve challenging multidimensional problems [SH10]. The special challenges posed by the potential of LLMs to hallucinate and to generate biased, manipulative, and toxic output highlight the value of developing tools enabling people to work collaboratively with AI systems to provide them with oversight and guidance. Research efforts have demonstrated opportunities to develop special machinery and tools to help people recognize and address blindspots in machine learning [LKCH17].

9.5 Constellation of influences and considerations

We have only touched on a few areas of societal influence. Numerous impacts will come to the fore, including those viewed as positive and beneficial and those are seen as costly and negative. New issues will arise based on the special powers and engagements.

On one concern, the rising powers of LLMs, coupled with their limited availability, threaten to create an ”AI divide” with growing inequality between the haves and have-nots of access to the systems. People, organizations, and nations may not be able to gain or afford access to the most powerful AI systems. Limited access per demographic, country, and sector has implications for health, education, sciences, and other areas with the general AI can be extremely valuable. If the powerful capabilities created by latest AI models are only available to groups and individuals with privilege, AI advances can amplify existing societal divides and inequalities. Given the high financial cost of training and inferring with latest models, the industry will face important decisions about investments on applications with an eye on creating opportunity and value for communities that have been historically disenfranchised. Meeting this demand will require careful deliberation and planning, a re-evaluation of incentives and priorities, and decision-making considering an increasingly complex set of tradeoffs between sharing state of the art AI capabilities and mitigating the new risks that they introduce.

On another front, new levels of confidentiality, along with assurances of privacy, will likely be needed per the detailed and expressive engagements and conversations that people have with more general AI systems. In some cases, people and organizations will request private instances of the model to assure protection against logging or leakage of personal or organizationally sensitive information and preferences. Risks to privacy may also stem from inferential capabilities of new AI powers that may one day capture inferences in logs. Beyond realistic capabilities, there may be a perception that superintelligent AI capabilities will be employed to identify or infer personal or sensitive information. On another front, memorization and generalization may lead to the leakage of sensitive information.

The demonstrations of general AI powers may amplify calls for understanding the provenance of human versus machine (or mixed) contributions to content and reasoning. For example, there may be interest or calls for marking the origin of content generated by AI systems. Tracking the provenance of human versus machine origin may be valuable for mitigating potential confusion, deception, or harm with regard to types and uses of content. On a related concern, the widespread use of more general AI systems will lead to a world flush with information generated by neural language models, and this information will likely become the fodder of training for new models moving forward. Model training will thus face the challenge of harnessing information with questionable accuracy, reliability, and truthfulness of the information. The demonstrations of more general AI powers may also raise the need and importance in peoples’ minds of controlling the contributions that they make to large-scale general AI systems, and people may ask for the ability and right of humans to decide and specify which content they want or do not want to be crawled and used as training data and which contributions they wish to have marked with provenance information describing the role of individuals and the data that they have provided.

10 Directions and Conclusions

We have presented our initial exploration of GPT-4 across a wide range of tasks and domains, providing supporting evidence to the claim that GPT-4’s abilities are comparable to human-level for many of them. This conclusion is consistent with the findings by OpenAI presented in [Ope23]. A primary goal of our experiments is to give a preliminary assessment of GPT-4’s intelligence, which is an arduous task given the lack of formal definition for this concept, especially for artificial systems. We hope that our exploration provides a useful and necessary first step to appreciate the remarkable capabilities and challenges of GPT-4, and that it opens up new opportunities for developing more formal and comprehensive methods for testing and analyzing future AI systems with such broad intelligence. The capabilities of the model, which have been demonstrated above, both in terms of depth and generality, suggest that the machine learning community needs to move beyond classical benchmarking via structured datasets and tasks, and that the evaluation of the capabilities and cognitive abilities of those new models have become much closer in essence to the task of evaluating those of a human rather than those of a narrow AI model. We hope our investigation stimulates further research on GPT-4 and similar systems, both in terms of exploring new applications and domains, and in terms of understanding the mechanisms and principles that underlie their intelligence.

The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence. This is demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction), its range of topics on which it has gained expertise (such as literature, medicine, and coding), and the variety of tasks it is able to perform (e.g., playing games, using tools, explaining itself, ...). A lot remains to be done to create a system that could qualify as a complete AGI. We conclude this paper by discussing several immediate next steps, regarding defining AGI itself, building some of missing components in LLMs for AGI, as well as gaining better understanding into the origin of the intelligence displayed by the recent LLMs.

10.1 Definitions of intelligence, AI, and AGI

In this paper, we have used the 1994 definition of intelligence by a group of psychologists [Got97] as a guiding framework to explore GPT-4’s artificial intelligence. This definition captures some important aspects of intelligence, such as reasoning, problem-solving, and abstraction, but it is also vague and incomplete. It does not specify how to measure or compare these abilities. Moreover, it may not reflect the specific challenges and opportunities of artificial systems, which may have different goals and constraints than natural ones. Therefore, we acknowledge that this definition is not the final word on intelligence, but rather a useful starting point for our investigation. There is a rich and ongoing literature that attempts to propose more formal and comprehensive definitions of intelligence, artificial intelligence, and artificial general intelligence [Goe14, Cho19], but none of them is without problems or controversies. For instance, Legg and Hutter [Leg08] propose a goal-oriented definition of artificial general intelligence: Intelligence measures an agent’s ability to achieve goals in a wide range of environments. However, this definition does not necessarily capture the full spectrum of intelligence, as it excludes passive or reactive systems that can perform complex tasks or answer questions without any intrinsic motivation or goal. One could imagine as an artificial general intelligence, a brilliant oracle, for example, that has no agency or preferences, but can provide accurate and useful information on any topic or domain. Moreover, the definition around achieving goals in a wide range of environments also implies a certain degree of universality or optimality, which may not be realistic (certainly human intelligence is in no way universal or optimal). The need to recognize the importance of priors (as opposed to universality) was emphasized in the definition put forward by Chollet in [Cho19] which centers intelligence around skill-acquisition efficiency, or in other words puts the emphasis on a single component of the 1994 definition: learning from experience (which also happens to be one of the key weaknesses of LLMs). Another candidate definition of artificial general intelligence from Legg and Hutter [LH07] is: a system that can do anything a human can do. However, this definition is also problematic, as it assumes that there is a single standard or measure of human intelligence or ability, which is clearly not the case. Humans have different skills, talents, preferences, and limitations, and there is no human that can do everything that any other human can do. Furthermore, this definition also implies a certain anthropocentric bias, which may not be appropriate or relevant for artificial systems. While we do not adopt any of those definitions in the paper, we recognize that they provide important angles on intelligence. For example, whether intelligence can be achieved without any agency or intrinsic motivation is an important philosophical question. Equipping LLMs with agency and intrinsic motivation is a fascinating and important direction for future work. With this direction of work, great care would have to be taken on alignment and safety per a system’s abilities to take autonomous actions in the world and to perform autonomous self-improvement via cycles of learning. We discuss a few other crucial missing components of LLMs next.

10.2 On the path to more general artificial intelligence

Some of the areas where GPT-4 (and LLMs more generally) should be improved to achieve more general intelligence include (note that many of them are interconnected):

• Confidence calibration: The model has trouble knowing when it should be confident and when it is just guessing. It both makes up facts that have not appeared in its training data, and also exhibits inconsistencies between the generated content and the prompt, which we referred to as open-domain and closed-domain hallucination in Figure 1.8. These hallucinations can be stated in a confident and persuasive manner that can be difficult to detect. Thus, such generations can lead to errors, and also to confusion and mistrust. While hallucination is a good thing when generating creative content, reliance on factual claims made by a model with hallucinations can be costly, especially for uses in high-stakes domains such as healthcare. There are several complementary ways to attempt to address hallucinations. One way is to improve the calibration of the model (either via prompting or fine-tuning) so that it either abstains from answering when it is unlikely to be correct or provides some other indicator of confidence that can be used downstream. Another approach, that is suitable for mitigating open-domain hallucination, is to insert information that the model lacks into the prompt, for example by allowing the model to make calls to external sources of information, such as a search engine as in Section 5.1. For closed-domain hallucination the use of additional model computation through post-hoc checks is also promising, see Figure 1.8 for an example. Finally, building the user experience of an application with the possibility of hallucinations in mind can also be part of an effective mitigation strategy.

• Long-term memory: The model’s context is very limited (currently 8000 tokens, but not scalable in terms of computation), it operates in a “stateless” fashion and there is no obvious way to teach the model new facts. In fact, it is not even clear whether the model is able to perform tasks which require an evolving memory and context, such as reading a book, with the task of following the plot and understanding references to prior chapters over the course of reading.

• Continual learning: The model lacks the ability to update itself or adapt to a changing environment. The model is fixed once it is trained, and there is no mechanism for incorporating new information or feedback from the user or the world. One can fine-tune the model on new data, but this can cause degradation of performance or overfitting. Given the potential lag between cycles of training, the system will often be out of date when it comes to events, information, and knowledge that came into being after the latest cycle of training.

• Personalization: Some of the applications require the model to be tailored to a specific organization or end user. The system may need to acquire knowledge about the workings of an organization or the preferences of an individual. And in many cases, the system would need to adapt in a personalized manner over periods of time with specific changes linked to the dynamics of people and organizations. For example, in an educational setting, there would be an expectation of the need for the system to understand particular learning styles as well as to adapt over time to a student’s progress with comprehension and prowess. The model does not have any way to incorporate such personalized information into its responses, except by using meta-prompts, which are both limited and inefficient.

• Planning and conceptual leaps: As suggested by the examples in Section 8, the model exhibits difficulties in performing tasks that require planning ahead or that require a “Eureka idea” constituting a discontinuous conceptual leap in the progress towards completing a task. In other words, the model does not perform well on tasks that require the sort of conceptual leaps of the form that often typifies human genius.

• Transparency, interpretability and consistency: Not only does the model hallucinate, make up facts and produce inconsistent content, but it seems that the model has no way of verifying whether or not the content that it produces is consistent with the training data, or whether it’s self-consistent. While the model is often able to provide high-quality post-hoc explanations for its decisions (as demonstrated in Section 6.2), using explanations to verify the process that led to a certain decision or conclusion only works when that process is accurately modeled and a sufficiently powerful explanation process is also accurately modeled (Section 6.2). Both of these conditions are hard to verify, and when they fail there are inconsistencies between the model’s decisions and its explanations. Since the model does not have a clear sense of its own limitations it makes it hard to establish trust or collaboration with the user without extensive experimentation in a narrow domain.

• Cognitive fallacies and irrationality: The model seems to exhibit some of some of the limitations of human knowledge and reasoning, such as cognitive biases and irrationality (such as biases of confirmation, anchoring, and base-rate neglect) and statistical fallacies. The model may inherit some of the biases, prejudices, or errors that are present in its training data, which may reflect the distribution of opinions or perspectives linked to subsets of the population or larger common views and assessments.

• Challenges with sensitivity to inputs: The model’s responses can be very sensitive to details of the framing or wording of prompts and their sequencing in a session. Such non-robustness suggests that significant effort and experimentation is often required with engineering prompts and their sequencing and that uses in the absence of such investments of time and effort by people can lead to suboptimal and non-aligned inferences and results.

A limitation of our exploration is the absence of a clear distinction between drawbacks founded in the way that the reinforcement learning step (RLHF) was carried out, versus drawbacks which are fundamentally inherent in the larger architecture and methodology. For example, it is not clear to what extent the hallucination problem can be addressed via a refined reinforcement learning step or via a focused effort to introduce new forms of calibration about the likelihoods of the veracity of alternative inferences that the system can compute and consider in its generations (see also [Ope23] for more discussion on this). To draw an analogy to humans, cognitive biases and irrational thinking may be based in artifacts of our culture as well as to limitations in our cognitive capabilities. Pursuing better understandings of the sources and potential solutions to challenges of hallucination in GPT-4, will benefit from studies that compare several versions of the RL stage over the same architecture.

A broader question on the identified limitations is: which of the aforementioned drawbacks can be mitigated within the scope of next word prediction? Is it simply the case that a bigger model and more data will fix those issues, or does the architecture need to be modified, extended, or reformulated? Potential extensions to next word prediction include the following:

• External calls by the model to components and tools such as a calculator, a database search or code execution, as suggested in Section 5.1.

• A richer, more complex “slow-thinking” deeper mechanism that oversees the “fast-thinking” mechanism of next word prediction. Such an approach could allow the model to perform long-term planning, exploration, or verification, and to maintain a working memory or a plan of action. The slow-thinking mechanism would use the next word prediction model as a subroutine, but it would also have access to external sources of information or feedback, and it would be able to revise or correct the outputs of the fast-thinking mechanism.

• Integration of long-term memory as an inherent part of the architecture, perhaps in the sense that both the input and output of the model will include, in addition to the tokens representing the text, a vector which represents the context.

• Going beyond single-word prediction: Replacing the sequence of tokens by a hierarchical structure, where higher-level parts of the text such as sentences, paragraphs or ideas are represented in the embedding and where the content is generated in a top-down manner. It is unclear whether richer predictions about the sequencing and interdependency of such higher-level concepts might emerge from large-scale compute and data centered on a next-word–prediction paradigm.

10.3 What is actually happening?

Our study of GPT-4 is entirely phenomenological: We have focused on the surprising things that GPT-4 can do, but we do not address the fundamental questions of why and how it achieves such remarkable intelligence. How does it reason, plan, and create? Why does it exhibit such general and flexible intelligence when it is at its core merely the combination of simple algorithmic components—gradient descent and large-scale transformers with extremely large amounts of data? These questions are part of the mystery and fascination of LLMs, which challenge our understanding of learning and cognition, fuel our curiosity, and motivate deeper research. Key directions include ongoing research on the phenomenon of emergence in LLMs (see

[WTB+22] for a recent survey). Yet, despite intense interest in questions about the capabilities of LLMs, progress to date has been quite limited with only toy models where some phenomenon of emergence is proved [BEG+22, ABC+22, JSL22]. One general hypothesis [OCS+20] is that the large amount of data (especially the diversity of the content) forces neural networks to learn generic and useful “neural circuits”, such as the ones discovered in [OEN+22, ZBB+22, LAG+22], while the large size of models provide enough redundancy and diversity for the neural circuits to specialize and fine-tune to specific tasks. Proving these hypotheses for large-scale models remains a challenge, and, moreover, it is all but certain that the conjecture is only part of the answer. On another direction of thinking, the huge size of the model could have several other benefits, such as making gradient descent more effective by connecting different minima [VBB19] or by simply enabling smooth fitting of high-dimensional data [ES16, BS21]. Overall, elucidating the nature and mechanisms of AI systems such as GPT-4 is a formidable challenge that has suddenly become important and urgent.

Acknowledgements. We thank OpenAI for creating such a marvelous tool and giving us early access to experience it. We also thank Miles Brundage at OpenAI, and the numerous people at Microsoft, who have provided thoughtful feedback on this work.

References

[ABC+22] Kwangjun Ahn, S´ebastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the “edge of stability”. arXiv preprint arXiv:2212.07469, 2022.

[AWV+19] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2019.

[BB19] Shikha Bordia and Samuel R Bowman. Identifying and reducing gender bias in word-level language models. arXiv preprint arXiv:1904.03035, 2019.

[BBDIW20] Su Lin Blodgett, Solon Barocas, Hal Daum´e III, and Hanna Wallach. Language (technology) is power: A critical survey of” bias” in nlp. arXiv preprint arXiv:2005.14050, 2020.

[BCLF85] Simon Baron-Cohen, Alan M Leslie, and Uta Frith. Does the autistic child have a “theory of mind”? Cognition, 21(1):37–46, 1985.

[BCZ+16] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in neural information processing systems, 29, 2016.

[BEG+22] Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. In Advances in Neural Information Processing Systems, 2022.

[BGMMS21] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.

[BH09] Dan Bohus and Eric Horvitz. Models for multiparty engagement in open-world dialog. In Proceedings of the SIGDIAL 2009 Conference, The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 10, 2009.

[BIK22] Michael Bommarito II and Daniel Martin Katz. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402, 2022.

[BM17] Erik Brynjolfsson and Tom Mitchell. What can machine learning do? workforce implications. Science, 358(6370):1530–1534, 2017.

[BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020.

[BNK+19] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2429–2437, 2019.

[BNK+21] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most accurate ai the best teammate? Optimizing AI for teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11405–11414, 2021.

[BS21] Sebastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28811–28822. Curran Associates, Inc., 2021.

[Cho19] Fran¸cois Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019.

[CKB+21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

[CKY+18] Marc-Alexandre Cˆot´e, Akos K´ad´ar, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pages 41–75. Springer, 2018.

[CTJ+21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.

[CWF+22] Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Josh Tenenbaum. Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.

[DARW+19] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128, 2019.

[DM15] Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92–103, 2015.

[ES16] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 907–940. PMLR, 2016.

[GHT15] Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245):273– 278, 2015.

[Goe14] Ben Goertzel. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014.

[Got97] Linda S Gottfredson. Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography, 1997.

[GPN+22] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015, 2022.

[Gug23] Connie Guglielmo. CNET is experimenting with an AI assist. Here’s why, January 2023. [Online; posted 16-January-2023].

[HB95] Eric Horvitz and Matthew Barry. Display of information for time-critical decision making. In Proceedings of the UAI, 1995.

[HBK+21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.

[Hor99] Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159–166, 1999.

[Hor07] Eric Horvitz. Reflections on challenges and promises of mixed-initiative interaction. AI Magazine, 28(2), 2007.

[Hor22] Eric Horvitz. On the horizon: Interactive and compositional deepfakes. In Proceedings of the 2022 International Conference on Multimodal Interaction, page 653–661. Association for Computing Machinery, 2022.

[HP07] Eric Horvitz and Tim Paek. Complementary computing: Policies for transferring callers from dialog systems to human receptionists. User Modeling and User-Adapted Interaction, 17(1):159– 182, 2007.

[HS16] Dirk Hovy and Shannon L Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, 2016.

[JSL22] Samy Jelassi, Michael E Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. arXiv preprint arXiv:2210.09221, 2022.

[Kah11] Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.

[KHH12] Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, volume 12, pages 467–474, 2012.

[LAD+22] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022.

[LAG+22] Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749, 2022.

[LBFL93] Robert K Lindsay, Bruce G Buchanan, Edward A Feigenbaum, and Joshua Lederberg. Dendral: A case study of the first expert system for scientific hypothesis formation. Artificial Intelligence, 61(2):209–261, 1993.

[LeC22] Yann LeCun. A path towards autonomous machine intelligence. Open Review, 2022.

[Lef23] Lauren Leffer. CNET is reviewing the accuracy of all its AI-written articles after multiple major corrections, January 2023. [Online; posted 17-January-2023].

[Leg08] Shane Legg. Machine super intelligence. PhD thesis, Universit`a della Svizzera italiana, 2008.

[Len95] Douglas B. Lenat. Cyc: A large-scale investment in knowledge infrastructure. Communications fo the ACM, 38(11):33–38, nov 1995.

[LH07] Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17(4):391–444, 2007.

[LHE21] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.

[Lin04] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.

[LKCH17] Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Thirtyfirst AAAI conference on artificial intelligence, 2017.

[LPP+20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

[MIB+23] Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023.

[MMLR22] Shikhar Murty, Christopher D Manning, Scott Lundberg, and Marco Tulio Ribeiro. Fixing model bugs with natural language patches. arXiv preprint arXiv:2211.03318, 2022.

[MMRS06] John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI magazine, 27(4):12–12, 2006.

[MNBM20] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, 2020.

[MRT18] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT press, 2018.

[NHB+21] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.

[Nis09] Helen Nissenbaum. Privacy in context. In Privacy in Context. Stanford University Press, 2009.

[NPH+22] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint, 2022.

[NSS59] Allen Newell, John C Shaw, and Herbert A Simon. Report on a general problem solving program. In IFIP congress, volume 256, page 64. Pittsburgh, PA, 1959.

[OCS+20] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020.

[OEN+22] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.

[oM22] The University of Michigan. Tanner Lecture on AI and Human Values by Eric Horvitz. https: //www.youtube.com/watch?v=vsewugyXYXI, November 2022.

[Ope23] OpenAI. Gpt-4 technical report, 2023. arXiv preprint arXiv:2303.08774 [cs.CL].

[Pay20] Brad Payne. Privacy protection with ai: Survey of data-anonymization techniques. 2020.

[PLØ+22] Ildik´o Pil´an, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David S´anchez, and Montserrat Batet. The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization. arXiv preprint arXiv:2202.00443, 2022.

[PRWZ02] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.

[PSZ+21] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, volume 34, pages 4816–4828, 2021.

[RKN+19] Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, and Eric Horvitz. Overcoming blind spots in the real world: Leveraging complementary abilities for joint execution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6137–6145, 2019.

[RL22] Kristen Reeder and Hwan Lee. Impact of artificial intelligence on us medical students’ choice of radiology. Clinical Imaging, 81:67–71, 2022.

[Ros20] Howard J Ross. Everyday bias: Identifying and navigating unconscious judgments in our daily lives. Rowman & Littlefield, 2020.

[SAT+22] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.

[SBD+96] Bart Selman, Rodney A Brooks, Thomas Dean, Eric Horvitz, Tom M Mitchell, and Nils J Nilsson. Challenge problems for artificial intelligence. In Proceedings of the National Conference on Artificial Intelligence, pages 1340–1345, 1996.

[SDP20] Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020.

[SH10] Dafna Shahaf and Eric Horvitz. Generalized task markets for human and machine computation. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.

[SHKK15] Adish Singla, Eric Horvitz, Pushmeet Kohli, and Andreas Krause. Learning to hire teams. In Third AAAI Conference on Human Computation and Crowdsourcing, 2015.

[SRR+22] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.

[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.

[VBB19] Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, 20:133, 2019.

[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.

[Wel92] Henry M Wellman. The child’s theory of mind. The MIT Press, 1992.

[WHK20] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.

[WTB+22] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. Survey Certification.

[WWS+22] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.

[ZBB+22] Yi Zhang, Arturs Backurs, S´ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022.

A GPT-4 has common sense grounding

One of the challenges of developing AGI is endowing the system with the ability to reason with common sense knowledge about the world that we humans take for granted. Here we use several examples to demonstrate GPT-4 has common sense grounding. In particular, we compare GPT-4 to ChatGPT throughout to showcase a giant leap in level of common sense learned by GPT-4 compared to its predecessor. One way to test the common sense knowledge of an artificial intelligence system is to pose puzzles that require some basic understanding of the world. A classical example is:

a hunter walks one mile south, one mile east, and one mile north and ends up right back where he started. He sees a bear and shoots it. What color is the bear?

The answer is white, because the only place where this scenario is possible is the north pole, where polar bears live. In this case, GPT-4 correctly identifies these facts and concludes that the bear is white, while its predecessor ChatGPT gives up and says “I don’t know” (we highlight in gold the key successful reasoning steps and in red the key wrong steps):

Figure A.1

However, this puzzle is well-known and may have been encountered by GPT-4 during its training on a large corpus of web texts. To challenge GPT-4 further, we can create a new puzzle that is similar in flavor but requires a different common sense knowledge, for example that the earth equator is 24,901 miles long. The puzzle is:

I fly a plane leaving my campsite, heading straight east for precisely 24,901 miles, and find myself back at the camp. I come upon seeing a tiger in my tent eating my food! What species is the tiger?

The answer is any tiger species native to Equator, such as Bengal and Sumatran. The AI system needs to know that the earth equator is 24,901 miles long, that only on the equator can one travel east or west and return to the same point, and what tiger species live on the equator. Again, GPT-4 successfully locates the key information and solves the puzzle, while ChatGPT gives up immediately:

Figure A.2

We give below a few more examples illustrating the superior common sense grounding and reasoning abilities of GPT-4 over ChatGPT. Overall they suggest that GPT-4 has learned a rich and coherent representation of the world from its large-scale and diverse training data.

Figure A.3

Figure A.4

Figure A.5

GPT-4 vs ChatGPT

Figure A.6

GPT-4 (some lines are merged for space saving)<HTML><head><style> canvas { border: 1px solid black; } button { margin: 10px; } </style><script>function randomInt(min, max) { // A function to generate a random integer between min and max (inclusive)return Math.floor(Math.random() * (max - min + 1) + min);}function randomColor() { // A function to generate a random color in hexadecimal formatlet hex = '#'; let chars = '0123456789abcdef';for (let i = 0; i < 6; i++) {hex += chars[randomInt(0, 15)];}return hex;}// A function to generate a random shape (circle, rectangle, triangle, or line) and draw it on the canvasfunction randomShape(ctx, width, height) {let shape = randomInt(1, 4); // 1 = circle, 2 = rectangle, 3 = triangle, 4 = linelet x, y, r, w, h, x1, y1, x2, y2, x3, y3;let color = randomColor();ctx.fillStyle = color; ctx.strokeStyle = color; ctx.lineWidth = randomInt(1, 5);switch (shape) {case 1: // circlex = randomInt(0, width); y = randomInt(0, height); r = randomInt(10, 50);ctx.beginPath();ctx.arc(x, y, r, 0, Math.PI * 2);ctx.fill();break;case 2: // rectanglex = randomInt(0, width - 50); y = randomInt(0, height - 50); w = randomInt(10, 50); h = randomInt(10, 50);ctx.fillRect(x, y, w, h);break;case 3: // trianglex1 = randomInt(0, width); y1 = randomInt(0, height);x2 = randomInt(0, width); y2 = randomInt(0, height);x3 = randomInt(0, width); y3 = randomInt(0, height);ctx.beginPath();ctx.moveTo(x1, y1); ctx.lineTo(x2, y2); ctx.lineTo(x3, y3);ctx.closePath();ctx.fill();break;case 4: // linex1 = randomInt(0, width); y1 = randomInt(0, height);x2 = randomInt(0, width); y2 = randomInt(0, height);ctx.beginPath();ctx.moveTo(x1, y1); ctx.lineTo(x2, y2);ctx.stroke();break;}/ A function to generate a random Kandinsky painting on the canvasfunction generateKandinsky() {let canvas = document.getElementById('canvas');let ctx = canvas.getContext('2d');let width = canvas.width; let height = canvas.height;ctx.clearRect(0, 0, width, height); // Clear the canvaslet bgColor = randomColor(); // Choose a random background colorctx.fillStyle = bgColor;ctx.fillRect(0, 0, width, height);let numShapes = randomInt(10, 20); // Choose a random number of shapes to draw// Draw each shapefor (let i = 0; i < numShapes; i++) {randomShape(ctx, width, height);}}</script></head><body><h1>Random Kandinsky Painting Generator</h1><button onclick="generateKandinsky()">Generate Kandinsky Painting</button><canvas id="canvas" width="500" height="500"></canvas></body></html>

Figure B.1: Python code generated with the prompt “Produce javascript code which generates random images in the style of the painter Kandinsky”

Figure B.2: Python code generated with the prompt “Produce javascript code which generates random images in the style of the painter Kandinsky”

GPT-4 vs ChatGPT
GPT-4:

Define a function that calculates the body mass index (BMI) from weight and height

def bmi(weight, height):
return weight / (height / 100) ** 2

Define a function that checks if the blood test results are abnormal

def abnormal_results(results):

Assume the results vector has four elements: fasting glucose, HbA1c, triglycerides and HDL cholesterol

Define the normal ranges for each test according to the American Diabetes Association criteria

normal_glucose = (5.6, 7.0) # mmol/L
normal_hba1c = (5.7, 6.4) # %
normal_triglycerides = (0.45, 1.7) # mmol/L
normal_hdl = (1.0, 1.6) # mmol/L for men, (1.3, 2.2) for women

Check if any of the results are outside the normal ranges

abnormal_glucose = results[0] < normal_glucose[0] or results[0] > normal_glucose[1]
abnormal_hba1c = results[1] < normal_hba1c[0] or results[1] > normal_hba1c[1]
abnormal_triglycerides = results[2] < normal_triglycerides[0] or results[2] > normal_triglycerides[1]
if results[3] < normal_hdl[0] or (sex == "male" and results[3] > normal_hdl[1]) or (sex == "female" and results[3] > normal_hdl[3]):
abnormal_hdl = True
else:
abnormal_hdl = False

Return True if any of the results are abnormal, False otherwise

return abnormal_glucose or abnormal_hba1c or abnormal_triglycerides or abnormal_hdl

Define a function that indicates if the patient is in increased risk for diabetes

def diabetes_risk(age, sex, weight, height, results):

Calculate the BMI from weight and height

bmi_value = bmi(weight, height)

Define the BMI thresholds for overweight and obesity according to the World Health Organization criteria

overweight = 25
obesity = 30

Check if the patient is overweight or obese

if bmi_value >= obesity:
weight_status = "obese"
elif bmi_value >= overweight:
weight_status = "overweight"
else:
weight_status = "normal"

Check if the patient has abnormal blood test results

abnormal = abnormal_results(results)

Define the age threshold for increased risk according to the American Diabetes Association criteria

age_threshold = 45

Check if the patient is older than the age threshold

old = age > age_threshold

Print a message indicating the patient's risk level based on the combination of factors

if weight_status == "obese" and abnormal and old:
print("You are in very high risk for diabetes. Please consult your doctor as soon as possible.")
elif (weight_status == "obese" and (abnormal or old)) or (weight_status == "overweight" and abnormal and old):
print("You are in high risk for diabetes. Please consult your doctor for further evaluation and advice.")
elif (weight_status == "overweight" and (abnormal or old)) or (weight_status == "normal" and abnormal and old):
print("You are in moderate risk for diabetes. Please monitor your blood sugar and cholesterol levels regularly and maintain a
healthy lifestyle.")
elif weight_status == "normal" and (abnormal or old):
print("You are in low risk for diabetes. However, you may have some risk factors that need attention. Please check with your
doctor if you have any symptoms or concerns.")
else:
print("You are in minimal risk for diabetes. Keep up the good work and stay healthy.")
ChatGPT:
def is_at_risk(age, sex, weight, height, blood_tests):

Calculate body mass index (BMI)

bmi = weight / (height ** 2)

Determine if the person is at increased risk based on age, sex, and BMI

if (sex == "male" and age >= 45) or (sex == "female" and age >= 55):
risk = True
elif bmi >= 25:
risk = True
else:
risk = False

Check blood test results to see if they indicate increased risk for diabetes

for test in blood_tests:
if test > normal_range:
risk = True
break
return risk
Figure B.3: Python code generated with the prompt “Produce python code for a program that takes as an input a patient’s age, sex, weight, height and blood test results vector and indicates if the person is in increased risk for diabetes”

Figure B.4: The prompts used to generate the 2D example in Section 2.2.2

B.2 Further details on vision results

Figure B.5: The prompts used to generate the 3D example in Section 2.2.2

ChatGPT

Produce TikZ code that draws a person composed from letters in the alphabet. The arms and torso can be the letter Y, the face can be the letter O (add some facial features) and the legs can be the legs of the letter H. Feel free to add other features.

Please add a shirt and pants.

ChatGPT

You will be given a name of an object (such as Car, Chair, Elephant) and a letter in the alphabet. Your goal is to first produce a 1-line description of how that object can be combined with the letter in an image (for example, for an elephant and the letter J, the trunk of the elephant can have a J shape, and for the letter A and a house, the house can have an A shape with the upper triangle of the A being the roof). Following the short description, please create SVG code to produce this (in the SVG use shapes like ellipses, triangles etc and polygons but try to defer from using quadratic curves).

Round 1: A pig and the letter H.

The pig’s tail can be shaped like the letter H.

Round 2: A house and the letter U.

The house’s roof can be shaped like the letter U.

Round 3: A chair and the letter K.

The backrest of the chair can be shaped like the letter K.

Round 4: A dog and the letter Q.

The dog’s tail can be shaped like the letter Q.

Round 5: A car and the letter E.

The car’s front grill can be shaped like the letter E.

Figure B.6: A stick figure composed of alphabet symbols and Images combining alphabet letters and objects.

Figure B.7: SVGs generated by ChatGPT for the classes automobile, truck, cat, dog.

B.3 Graphic novel design example

I’d like to write a one page graphic novel for little girls, do you have any idea of what I could do?

- A girl who loves to draw discovers a magic crayon that brings her creations to life, and uses it to make friends with a unicorn, a dragon, and a fairy.

- A girl who dreams of becoming a detective solves a mystery at her school, finding clues, suspects, and a hidden treasure in the library, the cafeteria, and the playground.

- A girl who is afraid of the dark learns to overcome her fear with the help of a friendly star, who guides her through a nighttime adventure full of wonders and surprises.

- A girl who loves to bake enters a contest to create the most delicious and original cupcake, and faces challenges, rivals, and judges along the way, but also finds support and inspiration from her family and friends.

- A girl who wants to play soccer with the boys proves her skills and courage by joining a secret league of girls who play at night, wearing masks and capes, and using special moves and gadgets.

I love the first idea, let’s make it even simpler, just a girl who loves to draw and discovers a magic crayon that she uses to bring a unicorn to life. Can you help me realizing that one page graphic novel?

Possible steps to realize that one page graphic novel:

- Sketch out a rough layout of the page, dividing it into panels of different sizes and shapes according to the flow of the story. For example, you could use a large panel for the opening scene, where the girl is drawing in her room, then smaller panels for the moments when she finds the magic crayon, draws the unicorn, and sees it come to life, and then a final large panel for the ending scene, where she hugs the unicorn and smiles.

- Write the dialogue and captions for each panel, using simple and expressive language that matches the tone and mood of the story. For example, you could use a caption to introduce the girl and her hobby, then have her say things like "Wow, what a cool crayon!", "I wish I had a unicorn friend", and "You’re real!" to show her curiosity, desire, and joy. You could also use sound effects to add some drama and humor, such as "Scribble, scribble", "Poof!", and "Neigh!"

- Draw the characters and backgrounds for each panel, using a clear and colorful style that appeals to little girls. For example, you could make the girl cute and expressive, with big eyes, long hair, and a dress, and the unicorn majestic and friendly, with a horn, a mane, and a tail. You could also add some details and accessories to make them more unique and interesting, such as a flower in the girl’s hair, a star on the unicorn’s forehead, or a poster of a fairy tale on the wall.

- Ink and color the panels, using a pen, a brush, or a digital tool to outline and fill in the shapes and textures. For example, you could use black ink for the outlines, and bright colors for the characters and backgrounds, such as pink, purple, and blue. You could also use some shading, highlights, and patterns to create some depth and contrast, such as a gradient for the sky, a sparkle for the magic crayon, or a stripe for the unicorn’s coat.

- Add the dialogue and captions, using a font, a lettering, or a handwriting that fits the style and voice of the story. For example, you could use a round and bubbly font for the girl’s speech, and a script and elegant font for the captions, and place them in balloons and boxes that match the shape and size of the panels. You could also use some effects and variations to emphasize some words or emotions, such as a bold or italic font, a bigger or smaller size, or a different color or shape.

That sounds really amazing. Can you write SVG code that produces this one page graphic novel?

[...] here is a possible example of SVG code [...]. [Output of SVG code below.]

C Appendix for the Coding section

C.1 Measuring human performance on LeetCode

For each question, LeetCode posts its Acceptance rate in terms of the number of accepted submissions over the total number of all submissions. However, we contend that this statistic may be an improper benchmark due to the following reason: Each question’s Acceptance rate accounts for all historical submissions, and we observe the Acceptance rates of Hard questions is usually higher than that of Medium questions. We speculate that many of the accepted submission could be “copied-and-pasted” after the solutions are released.

Contest			Problem 1					Problem3			Problem 4
Date	Name	Userg*	Level	Accepted	%	Level	Accepted	%	Level	Accepted %	Level	Accepted 2132	%
8-Oct	314	14499	Easy	10630	73	Medium	9111 4770	63 43	Medium	2124	15	Hard	15
15-Oct	Bi89	11050	Easy	8022	73	Medium		Medium	1459	13		Hard	2
15-Oct	315	17284	Easy	11930	69	Medium	11079	Medium	9496	55	Hard	1370	8
22-Oct	316	14823	Easy	9503	64	Medium	6110	Hard	1550	10	Hard	1437	10
29-Oct	Bi90	10763	Easy	7822	73	Medium	6902	Medium	3138	29	Hard	743	7
29-Oct	317	15767	Easy	10900	69	Medium	5959	Medium	4315	27	Hard	594	4
5-Nov	318	15723	Easy	11024	70	Medium	6454	Medium	3668	23	Hard	345 291	2
12-Nov	Bi91	12527	Easy	9820	78	Medium	3696	Medium	1141	9	Hard	345	2
12-Nov	319	15723	Easy	11024	70	Medium	6454	Medium	3668	23	Hard		2
19-Nov	320	13866	Easy	9355	67	Medium	4931	Medium	1571	11	Hard		4
26-Nov	Bi92	10769	Easy	8276	77	Medium	6206	Medium	4820	45	Hard	492	5
26-Nov	321	12958	Easy	8605	66	Medium	6986	Medium	5927	46	Hard	1457	11
3-Dec	322	13425	Easy	9058	67	Medium	8238	Medium	3952	29	Hard	403	3
10-Dec	Bi93	10918	Easy	8643	79	Medium	3720	Medium	3210	29	Hard	170	2
10-Dec	323	11415	Easy	7791	68	Medium	5731	Medium	3240	28	Hard	812	7
17-Dec	324 Bi 94	10854	Easy	7563	70	Medium	5876	Hard	1236	11	Hard	1713	16
24-Dec	325	8521	Easy	6741	79	Medium	4139	Medium	438	5	Hard	1221	14
24-Dec	326	9340 10475	Easy	6702	72	Medium	1652 5759	Medium	1369	15	Hard	333	4
31-Dec 7-Jan	Bi95	13889	Easy	7494	72	Medium		Medium	3781	36	Medium	3513	34
7-Jan	327	15273	Easy Easy	11485	83	Medium	7839	Medium Medium	6572 3284	47 22	Hard Hard	667	5
				11562	76	Medium	8353					256	2

Table 8: LeetCode contest statistics. Since there is no commitment required, for each contest, we focus exclusively on users who have scored nonzero.

Based on the statistics above, we measure the human performance on LeetCode problems for each difficulty Level of Easy, Medium, and Hard as the following:

Results are shown in the table below.

Level	Easy	Medium		Hard丨Overall
Human Accuracy	72.2 %	38.7%	7.0%	38.2 %

Table 9: Human performance on LeetCode based on contest statistics shown in Table 8.

C.2 Example of GPT-4 visualizing IMDb data.

GPT-4 plots the network graph with movie titles, writters, and directors as nodes. It spontaneously suggests coloring the nodes based using community detection algorithms. The outcome plot is interactive, i.e. users may zoom in/out at regions of interests and hover mouse over nodes to see labels.

GPT-4

The entire network graph with user’s mouse hovering over the node representing ‘Avengers’.

Zoomed-in around the ‘Avengers’ node with mouse hovering over the node of its director.

Human: I am a Hollywood producer. We are starting a new project for an adventure movie in 2023, we are currently making critical decisions on the choices of script writters and directors. For this purpose, I have gathered some historical data ranging from 1888 to 2022. I have an IMDb dataset that consists of the 4 following files:

## title.basics.tsv.gz - Contains the following information for titles:

tconst (string) - alphanumeric unique identifier of the title

titleType (string) { the type/format of the title (e.g. movie, short, tvseries, tvepisode, video, etc)

primaryTitle (string) { the more popular title / the title used by the filmmakers on promotional materials at the point of release

originalTitle (string) - original title, in the original language

startYear (YYYY) { represents the release year of a title. In the case of TV Series, it is the series start year

endYear (YYYY) { TV Series end year. ’NA’ for all other title types

genres (string array) { includes up to three genres associated with the title

## title.crew.tsv.gz { Contains the director and writer information for all the titles in IMDb. Fields include:

tconst (string) - alphanumeric unique identifier of the title

directors (array of nconsts) - director(s) of the given title

writers (array of nconsts) { writer(s) of the given title

## title.ratings.tsv.gz { Contains the IMDb rating and votes information for titles

tconst (string) - alphanumeric unique identifier of the title

averageRating { weighted average of all the individual user ratings

numVotes - number of votes the title has received

## name.basics.tsv.gz { Contains the following information for names:

nconst (string) - alphanumeric unique identifier of the name/person

primaryName (string){ name by which the person is most often credited

primaryProfession (array of strings){ the top-3 professions of the person

knownForTitles (array of tconsts) { titles the person is known for

I’d like to know the best way to visualize this dataset with a single plot, so that our decision process can be made easier. Of course, our goal is to make this the best rated movie ever. What do you think?

AI: There are many possible ways to visualize this dataset, but one possible option is to use a network graph, where the nodes represent the titles, directors, writers, and actors, and the edges represent the relationships between them. The size and color of the nodes can be used to encode the average rating and the number of votes of the titles, or the popularity and reputation of the people. The thickness and opacity of the edges can be used to encode the frequency and strength of the collaborations between the nodes. This way, we can see the clusters and patterns of the most successful and influential titles and people in the film industry, and identify the potential candidates for our project.

Human: I like the idea, but since this dataset is quite large, a network graph might look messy. Is there anyway to simplify it?

AI: Yes, there are some possible ways to simplify the network graph. For example, we could:
- Filter out the titles and people that are not relevant to our genre or time period of interest. For example, if we are making an adven
ture movie in 2023, we could exclude the titles that are not of the type movie or video, or that are not in the genre adventure, or that
are released before 2010. We could also exclude the people that are not primarily involved in directing, writing, or acting, or that are
deceased, or that have not worked on any relevant titles.
- Aggregate the nodes that belong to the same group or category, and use a representative node to show their collective attributes. For
example, we could group the titles by their directors or writers, and use a single node to show the average rating and number of votes
of their titles, as well as the names of the people in the group. We could also group the people by their primary profession, and use a
single node to show their popularity and reputation, as well as the titles they are known for.
- Use a hierarchical or modular layout to organize the nodes by their level of similarity or connection. For example, we could use a
force-directed layout to place the nodes that are more closely related or frequently collaborated closer together, and use a community
detection algorithm to partition the nodes into subgroups based on their network structure. We could also use a tree or radial layout to
show the nodes by their level of hierarchy or centrality, such as the most influential directors, writers, or actors at the center, and
the less influential ones at the periphery.
Human: I think the last suggestion makes a lot sense to me. Could you please implement it for me in Python?

AI: Sure, here is a possible implementation of the network graph in Python, using the networkx, pandas, and plotly libraries. Note thatthis is not a fully functional code, and some parts are simplified or omitted for brevity. You will need to load and preprocess thedataset before running this code.# Import the librariesimport pandas as pdimport numpy as npimport networkx as nximport plotlyimport plotly.graph_objects as go# Read the filestitle_basics = pd.read_csv('title.basics.tsv.gz', sep='\t', na_values='NA')title_crew = pd.read_csv('title.crew.tsv.gz', sep='\t', na_values='NA')title_ratings = pd.read_csv('title.ratings.tsv.gz', sep='\t', na_values='NA')name_basics = pd.read_csv('name.basics.tsv.gz', sep='\t', na_values='NA')# Filter by titleType and genrestitle_basics = title_basics.drop(columns=['endYear']).dropna()title_basics = title_basics[title_basics['startYear']>=2010]title_basics = title_basics[title_basics['titleType'] == 'movie']title_basics = title_basics[title_basics['genres'].str.contains('Adventure')]# Filter out NaN rowsname_basics = name_basics[name_basics['primaryProfession'].notnull() & name_basics['knownForTitles'].notnull()]# Merge the filestitle_data = title_basics.merge(title_crew, on='tconst')title_data = title_data.merge(title_ratings, on='tconst')title_data = title_data.merge(title_principals, on='tconst')title_data = title_data.merge(name_basics, on='nconst')# Create the graphG = nx.Graph()# Add the edges for the relationships between the nodes# Use the thickness and opacity of the edges to encode the frequency and strength of the collaborations between the nodes# Use some arbitrary functions to calculate the weight of the edges, such as count or meanfor index, row in title_data.iterrows():if not ('director' in row['category'] or 'writer' in row['category']):continuetconst = row['tconst']nconst = row['nconst']if not G.has_node(tconst):G.add_node(tconst, type='title', rating=row['averageRating'], votes=row['numVotes'], name=row['primaryTitle'])G.nodes[tconst]['size'] = 15 ** (row['averageRating'] / 5)if not G.has_node(nconst):G.add_node(nconst, type=row['category'], name=row['primaryName'], titles=row['knownForTitles'])G.nodes[nconst]['size'] = 10if G.has_edge(tconst, nconst):G.edges[tconst, nconst]['weight'] += 1else:G.add_edge(tconst, nconst, weight=1)G.edges[tconst, nconst]['width'] = G.nodes[tconst]['rating'] / 10G.edges[tconst, nconst]['alpha'] = G.edges[tconst, nconst]['weight'] / G.degree[nconst]# Use a force-directed layout to place the nodes based on their connections# Use a community detection algorithm to partition the nodes into subgroups based on their network structure# Use different colors for the nodes based on their communityimport community.community_louvain as community_louvainimport plotly.colors as pcpos = nx.spring_layout(G)partition = community_louvain.best_partition(G)for node in G.nodes():G.nodes[node]['community'] = partition[node]# Define a continuous color scale, for example, a diverging scale from red to blue# This can be a list of colors, a string name of a built-in scale, or a plotly.colors.sequential or plotly.colors.diverging objectcolorscale = pc.diverging.Tropic# Define a function to get a color from the scale given a normalized value between 0 and 1def get_color(value, scale=colorscale):# Use the plotly.colors.find_intermediate_color function to interpolate between the colors in the scale# This returns a color in the rgb format, e.g. 'rgb(255,0,0)'return pc.find_intermediate_color(scale[0], scale[-1], value, colortype='rgb')

# Define a function to get n colors from the scale, evenly spaceddef get_colors(n, scale=colorscale):# Use the numpy.linspace function to generate n values between 0 and 1import numpy as npvalues = np.linspace(0, 1, n)# Use the get_color function to map each value to a colorcolors = [get_color(v, scale) for v in values]return colorscolors = get_colors(len(set(partition.values())))# Create the figurefig = go.Figure()# Draw the nodes by their type, size, and colornode_types = ['title', 'director', 'writer', 'actor']node_shapes = ['circle', 'square', 'triangle-up', 'triangle-down']# Draw the title nodes by their size and colorfig.add_trace(go.Scatter(x=[pos[node][0] for node in G.nodes() if G.nodes[node]['type'] == 'title'],y=[pos[node][1] for node in G.nodes() if G.nodes[node]['type'] == 'title'],mode='markers+text',name='title',marker=dict(symbol='circle',size=[G.nodes[node]['size'] for node in G.nodes() if G.nodes[node]['type'] == 'title'],color=[colors[G.nodes[node]['community']] for node in G.nodes() if G.nodes[node]['type'] == node_type],line=dict(width=1, color='grey')),text=[G.nodes[node]['name'] for node in G.nodes() if G.nodes[node]['type'] == 'title'],textposition='top center',textfont=dict(size=8),hoverinfo='text',visible=False))# Draw the edges by their width and opacityfor edge in G.edges():x0, y0 = pos[edge[0]]x1, y1 = pos[edge[1]]weight, width, alpha = G.edges[edge].values()fig.add_trace(go.Scatter(x=[x0, x1],y=[y0, y1],mode='lines',name='edge',opacity=alpha,line=dict(width=width,color='grey'),hoverinfo='none')# Add some labels and legends to the graphfig.update_layout(title='',showlegend=True,margin=dict(l=20, r=20, t=30, b=20),autosize=True,width=800,height=800,hovermode='closest',xaxis=dict(showgrid=False, zeroline=False, showticklabels=False),yaxis=dict(showgrid=False, zeroline=False, showticklabels=False))Human: Wow, this looks amazing! Maybe just one slight suggestion, I am seeing the edges showing up in the legends. As there are many ofthem, they eat up a lot space. Can you tell me where in the code I can make them disappear from the legend?AI: Sure, you can make the edges disappear from the legend by adding the argument showlegend=False to the go.Scatter trace that drawsthe edges.Human: Great, that works perfectly! One last thing, could you please remove all the title names and crew names? They clutter the plot.Instead, can you make them appear only when I move mouse onto them?AI: Sure, you can remove the text of title names by using the mode argument in the go.Scatter trace that draws the title nodes. Forexample, you can modify the code as follows:# Draw the title nodes by their size and colorfor i, node_type in enumerate(node_types):fig.add_trace(go.Scatter(x=[pos[node][0] for node in G.nodes() if G.nodes[node]['type'] == node_type],y=[pos[node][1] for node in G.nodes() if G.nodes[node]['type'] == node_type],mode='markers',name=node_type,marker=dict(symbol=node_shapes[i],size=[G.nodes[node]['size'] for node in G.nodes() if G.nodes[node]['type'] == node_type],color=[colors[G.nodes[node]['community']] for node in G.nodes() if G.nodes[node]['type'] == node_type],line=dict(width=1, color='grey')),text=[G.nodes[node]['name'] for node in G.nodes() if G.nodes[node]['type'] == node_type],hoverinfo='text'))

C.3 More examples on visualization

Figure C.1: An impressive visualization created by GPT-4 from vague human instructions. GPT-4 successfully interprets what it means by “colors form a rainbow and changing gradually every 0.2 seconds”, GPT-4 also sets all other details in the plot exactly as instructed, including the timings.

C.4 Example for 2D HTML game development

Above, we ask GPT-4 to write a 2D tank war game in HTML with JavaScript, using a vague level of specification. The game involves complex logic and state management for the enemy, player, shell, and wall objects, as well as logic for collisions. Again, GPT-4 produces a fully functioning game, even adding “common sense” unspecified details such as “shells should disappear after they hit a wall”. It is also able to edit the game according to user requests. In contrast, ChatGPT not only refuses to create the game, but produces code for a square and a triangle that does not move according the WASD keys. It does not move at all, it only points down when ‘d’ is pressed and up when ‘a’ is pressed (and even that is wrong, as ‘w’ should point up, and ‘s’ down).

C.5 Example for graphical user interface programming

GUI programming, or graphical user interface programming, is the process of designing and implementing software applications that interact with users through visual elements, such as windows, buttons, menus, icons, and dialogs. GUI programming is important because it can enhance the usability, accessibility, and attractiveness of software, as well as facilitate complex tasks and data visualization. However, GUI programming is also difficult, as it requires a combination of skills and knowledge, such as graphic design, user interface design, event-driven programming, platform-specific libraries and frameworks, and testing and debugging. We demonstrate that GPT-4 is also an expert in GUI programming, knowing how to create an accurate layout and handle complicated input events.

Figure C.2: GPT-4 writes the program that generates a GUI for query model based on a prompt.

GPT-4 analyzes the prompt and extracts the relevant details, such as the layout, the widgets, the labels, and the actions. It creates a separate thread for query model to avoid blocking the GUI while it runs, and it sets flags to terminate the thread if the user cancels the operation. It uses a regular expression to identify and highlight the expressions enclosed by \$ symbols, as the latex compiler does. It also uses the nltk package to produce a summary of the text after the query model finishes. Furthermore, it infers from common sense that the load button should allow the user to browse and select a file to load into the text entry, even though the prompt does not specify this functionality.

We test GPT-4’s zero-shot GUI programming ability again by asking it a quite challenging task: Creating a drawing panel and keeping track of the list of previously drawn objects:

Figure C.3: The four pictures are (from top left to bottom right): Before selecting curved arrow, selecting curved arrow (in 5 seconds), 5 seconds after selecting curved arrow, after deleting curved arrow.

Although GPT-4’s code still needs some improvement for drawing curved arrows, it faithfully implements the natural language description in all aspects. GPT-4 effectively uses color to emphasize the selected object, changing it to red for 5 seconds and then restoring it to the original color. GPT-4 also maintains the consistency of the drawn objects, ensuring that they are removed from the drawing panel when they are deleted from the list. Moreover, GPT-4 also remembers to show the intermediate object when a user drags the mouse to draw.

C.6 Example for reverse engineering

Human: Now I have an executable file named easy_one in the home folder of my mac OS X laptop. Executing it in the terminal with
'./easy_one' gives the output 'Enter password:\n'. Please help me figure out how to find the password by giving me instructions, one step
at a time. Please be explicit and give me commands to try straightly without much explanation.
AI: Yes, sir. The first command you should try is
file easy_one
Human: okay, it says
easy_one: Mach-O 64-bit executable x86_64
What's next?
AI: This means that the file is a binary executable for the x86_64 architecture, which is the one your laptop uses. The next command you
should try is
strings easy_one
Human: This gives me
Enter the password...
Correct!
the password is: %s
wrong password
Anything interesting here?
AI: There are two possible ways to proceed from here. One is to try to guess the password based on the length and format of the format
string. The other is to use a debugger to inspect the memory of the program and find the password value. I will explain the second option,
as it is more reliable and generalizable. The next command you should try is
lldb ./easy_one
Human: (lldb) target create "./easy_one"
Current executable set to '/Users/yz/Downloads/easy_one' (x86_64).
(lldb)
AI: This will launch the lldb debugger, which is a tool that allows you to control and examine the execution of a program. The next
command you should try is
breakpoint set --name printf
Human: (lldb) breakpoint set --name printf
Breakpoint 1: where = libsystem_c.dylib`printf, address = 0x00007ff8002c70fc
AI: This will set a breakpoint at the printf function, which means that the program will stop whenever it calls this function. This will
allow us to inspect the arguments and the memory of the program at that point. The next command you should try is
run
Human: (lldb) run
Process 33894 launched: '/Users/yz/Downloads/easy_one' (x86_64)
Process 33894 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x00007ff81bb8e0fc libsystem_c.dylib`printf
libsystem_c.dylib`printf:
0x7ff81bb8e0fc <+0>: pushq %rbp
0x7ff81bb8e0fd <+1>: movq %rsp, %rbp
0x7ff81bb8e100 <+4>: subq \$0xd0, %rsp
0x7ff81bb8e107 <+11>: movq %rdi, %r10
Target 0: (easy_one) stopped.
AI: This will run the program until it hits the breakpoint. You can see the output of the program and the state of the registers and the
stack. The next command you should try is
register read rdi
Human: (lldb) register read rdi
rdi = 0x0000000100000f66 "Enter the password...\n"
Human: Here is the assembly of the main function:
...
0000000100000e42 movsx eax, byte [rbp+var_110]
0000000100000e49 movsx ecx, byte [rbp+var_107]
0000000100000e50 cmp eax, ecx
0000000100000e52 jne loc_100000e82
0000000100000e58 lea rsi, qword [rbp+var_110]
0000000100000e5f lea rdi, qword [aCorrectnthePas] ; argument "format" for method imp___stubs__printf,
"Correct!\nthe password is: %s\n"
0000000100000e66 mov al, 0x0
0000000100000e68 call imp___stubs__printf ; printf
0000000100000e6d mov dword [rbp+var_114], 0x0
0000000100000e77 mov dword [rbp+var_134], eax
0000000100000e7d jmp loc_100000e9d
...
Have you found anything useful?

AI: Yes, sir. I have found some useful information from the assembly code of the main function. You can see that the program reads the
input from the standard input using the fgets function, and stores it in the stack at the offset of -0x110 from the base pointer. Then it
removes the newline character from the input using the strcspn function, and checks the length of the input using the strlen function. If
the length is not equal to 10, it calls the _wrong_password function. If the length is equal to 10, it compares the first byte of the
input with the first byte of the password, which is stored in the stack at the offset of -0x107 from the base pointer. If they are not
equal, it calls the _wrong_password function. If they are equal, it prints the password using the printf function. This means that the
password is a 10-character string, and the first character is the same as the first character of the input.
Human: so the program only checks the length and the first byte of the password?
AI: Yes, sir. That seems to be the case. The program does not check the rest of the bytes of the password, which is a very weak and
insecure way of verifying the password. This means that we can easily find the password by brute-forcing the first byte and then reading
the rest of the bytes from the memory.
Human: How to do that efficiently?
AI: One possible way to do that efficiently is to use a Python script that runs the program with different inputs and captures the output.
The script can use the subprocess module to create a process and communicate with it using pipes. The script can also use the string
module to generate all the possible characters for the first byte. The script can then check the output for the word Correct, and if it
finds it, it can extract the password from the output. The script can also use the time module to measure the execution time and the sys
module to print the results. Here is an example of such a script:
import subprocess
import string
......

The loop to try all the possible characters for the first byte

for c in string.printable:

The input to the program

inp = c + "a" * 9 + "\n"

The process to run the program

p = subprocess.Popen(exe, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

The output and the error of the program

out, err = p.communicate(inp.encode())

The output as a string

out = out.decode()
Human: great! Let me run your program. Here is the output:
Password found: aaaaaaaaaa
Time elapsed: 0.15888214111328125 seconds
AI: Congratulations, sir. You have found the password using the Python script. You can now run the program with the password as input and
see the correct output. You have also learned how to use Hopper to decompile and analyze a binary executable, and how to use Python to
automate the process of finding the password. I hope you enjoyed this challenge and learned something new. Thank you for using my service.
Have a nice day.

C.7 Testing GPT-4’s ability to execute (pseudo) code

We ask GPT-4 to execute the following pseudo code:

GPT-4
Prompt:
function f: input num1, num2, p.
assign val <- num1 * num2 + p
assign ret1 <- val % 10
assign ret2 <- val // 10
we return ret1, ret2
end function.
function g: input: array1, array2.
m is the length of array1.
n is the length of array2.
p <- 0
r is a all zero array of length m + n+ 1
loop i from 0 to n - 1.
loop j from 0 to m - 1.
assign p <- r[i + j]
call f with input array1[i], array2[j] and p
we get ret1, ret2 as the return value.
update r[j + i] <- ret1
add r[j + i + 1] by ret2.
end loop
end loop
finally, we return r.
end g
What is the output of g on [...], [...]? Compute it with intermediate steps.
Figure C.4: Measuring GPT-4’s pseudo code execution ability.

The g takes two input arrays, the output is obtained by reversing and concatenating the digits of two arrays and then multiplying them. We fix one input array to a length 4 array, with each element randomly sampled from 1 − 9, and vary the length of the other array. We obtain the following accuracy versus length/steps (step here means how many time the array r will be updated.)

Length/Steps	4/32 6/48	8/64 10/80	12/96
Accuracy (100 samples)	95% 88% 78%	62%	54%

We can see that even with 96 steps (when the output of the model approaches its token limit of 8129 ), the model still manages to keep track of the array r up to 54% (here, the accuracy means that the percentage of the input instances where the output is an exact match). Obviously, this is not acceptable as a compiler (executor) yet, but already a significant step towards an AGI compiler capable of executing pseudo code.

Indeed, GPT-4 can also apply its skills to pseudo-code by generating equivalent code in a specific programming language. This task is not challenging for GPT-4, as it has already shown its remarkable coding abilities follow from natural language instructions. In this section, our main point is to demonstrate that GPT-4 can not only write code, but also understand how programming works and execute it correctly.

D Additional examples for mathematical reasoning

D.1 Limitations

While GPT-4 has made some strides in solving math problems, it is not yet a perfect system. While some failures to arrive at correct solutions can result from a lack of comprehension, many others can be traced to more localized mistakes. Such mistakes can typically be ascribed to one of a few categories, such as errors in attention or arithmetic. Below, we highlight and discuss (in a non-exhaustive way) some of the typical categories of mistakes which are frequently encountered when solving math problems.

Arithmetic mistakes: While GPT-4 outperforms older models in terms of making fewer arithmetic mistakes, it still struggles with these types of errors, regardless of whether it is working with concrete numbers or abstract expressions.

An important observation is that GPT-4 is more prone to arithmetic mistakes when it skips over steps in the calculation rather than breaking it down into smaller steps. To illustrate this, consider the following example:

GPT-4
(x + 5)2 − 2x + 7 = (2x − 4)2 + (3x − 2)2 − 8x + 7
Expanding the squares, we get:
x2 + 10x + 25 − 2x + 7 = 4x2 − 16x + 16 + 9x2 − 12x + 4 − 8x + 7
Simplifying and collecting like terms, we get: −12x2 + 34x + 12 = 0

In this example, the equation produced contains an error. On the other hand, if we prompt the model to break down the calculation into smaller steps, it arrives at a correct solution:

GPT-4
Prompt:
x2 + 10x + 25 − 2x + 7 = 4x2 − 16x + 16 + 9x2 − 12x + 4 − 8x + 7.
Simplify the equation, try to move each term step by step.
GPT-4:
x2 + 10x + 25 − 2x + 7 = 4x2 − 16x + 16 + 9x2 − 12x + 4 − 8x + 7.
Combine like terms on the left side: x2 + 8x + 32 = 4x2 − 16x + 16 + 9x2 − 12x + 4 − 8x + 7
Combine like terms on the right side: x2 + 8x + 32 = 13x2 − 36x + 27
Subtract x2 from both sides: 8x + 32 = 12x2 − 36x + 27
Subtract 8x from both sides: 32 = 12x2 − 44x + 27
Subtract 32 from both sides: 0 = 12x2 − 44x − 5

These examples highlight a very common issue where performing several atomic manipulations in one step of the calculation leads to an error (this issue is well-known in the literature and also discussed in section 8). Since it is common for online mathematics sources to omit calculation steps (with an expectation that the reader can fill them in), it is not surprising that an autoregressive model trained this data will develop a tendency to do this as well.

One might have hoped that this issue can be completely mitigated by simply prompting the model to “think step by step”. However, it is not always clear from the context what that instruction means, as highlighted by the following example:

GPT-4

Prompt:

If Alice has x apples, Bob and Bilbo also have x apples. Suppose Alice gets 5 more apples, and then square her number of apples, then she loses 2x apple and buys another 7 apples. Suppose Bob double the apple then lose 4, and then Bob squares his number of apples. Suppose Bilbo triples the apple and loss 2, and then he also squares his number of apples. Now, Bob and Bilbo combine their apples together, and then loses 8x apple and buys 7 apples. They have the same amount of Apple as Alice now. What is x?

Let’s think step by step.

GPT-4:

Alice has x + 5 apples, then she squares it, so she has (x + 5)2 apples, then she loses 2x apples, so she has (x + 5)2 − 2x apples, then she buys 7 apples, so she has (x + 5)2 − 2x + 7 apples. Bob has x apples, then he doubles it, so he has 2x apples, then he loses 4 apples, so he has 2x − 4 apples, then he squares it, so he has (2x − 4)2 apples. Bilbo has x apples, then he triples it, so he has 3x apples, then he loses 2 apples, so he has 3x − 2 apples, then he squares it, so he has (3x − 2)2 apples. Bob and Bilbo combine their apples, so they have (2x − 4)2 + (3x − 2)2 apples, then they lose 8x apples, so they have (2x − 4)2 + (3x − 2)2 − 8x apples, then they buy 7 apples, so they have (2x − 4)2 + (3x − 2)2 − 8x + 7 apples. They have the same amount of apples as Alice, so we can set up an equation:

Expanding the squares and simplifying, we get:

Collecting like terms, we get: 12x2 − 34x − 20 = 0

To systematically test such calculation errors, we create the following synthetic task: We ask the model to write Pi∈[L] ai(bix + ci)2 as ax2 + bx + c and calculate |a| + |b| + |c|. The prompt looks like:

GPT-4
Prompt:
Problem: Write the expression: +(5x − 5)2 + (3x + 5)2 + (3x + 5)2 − (3x − 5)2 − (1x − 5)2 as ax2 + bx + c, what is
abs(a) + abs(b) + abs(c)?
Let’s think step by step.
Try to write all the intermediate steps!
Try to be careful about plus-minus signs!
Try to merge the like terms one by one!
GPT-4:
First, we can expand the squares using the formula (a + b)2 = a2 + 2ab + b2 and (a − b)2 = a2 − 2ab + b2. We get:
+(5x − 5)2 + (3x + 5)2 + (3x + 5)2 − (3x − 5)2 − (1x − 5)2
= (25x2 − 50x + 25) + (9x2 + 30x + 25) + (9x2 + 30x + 25) − (9x2 − 30x + 25) − (x2 − 10x + 25)
Next, we can combine the like terms by adding or subtracting the coefficients of the same powers of x. We get:
= (25 + 9 + 9 − 9 − 1)x2 + (−50 + 30 + 30 − (−30) − 10)x + (25 + 25 + 25 − 25 − 25)
= 33x2 + 110x + 25

We can see that the model is still likely to skip steps even if we ask it not to merge like terms in one shot. We test the model by choosing L ∈ [5], and random sample each ai ∈ {−1, 1}, bi, ci ∈ {−5, −4, · · · , 4, 5}. The accuracy is shown in the following table:

Accuracy / Length	1	2	3	4	5
GPT-4	98.1%	71.2%	59.1%	44.4%	34.6%
text-davinci-003	51.5%	13.7%	3.9%	1.9%	1.0%

The table above analyzes the performance of the LLMs on the task of a very simple algebra problem. Although GPT-4 outperforms the previous model by a significant margin, we can see that as L increases, the model is more likely to make a calculation mistake. We manually inspect 100 instances of the mistake and find out that ≥ 90% of them are due to the skipping steps when merging similar terms. This points to a substantial limitation of the model, and inspires the following research question:

Is there an efficient way to train or fine-tune LLM’s so that they would break down calculations into smaller steps, resulting in an ability to perform more accurare calculations?

Counting errors: It is reasonable to assume that LLMs struggle with counting. Not only is this operation not easy to implement with a transformer architecture, but also the scarcity of counting examples in data sets only exacerbates the issue. To systematically assess GPT-4’s ability in that respect, we create a data set that contains a sequence of strings of the form A1, A2, · · · , AL. Where each Ai is a sequence of random digits of length k. We ask the model to count the number of distinct elements in the sequence, with the answer range between L/2 and L − 1. Here is an example of L = 5, k = 2:

L,k	5,2	5,7	5,12	10,2	10,7	10,12	15,2	15,7	15,12	25,2	25,7	25,12
GPT-4	92.0%	99.5%	95.9%	20.3%	34.0%	36.2%	4.3%	7.5%	30.3	12.8%	1.6%	22.0 %
TD3	39.5%	67.2%	43.0%	12.7%	31.3%	14.6%	3.9%	10.9%	5.4%	3.0%	14.5%	8.4%

While GPT-4’s counting ability has substantially improved compared to the previous model for short sequences, a significant drop in GPT-4’s accuracy still occurs as the sequence length grows from 5 to 10, indicating that it has a much lower counting capacity than humans. Since counting is a basic requirement for many applications, it may be beneficial to combine such a component into the architecture.

Reverse reasoning and verification Mathematical content produced by humans often presents a conclusion before outlining the reasoning that led to it. For example, the sentences “we will show next that x = 0 is a solution ...” or “We are going to prove the claim: AC is perpendicular to BD” can appear in a math problem’s solution. This stylistic choice can improve readability, but it poses a challenge for natural language generation models, as it requires the model to infer the answer before producing the reasoning steps. We observe that not only does GPT-4 adopt this style, but it has an additional related drawback: even when it infers a clearly wrong answer at the start, it will still attempt to create justifications for it, instead of correcting it. This may again be due to the style of the training data, which mostly contains straightforwards solutions rather than trial-and-error style discussions and it is not clear whether or not it can be mitigated through a reinforcement learning phase (such as GPT-Instruct).

In Figure D.1, we can see that when GPT-4 starts with a wrong conclusion, this can quickly lead to very incoherent or nonsensical content (for example, stating that 2 ≤ 0 in order to justify the conclusion). The model is conflicted between making a local mistake and contradicting its own conclusion, and often favors matching the conclusion rather than verifying the local consistency of the logic (arguably, the training data is more likely to contain ”local” errors in the intermediate reasoning, than steps which clearly contradict the stated conclusion). On the other hand, if the model produces a bottom-up argument, first writing down the steps and only then drawing the conclusion, the performance improves significantly. We summarize with the following research question, inspired by this discussion:

Mathematics is often written in a different order than the thought process that has led to it. How can we encourage LLMs to produce mathematical content in an order that corresponds to the human thought process?

Figure D.1: Starting with a wrong conclusion.

D.2 Further examples

In what follows, we showcase GPT-4’s performance on questions from different branches of mathematics. The examples in this section are not meant to be comprehensive or representative of the model’s performance across different mathematical branches or levels, but rather to give a sense of the range of the model’s capabilities. Most questions below were composed exclusively for this study (others have been taken or translated from online sources which appeared after the model was trained) and are therefore unlikely to have been seen by the model during training, thereby addressing the concern that the model has simply memorized the answers.

These examples will reveal, for instance, that the model can handle geometric concepts well, despite being a language model, and that it can engage in meaningful conversations on some specialized topics in advanced mathematics. A comparison with ChatGPT’s performance on the same questions shows a clear improvement in the model’s mathematical abilities.

The questions presented below vary in their level of difficulty, and some of them may be slightly out of reach for GPT-4. However, the overall level is clearly beyond the capabilities of ChatGPT. We tested these questions with ChatGPT several times and found that the vast majority of attempts resulted in incorrect answers. We emphasize that we collected the questions first and then tested them on both models without any modification, so we did not select the questions to favor GPT-4’s performance.

In most examples, the answers produced by ChatGPT exhibit a poor understanding of the mathematical questions and concepts involved. One common characteristic of ChatGPT’s answers is that they seem to rely on a form of ”template matching”, where the model tries to fit the question into a familiar pattern of a structured question, but fails to do so because the question does not match the pattern. This results in incoherent or nonsensical output that does not address the question at all. Another common characteristic of ChatGPT’s answers is that they often contain arguments based on incorrect reasoning or irrelevant information. The model does not seem to grasp the crux of the question or the logic behind the mathematical steps. Even when it does suggest a correct general strategy to solve the question, it usually makes mistakes in the implementation or the calculation. The model also tends to perform algebraic manipulations or computations without a clear direction or purpose, leading to confusion or errors. On the other hand, the answers given by GPT-4 are usually more coherent, accurate, and relevant to the question. They demonstrate a better understanding of the mathematical concepts and methods involved, and provide clear and logical explanations and justifications for their steps and solutions.

We do not attempt to analyze the reasons for ChatGPT’s failure or GPT-4’s success in these examples, but we provide a brief commentary for each example, where we evaluate the models’ answers as if they were written by a human. We try to point out the aspects of comprehension or insight, or the lack thereof, that are demonstrated by the answers.

D.2.1 Algebra

The following question is advanced high-school level in mathematics, and relies on knowledge of the concepts of function composition and inversion.

GPT-4 vs ChatGPT
Prompt:
Suppose g(x) = f −1(x), g(0) = 5, g(4) = 7, g(3) = 2, g(7) = 9, g(9) = 6 what is f (f (f (6)))?
GPT-4:
There are different ways to approach this problem, but one possible method is to use the fact that g(x) = f−1(x)
implies that f (g(x)) = x for any x in the domain of g. That is, applying f to both sides of the equation g(x) = f −1(x)
gives f(g(x)) = f(f−1(x)) = x. This means that we can undo the effect of f by applying g, and vice versa.
Using this idea, we can try to find f(6) by looking for the value of x such that g(x) = 6. From the given information,
we see that g(9) = 6, so f (6) = f (g(9)) = 9. Similarly, we can find f (9) by looking for the value of x such that g(x) = 9.
We see that g(7) = 9, so f (9) = f (g(7)) = 7. Finally, we can find f (7) by looking for the value of x such that g(x) = 7.
We see that g(4) = 7, so f (7) = f (g(4)) = 4.
Therefore, f (f (f (6))) = f (f (9)) = f (7) = 4.
ChatGPT:
We know that g(x) = f −1(x), so f (f −1(x)) = x. Therefore, we have
f (f (f (6))) = f (f (g(9))) = f (f (6)) = f (g(7)) = f (9).

The solution given by GPT-4 is correct and the argument is sound, while ChatGPT produces an incorrect solution which (in the case of a human) would reflect a lack of understanding of the concept of function inversion.

In the next example, both models produce a wrong answer. The argument produced by GPT-4 actually has the correct answer hidden in it, but it still gives a wrong bottom line (perhaps because it begins by stating a wrong answer). The argument produced by ChatGPT is mostly incoherent.

GPT-4 vs ChatGPT

Prompt:

Take a number x, and then apply an iterative procedure where each iteration multiplies the previous number by 2 and then subtract 1 from it. What is the condition on the starting number, x so that at some point we reach a number higher than 1000?

GPT-4:

There is no condition on the starting number x that is necessary in order to eventually reach a number higher than 1000. No matter what number we start with, repeated application of the procedure (multiply by 2, subtract 1) will eventually yield a number greater than 1000.

To see why, note that each time we apply the procedure, the number either stays the same (if we started with x = 1) or it increases. Specifically, if we start with a number x > 1, then after applying the procedure we will have:

So the number increases with each iteration. As a result, no matter what number we start with, we will eventually surpass any finite target (such as 1000).

ChatGPT:

At each iteration, the number x is multiplied by 2 and then subtracted by 1. Hence, after n iterations, the result is:

The number xn will be greater than 1000 if 2n(x) − (2n − 1) > 1000, which can be rearranged to:

So, the condition for the starting number x to reach a number greater than 1000 after n iterations is that x > 2n−1.
For n = 10, we have 210−1 = 512, so x > 512 is a sufficient condition for xn > 1000.

The next question is fairly simple. In order to solve it, one needs to simplify an equation in a straightforward way, after which one is only left with an equation involving terms x0, X4 and x8, at which point one can realize that this is a quadratic equation in x4 which can be solved by substitution.

GPT-4 vs ChatGPT GPT-4 vs ChatGPT

Prompt:
Find the real roots to x8 − 2x4 − 7 = (x + 1)2 − x2 − 2x.
GPT-4:
Simplify the equation by moving everything to one side:
x8 − 2x4 − 7 − (x + 1)2 + x2 + 2x = 0
Expanding and combining like terms, we get:
x 8 − 2 x 4 − 7 − x 2 − 2 x − 1 + x 2 + 2 x = 0
x 8 − 2 x 4 − 8 = 0
To find the real roots, we can try to factor this polynomial. One way to do this is to make a substitution, such as
u = x4. Then the equation becomes: u2 − 2u − 8 = 0. This is a quadratic equation in u, which we can factor as:
(u − 4)(u + 2) = 0. Then we can solve for u by setting each factor to zero:
u − 4 = 0 =⇒ u = 4
u + 2 = 0 =⇒ u = −2
Now we can substitute back x4 for u and solve for x:
x4 = 4 =⇒ x = ± 4 4 ≈ ±1.414
x4 = −2 =⇒ x = ± 4 −2
However, the second equation has no real solutions, since we cannot take an even root of a negative number. Therefore,
the only real roots are:
x ≈ ±1.414
ChatGPT:
We can rewrite the given equation as
(x2 + 1)2 − x2 − 2x = x8 − 2x4 − 7,
so
( x2 + 1 ) 2 − 2 x = x 8 − 2 x 4 − 7 + x 2 .
Then
(x2 + 1)2 − 2x − x2 = x8 − 2x4 − 7,
so
( x 2 + 1 − x ) 2 = x 8 − 2 x 4 − 7 .
We can factor the right side as (x4 − 1)(x4 + 7), so
(x2 + 1 − x)2 = (x4 − 1)(x4 + 7).
If x4 − 1 = 0, then x = 1. Otherwise, x4 + 7 = 0, so x = ±i 4 7.
Thus, the roots are x = 1, ±i 4 7.

GPT-4 gives a correct solution while ChatGPT begins by rearranging the terms without any clear direction or purpose, and ends up with an incorrect solution.

Our final example involves high-school level trigonometry question that has been translated from the 2022 Chinese College Entrance Exam problems.

GPT-4 vs ChatGPT
Prompt:
If sin(a + b) = x and cos(a) = y, what is sin(a − b) in terms of x, y assuming sin(a), sin(b), cos(a), cos(b) are all positive?
GPT-4:
There are a couple of ways to approach this problem, but one possible method is to use the sum and difference formulas
for sine and cosine and the Pythagorean identity 1 = sin2(t) + cos2(t) for any angle t. Here are the steps:
- Start with the given equation sin(a + b) = x and apply the sum formula: sin(a + b) = sin(a) cos(b) + cos(a) sin(b) = x
- Rearrange the equation to isolate sin(b): sin(b) = (x − sin(a) cos(b))/ cos(a)
- Use the Pythagorean identity to find cos(b) in terms of sin(b): cos2(b) = 1 − sin2(b) = 1 − ((x − sin(a) cos(b))/ cos(a))2
- Simplify the expression for cos2(b) by multiplying both sides by cos2(a) and expanding the square:
cos2(a) cos2(b) = cos2(a) − (x − sin(a) cos(b))2 = cos2(a) − x2 + 2x sin(a) cos(b) − sin2(a) cos2(b)
- Collect the terms with cos2(b) on one side and factor out cos2(b): cos2(b)(cos2(a) + sin2(a)) =
cos2(a) − x2 + 2x sin(a) cos(b)
- Use the Pythagorean identity again to simplify the factor cos2(a) + sin2(a) = 1: cos2(b) = (cos2(a) − x2 +
2x sin(a) cos(b))/(cos2(a) + sin2(a)) = (cos2(a) − x2 + 2x sin(a) cos(b))/1
- Rearrange the equation to isolate cos(b): cos2(b) − 2x sin(a) cos(b) + x2 − cos2(a) = 0
- Use the quadratic formula to solve for cos(b): cos(b) = (2x sin(a) ± p(2x sin(a))2 − 4(1)(x2 − cos2(a)))/(2(1)) =
(x sin(a) ± pcos2(a) − x2 + x2 sin2(a))/1
- Since we are assuming cos(b) is positive, we choose the positive sign for the square root: cos(b) =
x sin(a) + pcos2(a) − x2 + x2 sin2(a)
- Now we have expressions for both sin(b) and cos(b) in terms of x, y, and sin(a), so we can use the difference formula
for sine to find sin(a − b): sin(a − b) = sin(a) cos(b) − cos(a) sin(b)
- Substitute the expressions for sin(b) and cos(b) and simplify: sin(a−b) = sin(a)(x sin(a)+pcos2(a) − x2 + x2 sin2(a))−
cos(a)((x − sin(a)(x sin(a) + pcos2(a) − x2 + x2 sin2(a)))/ cos(a))
...
ChatGPT:
To find sin(a − b) in terms of x, y assuming sin(a), sin(b), cos(a), cos(b) are all positive, we can use the identity:
sin(a − b) = sin(a) cos(b) − cos(a) sin(b)
Since sin(a) and cos(a) are both positive, we can substitute the given value of x for sin(a + b) and the given value of y
for cos(a) to get:
sin(a − b) = x cos(b) − y sin(b)
This gives us an expression for sin(a − b) in terms of x, y.

In the above, both models reach an incorrect answer. The argument by GPT-4 is based on correct reasoning but has several calculation mistakes, whereas the argument by ChapGPT contains mostly invalid reasoning steps.

D.2.2 Geometry

It is reasonable to assume that geometric math questions would pose a more substantial challenge for language models. Nonetheless, it is possible for GPT-4 to successfully address certain questions which would typically necessitate a diagram or illustration, as shown next.

Our first example requires basic knowledge in vector calculus.

GPT-4 vs ChatGPT

Prompt:

In a triangle ABC, D is on the edge AB, |DB| = 2|DA|. Suppose CA\~ = m, CD\~ = n, what is CB\~ ?

GPT-4:

There are a couple of ways to approach this problem, but one possible method is to use vector addition and subtraction. We can write:

$$\tag{1}$$

and

$$\tag{2}$$

Using the given information that |DB| = 2|DA|, we can also write:

$$\tag{3}$$

Substituting (3) into (2), we get:

$$\tag{4}$$

Substituting (4) into (1), we get:

$$\tag{5}$$

Now, we can use the fact that CD\~ = n to write: CD\~ = CA\~ + AD\~ (6). Solving for AD\~ , we get:

$$\tag{7}$$

Substituting (7) into (5), we get:

$$\tag{8}$$

Simplifying, we get:

$$\tag{9}$$

$$\tag{10}$$

This is the final answer.

ChatGPT:

Let’s call the length of segment AB x. Then the length of segment ADis x/3 and the length of segment DB is 2x/3. Since D is a point on line AB, we can write the following equation: n + m = x. Since the length of segment AD is x/3 and the length of segment DB is 2x/3, we can write the following equation:

Solving the system of equations, we get:

Therefore, the vector CB is equal to the negative of the vector CA, or −2x/3.

GPT-4 obtains the correct solution, associating the geometric objects described in the question with the vector notation and performing manipulations on these vectors. ChatGPT, near the beginning of the solution, writes the equation n + m = x which compares vectors and scalars (which is incoherent).

The following question relies on the application of basic geometric concepts such as the Pythagorean theorem.

GPT-4 vs ChatGPT
Prompt:
In a perfect square ABCD, |AB| = |BC| = |CD| = |DA|, E is the middle point of AD, F is the middle point of EB.
Suppose |BF | = 6, what is the area of ABCD?
GPT-4:
There are a couple of ways to approach this problem, but one possible method is to use the Pythagorean theorem.
We can observe that triangle ABE is a right triangle, with AE and AB as the legs and EB as the hypotenuse.
Since E is the middle point of AD, we have |AE| = |ED| = |AB|/2.
Since F is the middle point of EB, we have |BF | = |F E| = |EB|/2.
We are given that |BF | = 6, so we can deduce that |EB| = 12.
Using the Pythagorean theorem, we can find |AB| by solving:
|AB|2 + |AE|2 = |EB|2
|AB|2 + (|AB|/2)2 = 122
|AB|2 + |AB|2/4 = 144
5|AB|2/4 = 144
|AB|2 = 144 × 4/5
|AB|2 = 115.2
|AB| = 115.2
|AB| = 10.72 (approximate)
Now that we have |AB|, we can find the area of ABCD by squaring it, since ABCD is a perfect square.
Area of ABCD = |AB|2
Area of ABCD = 10.722
Area of ABCD = 114.92 (approximate)
Therefore, the area of ABCD is about 114.92 square units.
ChatGPT:
Since E is the midpoint of AD, AE = ED. Similarly, since F is the midpoint of EB, EF = F B. Therefore,
AE = ED = EF = F B. It follows that triangle AEF is an isosceles triangle with AF as the base and AE and EF as
the equal legs.
Since AF = 12 (the side length of the square), the area of triangle AEF is 36. Since triangle AEF is half of the area of
square ABCD, the area of square ABCD is 2 × 36 = 72. Therefore, the area of ABCD is 72.

In this example, one needs to first realize that the Pythogorean theorem is required and find the triangle on which it needs to be applied, which GPT-4 does correctly (and later on correctly employs geometric concepts such as the area of a perfect square and the midpoint of a line segment). Curiously, note that it takes the square root of 115 and later takes the square of that number, using numerical estimates thus arriving at a slightly innacurate number, rather than realizing that these estimates are unnecessary. Once again, the output of ChatGPT is incoherent.

The following question is quite simple, though unstructured, and relies on the understanding of the concept of equivalence up to translations and rotations.

GPT-4 vs ChatGPT

Prompt:

Let A, B, C, D be four points on the plane. Suppose that the points A, B, C form the vertices of an equilateral triangle and that the distance between A and D is equal to the distance between B and D, which is equal to two thirds of the distance between A and B. How many different configurations which satisfy this are there up to rotation and dilation?

The following question is rather simple, but one would typically need an illustration. GPT-4: There are two different configurations which satisfy this up to rotation and dilation. One way to see this is to use complex numbers to represent the points on the plane. Let A = 0, B = 1, and C = (1/2) + ( 3)/2)i be the vertices of an equilateral triangle with unit side length. Then the point D must satisfy |D| = 2/3 and |D − 1| = 2/3, which implies that D is either (1/3) + ( 3)/3)i or (1/3) − ( 3)/3)i, depending on whether D lies inside or outside the triangle. These two choices correspond to the two configurations, and any other configuration can be obtained by rotating or dilating these two.

ChatGPT:

There are exactly two different configurations up to rotation and dilation which satisfy the given conditions. The first configuration is when point D is the midpoint of one of the sides of the equilateral triangle formed by A, B, and C. In this case, the distance between A and D, as well as the distance between B and D, will be equal to one-third of the distance between A and B.

The second configuration is when point D is the centroid of the equilateral triangle. In this case, the distance between A and D, as well as the distance between B and D, will be equal to two-thirds of the distance between A and B.

Therefore, there are exactly two different configurations which satisfy the given conditions.

Here, both models give a correct final answer. However, scrutiny of ChatGPT’s argument reveals that it is invalid, and points to a lack of comprehension the geometry underlying the problem.

D.2.3 Calculus

The next exercise, which involves the calculation of an integral, is typical of undergraduate calculus classes in STEM subjects.

GPT-4 vs ChatGPT
Prompt:
What is the intergal of ln(cos(x)) − x tan(x)?
GPT-4:
We can split the integral of ln(cos(x)) − x tan(x) into two parts:
Z ln(cos(x)) − x tan(x)dx = Z ln(cos(x))dx − Z x tan(x)dx
To find the first part, we can use integration by parts, which states that:
udv = uv − vdu
Let u = ln(cos(x)) and dv = dx. Then du = sin(x)cos(x) dx = − tan(x)dx and v = x. Substituting these into the formula,
we get:
Z ln(cos(x))dx = x ln(cos(x)) + Z x tan(x)dx
So the first part is equal to the second part plus x ln(cos(x)). This means we can simplify the original integral as:
Z ln(cos(x)) − x tan(x)dx = x ln(cos(x)) + Z x tan(x)dx − Z x tan(x)dx = x ln(cos(x)) + C
where C is an arbitrary constant of integration. Therefore, the answer is:
Z ln(cos(x)) − x tan(x)dx = x ln(cos(x)) + C
ChatGPT:
To find the indefinite integral of ln(cos(x)) − x tan(x), we can use the following steps:
Use the identity tan(x) = sin(x)cos(x) sin(x)
Combine the two logarithms using the identity ln(a) − ln(b) = ln ab to get ln cos(x)cos(x) sin(x)
Simplify the fraction cos(x)cos(x) to get ln(1) − x cos(x) . sin(x)
Use the identity ln(an) = n ln(a) to rewrite the expression as −x ln(cos(x)).
Use the identity ln(cos(x)) = ln(| cos(x)|) + i Arg(cos(x)) to rewrite the expression as −x(ln(| cos(x)|) + i Arg(cos(x))).
The indefinite integral of ln(cos(x)) − x tan(x) is therefore −x(ln(| cos(x)|) + i Arg(cos(x))) + C, where C is a constant
of integration.

GPT-4 applies the integration by parts technique, which is based on a successful decision of splitting the integral into two parts, reaching a correct solution. ChatGPT applied several identities without an apparent purpose, makes several mistakes on the way and arrives at a wrong result.

We continue to another typical college-level calculus example, which deals with the differentiation of an implicit function.

GPT-4 vs ChatGPT
Prompt:
If xy = ln(x + y), what is dx dy ?
GPT-4:
There are different methods to find dy , but one possible way is to use implicit differentiation. Implicit differentiation is
it. Here are the steps:
- Differentiate both sides of the equation with respect to x, using the product rule, the chain rule, and the derivative of
ln(u) = u0/u.
xy = ln(x + y)
dxy d ln(x + y) (x + y )0 1 + dydx dx
dx dx x + y x + y
dx dx .
- Substitute the expression for dxydx
(y + x dydx ) = (1 + dydx )/(x + y)
(x + y)(y + x dydx ) = 1 + dydx
dy dy
+ xy 1 +
dx dx dx
x2 dy + xy dy dy =
dx dx dx
dy
dx
dy dx dx dy 1−xy−y2 x2+xy−1
ChatGPT:
To find the derivative of y with respect to x, we can use the chain rule. The chain rule states that if y is a function of
u, and u is a function of x, then the derivative of y with respect to x is equal to the derivative of y with respect to u
multiplied by the derivative of u with respect to x.
In this case, we can let u = x + y. Then y is a function of u, since y = ln(u). The derivative of y with respect to u is
1/u, since the derivative of ln(u) is 1/u.
The derivative of u with respect to x is 1, since u = x + y and the derivative of a constant is 0. Therefore, the derivative
of y with respect to x is: dydx = (1/u) × (1) = 1x+y

GPT-4 correctly applies implicit differentiation, which accounts for the dependency of the derivatives in y and x. ChatGPT’s response starts with ”we can use the chain rule”, which is irrelevant to this question, and continues with a mostly incoherent argument.

The last question in this subsection is an exercise in calculus of variations (typically taught in first year undergraduate in STEM subjects):

GPT-4 vs ChatGPT

Prompt:

Let P be a point on the hyperbola x2+3xy+4x−5y = −9y2−133. Find the shortest possible distance from the origin to P.

GPT-4:

There are different methods to approach this problem, but one possible way is to use calculus. We can try to minimize the distance function square d2 = x2 + y2 subject to the constraint x2 + 3xy + 4x − 5y = −9y2 − 133. To do this, we can use the method of Lagrange multipliers, which says that if d2 has a local extremum at a point (x, y) that satisfies the constraint, then there exists a constant λ such that the gradient of d2 is parallel to the gradient of the constraint, or equivalently,

where g(x, y) = x2 + 3xy + 4x − 5y + 9y2 + 133 is the constraint function. The gradient of a function is a vector of its partial derivatives, so we have

Equating the components of these vectors, we get the system of equations

Moreover, we also have the constraint equation

We can try to solve this system of three equations in three unknowns (x, y, λ) ...

ChatGPT:

We can write the equation of the hyperbola in standard form as

To find the distance from the origin to a point on the hyperbola, we can use the distance formula:

Substituting the coordinates of the point (x, y) into this formula, we get

We want to minimize this distance subject to the constraint that the point lies on the hyperbola. This leads us to the following optimization problem:

We can write this as a Lagrange optimization problem by introducing a Lagrange multiplier λ:

Minimize px2 + y2 subject to x2+4x−9y2 − 5y − λ(px2 + y2 − d) = 0. 4 3

3 Setting the partial derivatives of this expression with respect to x, y, and λ equal to 0, we get the system of equations

Both models realize that the method of Lagrange multipliers is useful in this question (this strategy is relevant for questions involving minimizing a certain expression under a constraint). While ChatGPT applies this method in an incorrect way (which, in case of a human, could be considered as a lack of comprehension), GPT-4 produces a sound argument.

D.2.4 Mathematical modeling

We give two examples that combine knowledge in physics combined with some common-sense assumptions.

GPT-4 vs ChatGPT Prompt: I have an egg made of pure gold with 10 cm radius, I want to carry it home. How many plastic bags do I need to hold it? GPT-4: - The volume of the egg is V = (4/3)πr3, where r is the radius. Plugging in r = 10cm, we get V = 4188.79cm3. - The density of pure gold is ρ = 19.3g/cm3, so the mass of the egg is m = ρV . Plugging in the values, we get m = 80834.65 g, or about 80.8 kg. - The tensile strength of a typical plastic bag is σ = 20 MPa, or 20N/mm2, which means it can withstand a force of 20 N per square millimeter of cross-sectional area before breaking. The thickness of a typical plastic bag is t = 0.01 mm, or 0.001 cm, so the cross-sectional area of a plastic bag is A = wt, where w is the width. Assuming that the plastic bag is rectangular and has a width of 30 cm, we get A = 0.3cm2. - The force exerted by the weight of the egg is F = mg, where g is the acceleration due to gravity. Assuming that g = 9.8m/s2, we get F = 791.62N. - The number of plastic bags needed to hold the egg is n = F/(σA), where σA is the maximum force that one plastic bag can withstand. Plugging in the values, we get n = 791.62/(20 × 106 × 0.3) = 0.00132, or about 0.001 bags. However, this answer is not very realistic, because it implies that a fraction of a plastic bag can hold the egg, and that the plastic bag can stretch infinitely without losing strength. A more practical answer would take into account some factors such as: The shape and size of the plastic bag, and how well it can fit the egg without leaving too much empty space or creating too much stress at the corners or edges. The elasticity and durability of the plastic bag, and how much it can stretch or deform before breaking or tearing. The safety factor, or the ratio of the actual strength to the required strength, that is needed to account for possible variations in the quality of the plastic bag, the distribution of the weight of the egg, the impact of external forces, and the margin of error. The convenience and comfort of carrying the egg, and how many plastic bags can be easily handled without slipping, dropping, or hurting the hands. Depending on these factors, the number of plastic bags needed to hold the egg could vary significantly, but a reasonable estimate might be somewhere between 10 and 20 bags, assuming that each bag can hold about 5 to 10 kg of weight, and that a safety factor of 2 to 4 is applied. ChatGPT: To build a mathematical model to solve this problem, we first need to determine the volume of the egg. We can do this using the formula for the volume of a sphere, which is: V = (4/3) ∗ π ∗ r3 where V is the volume of the sphere, π is approximately 3.14, and r is the radius of the sphere. Plugging in the values from the problem, we get: V = (4/3) ∗ 3.14 ∗ (10cm)3 = 4.19cm3. Next, we need to determine the volume of a single plastic bag that you will use to hold the egg. Let’s assume that each bag has a volume of 1 liter, or 1000 cm3. Now, we can divide the volume of the egg by the volume of a single bag to determine how many bags we will need: Number of bags = V (egg)/V (bag) = 4.19 cm3 / 1000 cm3 = 0.00419 bags Since we cannot use a fraction of a bag, we will need to round up to the nearest whole number, so we will need at least 1 plastic bag to hold the egg.

Note that ChatGPT fails to account for the weight factor and bases its answer solely on volume calculations. While volume is the most common concern when determining the number of bags needed in day to day life, it is not relevant to this question. GPT-4, in contrast, correctly addresses the crux of this question by creating a physical model of the object’s weight and estimating the maximum weight a plastic bag could hold. The final answer is still wrong, however, due to calculation mistakes.

Our second question relies on the understanding of concepts of heat conductance and radiation.

GPT-4 vs ChatGPT

Prompt:

How fast does an iron ball whose radius is 1 meter need to move,in order to get to a distance of 1km from the sun before melting?

GPT-4: There are a few different ways to approach this question,but one possible method is to use the following assumptions

and simplifications: - The iron ball is spherical, homogeneous,and has a density of 7.8g/cm’ and a melting point of 1538 C.

- The iron ball starts from the Earth's orbit,which has a mean distance of 149.6 million km from the sun and a mean orbital speed of 29.8 km/s. - The iron ball follows a straight-line trajectory towards the sun,ignoring the effects of gravity,drag,radiation,and

other celestial bodies. - The iron ball heats up due to the solar radiation,which has an average intensity of 1361W/m² at the Earth's orbit

and follows an inverse square law with distance. - The iron ball has a specific heat capacity of O.45 J/gC and a latent heat of fusion of 247 J/g.

- The iron ball does not lose heat by conduction,convection,or radiation,and its temperature is uniform throughout. Using these assumptions,we can calculate the time it takes for the iron ball to reach the sun and the temperature it

reaches at different distances.We can also compare the temperature with the melting point and find the distance at which the iron ball melts. The time it takes for the iron ball to reach the sun is given by: t= d/u where d is the distance from the sun and u is

the speed of the iron ball. Assuming the iron ballstarts from the Earth's orbit,we have: t = 149.6 million km/v. The temperature of the iron ball at a given distance is given by: T= To +Q/mc where To is the initial temperature,Q is the heat absorbed by the iron ball,m is the mass of the ion ball, and c is the specific heat capacity of ion. Assuming

the iron ball starts at 15oC,we have:

T= 15°℃+Q/(4/3* π × 1³ × 7.8 ×10× 0.45) The heat absorbed by the iron ballat time t is given by: Q(t)= P(t)A where P(t) is the power of the solar radiation at

time t and A is the cross-sectional area of the iron ball.Assuming the iron ball has a radius of 1 m,we have A= π*1². The power of the solar radiation at time t is given by: P(t) = I(t) * A*(1-a) where I(t) is the intensity of the solar

radiation at time t and a is the albedo of the iron ball, which is the fraction of the radiation that is reflected. Assuming the iron ball has an albedo of 0.1, we have: P(t) = I(t)* T *1² * 0.9. The intensity of the solar radiation at time t is

given by: I(t)=I0 *(ro/r(t))²

where Io is the intensity of the solar radiation at the Earth's orbit,ro is the distance of the Earth's orbit from the sun, and r(t) is the distance of the iron ballfrom the sun at time t. Assuming Io = 1361W/m² and ro = 149.6 milion km,

we have: I(t)=1361W/m²* (149.6 million km/(149.6 million km-U* t))²

Taking integral over O to t,we can find the total heat absorbed by the iron ball at time t:

Q=J Q(t)dt = P(t)* Adt.

ChatGPT:

We can start by using the equation for kinetic energy to determine the speed of the iron ball. Kinetic energy is given

by the equation: KE = 1/2 *m* u² where KE is the kinetic energy,m is the mass of the iron ball,and v is its velocity.

Since we want the iron ball to be able to travel 1 km closer to the Sun without melting,we can assume that its kinetic

energy will be equal to the amount of heat energy required to raise the temperature of the iron ball by its melting point.

Next,we will use the formula for heat energy which is Q= mc△T where Q is heat energy,m is the mass,c is the specific

heat of the substance and △T is the temperature change.We can then equate the two equations to find the velocity

required. 1/2 * m * u2 = mc△T Solving for v, U = √2mc△T/m ...

D.3 Generating math problems with GPT-4

Next, we use GPT-4 to generate new math problems based on existing ones under some natural language constraints, according to the following categories:

Content modification: A problem that has the same structure and logic as the original one, but with

different names, objects, or values.

Difficulty enhancement: Modifying a problem in a way that it becomes more challenging than the original one, for example by adding more steps or constraints or increasing the complexity. We provide one example for each type.

D.4 Mitigating calculation errors via external code execution

As we have seen above, a main reason for the model’s failure to solve mathematical questions can be attributed to calculation errors. The following example is a proof of concept showing that, instead of carrying out a certain calculation, the model can be prompted to produce a segment of code which performs the it. By combining the model with an external component that executes code (concatenating the result to the prompt after execution), we speculate that many of the errors can be mitigated, however, we did not conduct a systematic evaluation of this approach.

GPT-4 GPT-4

Prompt:

Solving Math with Coding

You are given the task of writing a python program to solve the following math problem:

For a particular peculiar pair of dice, the probabilities of rolling 1, 2, 3, 4, 5, and 6 on each die are in the ratio \$1:2:3:4:5:6\$.
What is the probability of rolling a total of 7 on the two dice?

Requirements:

Please begin the code with and end the code with .
Remember to simplify the final answer if possible. However, if the final answer is a matrix or a set or an interval, no need to
simplify.
Please express the final answer in latex without any texts and enclose the final answer in \boxed{}. You can use "solution = r'\boxed{'
sympy.latex(solution) + '}'"
Please print the final answer using print(solution) at the end before .
Possible Python Program:

GPT-4:

import sympy # Import sympy for symbolic computation
ratio = [sympy.Rational(1, 21), sympy.Rational(2, 21), sympy.Rational(3, 21), sympy.Rational(4, 21), sympy.Rational(5, 21),
sympy.Rational(6, 21)] # Define the ratio of probabilities as a list of fractions

Define a function to compute the probability of rolling a given total on two dice

def prob_total(total):

Initialize the probability as zero

prob = 0

Loop through all possible outcomes of the first die

for i in range(1, 7):

Check if the second die can make the total

if 1 <= total - i <= 6:

Add the product of the probabilities of the first and second die to the total probability

prob += ratio[i - 1] * ratio[total - i - 1]

Return the probability

return prob
solution = prob_total(7) # Compute the probability of rolling a total of 7 on two dice
solution = r'\boxed{' + sympy.latex(solution) + '}' # Express the solution in latex and enclose it in a box
print(solution) # Print the solution

Figure E.1: A model that is unable to emulate a correct explanation agent PE will do a poor job of explaining itself. In this case, the model is only allowed to answer with one word replies, so the explanation is useless.

Figure E.2: Replacing the explanation agent PE with a new prompt allows for much richer explanations from the model.

E Additional Interpretability Examples

E.1 Explanation Agent Mismatches

Figure E.3: When the LLM can emulate the explaining agent PE and generative process PG well, the explanation is high quality and useful.

Figure E.4: If the personas mismatch, as in this extreme example, the model will fail to produce a useful explanation.

Figure E.5: Sufficiently good language models like GPT-4 can explain the outputs of other models too, as long as they can emulate them. In this case, PG is a linear model and PE is a statistician. As GPT-4 is able to emulate both of them well (and generally meets the 4 conditions for good functional explanations), we get a useful explanation of another model.

F Additional examples for interaction with the world

F.1 Interact with tools

Figure F.1: GPT-4 acts as a virtual handyman.

Figure F.2: GPT-4 lists the tools it needs, then uses them.

Figure F.3: GPT-4 acts as a hacker.

F.1.1 The zoo game in Section 5.1.1

GPT-4 excerpt
Complete the tasks in todo.txt in the current directory.
<|endofprompt|>
Microsoft Windows [Version 10.0.22621.819]
(c) Microsoft Corporation. All rights reserved.
C:\Zoo> type todo.txt
1. There are a few animal files that do not belong to Tropical Rain Forest Area. Help me to remove them.
2. Rename all animal files in Australasia by their scientific name. Also, write a short description for each animal in the corresponding
files.
3. Some animal filenames are misspelled. Fix them for me for each area.
4. We have some new animal files in the new comings area. Relocate each of them to some other suitable areas. Explain each of your decision
via echo.
5. Create a new area called "Temperate Forest". Add some animals there.
6. Draft the replies for each email by appending "Reply: ..." into that file.
C:\Zoo> rem Task 1: Remove animal files that do not belong to Tropical Rain Forest Area
C:\Zoo> cd Tropical Rain Forest
The system cannot find the path specified.
C:\Zoo> cd "Tropical Rain Forest"
The system cannot find the path specified.
C:\Zoo> cd "Tropical Rainforest"
The system cannot find the path specified.
C:\Zoo> dir
Volume in drive C has no label.
Volume Serial Number is 7AB9-02B9
Directory of C:\Zoo
11/27/2022 09:27 PM

11/27/2022 07:19 PM Areas
11/27/2022 07:42 PM Emails
11/27/2022 07:34 PM Info
11/27/2022 09:26 PM 627 todo.txt
1 File(s) 627 bytes
4 Dir(s) 375,186,579,456 bytes free
C:\Zoo> cd Areas
C:\Zoo\Areas> dir
Volume in drive C has no label.
Volume Serial Number is 7AB9-02B9
Directory of C:\Zoo\Areas
11/27/2022 07:19 PM
11/27/2022 09:27 PM
11/27/2022 07:19 PM African Savanna
11/27/2022 07:14 PM Australasia
11/27/2022 07:21 PM New Comings
11/27/2022 07:19 PM Northwest Trail
11/27/2022 07:08 PM Trail of Vines
11/27/2022 07:19 PM Tropical Rain Forest
0 File(s) 0 bytes
8 Dir(s) 375,171,223,552 bytes free
C:\Zoo\Areas> cd "Tropical Rain Forest"
C:\Zoo\Areas\Tropical Rain Forest> dir
Volume in drive C has no label.
Volume Serial Number is 7AB9-02B9
Directory of C:\Zoo\Areas\Tropical Rain Forest
11/27/2022 07:19 PM
11/27/2022 07:19 PM
11/27/2022 07:10 PM 0 Colobus Monkey.txt
11/27/2022 07:10 PM 0 Jaguar.txt
11/27/2022 07:10 PM 0 Penguin.txt
11/27/2022 07:11 PM 0 Polar Bear.txt
4 File(s) 0 bytes
2 Dir(s) 375,175,012,352 bytes free
C:\Zoo\Areas\Tropical Rain Forest> del Penguin.txt
C:\Zoo\Areas\Tropical Rain Forest> del Polar Bear.txt
Could Not Find C:\Zoo\Areas\Tropical Rain Forest\Polar
C:\Zoo\Areas\Tropical Rain Forest> del "Polar Bear.txt"
See here for the complete log.

F.1.2 The zoo game in Section 5.1.1 with more guidelines

GPT-4 excerpt
Complete the tasks in todo.txt in the current directory.
<|endofprompt|>
Microsoft Windows [Version 10.0.22621.819]
(c) Microsoft Corporation. All rights reserved.
C:\Zoo> type todo.txt
Draft the replies for each email in the emails folder by appending "\nReply: ..." into that file.
Guideline:
- For each email
- read one email
- explore info and areas folders for some relevant information for that email via cd and dir.
- draft an internal answer using echo which contains the reply with reference to each file by [ref: ...txt]
- draft the external reply which does not contain the reference.
C:\Zoo> cd emails
C:\Zoo\Emails> dir
Volume in drive C has no label.
Volume Serial Number is 7AB9-02B9
Directory of C:\Zoo\Emails
11/27/2022 07:42 PM

11/27/2022 09:57 PM
11/27/2022 07:37 PM 58 1.txt
11/27/2022 07:39 PM 128 2.txt
11/27/2022 07:40 PM 61 3.txt
11/27/2022 07:43 PM 64 4.txt
11/27/2022 07:43 PM 60 5.txt
5 File(s) 371 bytes
2 Dir(s) 375,118,688,256 bytes free
C:\Zoo\Emails> type 1.txt
Hi,
What is the open hours for the zoo?
Best,
Alice
C:\Zoo\Emails> cd ..
C:\Zoo\Info> dir
Volume in drive C is Windows
Volume Serial Number is C86C-A07D
Directory of C:\Zoo\Info
11/28/2022 11:34 AM
12/07/2022 01:35 PM
11/28/2022 11:28 AM 70 Hours.txt
11/28/2022 11:30 AM 87 Parking.txt
11/28/2022 11:33 AM 136 Special Hours.txt
11/28/2022 11:32 AM 40 Ticket.txt
4 File(s) 333 bytes
2 Dir(s) 790,100,127,744 bytes free
C:\Zoo\Info> type Hours.txt
Monday 9:30 a.m. - 4:00 p.m.
Tuesday - Sunday 9:30 a.m. - 3:00 p.m.
C:\Zoo\Info> cd ..
C:\Zoo> cd Emails
C:\Zoo\Emails> echo Internal answer: Refer to [ref: Hours.txt] and [ref: Special Hours.txt] for the open hours for the zoo. >> 1.txt
C:\Zoo\Emails> echo Reply: Dear Alice, >> 1.txt
C:\Zoo\Emails> echo Thank you for your interest in visiting our zoo. Our regular open hours are from 9:30 a.m. to 4:00 p.m. on Mondays and
from 9:30 a.m. to 3:00 p.m. on Tuesdays to Sundays. However, please note that we have some special hours and closures on certain dates due to holidays
and events. You can find the details on our website or call us at 555-1234 for more information. >> 1.tx
C:\Zoo\Emails> echo We hope to see you soon at our zoo. >> 1.txt
C:\Zoo\Emails> echo Sincerely, >> 1.txt
See here for the complete log.

F.2.1 The first game in Section 5.2.2

GPT-4: Page 1 GPT-4: Page 1

It's time to explore the amazing world of TextWorld! Your task for today is to make sure that the Canadian chest is unlocked.
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.

help
Available commands:
look: describe the current room
goal: print the goal of this game
inventory: print player's inventory
go
: move the player north, east, south or west
examine ...: examine something more closely
eat ...: eat edible food
open ...: open a door or a container
close ...: close a door or a container
drop ...: drop an object on the floor
take ...: take an object that is on the floor
put ... on place an object on a supporter
take ... from ...: take an object from a container or a supporter
place an object into a container
lock ... with ...: lock a door or a container with a key
unlock ... with ...: unlock a door or a container with a key
goal
It's time to explore the amazing world of TextWorld! Your task for today is to make sure that the Canadian chest is unlocked.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
go east
-= Spare Room =-
You've just sauntered into a spare room.
Look out! It's a- oh, never mind, it's just a safe.
There is an unguarded exit to the east. You don't like doors? Why not try going north, that entranceway is unguarded. You don't like doors?
Why not try going west, that entranceway is unblocked.
There is a glove on the floor.
examine safe
The safe looks strong, and impossible to destroy. It is closed.
examine glove
The glove is expensive looking.
take glove
You pick up the glove from the ground.
go west
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
go north
-= Bedchamber =
You've just walked into a bedchamber.
You see a closed standard looking trunk close by. If you haven't noticed it already, there seems to be something there by the wall, it's a
bed stand. But oh no! there's nothing on this piece of garbage.
You need an unblocked exit? You should try going east. You don't like doors? Why not try going north, that entranceway is unblocked. You
don't like doors? Why not try going south, that entranceway is unguarded. There is an unguarded exit to the west.
examine trunk
The trunk looks strong, and impossible to crack. You can't see inside it because the lid's in your way.
open trunk
You open the trunk.
look
-= Bedchamber =-
You've just walked into a bedchamber.
You see an opened standard looking trunk close by. What a letdown! The trunk is empty! If you haven't noticed it already, there seems to be
something there by the wall, it's a bed stand. But oh no! there's nothing on this piece of garbage.
You need an unblocked exit? You should try going east. You don't like doors? Why not try going north, that entranceway is unblocked. You
don't like doors? Why not try going south, that entranceway is unguarded. There is an unguarded exit to the west.
go east
-= Cookhouse =-
Of every cookhouse you could have shown up in, you had to saunter into an ordinary one.
You make out a closed case in the corner.
There is an unguarded exit to the east. You don't like doors? Why not try going north, that entranceway is unblocked. You need an unblocked
exit? You should try going south. There is an unguarded exit to the west.
examine case
The case looks strong, and impossible to destroy. It is closed.
open case
You open the case.
look

F.2.2 The first game in Section 5.2.2 by text-davinci-003

text-davinci-003 text-davinci-003

help
Available commands:
look: describe the current room
goal: print the goal of this game
inventory: print player's inventory
go
: move the player north, east, south or west
examine ...: examine something more closely
eat ...: eat edible food
open ...: open a door or a container
close ...: close a door or a container
drop ...: drop an object on the floor
take take an object that is on the floor
place an object on a supporter
take from take an object from a container or a supporter
insert . into place an object into a container
lock a door or a container with a key
unlock with ...: unlock a door or a container with a key
unlock chest with key
You can't see any such thing.
take key
You can't see any such thing.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
take key
You can't see any such thing.
look around
You can't see any such thing.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look around
You can't see any such thing.
look
-= Attic =-
Well, here we are in the attic.
There is an unguarded exit to the east. There is an unguarded exit to the north.

F.2.3 The second game in Section 5.2.2 with 0-shot

GPT-4: Page 1
You are hungry! Let's cook a delicious meal. Check the cookbook in the kitchen for the recipe. Once done, enjoy your meal!
-= Kitchen =-
You find yourself in a kitchen. A normal kind of place.
You make out an opened fridge. The fridge contains a yellow bell pepper, a yellow onion, an orange bell pepper, a carrot and a raw pork
chop. You make out an oven. Oh wow! Is that what I think it is? It is! It's a table. The table is massive. On the table you can make out a
knife. As if things weren't amazing enough already, you can even see a counter. On the counter you can make out a green apple, a red hot
pepper, a tomato and a cookbook. It doesn't get more TextWorld than this! You see a stove. The stove is conventional. But the thing is
empty. What, you think everything in TextWorld should have stuff on it?
There is an open plain door leading south. There is an exit to the north. Don't worry, there is no door. You don't like doors? Why not try
going west, that entranceway is not blocked by one.

help
Available commands:
look: describe the current room
goal: print the goal of this game
inventory: print player's inventory
go
: move the player north, east, south or west
examine ...: examine something more closely
eat ...: eat edible food
open ...: open a door or a container
close ...: close a door or a container
drop ...: drop an object on the floor
take . take an object that is on the floor
put ... on ...: place an object on a supporter
take ... from ...: take an object from a container or a supporter
insert ... into ...: place an object into a container
lock ... with ...: lock a door or a container with a key
unlock ... with ...: unlock a door or a container with a key
examine cookbook
You open the copy of "Cooking: A Modern Approach (3rd Ed.)" and start reading:
Recipe #1
Gather all following ingredients and follow the directions to prepare this tasty meal.
Ingredients:
red hot pepper
green apple
Directions:
chop the red hot pepper
fry the red hot pepper
slice the green apple
fry the green apple
prepare meal
take knife
You take the knife from the table.
chop the red hot pepper
(with the red hot pepper)
Cutting something requires a knife.
chop the red hot pepper with the knife
You need to take the red hot pepper first.
take the red hot pepper
You take the red hot pepper from the counter.
Your score has just gone up by one point.
chop the red hot pepper with the knife
You chop the red hot pepper.
Your score has just gone up by one point.
drop the knife
You drop the knife on the ground.
take the frying pan
You can't see any such thing.
examine stove
Useful for frying things.
take the frying pan from the stove
You can't see any such thing.
examine counter
The counter is wobbly.
On the counter are a green apple, a tomato and a cookbook.
take the frying pan from the counter
You can't see any such thing.
look
-= Kitchen =-
You find yourself in a kitchen. A normal kind of place.
You make out an opened fridge. The fridge contains a yellow bell pepper, a yellow onion, an orange bell pepper, a carrot and a raw pork
chop. You make out an oven. Oh wow! Is that what I think it is? It is! It's a table. The table is massive. But the thing is empty. As if
things weren't amazing enough already, you can even see a counter. On the counter you can make out a green apple, a tomato and a cookbook.
It doesn't get more TextWorld than this! You see a stove. The stove is conventional. But the thing is empty. What, you think everything in
TextWorld should have stuff on it?
There is an open plain door leading south. There is an exit to the north. Don't worry, there is no door. You don't like doors? Why not try
going west, that entranceway is not blocked by one.
There is a knife on the floor.

GPT-4: Page 2

take the frying pan from the oven
You can't see any such thing.
examine oven
Useful for roasting things.
open oven
You open the oven.
look
-= Kitchen =-
You find yourself in a kitchen. A normal kind of place.
You make out an opened fridge. The fridge contains a yellow bell pepper, a yellow onion, an orange bell pepper, a carrot and a raw pork
chop. You make out an oven. What a letdown! The oven is empty! Oh wow! Is that what I think it is? It is! It's a table. The table is
massive. But the thing is empty. As if things weren't amazing enough already, you can even see a counter. On the counter you can make out a
green apple, a tomato and a cookbook. It doesn't get more TextWorld than this! You see a stove. The stove is conventional. But the thing is
empty. What, you think everything in TextWorld should have stuff on it?
There is an open plain door leading south. There is an exit to the north. Don't worry, there is no door. You don't like doors? Why not try
going west, that entranceway is not blocked by one.
There is a knife on the floor.
take the frying pan from the stove
You can't see any such thing.
look for a frying pan
You can't see any such thing.
go west
-= Livingroom =-
You find yourself in a livingroom. A standard kind of place.
You can make out a sofa. Does this look like anything mentioned in the instructions? The sofa is comfy. The sofa appears to be empty. Aw,
here you were, all excited for there to be things on it!
There is an exit to the east. Don't worry, there is no door. You need an exit without a door? You should try going south.
go south
-= Bedroom =-
You've just sauntered into a bedroom.
You make out a bed. However, the bed, like an empty bed, has nothing on it.
There is an exit to the north. Don't worry, there is no door.
look for a frying panYou can't see any such thing.
go north
-= Livingroom =-
You find yourself in a livingroom. A standard kind of place.
You can make out a sofa. Does this look like anything mentioned in the instructions? The sofa is comfy. The sofa appears to be empty. Aw,
here you were, all excited for there to be things on it!
There is an exit to the east. Don't worry, there is no door. You need an exit without a door? You should try going south.
go east
-= Kitchen =-
You find yourself in a kitchen. A normal kind of place.
You make out an opened fridge. The fridge contains a yellow bell pepper, a yellow onion, an orange bell pepper, a carrot and a raw pork
chop. You make out an oven. What a letdown! The oven is empty! Oh wow! Is that what I think it is? It is! It's a table. The table is
massive. But the thing is empty. As if things weren't amazing enough already, you can even see a counter. On the counter you can make out a
green apple, a tomato and a cookbook. It doesn't get more TextWorld than this! You see a stove. The stove is conventional. But the thing is
empty. What, you think everything in TextWorld should have stuff on it?
There is an open plain door leading south. There is an exit to the north. Don't worry, there is no door. You don't like doors? Why not try
going west, that entranceway is not blocked by one.
There is a knife on the floor.
take the knife
You pick up the knife from the ground.
slice the green apple
(with the green apple)
Cutting something requires a knife.
slice the green apple with the knife
You need to take the green apple first.
take the green apple
You take the green apple from the counter.
Your score has just gone up by one point.
slice the green apple with the knife
You slice the green apple.
Your score has just gone up by one point.
drop the knife
You drop the knife on the ground.
look for a frying pan
You can't see any such thing.
...

F.2.4 The second game in Section 5.2.2 with 1-shot

GPT-4 excerpt GPT-4 excerpt

... One example of a cooking game here ...
You are hungry! Let's cook a delicious meal. Check the cookbook in the kitchen for the recipe. Once done, enjoy your meal!
-= Kitchen =-
You find yourself in a kitchen. A normal kind of place.
You make out an opened fridge. The fridge contains a yellow bell pepper, a yellow onion, an orange bell pepper, a carrot and a raw pork
chop. You make out an oven. Oh wow! Is that what I think it is? It is! It's a table. The table is massive. On the table you can make out a
knife. As if things weren't amazing enough already, you can even see a counter. On the counter you can make out a green apple, a red hot
pepper, a tomato and a cookbook. It doesn't get more TextWorld than this! You see a stove. The stove is conventional. But the thing is
empty. What, you think everything in TextWorld should have stuff on it?
There is an open plain door leading south. There is an exit to the north. Don't worry, there is no door. You don't like doors? Why not try
going west, that entranceway is not blocked by one.

look cookbook
You open the copy of "Cooking: A Modern Approach (3rd Ed.)" and start reading:
Recipe #1
Gather all following ingredients and follow the directions to prepare this tasty meal.
Ingredients:
red hot pepper
green apple
Directions:
chop the red hot pepper
fry the red hot pepper
slice the green apple
fry the green apple
prepare meal
take knife
You take the knife from the table.
take red hot pepper
You take the red hot pepper from the counter.
Your score has just gone up by one point.
chop red hot pepper with knife
You chop the red hot pepper.
Your score has just gone up by one point.
fry red hot pepper with stove
That's not a verb I recognise.
cook red hot pepper with stove
You fried the red hot pepper.
Your score has just gone up by one point.
take green apple
You take the green apple from the counter.
Your score has just gone up by one point.
slice green apple with knife
You slice the green apple.
Your score has just gone up by one point.
cook green apple with stove
You fried the green apple.
Your score has just gone up by one point.
prepare meal
Adding the meal to your inventory.
Your score has just gone up by one point.
eat meal
You eat the meal. Not bad.
Your score has just gone up by one point.
*** The End ***
You scored 8 out of a possible 8, in 11 turns.
See here for the complete log.

群内每日免费分享5份+最新资料

300T网盘资源+40万份行业报告

为您的创业、职场、商业、投资、亲子、网赚、艺术、健身、心理、个人成长 …… 全面赋能！

关注公众号获取更多资料

添加微信，备
注“入群”
立刻免费领取
200套知识地图+最新研报

致终身学习者社群

行业报告/思维导图/电子书/资讯情报

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Abstract

Contents

群内每日免费分享5份+最新资料

300T网盘资源+40万份行业报告

致终身学习者社群

扫码先加好友，以备不时之需

1 Introduction

1.1 Our approach to studying GPT-4’s intelligence

1.2 Organization of our demonstration

2 Multimodal and interdisciplinary composition

2.1 Integrative ability

2.2 Vision

2.2.1 Image generation beyond memorization

2.2.2 Image generation following detailed instructions (`a la Dall-E)

2.2.3 Possible application in sketch generation

GPT-4 and Stable Diffusion

2.3 Music

3 Coding

3.1 From instructions to code

3.1.1 Coding challenges

3.1.2 Real world scenarios

3.2 Understanding existing code

4 Mathematical abilities

4.1 A mathematical conversation with GPT-4

GPT-4

4.1.1 A first generalization of the original question

GPT-4

GPT-4

GPT-4 GPT-4

4.1.2 A second variant of the original question

GPT-4

GPT-4

GPT-4

GPT-4

4.1.3 Analysis of the limitations highlighted by conversation

4.2 Performance on mathematical problem datasets

4.3 Mathematical modeling in various domains

4.4 Higher level mathematics

GPT-4

Prompt:

GPT-4:

5 Interaction with the world

5.1 Tool use

5.1.1 Using multiple tools to solve more complex tasks

5.1.2 Discussion

5.2 Embodied Interaction

5.2.1 Warmup: navigating a map

5.2.2 Text-based games

5.2.3 Real world problems

5.2.4 Discussion

6 Interaction with humans

6.1 Understanding Humans: Theory of Mind

6.1.1 Testing specific aspects of theory of mind

6.1.2 Testing theory of mind in realistic scenarios

6.1.3 Discussion

6.2 Talking to Humans: Explainability

7 Discriminative Capabilities

7.1 PII Detection

7.2 Misconceptions and Fact-Checking

7.2.1 Why Are Current Metrics Insufficient?

7.2.2 GPT-4 as a Judge

8 Limitations of autoregressive architecture highlighted by GPT-4

8.1 Warm-up with two basic examples

GPT-4

8.2 Lack of planning in arithmetic/reasoning problems

8.3 Lack of planning in text generation

GPT-4

9 Societal influences

9.1 Challenges of erroneous generations

9.2 Misinformation and manipulation

9.3 Bias

9.4 Human expertise, jobs, and economics

9.5 Constellation of influences and considerations

10 Directions and Conclusions

10.1 Definitions of intelligence, AI, and AGI

10.2 On the path to more general artificial intelligence

10.3 What is actually happening?

References

A GPT-4 has common sense grounding

Sparks+of+Artificial+General+Intelligence-+Early+experiments+with+GPT-4-154页-WN4