Did you know that large language models were actually invented by linguists, to study the structure of language and how humans acquire and use it? It’s easy to forget, now that Language Is All You Need. But while everyone else is getting GPT to write code and trade stocks and hang their laundry, some psycholinguists are still fighting the good fight, trying to learn about language by studying our best models of it. In this session, well be looking at “Theory of Mind May Have Spontaneously Emerged in Large Language Models” (Kosinski 2023), a recent paper that uses standard false-belief tests to observe that as we progress from GPT-1 through to GPT-4, the models get better at theory of mind. We’ll discuss the pitfalls of applying experimental methods from human psychology to large language models, look at the steps Kosinski takes to avoid them, and speculate about how “spontaneous” this spontaneous emergence really is.
Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy, self-consciousness, and morality. We tested several language models using 40 classic false-belief tasks widely used to test ToM in humans. The models published before 2020 showed virtually no ability to solve ToM tasks. Yet, the first version of GPT-3 ("davinci-001"), published in May 2020, solved about 40% of false-belief tasks-performance comparable with 3.5-year-old children. Its second version ("davinci-002"; January 2022) solved 70% of false-belief tasks, performance comparable with six-year-olds. Its most recent version, GPT-3.5 ("davinci-003"; November 2022), solved 90% of false-belief tasks, at the level of seven-year-olds. GPT-4 published in March 2023 solved nearly all the tasks (95%). These findings suggest that ToM-like ability (thus far considered to be uniquely human) may have spontaneously emerged as a byproduct of language models' improving language skills.