Recently, SERI-MATS participants Jessica Rumbelow and Matthew Watkins found that Chat-GPT has mysterious failures when fed odd tokens like “ SolidGoldMagikarp” or “PsyNetMessage” [1, 2]. We’ll be taking a technical deep dive into how large language models work, explaining exactly why this happens (and speculating about what can / should be done about it).