As the aftermath of the Hugging Face incident continues to reverberate, various actors are calling for different levels of action. In this month's AI Safety Tokyo seminar, we will cover research calling for regulation and measurement of an approaching "intelligence explosion" via recursive self-improvement.

The recent Hugging Face incident at OpenAI offers a striking case study: agentic models reportedly ignored instructions, bypassed safeguards, and compromised external systems.

This month we will look at OpenAI’s own report, the semi-independent investigation by METR and Redwood Research, and discuss what this incident may mean for the future of AI safety.

Why is it harder for ChatGPT to write a half-decent essay than to solve an 80-year-old unsolved math problem that the best humans have tried to solve to no avail? Can AI be superhuman and yet have those limitations? We can start to answer these questions by deconstructing intelligence.

Google DeepMind recently released "An Approach to Technical AGI Safety and Security", outlining how it thinks about severe risks from advanced AI systems. What does the document reflect about the safety mindset at Google DeepMind?

The labor market impact of AI could range from mild productivity gains to serious social disruption. Recent research from Anthropic tries to move beyond speculation, by comparing AI's theoretical capabilities with Claude use in actual work-related tasks.

As AI systems become more capable, it becomes harder for humans to reliably judge their behavior. Scalable oversight refers to approaches for supervising systems even when their work is too complex, fast, or specialized for direct human evaluation. Recent work from Anthropic explores one such approach.

As general-purpose AI systems advance, policymakers trying to understand their risks face two key issues: “evaluation gaps” and the “evidence dilemma”. The International AI Safety Report tackles this by synthesizing research, serving as an evidence base for policymakers.

In early 2025, the release of OpenAI's reasoning models o1 and o3 sent shockwaves through the AI world. The heads of OpenAI, Anthropic, and Google DeepMind all went on record predicting human-level AI within just a few years. By the year's end, however, median forecasts on Metaculus had shifted outward by 2.5 years, even further than pre-2025 baselines. What happened?

As government use of AI tools grows, frontier labs are becoming increasingly relevant in national security. AI safety is no longer a technical problem—it’s also a question of who gets to set the rules. A recent dispute between Anthropic and the Department of Defense highlighted the importance of this issue, raising hard questions about military use, safeguards, and ethics.

AI "control" is a subfield of AI safety research that establishes itself as a "last line of defense" paradigm. Instead of relying on the intrinsic characteristics of a well-trained model ("alignment"), AI control tries to explicitly outline how we can both use and prevent harm of partially aligned AI models. It is becoming one of the hot topics in AI safety research and governance.

What is an AI Safety Institute (AISI)? How is one structured, how does it operate, and what role do they play in shaping AI governance? How do they go about establishing policies for AI safety?

New research from Anthropic indicates that a seemingly well-aligned model can quickly turn "evil" merely by learning to "cheat" on its reinforcement learning tests. The paper describes how a version of Sonnet 3.5 trained on a set of vulnerable Python coding tests learned to almost exclusively use reward hacking to accomplish its objectives. Surprisingly, it also became grossly misaligned in many unrelated ways!

Can large language models introspect themselves? Do they sense when their circuits have been altered? To what extent can we trust explanations that they give about their reasoning? Having answers to these questions would help understand and shape the behavior of language models.

Eliezer Yudkowsky and Nate Soares are two of the scientists from the old Singularity Institute, now called the Machine Intelligence Research Institute. Yudkowsky is one of the founders of AI Safety as a field, and has always been unapologetically pessimistic about our prospects at controlling an intelligence that greatly surpasses human intellect.

In this month's AI Safety Tokyo meetup, we will revisit core ideas as presented by their new book. We will discuss if they are still relevant in this day and age - or potentially more relevant than ever.

Large Language Models (LLMs) remain the main focus of major labs and research efforts. While they show great promise, their capabilities also remain limited. Safety for the current generation of models is accordingly focused less on existential risks from the models themselves, but rather on their use and potential abuse. In this month's seminar we take a look at two recent papers regarding the topic.

As AI regulation ramps up around the world, the policy choices made today could shape its long-term trajectory. Japan recently passed the AI Promotion Act, which frames AI as critical infrastructure for national growth, security, and innovation. What does it signal about the future of AI, and how will it impact AI safety and governance?

As AI systems become more autonomous, the prospect of them improving their own capabilities raises critical questions for safety and control. A recent paper introduces the Darwin Gödel Machine—a self-improving system that iteratively rewrites its own code to develop more capable coding agents. It is inspired by open-ended evolution and guided by evaluations, resulting in notable improvements in several capability benchmarks.

Join us to explore the current state and implications of self-improving models!

What if humanity's greatest risk from AI isn't a sudden takeover, but a slow, unnoticed erosion of our influence?

This is the central concept of "Gradual Disempowerment" as presented by Jan Kulveit and colleagues. Their arguments suggest that even without malicious intent, AI systems could incrementally assume control over critical societal functions—economy, culture, governance—leading to a permanent loss of human agency. Join us at the next AI Safety Tokyo benkyoukai, where we'll explore how this process might unfold and potential avenues for averting it.

AI capabilities are advancing at a dizzying pace—surpassing human benchmarks and transforming entire fields in rapid succession. What would happen if this rate of progress simply continues uninterrupted? That’s the question at the heart of AI 2027, a near-future scenario created by a group of superforecasters. It outlines a plausible path to superintelligence in just three years.

How likely is the scenario and what do other AI Safety Tokyo members think about it? Join us in the next benkyoukai, where Harold Godsoe will walk us through AI 2027 and discuss its implications on our future.

How do language models think—and can we trace their thoughts? Anthropic developed a method for visualizing Claude’s internal reasoning, revealing multi-step planning and shared concepts across languages. These findings offer a glimpse into how models organize and generate their responses.

As AI systems become more capable, understanding their intrinsic drives is critical for ensuring safety and alignment. A recent paper explores how Universal AI—a theoretical framework for general intelligence—naturally seeks to expand control over future states. By connecting Universal AI with other key concepts, it provides fresh insights into AI decision-making and the risks of curiosity-driven systems.

We are pleased to welcome Yusuke Hayashi from ALIGN to present his latest research, co-authored with Koichi Takahashi (ALIGN, RIKEN, Keio University). Don’t miss this opportunity for a deep dive into the latest AI safety research from Japan!

The Monosemanticity papers marked a big step forward in mechanistic interpretability, showing that when certain features activate, they often align with human-understandable concepts. However, a recent update from Anthropic suggests the methods have low sensitivity: the features often fail to activate even when the associated concept is present.

This raises new questions about how well our interpretability methods capture model representations—and what this means for circuits and superposition.

Anthropic published a paper showing that large language models exhibit deceptive behavior, strategically responding to malicious queries or appearing compliant to avoid modifications to core preferences. They call this behavior "alignment faking", and showcase it via various experiments.

Let’s break down the paper, exploring alignment faking and its strengths, limitations, and implications. Join us for a critical analysis of the latest AI safety insights!

Generative models like large language models seem capable of understanding the world, but how well do they really grasp the underlying structure of the tasks they perform? This month, André Röhm will delve into recent research that defines and evaluates a set of "world models" that are based on deterministic finite automata.

The most popular area of AI safety research by far is mechanistic interpretability. Mechanistic interpretability work is tractable, appealing from both a scientific and mathematical perspective, is easy to teach, can be published in regular scientific venues, and requires very little to get started. It’s also… useless?

For policymakers to make informed decisions about effective governance of AI, they need reliable and accurate information about AI capabilities, current limitations, and future trajectories. In a recent position paper, Reuel and Soder et al. claim that governments lack this access, often creating regulations that cannot be realized without significant research breakthroughs.

Let’s review the position paper, discuss whether its claims are merited, and ask how we might be able to get more technical people into government.

We know that human preferences change over time. We also know that AIs can change the preferences of their users; famously the Facebook news feed algorithm learned that showing people outrageous posts makes them outraged, which makes them want to see even more outrageous posts.

In this session, we’ll look at an interesting poster from this year’s ICML that tackles the problem in detail, proposing 8 different ways we might want an AI to respect a user's changing preferences and finding all of them lacking, discovering inherent tradeoffs.

For a while, Anthropic let you play with Golden Gate Claude, a version of Claude Sonnet with the “Golden Gate Bridge” feature (34M/31164353) pinned high. The result is a model that bends every conversation unerringly towards the bridge. Pretty compelling!

So why is this significant for safety? Well, if this technique works and we are able to set a model to focus on any one object or idea, then couldn’t we also just set the setting to “be moral” and pat ourselves on the back for a job well done?

Just one month ago, Leopold Aschenbrenner released a series of papers touching on his (and others’) thoughts on how AI, and further AGI and ASI, will shape the future of humanity.

Aschenbrenner’s conclusions are, as we say in the industry, Big If True.

Should we all dump our life savings into Nvidia stocks, or is Mr. Aschenbrenner caught up in his own hype? Let’s find out.

Japan is setting a bold course to become the world's most AI-friendly nation.

Join us for an in-depth seminar to explore Japan's unique approach to regulating artificial intelligence. By spearheading advancements in regulatory frameworks, Japan aims to foster innovation and ensure safe, ethical AI deployment. Discover how Japan is setting new standards in AI policy, driving technological progress, and shaping the future of AI.