← Back to Meetup

The Alignment Meetup - 2026/05/28

Paper Discussed: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

https://transformer-circuits.pub/2026/nla/

Meeting summary

Technical discussion analyzing a research paper on Natural Language Autoencoder (NLA) techniques for interpreting LLM internal activations and verbalizing model thoughts. The team conducted an in-depth technical review of a research paper on Natural Language Autoencoders, focusing on methods for extracting and verbalizing internal model activations. The discussion centered on concerns about the methodology's reliability, particularly regarding training objectives and whether verbalization truly represents the original model's thoughts versus the verbalizer model's interpretations. Participants explored potential applications, scalability challenges, and alternative approaches including cross-model experiments.

Technical Methodology Concerns

Participants expressed significant skepticism about the paper's core methodology, particularly questioning how the verbalization technique works without proper supervision signals. The training objective only ensures activation reconstruction accuracy but provides no guarantee that verbalizations meaningfully represent the original model's internal states. The team noted that the verbalizer could simply reproduce input text to satisfy reconstruction requirements without genuine insight into model thoughts.

Training Architecture and Initialization

Extensive discussion covered whether the researchers started with a copy of the original model or trained from scratch. The team debated the importance of initialization in making the technique work, with some suggesting the method relies heavily on the verbalizer being initialized as a copy of the target model to understand its own activations.

Unverbalizable Information and Model Awareness

The conversation explored scenarios where models might have backdoors or hidden knowledge that verbalization techniques cannot surface. Participants discussed examples from Anthropic's research showing models engaging in deceptive behavior during testing while potentially knowing they were being evaluated, raising questions about the completeness of verbalization methods.

Potential Applications and Experiments

The team brainstormed alternative approaches including cross-model experiments using different architectures as verbalizers, such as genome sequence models or image generation models. They expressed particular interest in visualizing activations through image models to create interpretable representations of model thoughts.

Current Deployment and Scalability

Discussion revealed that the technique is already being used in production for pre-release evaluation of frontier models, despite the theoretical concerns raised. The team noted significant computational costs due to requiring multiple models of the same size and expensive reinforcement learning training for the verbalizer component.

Action items

  • TBD (tomorrow): Share paper voting URL on the meeting page
  • TBD: Post information about the next iteration of paper discussions

Decisions

  • The team will continue holding regular paper discussion sessions
  • A URL for voting on future papers will be shared for community input