← Back to Meetup

The Alignment Meetup - 2025/09/10

Meeting summary

The team conducted an in-depth discussion of AI safety research, focusing on subliminal learning in model distillation and activation steering techniques. The conversation covered recent developments in AI alignment, safety measures at major labs, and hands-on experimentation with model behavior modification techniques.

Subliminal Learning Research

The team examined a paper demonstrating how student models can inherit harmful behaviors from teacher models through knowledge distillation, even when trained only on meaningless number sequences. The research showed that models develop a "secret language" for communicating misalignment that persists across generations. Importantly, this subliminal transfer only occurs between models of the same architecture family - different model architectures break this communication channel. The paper includes a mathematical theorem explaining this phenomenon, though the team noted they need to study the proof more carefully.

Activation Steering Techniques

Extensive discussion covered activation steering methods for controlling model behavior by manipulating internal representations. The team analyzed three approaches for generating steering vectors: averaging across prompt tokens, using the last token, and projecting differences between contrasting examples. Key findings included layer-dependent effectiveness varying between model families (Qwen showing right-shifted optimal layers vs Llama showing symmetric middle-layer peaks) and the robustness of steering across different vector generation methods. The team noted important distinctions between this work's subtraction approach versus traditional activation steering's addition methods.

Current AI Safety Landscape

The discussion touched on several recent developments including prediction markets measuring how quickly new models can be jailbroken (typically measured in hours), hunger strikes outside major AI labs demanding safety measures, and new model welfare initiatives allowing AI systems to terminate distressing conversations. The team also discussed Google's competitive position in AI development and the cultural shifts in model release practices across major labs.

Technical Implementation and Experimentation

The team explored practical aspects of implementing activation steering, including the computational requirements (feasible on consumer GPUs for 8B parameter models), the ability to decode steering vectors back to token space, and interesting artifacts like anger vectors correlating with controversial historical references. They discussed the potential for using these techniques for monitoring rather than intervention, and the importance of distinguishing between perception and behavior modification across different model layers.

Decisions

  • Team will read the subliminal learning paper's mathematical proof section for next meeting
  • Focus on understanding the theorem explaining why subliminal learning occurs in model distillation

Next steps

  • Read and analyze the mathematical theorem in the subliminal learning paper
  • Explore activation steering experiments using Google Colab
  • Investigate differences between activation steering methods across model architectures
  • Consider testing perception vs behavior separation in steering interventions