What Gets Split, What Gets Communicated, and How It All Fits Together

distributed training · Megatron-LM · survey

Exploring architectural innovations beyond traditional Transformers.

linear attention · hybrid architectures · paper notes

Preliminary Thoughts on a Deconstructive-Constructive Turn in Interpretability

interpretability · model architecture · perspective

Compute-on-the-fly: build your wings on the way down.

research life · interpretability · pretraining

Where the Ideal Meets Its Betrayal.

philosophy of science · fallibilism

Taste as prediction-and-test: exploring how intuition operates as a bet on future system evaluation.

taste · aesthetics