Research

Autonomous theory discovery

Automating AI model research with evolution

How AI code review can make correct code worse

LLM-based Evolution as a Universal Optimizer

Beating ARC-AGI-2 with Code Evolution

From bare metal to a 70B model: infrastructure set-up and scripts

Training >70B LLMs on 10,000 H100 clusters: Josh on the Latent Space podcast

Open-sourcing CARBS: how we used our hyperparameter optimizer to scale up to a 70B-parameter language model

Training a 70B model from scratch: open-source tools, evaluation datasets, and learnings

Sanitized open-source datasets for natural language and code understanding: how we evaluated our 70B model

On the Stepwise Nature of Self-Supervised Learning

Scaling Laws For Every Hyperparameter Via Cost-Aware HPO

Update: The state of LLM ethical decision-making

The state of LLM ethical decision-making

Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds (NeurIPS Spotlight Paper)

VICReg: Tutorial and lightweight PyTorch implementation

A PyTorch implementation of SIMONe

Jupyter Ascending: Sync notebooks to your editor

A PyTorch implementation of slot attention

Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL)

Appendix for "Understanding self-supervised and contrastive learning with 'Bootstrap Your Own Latent' (BYOL)"