Research

How AI code review can make correct code worse

LLM-based Evolution as a Universal Optimizer

Beating ARC-AGI-2 with Code Evolution

From bare metal to a 70B model: infrastructure set-up and scripts

Training >70B LLMs on 10,000 H100 clusters: Josh on the Latent Space podcast

Open-sourcing CARBS: how we used our hyperparameter optimizer to scale up to a 70B-parameter language model

Training a 70B model from scratch: open-source tools, evaluation datasets, and learnings

Sanitized open-source datasets for natural language and code understanding: how we evaluated our 70B model

On the Stepwise Nature of Self-Supervised Learning

Scaling Laws For Every Hyperparameter Via Cost-Aware HPO

Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL)