Noah Shpak is a machine-learning infrastructure engineer and Lance maintainer who builds systems for understanding, improving, and training foundation models on large datasets. His work connects the composition of training data to measurable model behavior and the experience of people using AI products.
After studying at the University of Pennsylvania, Shpak worked at Twitter before joining Character.AI, where he led the AI data platform. His team supported both foundation-model development and a consumer product, linking academic benchmarks with user-engagement experiments. He has subsequently been associated with Thinking Machines Lab and remains involved with Lance, an open-source data format for AI workloads.
At Character.AI, Shpak built research infrastructure combining Apache Spark, Trino, SQL, and GPU-backed services for prompting, embeddings, and classification. The platform helped researchers inspect training mixtures, enrich examples, generate synthetic data, and materialize reproducible datasets. His work integrating Trino with Lance brought familiar SQL analytics to AI-oriented data storage.
- Evaluation-driven dataset curation: Clean data and targeted evaluations guide dataset selection. Post-training requires examining individual examples, including task difficulty and desired behavior, instead of relying solely on broad dataset categories.
- Training-mixture analytics: Token counts, example length, programming-language categories, task complexity, retrieval, and clustering help researchers select data distributions that match specific evaluations.
- Synthetic data with human oversight: Generated examples provide early experimental signals; human labeling and rewriting improve classifiers and correct weak or defective training material.
- Efficient row-reference shuffling: Lance enables training systems to shuffle references to dataset rows without repeatedly moving the underlying examples. Separating dataset materialization from training reduces operational friction as data becomes larger and multimodal.
Shpak’s AI Engineer World’s Fair appearance captures his practical priority: giving researchers enough visibility into training data to improve model behavior while keeping experiments and GPU-backed training moving quickly.
