Yineng Zhang is senior director of inference at Together AI, a core maintainer of SGLang, and a co-creator of TokenSpeed. He builds the open-source systems that make advanced language models faster and more practical to run, from GPU kernels and attention mechanisms to distributed serving infrastructure.
Earlier in his career, Zhang optimized recommendation-ranking models and language-model inference at Meituan. After leaving, he joined the SGLang project, working with its creator, Lianmin Zheng, and contributing to FlashInfer, the attention and sampling infrastructure underpinning its runtime.
At Baseten, he helped deploy newly released open models, co-authoring Qwen 3 deployment benchmarks covering mixture-of-experts architectures, tensor and expert parallelism, FP8 quantization, and latency-throughput tradeoffs. He also contributed to SGLang’s DeepSeek-V3 and DeepSeek-R1 work. He subsequently joined Together AI as a principal AI researcher before becoming its inference leader.
Zhang also serves on the LightSeek Foundation’s governing board and helped create TokenSpeed, an inference engine developed with contributors from Together AI, NVIDIA, AMD, and other organizations.
- Inference performance depends on the complete serving stack. Zhang connects GPU kernels, memory limits, model architecture, request scheduling, and deployment configuration. His SGLang workshop demonstrates how exceeding an NVIDIA L4’s default captured batch size disables CUDA Graphs during decoding, and how adjusting that limit restores the optimization.
- Speculative decoding must match production workloads. He tunes EAGLE-style decoding across draft depth, candidate branching, verification budgets, and batch sizes, comparing acceptance rates and throughput against representative prompts.
- FlashInfer turns kernel optimization into serving infrastructure. He co-authored FlashInfer, covering customizable attention kernels, heterogeneous KV caches, load-balanced scheduling, and CUDA Graph compatibility, and FlashInfer-Bench, which evaluates GPU kernels against real serving traces.
- KV-cache architecture determines distributed serving efficiency. Zhang co-authored Mooncake research on cache-centric, disaggregated inference, extending his work from individual kernels to system-wide coordination.
