Pankaj Gupta is a co-founder of Baseten, the inference infrastructure company helping teams deploy and scale machine-learning models. His work spans the systems challenges that determine whether those models become useful products: hardware-specific optimization, latency, operating costs, autoscaling, and reliable production deployment.
Previously a software engineer at Uber, Gupta co-founded Baseten in 2019 with Tuhin Srivastava, Amir Haghighat, and Philip Howes. He helped develop Truss, Baseten’s open-source model-serving framework, which packages model code, dependencies, and configuration while allowing deployments to use different inference engines and hardware.
- Hardware-aware inference optimization. Gupta approaches inference as a complete systems problem, from CUDA kernels and GPU architecture to request batching and serving infrastructure. His work on NVIDIA H100 performance demonstrates why hardware-specific compilation with TensorRT can outperform generic deployment strategies. He also recognizes the tradeoff: simpler tools such as vLLM may suit smaller deployments, while intensive optimization becomes more valuable for expensive, high-volume workloads.
- FP8 quantization with quality safeguards. Gupta has examined how FP8 improves inference efficiency while preserving the numerical range needed for sensitive model components. Quantizing both model weights and the KV cache can reduce memory pressure and accelerate computation, but calibration and output-quality checks remain essential.
- Production-realistic inference benchmarking. Gupta evaluates time to first token, per-user generation speed, aggregate throughput, concurrency, and GPU-memory use together. His analysis of Llama 3.3 70B on NVIDIA GH200 hardware connects serving performance to memory capacity and CPU-GPU interconnects; his work on speculative decoding examines another route to lower latency.
- Autoscaling designed for model inference. Gupta has described adapting Knative for long-running, streaming inference requests, with dynamically adjustable capacity, request queuing, and faster cold starts. Efficient scale-to-zero depends on loading large models quickly enough to prevent incoming requests from timing out.
At AI Engineer World’s Fair 2024, Gupta and Philip Kiely led a hands-on TensorRT-LLM workshop covering engine compilation, tensor parallelism, quantization, deployment, and benchmarking. His subsequent overview of Baseten’s performance-engineering work extends these concerns across language models, speech systems, developer tooling, and production infrastructure.
