Mark Moyou is an NVIDIA data scientist and solutions architect specializing in production language-model inference: the GPU memory, workload patterns, latency, and costs that determine whether AI applications can operate at scale.
His career began in computer vision and geometric machine learning. He earned a doctorate at Florida Institute of Technology in 2017, studying shape classification, retrieval, and registration, and coauthored research on affine-invariant feature matching and LiDAR-based railroad-bridge inspection. He also helped organize PyData Miami 2019. GPU-based research and a former labmate’s referral led him to NVIDIA in 2021, where he worked with retail customers on data processing, computer vision, recommendation systems, and transformer applications before concentrating on generative-AI infrastructure.
- KV-cache optimization determines inference economics. Each prompt and generated token creates attention state that competes with model weights for finite GPU memory. Moyou emphasizes paged and quantized caches, smaller model footprints, and lower-precision inference because freeing memory increases the number of requests a system can serve.
- Workload-aware inference benchmarking beats isolated throughput figures. He measures prompt and response length distributions, concurrency, time to first token, inter-token latency, and completion time under realistic load. Long-input, short-output applications impose different memory and scheduling demands from services generating lengthy responses, even when both use the same model.
- Optimization spans the entire serving stack. His explanation of production inference workloads connects attention calculations and GPU memory budgets to quantization, in-flight batching, tensor and pipeline parallelism, TensorRT-LLM, Triton Inference Server, and NVIDIA NIM.
- Traffic-aware inference engines can adapt to changing demand. At AI Engineer World’s Fair 2024, Moyou proposed tuning inference engines to observed request distributions and revisiting their configurations as traffic changes throughout the day. Agent-generated workloads, he argues, could make this flexibility increasingly important.
In 2025, Moyou coauthored NEMO-4-PAYPAL, applying NVIDIA NeMo and a fine-tuned Nemotron small language model to commerce-agent search and discovery. His reflection on four years at NVIDIA traces his path from academic GPU experiments to enterprise AI and emphasizes open-source contribution, professional community, and practical collaboration.
