Jyh-Jing Hwang is a Google DeepMind research scientist focused on Gemini post-training and the first author of EMMA, Waymo’s experimental foundation model for autonomous driving. His research addresses how machines interpret visual environments, anticipate unusual hazards, and make driving decisions that can be evaluated and understood.
Hwang studied electrical engineering at National Taiwan University, conducted research at Academia Sinica, and completed a doctorate in computer and information science at the University of Pennsylvania in 2020. His early work included research experience at Google AI and Facebook AI Research and SegSort, a 2019 approach to semantic image segmentation. He joined Waymo in 2020, became a research scientist and technical lead manager, and taught machine learning and computer vision through Penn’s online MCIT program.
At Waymo, Hwang’s CramNet combined camera imagery and radar using ray-constrained cross-attention, improving three-dimensional object detection and resilience when either sensor fails. His subsequent work extended from sensor fusion into foundation-model planning and rigorous evaluation.
- Foundation-model driving. As first author of EMMA, Hwang helped adapt Gemini to predict driving trajectories from surrounding camera images and written navigation instructions. The experimental system learns from recorded driving trajectories without high-definition maps and jointly supports motion planning, object detection, and road-graph estimation.
- Safety-critical long-tail scenarios. Hwang studies rare situations that defeat routine assumptions, including traffic controllers overriding red lights and scooter riders falling on wet roads. The WOD-E2E benchmark, which he coauthored, provides 4,021 challenging driving segments and a Rater Feedback Score based on human judgments of driving behavior.
- Interpretable motion planning. Hwang has explored prompting driving models to identify relevant road users, anticipate their behavior, and describe a high-level maneuver before generating a trajectory, making end-to-end decisions easier to inspect. His AI Engineer World’s Fair talk also demonstrated how generated video can test driving performance under changing weather and lighting.
- Generated-video evaluation. In Drive&Gen, Hwang and collaborators investigated joint evaluation of driving models and generated driving video, exposing how camera-based planning changes when simulated environmental conditions deteriorate.
