AI Engineer World's Fair 2024

Unveiling the latest Gemma model advancements

Kathleen Kenealy· Staff Research Engineer, DeepMind16:25

Loading slides…

Read the talk

Unveiling the latest Gemma model advancements

Gemma’s launch-day story connects safety and deployment choices with specialized models, multimodal checkpoints, and the performance claims behind Gemma 2.

From a talk by Kathleen Kenealy

Before you start: Familiarity with language-model inference, fine-tuning, and parameter counts will help; no knowledge of Gemma is required.

What can developers build when they have the model?

How do you put stronger language models into the hands of developers and researchers who want to build on them? That was the starting problem for Gemma. Kathleen Kenealy, the Gemma team’s technical lead at Google DeepMind, describes the project’s goal as enabling community projects, research, and experimentation. The first models launched in February 2024; the work already emerging around them provided the backdrop for the next releases.

Gemma extends a longer sequence of contributions across the machine-learning stack: Transformer research, SentencePiece, and BERT; TPU hardware; and frameworks including TensorFlow and JAX. The family brings that work into lightweight models built from the research and technology behind Gemini. Kenealy uses the term “open source”; more precisely, these are open models distributed under Gemma-specific licensing terms. Access to the models creates opportunities to inspect, adapt, and deploy them, while their license remains a separate consideration.

Slide titled “Gemma open models” with a short description beside a large Gemma logo.Open full source frame
Gemma introduces its open models as a lightweight family built from Gemini research and technology.
0:300:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:30 · section reference included

Safety during training, flexibility during deployment

Safety enters the development process before the final evaluation. Kenealy describes three stages:

  1. Manually inspect training datasets for quality and safety.
  2. Evaluate safety during early experiments and ablations, using the results to help select training methods.
  3. Evaluate the final models using the safety evaluations applied to Gemini.

These are development practices, not a guarantee that every customized application will remain safe. Kenealy offers a broader assurance about customized models, but the Gemma 2 model card calls for monitoring during fine-tuning and safeguards suited to the application.

The next design objective is performance relative to size: making a smaller model useful enough to compete with larger alternatives. That objective sits alongside portability. Gemma targets TPUs, GPUs, and local devices, with support across TensorFlow, JAX, Keras, PyTorch, Ollama, and Transformers. These options let a developer choose an execution environment without first committing to a single framework.

Model access and broad distribution complete the picture. The practical promise is that developers can customize Gemma using tools they already prefer, rather than build their application around one hosted interface. Performance matters, but so do the ability to obtain the weights and the freedom to decide where the application runs.

Four labeled panels under “The power of Gemma”: Responsible by design, Unmatched performance at size, Extensibility and optimization, and Community-driven innovation.Open full source frame
The power of Gemma: responsibility, performance, extensibility, and community-driven innovation.
3:183:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:18 · section reference included

Language, code, and recurrent inference

The initial family grew along three distinct directions:

ModelPurpose
Gemma 1.0Foundational language models
CodeGemmaSpecialization for code generation and evaluation
RecurrentGemmaAn architecture targeting more efficient inference, especially at long contexts

Kenealy presents CodeGemma as a fine-tuned extension of Gemma 1.0. RecurrentGemma changes the architecture instead: although described here as a state-space model, its Griffin architecture combines local attention with linear recurrences. The distinction matters because it pursues inference efficiency through the model’s structure, rather than only through specialization of its training.

All three branches had received updates by the time of the talk. Gemma 1.1 improved instruction following in chat, CodeGemma received further code-performance improvements, and RecurrentGemma expanded from its original 2B size to include a 9B model. These are separate changes: better conversational behavior, stronger code capabilities, and another model size for recurrent inference.

Three horizontally arranged model labels—Gemma, CodeGemma, and RecurrentGemma—with an “Updated!” badge above each.Open full source frame
Gemma, CodeGemma, and RecurrentGemma are each marked “Updated!”
5:576:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:57 · section reference included

PaliGemma connects vision to a language decoder

Multimodality was the most requested addition from the community. PaliGemma addresses it by combining a SigLIP vision encoder with a Gemma 1.0 text decoder. That pairing supports visual question answering, image and video captioning, object detection, and segmentation. The same broad vision-and-language foundation can therefore be adapted to tasks that ask for descriptions, answers, or information about objects.

Checkpoint choice determines the starting point for that adaptation:

  • Pretrained weights: A foundation for task-specific fine-tuning.
  • Fine-tuned variants: Models already targeted toward particular tasks, including the detection and segmentation uses Kenealy describes.
  • Transfer checkpoints: Models specialized for academic benchmarks.

The talk calls PaliGemma a 2B model. That number identifies its Gemma language component; the complete original vision-language model is 3B, as clarified in the later PaliGemma paper. Pretrained weights should be treated as a tuning starting point, not as interchangeable with a checkpoint already adapted to the intended task.

6:587:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:58 · section reference included

Gemma 2 fits into existing development workflows

Gemma 2 arrived on the day of the presentation, following work informed by community feedback about Gemma 1.0 and 1.1. The launch introduced 9B and 27B models. Kenealy described both as outperforming models two to three times larger, a claim about the launch evaluations rather than a promise of superiority on every task.

Adoption was meant to preserve the developer’s existing workflow. Gemma 2 used the same tooling and broad framework ecosystem as Gemma 1, with Kenealy describing a switch requiring only a few lines of code. Expanded documentation covered inference as well as advanced, efficient fine-tuning.

Fine-tuning was also a model-development objective. The team studied how core modeling decisions affected downstream customization, aiming to make the resulting models easier to adapt. The talk does not specify those architectural or training choices; its emphasis is on making customization an expected use of the model, supported from launch.

Gemma 2 logo beside “Built for developers and researchers” and four bullets: Same tooling as Gemma 1, Broad framework compatibility, Effortless deployment, and Flexible fine-tuning.Open full source frame
Gemma 2 highlights familiar tooling, framework compatibility, deployment, and fine-tuning.

For immediate experimentation, the launch offered Gemma 2 27B in Google AI Studio: select the model and begin trying prompts before setting up a local inference environment. This was the hosted entry point available at the June 2024 launch, not a statement about today’s AI Studio model menu.

8:238:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:23 · section reference included

Benchmarks and human preference answer different questions

Kenealy reports that Gemma 2 27B performed in the same ballpark as Llama 3 70B and exceeded Grok on many benchmarks. The presentation does not provide benchmark names or scores for that comparison. Its useful claim is competitive capability at a smaller parameter count, not uniform dominance across evaluations.

Academic benchmarks are only one part of the evaluation because they may not predict how a model behaves in an application. Kenealy reports that extensive human evaluations strongly preferred Gemma over other open models, including larger ones. She also describes Gemma 27B as the top open model of its size, ranking above named Llama, Nemotron, Grok, and Claude 3 entries. These are launch-time claims; the talk supplies neither preference percentages nor enough comparator-version detail to reconstruct the ranking.

In that LMSYS comparison, Kenealy identifies Yi-Large as the only model she classifies as open that ranked above Gemma 27B. That classification needs care: the later Gemma 2 technical report marks Yi-Large as non-open, and its later leaderboard snapshot is not the conference-day ranking. The practical motivation for discussing human preference remains clear: chat quality needs evaluation beyond academic test scores.

Against earlier Gemma releases, the reported improvements include instruction following, creativity, and factuality. Those dimensions give developers more specific behaviors to examine than a single overall rank: whether the model follows a requested format, produces useful alternatives, and keeps its answers grounded in facts.

11:2911:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:29 · section reference included

From trying a model to sharing an application

The Gemma Cookbook provides the next step beyond a prompt experiment. At the time of the talk, Kenealy described 20 recipes ranging from introductory examples to advanced applications. The repository was also accepting pull requests, giving developers a way to contribute their own applications and make them available to the wider community.

Research support accompanied the release. Kenealy encouraged applications for GCP credits from projects of any size; these were applications for support, not guaranteed allocations. The launch program had an August 9, 2024 application deadline. Guides, tutorials, and Colabs covered inference, fine-tuning, and evaluation across frameworks, while downloadable models supported work outside the hosted environment. Discord and other social channels provided ways to discuss what developers were building.

13:2813:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:28 · section reference included

Closing the gap for open-model research

Kenealy closes with the reason she had spent nearly two years on Gemma. As an academic researcher, she had been disappointed by the gap between open foundational language models and rapidly improving proprietary systems. Gemma was an opportunity to build something she wished had been available during that work: stronger models that researchers and developers could use as the basis for their own projects.

The request is not simply to try the release. It is to report what works, what falls short, and what the team should improve next—through GitHub, Hugging Face, social channels, or direct conversation. Community feedback had already helped shape Gemma 2; the applications and research built with it would provide the next set of concrete problems to solve.

15:0315:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:03 · section reference included

Resources

From the talk

  • Griffin-based model implementations with JAX and PyTorch sampling examples and a fine-tuning tutorial.

  • Overview of coding models for completion, generation and instruction-following applications.

  • Practical notebooks and application examples for Gemma, with archived examples and contribution guidance.

Updates since the talk

Transcript download
Full transcript

Loading transcript…