Scaling AI in Education: A Khanmigo case study

How can one-on-one tutoring reach students whose schools cannot afford to provide it? At Khan Academy, that question sits inside a larger mission: free, world-class education for anyone, anywhere. Shawn Jansepar, who introduces himself as leading AI and learning while also serving as a product leader, describes the scale already behind that mission. He reports 140 million registered users, seven billion learning minutes in 2021–2022, and 1.5 million very active learners using Khan Academy for two hours per month. At the time of the talk, he reports about 200,000 paid Khanmigo learners and teachers, approximately half through school districts and half paying independently.

Access matters as much as aggregate reach. Khan Academy prioritizes historically under-resourced learners through district partnerships, including schools with higher participation in free and reduced-price lunch programs. The opening chart illustrates the unfinished learning problem: Jansepar describes students as largely grade-level proficient in third grade, but an average of 1.1 grade levels behind by eighth grade. The proposed response is to make individual tutoring available at a scale that conventional staffing cannot support.

Khanmigo is a tutor for learners and an assistant for teachers. That role extends an existing direction rather than replacing it. Self-paced videos and practice already let learners get explanations on their own schedule; Jansepar recalls Sal Khan’s story that his cousins preferred the automated version of their cousin. LLMs add another way to move toward individualized tutoring. Borrowing a framing he attributes to Notion’s CEO, Jansepar treats the opportunity as an interface revolution as well as a technology revolution: the model becomes useful through an educational experience built around it.

A classroom tutor needs more than a correct final answer. Khanmigo’s design combines Socratic questions, mathematical accuracy, awareness of the learner’s work, integration with course content, multilingual support and text-to-speech. The side-by-side demonstration makes the first two requirements concrete. The shown ChatGPT response moves toward answering the question, which can be useful for an older learner who already knows how to direct their learning. For classroom students, Khanmigo instead asks questions that require the learner to work toward the answer.

The more consequential difference appears when the student distributes negative four incorrectly. In the displayed exchange, ChatGPT praises the distribution as correct. Khanmigo identifies a distribution mistake, backs up and asks a guiding question. This single example exposes two separate failure modes: supplying an answer too readily, and reinforcing an incorrect intermediate step. Accurate tutoring requires judging the student’s work, not merely solving the problem independently.

Embedding the tutor inside an exercise gives it information a standalone chat window would otherwise need the student to supply. After an incorrect submission, the learner can ask for tutoring, and Khanmigo receives:

• The question: what the student was trying to solve.
• The solution: the step-by-step path through the problem.
• The student’s entry: the answer that needs diagnosis.

With that context, the tutor can analyze how the learner may have reached the wrong answer and offer a relevant hint. The timing is deliberate: this diagnostic intervention follows an incorrect submission; Khanmigo would not immediately dive into it before the student submits.

The interface also has to support the subject being taught. A math input widget makes complex equations practical to enter, and specialized graph rendering makes mathematical relationships easier to discuss. Essay writing needs a different interaction again. These are educational controls around the model, not details that a generic chat box automatically supplies.

The familiar essay workflow puts much of the feedback at the end: a student submits, a teacher marks the paper, and the student moves on without revisiting it. AI assistance can shorten that loop by making feedback available while revision is still possible. Jansepar imagines a future in which students might need to reach at least B-level quality before submitting because they have already iterated with an AI. That is a proposed future standard, not a requirement of the writing-coach preview.

The preview divides writing into stages, each with a different kind of help:

1. Outline. The student develops a thesis with feedback from the coach. The workflow does not let the student advance until the thesis is sufficiently strong.
2. Draft. The student writes the body of the essay and can ask for tips. If asked to write the next paragraph, the coach refuses.
3. Revise. The coach provides feedback on evidence, structure, style and tone, and the conclusion.

The boundary is consistent across the stages: the system helps the student improve the work while leaving the act of writing with the student.

Teacher visibility closes the loop. Teachers can inspect the conversation history to understand how an essay developed with AI assistance. Replay functionality is being built, and the system can flag a chunk of text that suddenly appears rather than emerging through the visible writing process. Jansepar illustrates this with a passage of roughly fifty words appearing at once. That gives the teacher process evidence to investigate, rather than just a polished final document. He also describes plans to bring similar support to Khan Academy’s coding platform.

Engagement features let students use energy points to customize Khanmigo, but classroom integration addresses a deeper access problem. A learner who finds Khan Academy through Google already has enough motivation to seek help. Reaching students who dislike learning or believe they are bad at math requires meeting them in the classroom, with a tight feedback loop between their activity and the teacher.

That connection turns concerns about AI into concrete product requirements. Inappropriate conversations trigger teacher notifications. A classroom snapshot uses Khan Academy activity data to summarize how a class is doing, then lets the teacher explore skills reports and graphs. From that overview, a teacher can ask for a progress report to Reese’s family using the learning data already available in the platform. The assistant becomes useful because it can move from a broad class question to a specific student communication without asking the teacher to reconstruct the underlying record.

Other teacher tools include lesson planning and exit tickets. Through Microsoft’s sponsorship, Jansepar says these tools are available free to US teachers, with expansion beyond the US still forthcoming in the talk. This offer concerns teacher tools, not free student tutoring.

The embedded classroom video shows how those product choices feel to learners. Miss Gregory’s class uses Khanmigo for stories and history, including learning about George Washington. A student explains that it helps them reach an answer rather than simply providing one. The clip then visits Ms. Barrera’s fifth-grade and Mr. Rodriguez’s sixth-grade classes, emphasizing personalized learning and adaptive feedback. One student compares the experience to a little person sitting beside them, or a teacher helping from behind. These are qualitative accounts of engagement and assistance, rather than measured learning gains.

Khanmigo did not emerge from a complete specification of the perfect tutor. Jansepar briefly describes requirements, estimates and a Gantt chart, then retracts the story as a joke. The actual process centered on rapid prototyping and iteration. He recommends Ken Kocienda’s Creative Selection for its account of Apple’s demo-driven development: invest in promising demos and stop the ones that do not work. At Khan Academy, the productive pairing was strong builders with domain experts who understood the customer, ideally working together inside the team rather than handing experimentation to a separate R&D group.

The OpenAI relationship began, in Jansepar’s account, with a request for AP Biology questions to test a new model. An initial September preview produced little headway until December, when the teams shifted to joint prototyping and red teaming and narrowed their use cases. Once tutoring felt promising inside Khan Academy, a company-wide hackathon generated additional features. The strongest ideas were selected for the March launch alongside GPT-4; Khan Academy’s launch retrospective dates that limited pilot to March 2023. Jansepar also reports that content-team members worked with OpenAI on reinforcement learning using 100 math-tutoring examples.

Moving quickly required explicit decision rules, not just enthusiasm for prototypes. Khan Academy resisted perfectionism, made reversible two-way-door decisions quickly, and spent more time on consequential one-way-door questions such as trust and safety. Learning scientists and educators supplied judgment where testing every decision would have slowed progress. Demo-driven iteration ran against fixed deadlines to counter Parkinson’s law—the tendency for work to fill the available time.

The organization also expanded architecture decision records into organization decision records. For a strategic shift or major process change, the record identifies the problem, driver, approver and contributors, then circulates for asynchronous reading. Changes to the core values made the desired behavior durable: taking a stand now included trusting intuition, and delivering wow included resisting perfectionism. Those expectations could then shape hiring and performance reviews, rather than remaining advice given during a single project.

The development process changed with those values. The previous process looked agile, with loops and arrows suggesting iteration, but required substantial documentation for every feature. That rigor could make sense for something headed to all users; it was expensive for an uncertain prototype. The replacement linked engineering rigor to confidence and release scope:

Teams could choose how to work within this framework. A common set of release expectations replaced a single heavyweight process for every stage of discovery.

An AI platform team sits between infrastructure and product, owning the developer and prompt-engineer experience. One concrete mechanism is a Go component interface: when developers conform to it, tracing and Langfuse become available through the shared infrastructure. The interface provides a common integration point instead of requiring each product team to build its own observability path.

The same team owns the AI router, hiding routing concerns from feature teams, and an extensible chat component with APIs for different product experiences. It also consults with teams that need AI expertise. A tentative Khanmigo-as-a-service effort extends the reuse boundary outside Khan Academy, exploring how third parties could embed the tutor instead of rebuilding one themselves.

Improving accuracy starts with giving the tutor somewhere to delegate mathematical work. Khanmigo can defer to a math agent that uses a basic calculator or Python for more advanced problems. The model does not have to handle every calculation through text generation alone.

The team also introduced a staged prompting process, which Jansepar calls chain-of-thought prompting. Rather than asking a model to help the student immediately, the workflow first asks how the student might have arrived at the submitted answer and what the correct solution path should have been. A separate model call receives that diagnostic package and decides how to tutor the student. Jansepar reports a substantial improvement from this separation, without giving a numerical result.

The handoff can be expressed in Go as two components with an explicit intermediate result. This illustrative interface keeps the student’s work attached to both calls and separates diagnosis from the learner-facing reply:

package tutoring

import "context"

type Attempt struct {
    Question      string
    StudentAnswer string
    SolutionSteps []string
}

type Diagnosis struct {
    LikelyStudentPath string
    CorrectPath       string
}

type Diagnoser interface {
    Diagnose(context.Context, Attempt) (Diagnosis, error)
}

type Tutor interface {
    Respond(context.Context, Attempt, Diagnosis) (string, error)
}

func Guide(
    ctx context.Context,
    attempt Attempt,
    diagnoser Diagnoser,
    tutor Tutor,
) (string, error) {
    diagnosis, err := diagnoser.Diagnose(ctx, attempt)
    if err != nil {
        return "", err
    }
    return tutor.Respond(ctx, attempt, diagnosis)
}

The important boundary is between analyzing the mistake and choosing the next teaching move. The diagnostic result informs the tutor; it is not automatically the response shown to the student.

Additional context includes step-by-step solutions and material supplied through retrieval-augmented generation, or RAG. A tutoring-accuracy dashboard monitors quality as prompts change, models are upgraded and other system changes ship. Its purpose is both to catch math regressions and to see whether accuracy is improving over time.

The next evaluation target is accurate tutoring itself. At the time of the talk, Khan Academy is preparing a benchmark dataset and paper intended to test whether a tutor correctly evaluates student work, rather than only whether a model can solve math or reason. Jansepar also describes exploration of Wolfram integration for harder mathematics and fine-tuning. These are next steps in the talk, distinct from the calculator, Python, staged prompting and monitoring already described.

The initial prompt code grew through fast experimentation and became tangled. Test coverage was uneven: math tutoring had substantial coverage, while activities such as talking to a historical figure could have no test cases at all. To evaluate a model change for those activities, someone who knew the experience would try it manually. Debugging failures often meant searching logs for the relevant completion calls.

Refactoring into components with defined inputs and outputs made smaller pieces easier to identify and test. The team began building a minimum viable test suite across all activities, alongside tools that let humans run manual cases quickly. It was also moving toward model-graded evaluations where appropriate, while retaining human review for scenarios that still needed it. Langfuse made execution traces available for inspecting a failure and following what happened through the system.

Automating the grader remains a separate reliability problem. Compared with the team’s own labels, Jansepar reports that AI graders catch math mistakes roughly 70% of the time. This is an internal estimate of error detection, not Khanmigo’s overall tutoring accuracy; the talk does not specify the evaluator, sample size or exact metric definition. More dependable model grading remains an open goal. Adding languages creates another pressure, because every new language can expand the testing and iteration workload. The team wants broader language support without letting that complexity grow unchecked.

Classroom deployment imposes three simultaneous constraints: the service must be reliable enough for daily use, affordable enough for lower-income districts, and fast enough to avoid frustrating students and teachers. Khanmigo already uses different models for different production workflows and falls back to shared capacity when dedicated capacity reaches its limits.

Better tutoring can also introduce more waiting. When additional model chains slowed the experience, Jansepar required a UI change that showed users that work was happening before the feature could ship. A teammate later reported that their child did not perceive the wait as a problem because Khanmigo was doing math. The anecdote is about making latency intelligible, not reducing elapsed time: a visible explanation helped the student understand the pause. Microsoft’s donation of Azure LLM compute also eased the cost of providing Khanmigo free to teachers.

The deployment roadmap adds provider redundancy and more efficient models. Jansepar describes planned load balancing between OpenAI and Azure, evaluation of GPT-4o and smaller models such as Phi-3, and collaboration with Microsoft on math tutoring. He calls one model under development Phi-3.14, describing it as small to medium in size and optimized for tutoring mathematics. That name refers to the work in development described in the talk, not an established public release.

Capacity utilization is the other side of affordability. At launch, shared OpenAI capacity could not provide the throughput and speed Khanmigo needed, so the team moved to a dedicated instance. Dedicated capacity supplied headroom, but it had to be provisioned for peak demand and did little work at night. Dynamic scaling up and down is the desired next step: preserve enough capacity for classroom peaks while avoiding idle compute during quieter hours.

The closing objective is to use AI to enhance human potential and human intelligence. For an educational system, the engineering work ultimately serves that purpose: making useful help available to more people so they can do more of the learning themselves.