Luminal - Search-Based Deep Learning Compilers - Joe Fioti

Deep learning works with scalars, vectors, matrices and tensors, combining addition, multiplication, matrix multiplication and element-wise functions. Why, then, are the libraries implementing it so complicated? Joe Fioti, a co-creator of Luminal, starts with that mismatch: a small mathematical foundation has produced an enormous software ecosystem. His proposed way back to simplicity is to make compilation—and specifically search—do more of the work.

Fioti describes PyTorch as having more than 1,200 operations and more than 15 data types, across targets including CPUs, CUDA, AMD, TPUs and NPUs. The implementation burden grows across combinations, not merely by adding features:

Supporting another operation potentially requires implementations for every relevant data type and device. A new device creates the corresponding problem across operations and types.

Fioti puts PyTorch at roughly three million lines of code. These figures are his scale estimates for the framework at the time, without a version or counting methodology. The practical concern is not size alone: a large implementation surface creates more opportunities for bugs and makes internals harder to understand, extend and debug.

Luminal starts from the other direction: what is the smallest vocabulary needed to represent useful models? Fioti describes a twelve-operation core and names the following mathematical operations:

The spoken list contains eleven entries; Luminal’s primitive-graph introduction includes Contiguous as the twelfth. That is the architecture described here, rather than the expanded primitive set in current project documentation. Fioti says this small vocabulary can represent language models, vision-language models, CNNs, RNNs and diffusion models.

Many apparently missing operations are compositions. Subtraction multiplies the second input by −1 and adds it; division multiplies by its reciprocal:

The distinction is between the operation a model author wants to express and the primitives the compiler must understand. A convenient frontend need not make every convenience a separate backend primitive.

Matrix multiplication becomes broadcasted multiplication followed by sum reduction. For A shaped [M, K] and B shaped [K, N], shape metadata presents them as [M, K, 1] and [1, K, N]. Broadcasting yields products indexed by [M, K, N]; reducing over K yields [M, N]:

This is a representation of the computation, not a requirement to materialize the entire broadcasted intermediate. Convolution follows the same approach: shape trackers describe pooled input windows, then a matrix multiplication combines those windows with the convolution kernel.

Dynamic frameworks made experimentation convenient when researchers were exploring RNNs, LSTMs and changing model structures. Luminal instead asks how much dynamism a particular workload actually needs. In the transformer workload Fioti describes, the changing quantities are sequence length and KV-cache length; the surrounding computation stays fixed. That makes a static representation useful even when some dimensions vary.

The representation is a directed acyclic graph. In the dense-layer example, one node loads the input tensor and another loads the weights. Their outputs feed an element-wise multiply, followed by a sum reduction—the matrix multiplication just decomposed above. Larger models produce larger graphs, but the same vocabulary still specifies their computations.

Fioti reports that Luminal’s implementation at the time was under 5,000 lines of code, without specifying the counting scope. The design goal is equally concrete: a developer should be able to understand the library’s core structure in an afternoon. A small graph language makes that plausible, but it does not yet make execution fast.

Fioti illustrates unoptimized execution with Llama 7B: generating a sentence could take roughly all day, with no hardware, precision or token count specified. Executing each primitive separately is not the intended endpoint. The graph first expresses the model correctly; compilers then transform it into faster graphs. A minimal representation and an efficient implementation are separate responsibilities.

The conventional stack he sketches starts with Hugging Face Transformers, passes through PyTorch, xFormers and handwritten kernels, and may call cuDNN or cuBLAS before reaching CUDA. Those kernels must accommodate many workloads. The resulting dependencies complicate both installation and tracing a bug through the stack.

In the path demonstrated here, Luminal’s graph and compilers directly generate CUDA code. That shortens the route from model representation to device execution:

The simplification does not eliminate optimization work. It concentrates that work in the transformation from a primitive graph to an efficient executable graph.

Why have compilers not already removed all this complexity? Fioti’s answer is that generating sophisticated kernels can require disproportionately sophisticated compiler logic. He uses square-or-cube growth, and a possible tenfold compiler expansion for a doubling of kernel complexity, as illustrations rather than measured scaling laws. The difficulty is especially acute for hardware startups: a promising device still needs software that can schedule and feed it effectively.

His hardware comparison follows the movement of responsibility from circuitry into software:

The intended tradeoff is simpler, more uniform hardware with better throughput and performance per watt for suitable workloads. But the work removed from hardware must be handled elsewhere, increasingly by the compiler.

This leads to the familiar VLIW, or Very Long Instruction Word, scheduling problem. Hardware can become simpler when the compiler decides ahead of time which work runs together and how resources are used. Beyond a certain level of scheduling complexity, however, writing a compiler that consistently makes good choices becomes the bottleneck.

Fioti turns to AlphaGo as an analogy. Its designers did not encode a perfect answer for every Go position; they explored board states using tree search guided by a neural network. Luminal substitutes logically equivalent GPU kernels for board states. Instead of prescribing one optimization sequence that must always produce fast code, simple rules generate alternatives and search chooses among them. The neural-network guidance belongs to the AlphaGo comparison; Fioti does not describe a corresponding learned guide in Luminal.

The implementation uses egglog to represent and expand those alternatives:

1. Convert the operation graph into egglog expressions.
2. Represent equivalent intermediate expressions compactly in e-graphs.
3. Repeatedly apply small rewrite rules that preserve the logical result.
4. Profile candidate kernels and select the fastest measured candidate.
5. Narrow exploration when the space becomes too large to profile exhaustively.

Fioti describes approximately 20–25 rewrite rules in this search setup. A rewrite need not make a kernel faster by itself: it may produce a slower candidate while opening a path to a better combination of transformations.

Runtime measurement supplies the performance judgment that the rewrite rules deliberately avoid making. Once profiling every possibility becomes infeasible, Fioti says Luminal uses techniques such as Monte Carlo tree search to narrow exploration. The division of labor is crucial: rules establish which alternatives preserve the computation, while search and profiling determine which implementations are worth pursuing.

Consider a tensor passed through sine and then exp2. With separate kernels, the GPU loads the input, computes sine, and writes the intermediate tensor to global memory. The next kernel reads that same intermediate, computes exp2, and writes the result. The arithmetic is simple; the extra write and read are avoidable.

Fioti characterizes data movement as roughly 99% of GPU time and energy; that is a broad claim, not a measured share for this example. The useful distinction is whether a workload is limited by memory traffic, arithmetic or latency, as described in NVIDIA’s GPU performance guide. For this dependent element-wise chain, kernel fusion removes the global-memory intermediate: load once, compute both functions, then store the final output.

The resulting CUDA computation can be expressed directly:

#include <cuda_runtime.h>
#include <math.h>

__global__ void sine_then_exp2(const float* input, float* output, int n) {
    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        const float intermediate = sinf(input[i]);
        output[i] = exp2f(intermediate);
    }
}

intermediate is local to the computation; there is no separate global-memory tensor between sine and exp2. The mathematical dependency remains intact while the memory path changes.

The larger graph Fioti next shows in the recording applies the same principle repeatedly. Many separate operations, each with an intermediate memory round trip, collapse into one kernel. Fioti reports that this fused kernel takes little longer than an individual kernel from the original graph. His explanation is that the additional arithmetic costs relatively little compared with repeatedly moving intermediate results through memory; the talk supplies no timing configuration for that comparison.

The more ambitious example is FlashAttention, the IO-aware exact-attention algorithm introduced by Tri Dao and colleagues in 2022. Fioti contrasts that date with the introduction of transformers in 2017: an important optimization took years to emerge after the underlying architecture. He reports that Luminal’s compiler can find the optimization through search rather than requiring the full algorithm to be manually encoded as a compiler pass.

The input is a naive multi-head attention graph. Repeated rewrites construct a large space of equivalent implementations; kernel profiling then selects an implementation that Fioti identifies as FlashAttention. He qualifies the claim of compiler uniqueness as being to the team’s knowledge. This is a reported prototype result, without a supplied search budget, numerical-tolerance specification or reproducible runtime comparison.

Later in the recording, Fioti shows an announcement slide with generated kernel code in green and the intermediate representation in white. It makes the output of the search concrete: an optimized representation becomes CUDA code, rather than stopping at an abstract graph transformation.

Search does not have to make every optimization decision. After selecting kernels, Luminal applies deterministic passes before final execution code. Fioti characterizes these as transformations that will not hurt, although their benefit varies. Buffer reuse is the first example: the complete graph exposes when each intermediate is needed and when its storage becomes available again.

In the demonstration, buffer 1 and buffer 3 have non-overlapping lifetimes. By the time buffer 3 is needed, the last use of buffer 1 has finished. They can therefore share an allocation instead of reserving two independent buffers. This is a change to the memory plan, not to the logical identity of the two intermediate results. The comparison below uses allocation-a and allocation-b as illustrative allocation names.

The next pass addresses launch overhead. Fioti’s illustrative baseline alternates CPU dispatch, GPU execution and a return to the CPU before the next dispatch. Luminal instead builds a queue ahead of execution so the GPU can proceed through successive kernels with less launch waiting.

That baseline should not be read as a requirement of ordinary CUDA: kernel launches are asynchronous with respect to the host, and operations in a stream can execute in order without a host wait after each kernel. The optimization target here is avoidable host submission and synchronization overhead; the talk does not identify the queue mechanism as CUDA Graphs.

Luminal began as an inference library, but its graph representation also provides an extension point for training. An external autograd crate takes the forward graph, derives a backward graph and attaches it. The combined graph then passes through the existing downstream compilers.

The compiler optimizes computations, whether they belong to inference or differentiation. Kernel search and the other graph transformations therefore apply to the backward pass as well. Fioti’s characterization of getting training for free refers to this reuse of the compilation machinery, not to eliminating the work of building autograd or running training.

Fioti describes external training support as unusual, qualifying his comparison with other libraries as being to his knowledge. The architectural opportunity is more important than the priority claim: contributors can supply alternative autograd engines, gradient sharding or specialized training arrangements without making each one part of Luminal’s core.

At the time of the talk, Fioti lists CPU, CUDA and Metal support. AMD, Tenstorrent, Groq and TPUs are proposed targets. The aim is to make more hardware usable through the same model representation and reduce dependence on CUDA.

Two further directions extend how much work the graph describes:

• Distributed inference and training: combine data, pipeline and tensor parallelism into full 3D distributed execution.
• GPU-resident reinforcement-learning environments: remove the back-and-forth between a model running on the GPU and an environment stepping on the CPU.

For the latter, the environment itself becomes a Luminal graph and passes through the same compiler flow as the model. Fioti says the team had done this for very simple environments; handling more complex ones remained an open goal. If both pieces fit that representation, the model’s forward pass and the environment step can run together on the GPU.

After a joke about a Dyson Sphere awaiting a Sequoia fundraise, Fioti turns to Luminal Cloud. The proposed deployment interface follows directly from treating a model as a portable graph:

1. Export the model graph using the talk’s graph.export interface.
2. Upload the resulting file.
3. Receive a serverless inference endpoint.

This is the workflow presented in the recording, rather than a current SDK call signature.

The service would handle optimization, batching, queuing and machine provisioning. Fioti describes billing only while the graph executes. The cloud proposition therefore extends the same separation used locally: the user supplies the computation, and the system takes responsibility for turning it into an efficient execution plan. His simplest-and-fastest positioning is a product ambition, not a demonstrated cloud comparison.

The closing invitation is practical: contribute ideas and pull requests to the Luminal repository, or bring an inference workload that can test the approach. Fioti connects the small core to the ability to develop new capabilities without the overhead of much larger frameworks. He invites startups and companies to discuss workloads through the Luminal site.