Continuously Monomorphized Systems

If AI for coding becomes more and more reliable, in the limit, what manner of system building would we converge to?

AI-Assisted Software Engineering

Looking back, how did we do additions with computers? A while ago we always wrote assembly, either by hand or using a C compiler, which is one instruction. Today we still do some of that, but we’re also using Python a lot, where operands likely sit on the heap, and the interpreter will run much more than one instruction for the addition itself. Apparently, addition has gotten so much inefficient!

However, the reason we use Python is that it is much easier to write code in those languages than in assembly, especially for complex things due to the higher-level abstractions they provide. Due to that, when a human looks at the code, it’s also much easier to understand and debug. So, the cost of doing the simplest thing (e.g., addition) can be much higher in those languages, but the net cost of doing complex things can be much lower (especially considering the time to ship new things).

Here, a very important property that enables this higher-level abstraction to be useful is it being not leaky. When we write code in Python, we generally don’t worry about how the interpreter is going to boil down our class hierarchy and method resolution into machine code; we only reason at the higher-level API. With the level of abstraction raised, the world of software engineering was able to produce more complex systems and generally do more cool stuff.

The way I view it, AI-assisted software engineering is a similar lift. In 2020, if you wanted to create a new static website for your project, you would have to set up a web framework, write a bunch of boilerplate code, and write the actual content, and still no one could figure out how to center a div. Or, if you find yourself repeating a similar command in your shell, you would briefly consider automating it with a shell script or a CLI tool, until you realize it’s unclear whether the amount of time you’ll spend on writing that thing is less than the amount of time you’ll save by using it.¹ AI-assisted software engineering is a lift in that, the time it takes to build many things that used to take a lot of human time and effort is now significantly reduced; you can ask for a website with a paragraph of natural langauge and get it working the next moment, or you can paste in part of your ~/.zsh_history and get a working CLI tool in minutes.

As of today, however, these AI-assisted tools are not completely reliable. In other words, they are still leaky; especially for tools that matter, or those with potentially large blast radius, you still have to review the code it generates. But in the limit of AI-assisted software engineering, it will probably approach non-leakiness.

Systems Today are Polymorphic

Now let’s switch to system building. Why do we build general systems in the first place? I believe that it is in large part because writing code has been expensive; a human had to write and maintain every line. Much of software engineering practice, like libraries, frameworks, code sharing, and dynamic dispatch, exists so that we write less code; to have concise representations of our intent in code in higher-level abstractions.

ML systems is no exception. One codebase runs every model, at every precision, on every accelerator, so that nobody has to write a system per workload. Identical PyTorch code running on both NVIDIA GPUs and AMD GPUs is a beautiful example of this. Things like PyTorch’s dispatcher allows you write one operator call (e.g., torch.mm), and the right kernel among the many registered ones is dispatched at runtime. This leads to not only runtime overhead² but also code complexity. Different models, precisions, and GPU types route to different backends through layers of dynamic dispatch, and reading the code tells you the list of what could run, and figuring out what actually runs in your environment takes effort.

This isn’t merely an issue of PyTorch of course. vLLM’s feature compatibility matrix shows that it’s trying to cram in all these features into a single codebase and some combinations are not supported, which is sometimes not a fundamental incompatibility but rather due to the difficulty of maintaining a single codebase that supports all combinations. vLLM’s hybrid KV cache manager reads each layer type’s attention spec (e.g., full attention, mamba, MLA) on startup, computes how many bytes its state needs per token, and adjusts block sizes until every layer type has the same bytes-per-page. This is complex runtime resolution logic; it is hard to understand, maintain, keep bug-free, generalize to new model architectures that come out every week, and keep it working for all existing models.

I would call the current systems we’re building polymorphized systems: a single system codebase that is polymorphic over many workloads, and the polymorphism is resolved at runtime.

Monomorphized Systems

I think most of this generalization is a compromise forced on us by the cost of producing code. However, if AI-assisted software engineering makes writing code fast, we ought to reconsider this compromise. We can couple the systems layer tightly with the application layer and build each system monomorphized for one workload, borrowing the compiler term for stamping out a specialized copy of generic code for each concrete type it is used with.

In such a monomorphized system, every choice that a general system defers to runtime dispatch or resolution is instead resolved and baked in at build time. Places that used to be polymorphic will be replaced by an interface (e.g., prefill attention) and a list of functionally interchangeable implementations (e.g., FlashAttention, FlashInfer, CuDNN), and exactly one of those implementations will be chosen and baked into the system (i.e., the interface will be monomorphized) at build time. Similarly, configurations are hard-coded, statically known shapes and precisions are baked in, and code paths that do not apply to the deployment are never generated in the first place.

If this is doable, there are many immediate benefits. Smaller binary size. Minimal dependencies (and surface for supply chain attacks for at least that specific monomorphization). No runtime dispatch overhead or resolution/warmup time on startup or first-run. Trivial to read and step through code for understanding, debugging, and profiling. No worries about code changes of one part of the system unintentionally breaking some completely unrelated part of the system due to a hidden coupling, because that unrelated part of the system doesn’t even exist in your monomorphized system.

Continuously Monomorphized Systems

But the world does not stay still; production systems sit on top of components that change at high velocity. Key dependencies like kernel libraries release constantly (e.g., FlashAttention is the fastest long context prefill attention kernel for GB300 one day, and FlashInfer may take over tomorrow), and hardware architectures, drivers, and CUDA versions keep shifting underneath as well. On top of that, the target workload the system was built for can itself change: the model, the precision, the request pattern, prefix cache hit rate, etc.

This is why monomorphization must be continuous, which I call continuously monomorphized systems. Each monomorphized system declares its system component dependencies and a workload signature. A centralized benchmarking authority (e.g., one per organization) continuously evaluates the performance of system components whenever a trigger occurs, such as a new release of a kernel library or a change in a workload pattern that corresponds to a workload signature. When the authority detects that, for a certain workload signature, the performance ranking among interchangeable components compatible with an interface has changed, it will signal the downstream monomorphized system to trigger re-monomorphization. Centralization is important here; this kind of continuous benchmarking is costly, but it only needs to happen once, rather than being repeated by every downstream system.

1. An exception is when you just enjoy building cute little tools, like myself. ↩

2. To be fair to PyTorch, in the common case, this runtime cost does not show up. This is because dispatch is asynchronous. As long as the CPU dispatching work to the GPU is not the bottleneck, the GPU will not be idle. Even when the CPU is slow, solutions like CUDA graphs can be used for roughly static/repeating sets of kernel. ↩