← Research

How early do we know it has to be CPython?

2026-07-30 · speed · 9 min

Most submitted Python cannot be compiled, and finding that out late costs five seconds of somebody else's compiler. Measuring the pipeline against a labelled corpus, and the static filter that came out of it.

mojojojo takes arbitrary Python and tries to make the numeric parts native. Most submitted Python is not numeric, so most of the time the answer is "this runs on CPython" — and the interesting engineering question is not how fast the compiled path is. It is how early we find out that there is no compiled path.

There are four places that can say no, and they do not cost the same:

The third row is the one that matters, and it is easy to miss why. In this service the sandbox has no compiler at all — a build is a request that the host fills afterwards. So a function that gets as far as the compiler and then fails does not slow down the run that submitted it. It spends five seconds of a shared machine on behalf of somebody who is no longer waiting, and it does that on every submission, for ever, because nothing was learned.

You cannot tune this without a corpus

"Does the filter work" is unanswerable in the abstract, so we wrote down 22 programs of the shape people actually submit — backtests, Monte Carlo, rolling statistics, and the glue code that surrounds them — and hand-labelled each with what should happen to it, by measurement rather than by reading the filter. Three labels: native, cpython, and either for cases too small to care.

The corpus is bench/corpus.py and the harness is bench/fallback_study.py. It reports, per case, what each stage said and what the two tiers actually measured:

bench/run.sh bench/fallback_study.py --compile --crossover

The first run was not flattering. Of 14 cases that must end on CPython, the filter caught 9. Five got a transpile they could never pass, and two of those went on to get a real compile.

What leaked, and why a node blacklist could not catch it

The old filter walked the AST and rejected a list of statement types — Try, With, Import, comprehensions, and so on. That is the obvious design and it misses two whole categories.

1. Free variables

WINDOW = 64

def score(xs) -> float:
    total = 0.0
    for i in range(WINDOW):
        total += xs[i]
    return total

Every node in that function is on the allowed list. It still cannot be compiled: the transpiler has no environment outside the function, so WINDOW is not a constant it can fold, it is Unsupported: unbound name WINDOW. And hoisting a window size to the top of the file is good Python, so this is not a rare shape — it is what carefully written code looks like.

Catching it is a scope computation, not a node test: collect the parameters, the assignment targets and the loop targets, then check that every name read is in that set. Annotations are excluded, or list[float] makes list look like a name the body reads.

2. Reachability through helpers

def helper(x):
    return sorted([x, 1.0])[0]

def process(xs) -> float:
    total = 0.0
    for i in range(len(xs)):
        total += helper(xs[i])
    return total

process is spotless. It is also uncompilable, because the transpiler emits a unit — every function it needs, together — or nothing, and helper is not in the subset. Worse, helper would be out even if its body were pure arithmetic: mojosub builds helper signatures from annotations, and an unannotated helper is simply absent from the unit, so the call cannot be emitted.

Deciding this needs the sibling functions' bodies, so the filter now receives name → node rather than a set of names, and recurses. Recursion in the user's code is treated as fine — a function already on the stack is being checked by the frame below it.

A distinction worth keeping

Doing the above surfaced a bug of our own. The filter required a loop, on the reasonable ground that a function with no loop has nothing to win. But that is a statement about whether compiling is worth it, not about whether it is possible — and a helper called from inside somebody else's loop is very much worth compiling with no loop of its own. So the two questions are now two functions: compilable() and eligible(), which is compilable() plus a reason to bother.

One case where the old filter was too strict

It also rejected functions taking no arguments, since there is no signature to specialise. Measured, that was a mistake:

def constant_work() -> int:
    total = 0
    for i in range(1000):
        total += i * i
    return total

It cannot be specialised, but it does not need to be. Because the compile cache is shared and persistent, the second submission of this program dispatches natively on the first call — mojosub's warm start looks the library up before running anything. Measured at 140x. Refusing it gave that away for nothing.

Result

The filter got about 30% more expensive and it is still four orders of magnitude cheaper than the thing it is protecting. The one case it does not catch cannot be caught statically at all: whether a compile pays depends on whether the argument arrives as a list or a numpy array, and that is a runtime fact. The next post is about what we found when we chased it.

What the exercise actually taught

Not the individual rules — those are ten lines each. It is that this class of bug is invisible to unit tests of the transpiler, which had thirty passing ones. The tiered design is built to swallow failures: anything that does not compile falls back to CPython and returns the right answer, so a function that should have been 300x and is 1x looks exactly like a function that was never eligible. Nothing is red. Nothing is slow enough to notice.

What found these was running realistic code end to end and reading what each stage said about it — and then writing the labels down, so the next change has something to be wrong against.

Reproducing

# the corpus, the stage-by-stage verdicts, and the disagreements
bench/run.sh bench/fallback_study.py

# plus real compiles and both tiers timed (slow: ~5s per eligible case)
bench/run.sh bench/fallback_study.py --compile

# the argument-shape measurements behind the next post
bench/run.sh bench/fallback_study.py --crossover --opsweep --copyback

The filter itself is runner/accel.py; each rule above is pinned by a named test in runner/tests/test_runner.py, including one that asserts every native-labelled case in the corpus still passes — the point is not to lose the 38x–423x kernels while tightening.

Measured on the machine that served this page: Intel Xeon E5-2697 v4 (Broadwell, AVX2), CPython 3.13, Mojo 1.0.0b3.dev2026072406. Numbers move between machines; the shape of the result is what we are claiming.