← Research

There is no compiler in the sandbox

2026-07-28 · containment · 11 min

How a multi-tenant service shares a compile cache without letting one tenant put code in another tenant's process — and the four isolation bugs we shipped and fixed on the way there.

mojojojo compiles your numeric loops to native code and caches the result by content hash, so the second person to submit a kernel gets native speed on their first call. That sentence contains a security problem large enough to sink the whole design:

If untrusted code can cause a shared library to be written, and untrusted code can load shared libraries out of a shared cache, then one tenant can put machine code in another tenant's process.

Content addressing does not save you by itself. A hash tells you which bytes you asked for; it says nothing about who produced them or whether the .so sitting at that path is really the compilation of that source.

The rule

Which raises the obvious question: mojosub decides what to compile at runtime, from the types it observes. How does a build happen in a place that cannot build?

A compile becomes a request

MOJOSUB_MOJO points at a program that is not a compiler. It accepts the real thing's arguments, writes the generated Mojo into the run's own request directory, and fails:

def main() -> int:
    argv = sys.argv[1:]
    if "--version" in argv:
        # mojosub hashes this into the cache key, so it has to be the version
        # of the compiler that will actually build the request.
        print(os.environ.get("MOJOJOJO_MOJO_VERSION", "mojojojo-host-compiler"))
        return 0
    source = next((a for a in argv[1:] if a.endswith(".mojo")), None)
    shutil.copyfile(source, os.path.join(requests, uuid.uuid4().hex + ".mojo"))
    print("mojojojo: queued for compilation on the host; this kernel runs "
          "natively from the next execution onward", file=sys.stderr)
    return 1

Failing is the correct behaviour, and it is free: mojosub already treats a toolchain failure as "stay on CPython", because that path has to work anyway for every function outside the compilable subset. So the sandbox needs no special case. The user's program runs, correctly, at interpreted speed, and says so in its trace.

Afterwards the host — as the service user, outside the jail — compiles what was requested and puts the result in the shared cache. The next run to hash that same source finds a library that our compiler built.

Only the return type crosses

A run does learn one useful thing that only running the function can reveal: its return type. An unannotated def kernel(n): has none, and the transpiler needs one before it can emit a signature. So promotion carries the learned type across and drops every verdict — verified, raced, retired.

That asymmetry is the whole game. A return type is a fact about the program, and a wrong one produces a unit that does not compile or does not match, which fails safely. A verdict is a judgement: "this compiled variant agrees with CPython", "this one is faster". Letting one tenant write those into shared state means letting them tell another tenant's run that a wrong answer is fine. So every run re-checks its own first native result against CPython, even when a thousand runs before it already did.

It costs one interpreted call per signature. It is the least interesting optimization in the system to give up.

Four isolation bugs we shipped

All four of these were live at some point. None was found by reasoning about the design; each was found by testing the property directly, and each now has a test that fails if it comes back.

1. Resource limits were being silently discarded

Limits were set before launching the jail — memory, CPU seconds, file size, process count — which is the obvious place. But the jail runs under sudo, and PAM resets the target user's limits on the way through. Every limit was gone by the time user code started.

They are now applied inside the jail, in the child, before a single line of the program is parsed. And only ever downward:

soft, hard = resource.getrlimit(what)
# Never raise a limit, only lower it: a warm slot may already be
# more restricted than this run asked for.
target = value if hard in (resource.RLIM_INFINITY,) else min(value, hard)
resource.setrlimit(what, (target, target))

Two related things we learned the hard way. RLIMIT_AS breaks numpy — OpenBLAS reserves an enormous amount of virtual address space per thread and a virtual-size cap kills it long before it touches real memory; the limit that works is RLIMIT_DATA. And RLIMIT_NPROC is per-uid, not per-process, so applying it before the jail has switched users limits the service.

2. Caller-supplied environment merged the wrong way

The API lets you pass environment variables. They were merged over ours, which means a caller could set MOJOSUB_MOJO, MOJOSUB_CACHE, PYTHONPATH or LD_PRELOAD and take the sandbox apart from the inside. Caller variables now merge under a reserved set that always wins.

A related fact that cost an afternoon: sudo scrubs the environment, so the jail rebuilds it with /usr/bin/env -i K=V … inside the namespace. Without that the program runs but has no trace file, no acceleration and no timings — it works, and quietly does none of what it is for.

3. The hidden tree was too hidden

The jail hides /home, /root, /opt, /var/log and everything under the data disks by mounting an empty tmpfs over them, then binding back a short allowlist. Two mistakes:

• The covering tmpfs was mode 000. Where an allowlisted path sits underneath one, 000 blocks traversal and the path is unreachable — the interpreter disappears. It is 0111: traversable, not listable.
• Sources have to be staged aside before their parent is overmounted. Bind a path onto itself after its parent is covered and you bind the empty replacement over the real thing.

The allowlist lives in the root-owned jail script, which is the only place it is trustworthy, and it is four entries long. Adding one is a security decision.

A related trap, since the sandbox user has to be able to execute the interpreter: uv's managed interpreters default to ~/.local/share/uv, inside the hidden /home. UV_PYTHON_INSTALL_DIR has to point somewhere world-readable or every jailed run dies with Permission denied on Python itself.

4. Cache seeding was whole-cache

A run gets a private directory of symlinks into the shared cache. It was being seeded with everything, so a run could enumerate every kernel every other tenant had ever compiled — a side channel telling you what other people are computing. Seeding is now per-program: a run is handed links for the units its own source hashes to, and nothing else.

What separates two runs is a name, not a mode

Worth stating plainly because it looks like a mistake in the source. The pool socket directory is 0666, and pool/ and runs/ are 0733 — traversable, not listable.

They have to be. A warm slot establishes its mount namespace before the runs it will serve exist, so the runs root itself must be bound in; a per-run bind would be invisible to it. What keeps one run out of another is that run directories are named with 32 random hex digits and the directory cannot be listed. Unguessable names are the mechanism. The mode is what makes them work.

Why warm interpreters do not leak

Cold start is about 220 ms and most of it is interpreter startup, the sandbox, and importing numpy. So runs land on a pre-warmed interpreter that already has all three, and each run is a fork() of it — about 60 ms end to end.

A pool of processes reused across tenants is exactly the shape of an information leak, so: the fork is the unit of reuse, not the process. A run mutates its own copy-on-write address space and exits. Nothing is handed back. The parent never executes user code.

That warmth caused one bug worth knowing about, though it is a correctness one rather than a security one. A pool slot imports mojosub before any run exists, so the cache root and the memoised toolchain lookup bind to the pool's environment. Without re-reading them per run, every kernel looks new for ever and nothing is ever native — the fast path silently switches itself off, and the only symptom is that cache.compiled stays 0.

What billing has to do with containment

One more, because it is the same class of thinking. We bill per millisecond, which makes the clock an attack surface in the honest direction: whatever we choose to include, users pay for.

So the compiler is not billed. It runs on our clock — the clock stops before the warm phase where in-flight compiles finish, and compile time that overlapped the run is subtracted, capped at half of it so a compiling run is discounted rather than given away. Queue time, environment builds and cold start are not billed either.

That is not generosity, it is the only defensible line: none of it is work the user asked for, and all of it is work we chose to do to make the next run faster.

The test that matters most

runner/tests/test_platform.py pins the promotion rule: the learned return type crosses, every verdict is dropped. It is the one place where a plausible-looking refactor — "why re-verify what a thousand runs already verified?" — turns a performance optimization into a mechanism for one tenant to certify another tenant's wrong answer.

If a change makes those tests awkward, the change is wrong.

The jail is runner/mojojail, a root-owned shell script installed by deploy.sh; it is the only thing here that runs as root and all it does with that is drop privileges. The boundary is in runner/hostcompile.py and runner/mojo_stub.py, and the isolation properties above are tested in runner/tests/test_security.py and security_test.go.

Measured on the machine that served this page: Intel Xeon E5-2697 v4 (Broadwell, AVX2), CPython 3.13, Mojo 1.0.0b3.dev2026072406. Numbers move between machines; the shape of the result is what we are claiming.