Compiler Choice Is a Systems Decision

When a document project starts failing on CI but works on your laptop, the usual suspects are the usual: paths, permissions, cache. I spent this summer collecting evidence for a different suspect: the compiler, and how it interacts with the machine it runs on. I compiled 1,784 real documents with five engines on three operating systems, and the results changed how I think about the question “which engine should we use?” The answer has less to do with typography than most of us assume. It behaves like a systems decision, closer to choosing a dependency manager than choosing a font.

Here is what I measured, and what it showed.

What I measured

The corpus is Engine-Transfer-Bench: 809 LaTeX documents and 975 Typst documents taken from real public sources, GitHub templates, Overleaf gallery projects, and Typst’s package universe. No toy documents written for the benchmark. The sources were never modified. Each compilation ran in a throwaway copy of the project directory, with a 60 second timeout that multi-pass LaTeX runs had to share. Success means the engine exited cleanly and produced a valid PDF.

I ran this in three places: a macOS arm64 laptop with BasicTeX plus package collections, Ubuntu x64 on GitHub Actions with the apt TeX Live subset, and Windows x64 on GitHub Actions with basic MiKTeX and auto-install switched on. Four LaTeX engines (Tectonic 0.17.0, pdfLaTeX, XeLaTeX, LuaLaTeX on TeX Live 2025) and Typst 0.15.1, 4,211 compilations per host.

The full dataset, harness, and paper are public: the preprint is arXiv:2608.18329, the replication package is on Zenodo (DOI 10.5281/zenodo.21831918), and the cross-OS harness is on GitHub.

The headline table

Success rates, same seeds, same harness, three operating systems:

Engine	macOS	Ubuntu	Windows	Gap	Worked on all 3
Tectonic	97.0	96.3	97.2	0.9	93.4%
Typst	95.2	83.0	83.0	12.2	83.0%
pdfLaTeX	88.8	76.6	87.9	12.2	72.4%
XeLaTeX	88.8	74.3	91.3	17.0	70.5%
LuaLaTeX	93.4	73.8	90.7	19.6	70.1%

Rates in percent. Gap is the spread between the best and worst host.

Two things jump out. Tectonic barely notices which OS it is on. The classic TeX engines swing by up to 19.6 percentage points across hosts, which is the difference between a build that mostly works and one that fails a quarter of the time.

Classic engines track the distribution, not the OS

The interesting part is where the classic engines lose. It is not random. Ubuntu’s failure concentration sits in missing_package and font errors, because the apt TeX Live subset installs a fixed set of packages and nothing more. Windows runs basic MiKTeX with AutoInstall=1, so when a document asks for a package that is missing, MiKTeX fetches it mid-compile. macOS sits in between with BasicTeX plus manually added collections.

Same documents. Same engine binaries in spirit. What differs is the package acquisition policy of the distribution around the engine. LuaLaTeX does not get worse between macOS and Ubuntu; the Ubuntu environment just stops supplying it with packages.

If you have ever said “XeLaTeX is flaky”, the data suggests a correction: XeLaTeX is not flaky, your package set is incomplete, and the engine takes the blame. That is why the same document can compile for one teammate and fail for another. It is also why “works on my machine” survives so stubbornly in document builds: the machine really is a variable.

Portability is a property you can choose

Tectonic’s stability is not magic. It ships a bundled package set inside its own versioned bundle and fetches anything else on demand, so its effective environment moves with it. The cost is real too: first compiles need network access, and an air-gapped CI runner would tell a different story. On a normal GitHub Actions runner, though, the trade lands at a 0.9 point spread across three operating systems, and 93.4 percent of documents succeed on every host.

That is the systems framing I mean. When you pick an engine, you are also picking an environment contract: who supplies packages, when, and what happens when one is missing. Tectonic and container-pinned full TeX Live both answer that question explicitly. A bare “we use XeLaTeX” answers nothing, and the 12 to 20 point swings in the table are the price.

Success is not fidelity

One more finding, because it undercut my own assumptions. For 702 documents that all four TeX engines can compile, I compared the text layers of the outputs across engines. A quarter of them, 179 of 702, fall below a 0.95 token-level similarity threshold in at least one engine pair. Same “success” verdict, meaningfully different PDFs. Missing glyphs, silent fallbacks, and spacing differences accumulate quietly.

So even the success column above is a floor, not a guarantee. If your pipeline validates “compiled without error” and ships the PDF, engine-dependent output differences can pass straight through. The benchmark’s similarity metric is coarse (the paper has the details), but a 25 percent flag rate is enough to want a diff step in any document pipeline that matters.

When the choice actually does not matter

To be fair to the classic engines: on those 702 engine-portable documents, all four TeX engines succeed on all of them. When your package environment is complete, engine choice stops being a reliability question entirely, and you are choosing on other grounds. On my hardware, Tectonic’s median compile time was about a third of LuaLaTeX’s (median ratio 3.24, Cliff’s delta 0.79), which matters at CI scale and not at all for a one-off thesis chapter.

What this does not prove

Wall-clock times are not compared across the three hosts, because they run on different VMs and hardware; the latency comparison above is same-machine only. Success rates are single-attempt results with a 60 second budget, so they measure document-environment fit, not what happens under retry or load. And the corpus skews toward templates and small-to-medium documents, so a different corpus would shift the absolute numbers.

The rule I take away

Pick your document engine the way you pick a dependency manager: by what it does to your build environment, not by taste or default. If your documents build in more than one place, a CI run, a colleague’s laptop, a container, prefer an engine whose package story travels with it, or pin a full distribution deliberately. If a build breaks only on one OS, check the distribution policy before you blame the engine or the document.

Every number above is reproducible from the Zenodo package, and the cross-OS harness can run the whole campaign from a GitHub Actions workflow dispatch if you want to try it on your own corpus.