Improving Rust Compile Times By 71%

If you main­tain or work on a pro­ject in any com­piled lan­guage, par­tic­u­larly a lan­guage that is known for hav­ing a slow com­piler, I be­lieve it is crit­i­cal to reg­u­larly sched­ule time to do an analy­sis of your build process and make ad­just­ments if nec­es­sary.

Compile-time in­creases slowly as a pro­ject morphs and grows. Slow enough that it of­ten goes un­no­ticed. I call this “compiler creep” and be­lieve it to be sim­i­lar in many ways to lifestyle creep. By sched­ul­ing time to do a full ac­count­ing of where your CPU cy­cles are go­ing, you can pre­emp­tively save your­self (and your con­trib­u­tors) a lot of time.

Yesterday was the big day for me. Over the last few months, Harper’s com­pile time from scratch in GitHub ac­tions had bal­looned to a five full min­utes. On my per­sonal lap­top, it had grown to three min­utes. That may not sound like a lot, but I of­ten need to build from scratch when swap­ping be­tween branches or check­ing out a pull re­quest. Incremental build times had also grown to an un­jus­ti­fi­able du­ra­tion. Iterating upon unit tests was start­ing to be­come quite slow.

Harper’s low la­tency is one of the rea­sons many turn to it over the al­ter­na­tives. In the early days, we ac­com­plished that by crank­ing the LLVM op­ti­mizer to the max, re­duc­ing the num­ber of code­gen units to one and us­ing link-time op­ti­miza­tion to take care of the rest. At the time, I be­lieve this re­sulted in a ~30% per­for­mance up­lift. Not bad!

For the unini­ti­ated, a code­gen unit is the small­est unit of work that is passed to the LLVM com­piler. If you al­low the pro­ject to be split into more than one code­gen unit, you can al­low more than one com­piler to run at a time. This, there­fore, in­creases the num­ber of phys­i­cal cores that can be used at any given mo­ment dur­ing com­pi­la­tion. The down­side is that the com­piler misses out on op­ti­miza­tion op­por­tu­ni­ties at the bound­aries be­tween code­gen units.

So when I sat down yes­ter­day, I was hardly sur­prised to see that code gen­er­a­tion was the bot­tle­neck. The com­piler was spend­ing very lit­tle time type-check­ing and link­ing, and a lot of time gen­er­at­ing and op­ti­miz­ing ma­chine code.

My ini­tial thought was to re­duce the amount of code that needed op­ti­miz­ing. So I did what any­one else would do and went search­ing for un­nec­es­sary de­pen­den­cies and other kinds of dead code in the repos­i­tory. I found some, but not enough to make a mean­ing­ful dent in the com­pile time.

That’s when I took a look at our build con­fig­u­ra­tion and asked my­self: “Do we still need LTO? Could we in­crease our code­gen units?”

In the last few months, I’ve made some sig­nif­i­cant im­prove­ments to caching and pipelin­ing, which means the bot­tle­neck has shifted from com­pile-time op­ti­miza­tion to mem­ory band­width. We sim­ply don’t need the com­piler to do as much work any­more. So, I tried dis­abling LTO and re­vert­ing the code­gen units to their de­fault value. What hap­pened?

Clean builds went from five min­utes on CI to less than two. All in, that’s an im­prove­ment of 71%. On top of that, since it was the code gen­er­a­tion stage that was slow­ing down in­cre­men­tal builds as well, we saw a sim­i­lar im­prove­ment to test ex­e­cu­tion time. That means there is now less time be­tween mak­ing a change in the code, and know­ing how it af­fects our ex­ist­ing test cases. Since that’s a huge por­tion of a con­trib­u­tor’s time, im­prove­ments to in­cre­men­tal builds can be force-mul­ti­pli­ers for the en­tire pro­ject.

I was sur­prised such a sim­ple change would have such an out­sized im­pact on the de­vel­oper ex­pe­ri­ence. It felt too good to be true, so I re-ran our bench­marks to see if the change would have a neg­a­tive im­pact on Harper’s ac­tual per­for­mance. Our un­cached lint times took three per­cent longer than be­fore. I think that’s ac­cept­able.

Is there a les­son to be learned here? I think so. I im­plore you to take a half-hour this week to mess around with your com­pil­er’s set­tings. Even an im­prove­ment of a few sec­onds per it­er­a­tion can com­pound into hours of time sav­ings, both for you and your fel­low con­trib­u­tors. If you think your com­piler set­tings are al­ready per­fect, do it any­way. Your code or re­quire­ments may have changed.

This is great step in my re­cent work to make peo­ple’s first con­tri­bu­tions to Harper as fric­tion­less as pos­si­ble.