Avoid Complexity

For as long as I can re­mem­ber, I’ve told peo­ple that the real chal­lenge of soft­ware en­gi­neer­ing is­n’t writ­ing code. It is­n’t doc­u­men­ta­tion or CI pipelines ei­ther. No, the biggest chal­lenge in soft­ware en­gi­neer­ing is man­ag­ing com­plex­ity.

Think about it. The typ­i­cal soft­ware en­gi­neer is a sin­gle per­son on a vast team, writ­ing and edit­ing a sys­tem com­posed of hun­dreds, mil­lions, or even bil­lions of lines of code. I find that in­tim­i­dat­ing. How in the world are we sup­posed to build ro­bust and re­li­able sys­tems when fac­ing such com­plex odds?

I worry that, in my fas­ci­na­tion in the novel ap­pli­ca­tions of ma­chine learn­ing we hear about on a day-to-day ba­sis, I’ve got­ten dis­tracted from the beast of com­plex­ity. I’ve al­lowed the Harper code­base to ac­crue tech debt.

I re­cently watched DHH’s keynote from Rails World 2025. While I def­i­nitely do not agree with every­thing he has to say, I be­lieve he makes some poignant points about com­plex­ity. Some things, he says, which we con­sider nor­mal to­day would have been lu­di­crous ten years ago. Some­how, with com­puter hard­ware faster than ever, we’ve ar­rived at a point where 20 minute CI runs are ac­cept­able. Where a de­ploy to pro­duc­tion can take an hour and set­ting up a new en­vi­ron­ment can take days.

I want to change that—at least, for Harper.

While work­ing on sev­eral smaller pro­jects, I’ve been cut­ting the fat out of the Harper monorepo. That means re­mov­ing un­nec­es­sary build steps, par­al­leliz­ing in­di­vid­ual CI jobs, and re­mov­ing op­ti­miza­tion flags that don’t work.

Harper is­n’t at the 2005 stan­dard yet, but I want con­tribut­ing to be as easy as pos­si­ble. Ide­ally, it should­n’t take more than five min­utes to go from cloning the repo to com­mit­ting code. That starts with sim­pli­fy­ing the en­vi­ron­ment.

Addendum

I just came across this other blog post where they dis­cuss achiev­ing three times the per­for­mance for a quar­ter of the price by switch­ing from a com­plex AWS setup to a bare metal Hetzner server. An­other win for sim­pler sys­tems!