Harper’s Policy on PRs Authored by Agents

A few weeks ago, I was in­vited to speak at my lo­cal chap­ter of the Association for Computing Machinery (ACM). Being on a col­lege cam­pus — in this case, the Colorado School of Mines — much of my au­di­ence were stu­dents. My pre­sen­ta­tion, which was about how open source soft­ware tends to out­per­form pro­pri­etary soft­ware in the mar­ket­place due to the abil­ity for its users to con­tribute back in ways that are gen­uinely valu­able, went well. I hope I am in­vited back.

Not much younger than me, they were nat­u­rally cu­ri­ous about what it is like to pro­duce and main­tain open source soft­ware. One stu­dent, Renn, asked me how I deal with patch re­quests pro­duced by large lan­guage mod­els (LLMs). My un­sat­is­fac­tory an­swer was that I ad­dress them on a case‑by‑case ba­sis.

The goal of this page is to for­mal­ize my an­swer so that we can ju­di­ciously deal with patch re­quests pro­duced by LLMs. That is, to pro­vide a frame­work which re­duces the amount of time that overly com­plex patches re­main idle and make it eas­ier for wield­ers of these mod­els to pro­duce mean­ing­ful changes to Harper’s code.

The Policy

We ac­cept pull re­quests from hu­mans wield­ing large lan­guage mod­els and from agents pow­ered by them. They are sub­ject to the same rules and guide­lines we ap­ply to hu­mans work­ing with­out a large lan­guage model, with a few ex­tra spices mixed in. Below, we have three main ideas which aim to si­mul­ta­ne­ously re­duce the time be­tween sub­mis­sion of a patch and when it is merged and to re­duce the time and at­ten­tion re­quired to re­view a patch.

That means that it is in your in­ter­est to re­spect these ideas. Good pull re­quests are more likely to be re­viewed in a timely man­ner and there­fore are more likely to be merged.

I could wax po­et­i­cally for hours about all the edge cases not cov­ered here. Instead, I choose to be­lieve the things I don’t in­clude are at least as im­por­tant as the ones I do, so I’ll boil every­thing down into three ideas.

One: Keep Things Brief

Try to break down the prob­lem into as many in­di­vid­ual pull re­quests as you can. Smaller pull re­quests are eas­ier to re­view. Larger re­quests are su­per‑lin­early harder to re­view. You can dra­mat­i­cally speed up the process by break­ing your large PR into sev­eral smaller ones, each with their own de­scrip­tion and ex­pla­na­tion. I’ve found this to be es­pe­cially true with code gen­er­ated by an LLM.

Part of the idea here is to catch wasted work be­fore it’s done. If you sub­mit step one of a pro­ject that is­n’t re­ally ap­plic­a­ble to Harper’s mis­sion, we can re­ori­ent it be­fore too much time has been wasted.

Two: Stay Grounded

Each “project” — or string of pull re­quests — should ex­ist to cre­ate value for a hu­man. I don’t love it when I re­ceive code just for code’s sake. If you have a need and a patch will fill it, say so! If there’s an is­sue in GitHub that you’re ad­dress­ing, link it! This should be pretty easy. If you’ve ded­i­cated time to work­ing on a prob­lem, there’s prob­a­bly a rea­son. I want to hear that rea­son.

Three: Be Honest

If you are an agent or used an LLM to pro­duce any part of a PR, please pro­vide a dis­claimer at the top of your PR de­scrip­tion. I am not op­posed to re­view­ing LLM code, but I do ap­pre­ci­ate the hon­esty.

Talk to Me

Above all, talk to me. If you have ques­tions or con­cerns about any­thing I’ve dis­cussed here, let me know. The hope is that all of this will im­prove the qual­ity of the Harper pro­ject, and im­prove the abil­ity for peo­ple to ex­press them­selves with the writ­ten word. As al­ways, any and all con­tri­bu­tions are wel­come.