Using Codex is a Lot Like Baking

The biggest prob­lem with AI pro­gram­ming to­day is not what I ex­pected a few years ago. I truly did­n’t be­lieve they would be able to do any eco­nom­i­cally valu­able work.

For web de­vel­op­ment, mod­ern pro­gram­ming tools like Codex are quite good. They are able to un­der­stand a query and iden­tify the rel­e­vant ar­eas in a code­base. Then, they can make mod­i­fi­ca­tions and it­er­ate on the so­lu­tion un­til it fits the prompt. Frankly, I never ex­pected them to reach this point.

No, the pre­vail­ing prob­lem I’ve en­coun­tered with these tools is not their abil­ity to do work. Rather, the prob­lem comes from their in­abil­ity to ask ques­tions up-front and it­er­ate on their men­tal model be­fore start­ing to code.

Here’s what I mean.

Some Context

Harper’s land­ing page is pretty im­por­tant. It sets ex­pec­ta­tions for po­ten­tial users with a work­ing in-browser demo and re­duces un­pro­duc­tive traf­fic to our is­sue-tracker by an­swer­ing com­mon ques­tions. In the past, im­prov­ing the cap­i­tal-“Q” Quality of our land­ing page has had a sig­nif­i­cant im­pact on Harper’s growth rate.

What Did I Do?

Earlier this week, I wanted to make some sig­nif­i­cant changes to parts of this land­ing page. Since the com­plex­ity of this part of the code-base is rel­a­tively small, I wanted to try some­thing: a new (to me) way to work with OpenAI’s Codex.

Most of the mar­ket­ing ma­te­ri­als sur­round­ing these tools im­ply that hav­ing a con­ver­sa­tion with the “agent” is the in­tended work­flow. Sam Altman et al. make it seem like I’m sup­posed to boot up Codex and have a dis­cus­sion. At the end of that dis­cus­sion, the thing in my head is down on pa­per (or code).

I wanted to try some­thing new. In­stead of a con­ver­sa­tion, I wrote a de­tailed goal doc­u­ment. I usu­ally would­n’t do such a thing for such a triv­ial task, but as I said: this was an ex­per­i­ment.

To par­al­lel what real pro­ject doc would look like, I in­cluded what, why, and how the changes I wanted should be made. No­tably, I tried to an­tic­i­pate any pos­si­ble re­vi­sions or mis­in­ter­pre­ta­tions the agent might make, which I would later have to have a con­ver­sa­tion about. When I was done, I gave it to the agent and went to work on some­thing else. I came back an hour later, and it was done.

What’s the Big Deal?

Imagine this sit­u­a­tion. It may sound fa­mil­iar to you.

You sit down to ex­per­i­ment with an LLM. You want to see what all hype around vibe-cod­ing looks like. You tell the agent what you want. You wait a few min­utes for it to do its thing. The re­sult does­n’t look right, so you prompt it once more. You wait some more. The cy­cle con­tin­ues.

This is ex­tra­or­di­nar­ily in­ef­fi­cient. You’re cos­play­ing as a mi­cro­manag­ing boss.

If I had done used the con­ver­sa­tional mi­cro-man­ag­ing strat­egy when work­ing on the land­ing page, it prob­a­bly would have taken just as long (about an hour). The down­side, is that I don’t get to go work on some­thing else in the mean­time.

By writ­ing a plan­ning doc­u­ment for your agent, you can give your­self room to breath, sav­ing time and be­ing more pro­duc­tive as a re­sult. It’s a lot like bak­ing. You put the in­gre­di­ents to­gether, put them in the oven, then walk away and some­thing else while it cooks.