Generating Weir Code with LLMs

As you know, I’ve been work­ing on a small pro­gram­ming lan­guage called Weir for gen­er­at­ing cor­rec­tions to nat­ural lan­guage. For the cu­ri­ous, I highly sug­gest read­ing my pre­vi­ous blog posts on the sub­ject if you haven’t al­ready.

• Building the Weir Language
• Updates on the Weir Language

In one of those pre­vi­ous blog posts, I spec­u­lated that an LLM might be quite ef­fec­tive at gen­er­at­ing these rules, es­pe­cially if they are given ac­cess to tools that can val­i­date and run tests. To­day, while writ­ing some pub­lic doc­u­men­ta­tion for the lan­guage, I did a bit of ex­per­i­men­ta­tion. I wanted to know: Can an LLM si­mul­ta­ne­ously han­dle the com­plex chal­lenge of un­der­stand­ing a DSL’s syn­tax, the se­man­tics of the English lan­guage, and be able to val­i­date its own work with tests? More to the point: can it do a bet­ter job with a DSL than an es­tab­lished lan­guage like Rust?

For those who want to fol­low along at home, here’s the GPT I’ve been toy­ing around with.

Some Experimentation

I of­ten hear that one should avoid novel or ex­ceed­ingly old pro­gram­ming lan­guages when work­ing with LLMs, sim­ply be­cause these lan­guages aren’t well rep­re­sented in their train­ing data. Based on this hearsay, I did­n’t ex­pect much go­ing into this lit­tle ex­per­i­ment of mine.

First, I won­dered if it could han­dle the triv­ial is­sue of cor­rect­ing “as nau­seam” to the cor­rect form, “ad nau­seam”. It’s a com­mon typo, and it should be pretty sim­ple Weir rule.

My min­i­mal GPT spit this out:

expr main (as nauseam)

let message "Did you mean `ad nauseam`?"
let description "Corrects the common typo `as nauseam` to the proper Latin phrase `ad nauseam`."
let kind "Typo"
let becomes "ad nauseam"

test "He repeated the point as nauseam." "He repeated the point ad nauseam."
test "This argument has been made ad nauseam." "This argument has been made ad nauseam."

That looks pretty good! All the syn­tax is cor­rect, and it even in­cluded some (passing!) tests. Claude Code or Codex could do this us­ing our ex­ist­ing Rust in­fra­struc­ture, but it would have taken sev­eral min­utes to churn out a valid re­sult. GPT 5.2 Instant was able to do this in a mat­ter of sec­onds. Not to men­tion that it did­n’t need a copy of the en­tire Harper source code.

I wanted to try a prob­lem that was a bit harder to solve, so I asked it to ad­dress the prob­lem of dou­ble-neg­a­tives. Af­ter think­ing for a few min­utes, it came up with four in­di­vid­ual rules that each ad­dress dif­fer­ent cases.

expr main <(
  [(don't), (do not), (doesn't), (does not), (didn't), (did not),
   (can't), (cannot), (couldn't), (could not),
   (won't), (will not), (wouldn't), (would not),
   (shouldn't), (should not),
   (isn't), (is not), (aren't), (are not), (wasn't), (was not), (weren't), (were not),
   (ain't)]
  [$have, $get, $need, $want, $know, $see, $find, $say, $do]
  no
), no>

let message "Avoid a double negative."
let description "If the clause is already negated (e.g., `don't`), prefer `any` over `no` (e.g., `don't have any`)."
let kind "Usage"
let becomes "any"
let strategy "MatchCase"

test "I don't have no idea." "I don't have any idea."
test "She did not get NO response." "She did not get ANY response."
test "I have no idea." "I have no idea."

expr main <(
  [(don't), (do not), (doesn't), (does not), (didn't), (did not),
   (can't), (cannot), (couldn't), (could not),
   (won't), (will not), (wouldn't), (would not),
   (shouldn't), (should not)]
  [$do, $say, $see, $hear, $find, $know, $have, $get]
  nothing
), nothing>

let message "Avoid a double negative."
let description "If the clause is already negated, replace `nothing` with `anything` (e.g., `didn't do anything`)."
let kind "Usage"
let becomes "anything"
let strategy "MatchCase"

test "He didn't do nothing." "He didn't do anything."
test "I can't see NOTHING." "I can't see ANYTHING."
test "I did nothing." "I did nothing."

expr main <(
  [(don't), (do not), (doesn't), (does not), (didn't), (did not),
   (can't), (cannot), (couldn't), (could not),
   (won't), (will not), (wouldn't), (would not)]
  [$tell, $call, $see, $know, $say, $help, $find]
  nobody
), nobody>

let message "Avoid a double negative."
let description "If the clause is already negated, replace `nobody` with `anybody`."
let kind "Usage"
let becomes "anybody"
let strategy "MatchCase"

test "I didn't tell nobody." "I didn't tell anybody."
test "We can't find NOBODY." "We can't find ANYBODY."
test "Nobody called." "Nobody called."

expr main <(
  [(don't), (do not), (doesn't), (does not), (didn't), (did not),
   (can't), (cannot), (couldn't), (could not),
   (won't), (will not)]
  [$go, $run, $walk, $drive, $get, $look]
  nowhere
), nowhere>

let message "Avoid a double negative."
let description "If the clause is already negated, replace `nowhere` with `anywhere`."
let kind "Usage"
let becomes "anywhere"
let strategy "MatchCase"

test "She can't go nowhere tonight." "She can't go anywhere tonight."
test "I did not look NOWHERE else." "I did not look ANYWHERE else."
test "Nowhere is safe." "Nowhere is safe."

Again, I’m quite im­pressed. Ob­vi­ously, it’s hard to tell at first glance whether these rules would hold up in pro­duc­tion for tens of thou­sands of users. Nev­er­the­less, these rules look cor­rect, and my ini­tial test­ing shows that they work. The model was able to suc­cess­fully ap­ply more com­plex logic: the de­riva­tion syn­tax and the fil­ter syn­tax.

In my orig­i­nal an­nounce­ment blog post, I ex­plained that the tar­get au­di­ence for Weir were the mem­bers of large or­ga­ni­za­tions that wish to en­force styl­is­tic con­ven­tions or brand names through­out their work­force. To that end, I wanted to test an LLM against some of Automattic’s nomen­cla­ture. Af­ter read­ing though some of our pub­lic doc­u­men­ta­tion, it come up with some ad­mit­tedly use­ful rules:

expr main (wpvip)

let message "Use the official spacing for VIP."
let description "Expands `wpvip` to the official brand forms `WordPress VIP` or `WP VIP`."
let kind "Miscellaneous"
let becomes ["WordPress VIP", "WP VIP"]
let strategy "Exact"

test "This runs on wpvip." "This runs on WordPress VIP."

expr main (jet[-, ( )]pack)

let message "Use the official spelling `Jetpack`."
let description "Standardizes `jet pack` / `jet-pack` to `Jetpack`."
let kind "Miscellaneous"
let becomes "Jetpack"
let strategy "Exact"

test "Install jet pack for backups." "Install Jetpack for backups."
test "Install jet-pack for backups." "Install Jetpack for backups."
test "Install Jetpack for backups." "Install Jetpack for backups."

expr main (word[-, ( )]press)

let message "Use the official spelling `WordPress`."
let description "Standardizes `word press` / `word-press` to the product name `WordPress`."
let kind "Miscellaneous"
let becomes "WordPress"
let strategy "Exact"

test "I build sites with word press." "I build sites with WordPress."
test "I build sites with word-press." "I build sites with WordPress."
test "I build sites with WordPress." "I build sites with WordPress."

Again, I’m quite pleased. With just a quick search of our web­site, it was able to dis­cern what style we wished to en­force and wrote func­tion­ing Weir rules to do so. I can see this be­ing help­ful at any num­ber of busi­ness that com­mu­ni­cate reg­u­larly.

Conclusion

Overall, I’m quite pleased with how well these LLMs were able to write Weir code. I’ve yet to fi­nal­ize any doc­u­men­ta­tion on the more com­plex parts of Weir’s syn­tax (like our POS-tagging sys­tem), which means I haven’t yet been able to test them with any LLMs. Even so, I’m im­pressed.

I even tried it out on Mistral’s tiny and ul­tra-fast three bil­lion pa­ra­me­ter model. It per­formed al­most as well as OpenAI’s 5.2 Instant model, al­beit with­out nearly the same level of cre­ativ­ity. I think this some­what proves that mod­ern LLMs are able to gen­er­al­ize to novel lan­guages, which makes them ex­ceed­ingly use­ful for DSLs like Weir.

I’m look­ing for­ward to see­ing how peo­ple end up tak­ing ad­van­tage of this.