<h1>The Books I Read in January 2025</h1>
I only got to read two full books this month.
The ﬁrst was a short novel by Matt Haig with the ti­tle <a href="https://app.thestorygraph.com/books/d9c7ed04-6148-4e01-a118-d96cba16f507">The Midnight Library.</a>
It reads like and feels like a ther­apy ses­sion.
I quite en­joyed it, but I’m not sure it is for every­one.
The sec­ond book (and the one I got the most value from) was <a href="https://app.thestorygraph.com/books/2505f748-ac12-41fb-a411-0868cbe3694c">Working in Public: the Making and Maintenance of Open Source Software</a> by Nadia Eghbal.
Until now, I’ve largely or­ga­nized Harper in ways in­spired by pro­jects like Vite and Rust that I in­ter­act with on a daily ba­sis.
Eghbal gives a more holis­tic view than mine, writ­ten for an ex­ter­nal ob­server.
This book was es­pe­cially in­ter­est­ing due to the com­pre­hen­sive cov­er­age of “open” pro­jects with no code at all.
She seems to be­lieve the OSS soft­ware man­age­ment prob­lem is one of so­cial com­plex­ity, not tech­ni­cal com­plex­ity.
Furthermore, Nadia makes the case that syn­chro­nous com­mu­ni­ca­tion, while of­ten a bur­den on main­tainer at­ten­tion, is usu­ally nec­es­sary.

The Books I Read in January 2025

<h1>Transformation-based Learning for POS Tagging</h1>
<img src="/images/ctlm.webp" alt="CTLM">
<a href="https://writewithharper.com/">Harper</a> is cur­rently un­der­go­ing some pretty rad­i­cal changes when it comes to its lan­guage analy­sis. These im­prove­ments will im­prove the out­put of our ex­ist­ing rule en­gine, in ad­di­tion to mak­ing en­tirely new cor­rec­tions pos­si­ble. This post will cover our ex­ist­ing NLP pipeline, the re­cent changes and im­prove­ments to our ma­chine learn­ing ap­proach, and what will come next.
While AI is a com­mon topic of dis­cus­sion on­line, I don’t hear much about ac­tual ma­chine learn­ing. In that light, I hope this post piques some­one’s in­ter­est.
<h2>What is POS Tagging?</h2>
POS (Part-of-speech) tag­ging is the ﬁrst step of most NLP (Natural Language Processing) pipelines. For any gram­mar checker worth its salt, POS tag­ging is es­sen­tial. Apart from the ba­sic cor­rec­tions you’re ca­pa­ble of do­ing with sim­ple string ma­nip­u­la­tion, most gram­mar check­ing di­rectly or in­di­rectly de­pends on POS tag­ging. High-quality tag­ging re­sults in high-qual­ity sug­ges­tions.
What is POS tag­ging? It is the process of iden­ti­fy­ing which pos­si­ble de­f­i­n­i­tion of a word is be­ing used, based on the sur­round­ing con­text. For those un­fa­mil­iar with the ter­ri­tory, I’m cer­tain an ex­am­ple is the best way to ex­plain.
<blockquote>
“I am go­ing to go tan in the sun.”
</blockquote>
Here we have a sim­ple, English sen­tence. In this case, it is clear the word “tan” is be­ing used as verb. The lin­guists in the au­di­ence would point out that it is specif­i­cally in the ﬁrst-per­son fu­ture tense. Consider this sim­i­lar sen­tence:
<blockquote>
“I am al­ready very tan, so I will stay in­side.”
</blockquote>
In this sen­tence, the word “tan” is be­ing used as an ad­jec­tive. How can we tell?
As in­tel­li­gent hu­mans, some of whom have been speak­ing English their en­tire lives, it is easy for us to de­ter­mine which words are serv­ing which roles. It’s not as easy for a com­puter to do the same. From an al­go­rith­mic stand­point, there are a num­ber of ways to go about it, each with dif­fer­ing lev­els of “machine learn­ing” re­quired.
Before this week, Harper pri­mar­ily took a dic­tio­nary-based ap­proach. In short: we ship a “dictionary” of English words to the user’s ma­chine and use hash table lookups to de­ter­mine the pos­si­ble roles each word could as­sume. The au­thors to our rule en­gine could then use rudi­men­tary de­duc­tive rea­son­ing to nar­row the pos­si­bil­i­ties down. This strat­egy is re­mark­ably ef­fec­tive and it has scaled to tens of thou­sands of users with sur­pris­ingly few hic­cups.
That said, there are edge-cases and sys­tems (which I’ll cover next week when I dis­cuss chunk­ing) which re­quire ex­treme speci­ﬁcity from POS tags. My mis­sion: im­prove our POS tag­ging to in­crease the con­ﬁ­dence of Harper’s out­put and open the door for more ad­vanced al­go­rithms.
<h2>Why Transformation-based Learning?</h2>
The lit­er­a­ture high­lights four un­der­ly­ing ma­chine learn­ing model strate­gies that seem to work well for POS tag­ging.
<ul>
<li>Hidden Markov Models (probabilistic mod­els that pre­date the mod­ern deep learn­ing era)</li>
<li>Maximum Entropy Models (statistical mod­els closely re­lated to lo­gis­tic re­gres­sion).</li>
<li>Transformer-based Models (which use deep neural net­works)</li>
<li>Transformation-based Rule Models (which are based on learned rules)</li>
</ul>
While I heav­ily con­sid­ered us­ing a neural net­work (either via an HMM or MEM), I dis­carded the tech­nol­ogy for three rea­sons.
<ul>
<li>TRMs are typ­i­cally more ac­cu­rate (barely; mea­sured in ba­sis points).</li>
<li>TRMs are more amenable to ﬁne-tun­ing.</li>
<li>TRMs are ex­cep­tion­ally low-la­tency and can be com­pressed quite small.</li>
</ul>
Transformation-based learn­ing is re­mark­ably sim­ple. It boils down to just four steps:
<ul>
<li>Use a sim­ple, sto­chas­tic model to la­bel your data. This can be as sim­ple as tag­ging each to­ken (or other dis­crete com­po­nent) with that vari­ant’s most com­mon tag. It does­n’t need to su­per ac­cu­rate, just enough to es­tab­lish a base­line.</li>
<li>Identify the er­rors be­tween the tags in your canon­i­cal data and that which pro­duced by your base­line model.</li>
<li>Using a ﬁ­nite list of hu­man-de­ﬁned tem­plates, gen­er­ate can­di­date rules that trans­form the out­put of the base­line model into some­thing else. This is where the term “transformation-based” comes from.</li>
<li>Apply each of the can­di­date rules to the base­line mod­el’s out­put. Check if the re­sult is more ac­cu­rate than be­fore. If so, save the rule for fu­ture use.</li>
</ul>
These saved can­di­dates be­come your model.
<h2>POS-Tagging us­ing Transformation-based Learning</h2>
Let’s ap­ply these steps to build a POS-tagging sys­tem.
For our base­line model, we will just as­sign each word in our dataset the most com­mon POS tag as­so­ci­ated with that word. If the word is “tan”, we’ll as­sign it’s most com­mon POS tag (verb). It will of­ten be in­cor­rect, but those cases will be han­dled by our rules.
To iden­tify the base­line mod­el’s er­rors, we’ll use an off-the-shelf tree-bank from the Universal Dependencies pro­ject.
Our rule tem­plates will take the form of these <code>PatchCriteria</code>. By ini­tial­iz­ing our can­di­dates with any one of these enum vari­ants and ini­tial­iz­ing the child vari­ables to ran­dom val­ues, we can cover a good num­ber of cases.
<pre><code class="hljs language-rust">#[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)]
pub enum PatchCriteria {
 WordIsTaggedWith {
 /// Which token to inspect.
 relative: isize,
 is_tagged: UPOS,
 },
 AnyWordIsTaggedWith {
 /// The farthest relative index to look
 max_relative: isize,
 is_tagged: UPOS,
 },
 SandwichTaggedWith {
 prev_word_tagged: UPOS,
 post_word_tagged: UPOS,
 },
 WordIs {
 relative: isize,
 word: String,
 },
 /// Not applicable to the Brill Tagger, only the chunker
 NounPhraseAt {
 is_np: bool,
 relative: isize,
 },
 Combined {
 a: Box&#x3C;PatchCriteria>,
 b: Box&#x3C;PatchCriteria>,
 },
}
</code></pre>
Finally, we’ll ap­ply each of the hun­dreds of thou­sands of can­di­dates to our tree­bank to see if the re­sult of their trans­for­ma­tions have a lower er­ror rate than the base­line.
Here are a cou­ple of the can­di­dates we found:
<pre><code class="hljs language-json">[
 {
 "from": "PRON",
 "to": "SCONJ",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 0,
 "word": "that"
 }
 },
 "b": {
 "WordIsTaggedWith": {
 "relative": -1,
 "is_tagged": "VERB"
 }
 }
 }
 }
 },
 {
 "from": "PART",
 "to": "ADP",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 1,
 "word": "there"
 }
 },
 "b": {
 "AnyWordIsTaggedWith": {
 "max_relative": -4,
 "is_tagged": "NOUN"
 }
 }
 }
 }
 }
]
</code></pre>
That’s the whole process! With it, I was able to bring our pre­vi­ous ac­cu­racy all the way up to 95% (from 40%) with­out a mean­ing­ful change in lint­ing la­tency or com­piled bi­nary size.

Harper is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible.

Harper is cur­rently un­der­go­ing some pretty rad­i­cal changes when it comes to its lan­guage analy­sis. These im­prove­ments will im­prove the out­put of our ex­ist­ing rule en­gine, in ad­di­tion to mak­ing en­tirely new cor­rec­tions pos­si­ble.

Transformation Based Learning

<h1>Build a Wordle Solver Using Rust</h1>
<h2>The Game</h2>
<a href="https://www.nytimes.com/games/wordle/index.html">Wordle</a> is a rel­a­tively sim­ple game. If you have ever played Mastermind, it should sound fa­mil­iar. The goal is to ﬁg­ure out a mys­tery word with as few guesses as pos­si­ble. The mys­tery word changes each day. Here are two ex­am­ple guesses.
<img src="/images/wordle_example.png" alt="Two example guesses from the game Wordle">
After a guess, each let­ter’s color changes.
Green — The let­ter is cor­rect.
Yel­low— The let­ter ex­ists in the word, but not in that space.
Gray — The let­ter does not ex­ist in the word.
As you can see, there are a max­i­mum of six guesses. If you can­not ﬁnd the mys­tery word within six guesses, you lose. I have been com­pet­ing with my grand­mother each day to ﬁnd the word in as few guesses as pos­si­ble.
<img src="/images/xkcd_nerd_sniping.png" alt="XKCD Comic #356">
This to­tally nerd-sniped me. I felt an over­whelm­ing urge to build an app that could, con­ceiv­ably ﬁnd the mys­tery word in as few guesses as pos­si­ble.
<h2>How I Did It</h2>
First things ﬁrst, we need a list of English words. I ini­tially used the <a href="http://www.mieliestronk.com/wordlist.html">corn­cob</a> list, but I found greater suc­cess with <a href="https://github.com/dwyl/english-words/">dwyl’s</a> list.
For this pro­ject, I de­cided to use Rust, just be­cause I felt most con­ﬁ­dent in my abil­ity to make an MVP quickly.
Both the word lists I used are for­mat­ted as a se­quence of in­di­vid­ual words, sep­a­rated by <code>\n</code> char­ac­ters. On Windows (which is what I am us­ing), they also have pesky those lit­tle <code>\r</code> char­ac­ters.
Wordle is heav­ily fo­cused on let­ters. I can re­move items from the word list based on what let­ters I know aren’t in the mys­tery word (these are gray let­ters in-game), and I can re­move items based what let­ters I know are in the mys­tery word (the or­ange or green let­ters), but in a lot of cases that still leaves a lot of pos­si­ble words. I need to way to sort words based on how likely their let­ters are.
To do this, I count how fre­quently each let­ter ap­pears in the word list, and give each word a score based on how fre­quently its com­po­nents ap­pear.
The ﬁrst step in the pro­gram is to load the word list and count the let­ters:
<pre><code class="hljs language-rust">// Store the total number of times a letter appears.
let mut letter_scores = HashMap::new();
// The final list of words. It will make like easier later in the program to store the words as Vec&#x3C;char>.
let mut word_list = Vec::new();
let mut last_word = Vec::new();
let file = std::fs::read("corncob_lowercase.txt")?;

// Iterate through all the bytes in the wordlist file, ignoring all `\r` instances.
for letter in file {
 let letter = letter as char;
 match letter {
 '\n' => {
 word_list.push(last_word);
 last_word = Vec::new();
 }
 '\r' => (),
 _ => {
 let entry = letter_scores.entry(letter).or_default();
 *entry += 1;
 last_word.push(letter);
 }
 }
}
</code></pre>
Using the de­fault <code>HashMap</code> (which uses <code>SipHash</code>, which is­n’t great for sin­gle-char­ac­ter lookup), prob­a­bly is­n’t the best, per­for­mance-wise, but this is just a toy pro­gram, and does­n’t need to be the fastest thing in the world.
Next, we need to go through the word list, and elim­i­nate words that con­tain gray let­ters. Here is a func­tion that helps do that:
<pre><code class="hljs language-rust">fn matches_found(
 word: &#x26;[char],
 found: &#x26;[char],
 not: &#x26;[char],
 must: &#x26;[char],
 masks: &#x26;[Vec&#x3C;char>],
) -> bool {
 // Check if the word contains a letter we know *isn't* in the mystery. &#x3C;-- The gray letters.
 for c in not {
 if word.contains(c) {
 return false;
 }
 }

 // Check if the word contains the letters we don't know the positions of, but know they are in the mystery word.. &#x3C;-- The orange letters.
 let mut found_letters = 0;
 for c in must {
 if word.contains(c) {
 found_letters += 1;
 }
 }
 if found_letters &#x3C; must.len() {
 return false;
 }

 // Check if the word has letters we know exist in the word, but not at the right spots. &#x3C;-- The orange letters.
 for mask in masks {
 for i in 0..min(word.len(), mask.len()) {
 if word[i] == mask[i] {
 return false;
 }
 }
 }

 // Check if the word contains the already found (green) letters.
 for i in 0..min(word.len(), found.len()) {
 if found[i] != ' ' &#x26;&#x26; word[i] != found[i] {
 return false;
 }
 }

 true
}
</code></pre>
It ac­cepts a few dif­fer­ent char slices:
<ul>
<li>Word: the word we want to check.
Found: this is a slice con­tain­ing the let­ters we have found (the green ones), with “ “ (space) char­ac­ters in the lo­ca­tions we don’t know the char­ac­ter of.</li>
<li>Not: this is a slice con­tain­ing the let­ters we know aren’t in the mys­tery word.</li>
<li>Must: this is a slice con­tain­ing the let­ter we know are in the mys­tery word, but we don’t know the po­si­tion of.</li>
<li>Masks: this is a se­ries of masks. We re­move every word that has let­ters that match any mask here. This is use­ful for elim­i­nat­ing words in the wordlist that con­tain punc­tu­a­tion and for elim­i­nat­ing words that con­tain or­ange let­ters, but in po­si­tions we know they aren’t.</li>
</ul>
Now all we have to do is run each word in the word list and see if it matches our al­ready known char­ac­ters, up­dat­ing the con­tents of each slice with new in­for­ma­tion af­ter each guess.
<h2>Why You Should Care</h2>
This sounds like a use­less prob­lem. It is. There is no way this will ben­e­ﬁt any­one other than me, and I def­i­nitely won’t use this when I’m ac­tu­ally com­pet­ing with my grand­mother.
Then why did you do it?
Useless an­swers to use­less prob­lems are use­ful. They teach us how to im­prove, with­out the pres­sure of real stakes. They are also just plain fun.
It’s also a re­ﬂec­tion. How would you have ap­proached this prob­lem in the past? How has your think­ing im­proved. Maybe it’s a bit mag­nan­i­mous to say this lit­tle Wordle solver is the key to self re­ﬂec­tion, but I don’t think it’s that far off.
<hr>
<h2>A Reﬂection From Months Later</h2>
Hi! I am re­turn­ing to this pro­ject months later with a few thoughts.
When I ﬁrst wrote this ar­ti­cle, I com­pletely ne­glected to share my ﬁt­ness test for each word. In hind­sight, it’s a good thing I did­n’t. It was the ex­act method 3Blue1Brown de­scribed as “naive” in his (fantastic) <a href="https://youtu.be/v68zYyaEmEA">video</a> on this very topic.

I built a Wordle solver in Rust to beat my grandma. Follow the journey of how I did, and how I failed.

I built a Wordle solver in Rust to beat my grandma. Follow the jour­ney of how I did, and how I failed.

Build a Wordle Solver Using Rust

notifications

<h1>Notiﬁcations</h1>
I have found that the ﬁrst cou­ple hours of the day are my most pro­duc­tive.
I try to move te­dious tasks to later so I can fo­cus on solv­ing “hard prob­lems” that need my full at­ten­tion.
In the past, this has been hard to do for one rea­son: GitHub no­ti­ﬁ­ca­tions.
They pull me in and con­sume hours of my time.
Re­spond­ing to and work­ing with OSS con­trib­u­tors is im­por­tant to me, but that can hap­pen later in the day.
So I wrote a lit­tle ViolentMonkey script to hide the no­ti­ﬁ­ca­tions icon dur­ing these early morn­ing hours.
<pre><code class="hljs language-javascript">// ==UserScript==
// @name Hide Notification Indicator
// @namespace Violentmonkey Scripts
// @match https://github.com/*
// @grant none
// @version 1.0
// @author Elijah Potter
// @description 1/29/2025, 8:16:11 AM
// ==/UserScript==

function hideEm() {
 const matches = document.getElementsByClassName(
 "AppHeader-button--hasIndicator",
 );
 for (const element of matches) {
 element.remove();
 }
}

new MutationObserver(hideEm).observe(document, {
 childList: true,
 subtree: true,
});
</code></pre>

Elijah Potter

Notiﬁcations

Other Stuff

The Books I Read in January 2025

Transformation Based Learning

Build a Wordle Solver Using Rust