<h1>Transformation-based Learning for POS Tagging</h1>
<img src="/images/ctlm.webp" alt="CTLM">
<a href="https://writewithharper.com/">Harper</a> is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible. This post will cover our existing NLP pipeline, the recent changes and improvements to our machine learning approach, and what will come next.
While AI is a common topic of discussion online, I don’t hear much about actual machine learning. In that light, I hope this post piques someone’s interest.
<h2>What is POS Tagging?</h2>
POS (Part-of-speech) tagging is the first step of most NLP (Natural Language Processing) pipelines. For any grammar checker worth its salt, POS tagging is essential. Apart from the basic corrections you’re capable of doing with simple string manipulation, most grammar checking directly or indirectly depends on POS tagging. High-quality tagging results in high-quality suggestions.
What is POS tagging? It is the process of identifying which possible definition of a word is being used, based on the surrounding context. For those unfamiliar with the territory, I’m certain an example is the best way to explain.
<blockquote>
“I am going to go tan in the sun.”
</blockquote>
Here we have a simple, English sentence. In this case, it is clear the word “tan” is being used as verb. The linguists in the audience would point out that it is specifically in the first-person future tense. Consider this similar sentence:
<blockquote>
“I am already very tan, so I will stay inside.”
</blockquote>
In this sentence, the word “tan” is being used as an adjective. How can we tell?
As intelligent humans, some of whom have been speaking English their entire lives, it is easy for us to determine which words are serving which roles. It’s not as easy for a computer to do the same. From an algorithmic standpoint, there are a number of ways to go about it, each with differing levels of “machine learning” required.
Before this week, Harper primarily took a dictionary-based approach. In short: we ship a “dictionary” of English words to the user’s machine and use hash table lookups to determine the possible roles each word could assume. The authors to our rule engine could then use rudimentary deductive reasoning to narrow the possibilities down. This strategy is remarkably effective and it has scaled to tens of thousands of users with surprisingly few hiccups.
That said, there are edge-cases and systems (which I’ll cover next week when I discuss chunking) which require extreme specificity from POS tags. My mission: improve our POS tagging to increase the confidence of Harper’s output and open the door for more advanced algorithms.
<h2>Why Transformation-based Learning?</h2>
The literature highlights four underlying machine learning model strategies that seem to work well for POS tagging.
<ul>
<li>Hidden Markov Models (probabilistic models that predate the modern deep learning era)</li>
<li>Maximum Entropy Models (statistical models closely related to logistic regression).</li>
<li>Transformer-based Models (which use deep neural networks)</li>
<li>Transformation-based Rule Models (which are based on learned rules)</li>
</ul>
While I heavily considered using a neural network (either via an HMM or MEM), I discarded the technology for three reasons.
<ul>
<li>TRMs are typically more accurate (barely; measured in basis points).</li>
<li>TRMs are more amenable to fine-tuning.</li>
<li>TRMs are exceptionally low-latency and can be compressed quite small.</li>
</ul>
Transformation-based learning is remarkably simple. It boils down to just four steps:
<ul>
<li>Use a simple, stochastic model to label your data. This can be as simple as tagging each token (or other discrete component) with that variant’s most common tag. It doesn’t need to super accurate, just enough to establish a baseline.</li>
<li>Identify the errors between the tags in your canonical data and that which produced by your baseline model.</li>
<li>Using a finite list of human-defined templates, generate candidate rules that transform the output of the baseline model into something else. This is where the term “transformation-based” comes from.</li>
<li>Apply each of the candidate rules to the baseline model’s output. Check if the result is more accurate than before. If so, save the rule for future use.</li>
</ul>
These saved candidates become your model.
<h2>POS-Tagging using Transformation-based Learning</h2>
Let’s apply these steps to build a POS-tagging system.
For our baseline model, we will just assign each word in our dataset the most common POS tag associated with that word. If the word is “tan”, we’ll assign it’s most common POS tag (verb). It will often be incorrect, but those cases will be handled by our rules.
To identify the baseline model’s errors, we’ll use an off-the-shelf tree-bank from the Universal Dependencies project.
Our rule templates will take the form of these <code>PatchCriteria</code>. By initializing our candidates with any one of these enum variants and initializing the child variables to random values, we can cover a good number of cases.
<pre><code class="hljs language-rust">#[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)]
pub enum PatchCriteria {
 WordIsTaggedWith {
 /// Which token to inspect.
 relative: isize,
 is_tagged: UPOS,
 },
 AnyWordIsTaggedWith {
 /// The farthest relative index to look
 max_relative: isize,
 is_tagged: UPOS,
 },
 SandwichTaggedWith {
 prev_word_tagged: UPOS,
 post_word_tagged: UPOS,
 },
 WordIs {
 relative: isize,
 word: String,
 },
 /// Not applicable to the Brill Tagger, only the chunker
 NounPhraseAt {
 is_np: bool,
 relative: isize,
 },
 Combined {
 a: Box&#x3C;PatchCriteria>,
 b: Box&#x3C;PatchCriteria>,
 },
}
</code></pre>
Finally, we’ll apply each of the hundreds of thousands of candidates to our treebank to see if the result of their transformations have a lower error rate than the baseline.
Here are a couple of the candidates we found:
<pre><code class="hljs language-json">[
 {
 "from": "PRON",
 "to": "SCONJ",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 0,
 "word": "that"
 }
 },
 "b": {
 "WordIsTaggedWith": {
 "relative": -1,
 "is_tagged": "VERB"
 }
 }
 }
 }
 },
 {
 "from": "PART",
 "to": "ADP",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 1,
 "word": "there"
 }
 },
 "b": {
 "AnyWordIsTaggedWith": {
 "max_relative": -4,
 "is_tagged": "NOUN"
 }
 }
 }
 }
 }
]
</code></pre>
That’s the whole process! With it, I was able to bring our previous accuracy all the way up to 95% (from 40%) without a meaningful change in linting latency or compiled binary size.

Harper is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible.

Harper is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible.

Transformation Based Learning

<h1>Why Rust Might Attract More Developers Than Java and JavaScript</h1>
Rust is undeniably a greatly appreciated language, after all, it has placed as "#1 Most Loved Programming Language" on the StackOverflow Developer Survey for 6th year running.
Reading this, you are probably very aware of why so many people like it, but I will be going over those reasons anyway, at least before I reveal its secret weapon.
It’s fast, very fast. While it may not be a great benchmark, The Computer Language Benchmarks Game, measures that it falls just a hair behind the fastest language, C.
That makes sense, given that Rust was meant to replace C/C++. Even better than the benchmark, it has been shown that in many situations, the Rust compiler is able to optimize code significantly better than C or C++, just because it has more information about the developers intent.
It’s consistent. For more than pretty much any other language, Rust developers can expect that, if their code compiles, it works. When it doesn’t compile, the compiler gives better feedback and advice than any other language. It can even check if your arithmetic is wrong.
It’s complicated, but only where it needs to be. The language itself, as well as its standard library are built very consciously, by its users. Most places where you might find you have to write tedious boilerplate, Rust makes it easy. It also gives you the freedom to write code that does exactly what you want, exactly how you want. We will get back to this later.
<h2>The Problem</h2>
When starting a new project what is the first thing you are going to do? It depends on what stack you are using.
If you want to use NodeJS, it might look something like:
<pre><code class="hljs language-bash">mkdir new_project
touch index.js
npm install
</code></pre>
That is just the bare minimum. If you want to add any dependencies, you have to go through an unreasonable number of hoops, and the standard package repository, npm, is known for being quite insecure, not to mention it being controlled by a private corporation.
If you want to use Java, you have to use some kind of template, otherwise it’s impossible to create a project from scratch. Even then, you have to decide between two or three package managers, and deal with it when libraries don’t support your choice.
If you want to use C or C++ forget it. As a beginner, you are going to be stuck figuring out how to use the local linker and compiler to get any library to work.
<h2>The Rusty Solution</h2>
Cargo is one of the simplest, easiest to use build tools out there. It does everything you might need, and it may be why Rust may soon become a major player. 99% of the time, you can just copy and paste a given library’s string directly from <a href="https://crates.io">crates.io</a>, which is a repository owned and operated by the Rust Foundation, and you are off to the races.
Testing is easy too, just run <code>cargo test</code>, and you're done. It covers both unit tests, and integration tests if you need it.
`cargo doc`` provides a consistent, and easy way to document code. It’s also great for users, allowing developers access to a straightforward representation of what a library looks like.
It’s extensible, some of the greatest binary crates out there exist to extend the functionality. So, in the few areas where <code>cargo</code> on its own doesn’t cut it, there are additional crates to fill the gaps.
<h2>Closing</h2>
At least for the moment, of all the great reasons to use Rust, <code>cargo</code> will be the most significant reason it will increase in popularity.
<img src="/images/most_loved_stackoverflow_2021.png" alt="Most loved programming languages, according to the 2021 StackOverflow Developer Survey">
Rust can be tough to learn, especially for people who are used to dynamic languages like JavaScript. But what Rust has beat for most languages is the build system. <code>cargo</code> alone makes Rust more approachable than C/C++, Java and many others.
More and more new developers will be going to Rust instead of those other languages, because it is easier to start with, easier to learn, and has a great community to request help from.

A key part of Rust is far better than what JavaScript has to offer.

A key part of Rust is far better than what JavaScript has to offer.

Why Rust May Be More Attractive than JavaScript

<h1>The Best Books I Read This Year</h1>
This year I've been reading a lot more.
Is there a particular reason?
Other than simply admiring those that do, no.
I have found some gems of literature that I would like to share.
<h2><a href="https://app.thestorygraph.com/books/d0a652d7-2461-46ce-9bf8-b684c45a67ea">Social Intelligence by Daniel Goleman</a></h2>
Social Intelligence is truly a treasure.
I heard of Daniel Goleman's work first from an episode of <a href="https://www.youtube.com/watch?v=-hoo_dIOP8k">Talks at Google</a> and later from psychologists who raved about his previous book, Emotional Intelligence.
It gives a deep dive into the subconscious and lightning-fast systems the underlay every social interaction.
As I read it, I found myself analyzing my own behavior, as well as the behavior of others.
As someone who has been overthinking my own social life from a young age, Daniel's advice to let my brain go to its default mode was relieving.
If you're the type to feel anxious in social situations, this book is perfect for you.
<h2><a href="https://app.thestorygraph.com/books/d57e098f-82dc-41f1-94f3-8fcb02dfab1b">Tomorrow and Tomorrow and Tomorrow by Gabrielle Zevin</a></h2>
Tomorrow and Tomorrow and Tomorrow is probably one of the best fiction books I've read.
Period.
At first glance, it is just the story of the careers of two game developers and their producer.
But if you dig a little bit deeper, it reveals itself to be the story of how the relationships between people who work together can be incredibly deep and complex.
Specifically, how the passion to create something can both bond and break people.
I don't think it is possible for me to accurately describe the emotional rollercoaster this book took me on.
Which is why you should read it!
<h2><a href="https://app.thestorygraph.com/books/73e8dabf-d6bc-432e-98f0-a48e30a470ad">Why We Sleep by Matthew Walker</a></h2>
I was curious why some people in my life sleep very little and other sleep quite a bit, both with varying levels of general wakefulness.
So, I ran a search: "Why do we sleep?"
To my surprise, I discovered someone wrote a book with almost that exact title.
Since I've read it, I've increased the duration of my sleep by nearly two hours.
This practice has had an outsize impact on my health and happiness.
I could not recommend it more.
I've <a href="./the_optimal_workspace">written about this book extensively</a>, so I won't repeat myself further.
<h2>That's It!</h2>
I could wax poetic about these books and how they've changed me—but I won't.
In reality, I read them because I enjoyed reading them.
In an age where people <a href="https://www.youtube.com/watch?v=A3wJcF0t0bQ">read less than ever</a>, I implore you: try picking up a book, you might just like it.

I might be starting to sound like a broken record.

I might be starting to sound like a broken record.

The Best Books I Read this Year

notifications

<h1>Notifications</h1>
I have found that the first couple hours of the day are my most productive.
I try to move tedious tasks to later so I can focus on solving "hard problems" that need my full attention.
In the past, this has been hard to do for one reason: GitHub notifications.
They pull me in and consume hours of my time.
Responding to and working with OSS contributors is important to me, but that can happen later in the day.
So I wrote a little ViolentMonkey script to hide the notifications icon during these early morning hours.
<pre><code class="hljs language-javascript">// ==UserScript==
// @name Hide Notification Indicator
// @namespace Violentmonkey Scripts
// @match https://github.com/*
// @grant none
// @version 1.0
// @author Elijah Potter
// @description 1/29/2025, 8:16:11 AM
// ==/UserScript==

function hideEm() {
 const matches = document.getElementsByClassName(
 "AppHeader-button--hasIndicator",
 );
 for (const element of matches) {
 element.remove();
 }
}

new MutationObserver(hideEm).observe(document, {
 childList: true,
 subtree: true,
});
</code></pre>

Elijah Potter

Notifications

Other Stuff

Transformation Based Learning

Why Rust May Be More Attractive than JavaScript

The Best Books I Read this Year