<h1>Transformation-based Learning for POS Tagging</h1>
<img src="/images/ctlm.webp" alt="CTLM">
<a href="https://writewithharper.com/">Harper</a> is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible. This post will cover our existing NLP pipeline, the recent changes and improvements to our machine learning approach, and what will come next.
While AI is a common topic of discussion online, I don’t hear much about actual machine learning. In that light, I hope this post piques someone’s interest.
<h2>What is POS Tagging?</h2>
POS (Part-of-speech) tagging is the first step of most NLP (Natural Language Processing) pipelines. For any grammar checker worth its salt, POS tagging is essential. Apart from the basic corrections you’re capable of doing with simple string manipulation, most grammar checking directly or indirectly depends on POS tagging. High-quality tagging results in high-quality suggestions.
What is POS tagging? It is the process of identifying which possible definition of a word is being used, based on the surrounding context. For those unfamiliar with the territory, I’m certain an example is the best way to explain.
<blockquote>
“I am going to go tan in the sun.”
</blockquote>
Here we have a simple, English sentence. In this case, it is clear the word “tan” is being used as verb. The linguists in the audience would point out that it is specifically in the first-person future tense. Consider this similar sentence:
<blockquote>
“I am already very tan, so I will stay inside.”
</blockquote>
In this sentence, the word “tan” is being used as an adjective. How can we tell?
As intelligent humans, some of whom have been speaking English their entire lives, it is easy for us to determine which words are serving which roles. It’s not as easy for a computer to do the same. From an algorithmic standpoint, there are a number of ways to go about it, each with differing levels of “machine learning” required.
Before this week, Harper primarily took a dictionary-based approach. In short: we ship a “dictionary” of English words to the user’s machine and use hash table lookups to determine the possible roles each word could assume. The authors to our rule engine could then use rudimentary deductive reasoning to narrow the possibilities down. This strategy is remarkably effective and it has scaled to tens of thousands of users with surprisingly few hiccups.
That said, there are edge-cases and systems (which I’ll cover next week when I discuss chunking) which require extreme specificity from POS tags. My mission: improve our POS tagging to increase the confidence of Harper’s output and open the door for more advanced algorithms.
<h2>Why Transformation-based Learning?</h2>
The literature highlights four underlying machine learning model strategies that seem to work well for POS tagging.
<ul>
<li>Hidden Markov Models (probabilistic models that predate the modern deep learning era)</li>
<li>Maximum Entropy Models (statistical models closely related to logistic regression).</li>
<li>Transformer-based Models (which use deep neural networks)</li>
<li>Transformation-based Rule Models (which are based on learned rules)</li>
</ul>
While I heavily considered using a neural network (either via an HMM or MEM), I discarded the technology for three reasons.
<ul>
<li>TRMs are typically more accurate (barely; measured in basis points).</li>
<li>TRMs are more amenable to fine-tuning.</li>
<li>TRMs are exceptionally low-latency and can be compressed quite small.</li>
</ul>
Transformation-based learning is remarkably simple. It boils down to just four steps:
<ul>
<li>Use a simple, stochastic model to label your data. This can be as simple as tagging each token (or other discrete component) with that variant’s most common tag. It doesn’t need to super accurate, just enough to establish a baseline.</li>
<li>Identify the errors between the tags in your canonical data and that which produced by your baseline model.</li>
<li>Using a finite list of human-defined templates, generate candidate rules that transform the output of the baseline model into something else. This is where the term “transformation-based” comes from.</li>
<li>Apply each of the candidate rules to the baseline model’s output. Check if the result is more accurate than before. If so, save the rule for future use.</li>
</ul>
These saved candidates become your model.
<h2>POS-Tagging using Transformation-based Learning</h2>
Let’s apply these steps to build a POS-tagging system.
For our baseline model, we will just assign each word in our dataset the most common POS tag associated with that word. If the word is “tan”, we’ll assign it’s most common POS tag (verb). It will often be incorrect, but those cases will be handled by our rules.
To identify the baseline model’s errors, we’ll use an off-the-shelf tree-bank from the Universal Dependencies project.
Our rule templates will take the form of these <code>PatchCriteria</code>. By initializing our candidates with any one of these enum variants and initializing the child variables to random values, we can cover a good number of cases.
<pre><code class="hljs language-rust">#[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)]
pub enum PatchCriteria {
 WordIsTaggedWith {
 /// Which token to inspect.
 relative: isize,
 is_tagged: UPOS,
 },
 AnyWordIsTaggedWith {
 /// The farthest relative index to look
 max_relative: isize,
 is_tagged: UPOS,
 },
 SandwichTaggedWith {
 prev_word_tagged: UPOS,
 post_word_tagged: UPOS,
 },
 WordIs {
 relative: isize,
 word: String,
 },
 /// Not applicable to the Brill Tagger, only the chunker
 NounPhraseAt {
 is_np: bool,
 relative: isize,
 },
 Combined {
 a: Box&#x3C;PatchCriteria>,
 b: Box&#x3C;PatchCriteria>,
 },
}
</code></pre>
Finally, we’ll apply each of the hundreds of thousands of candidates to our treebank to see if the result of their transformations have a lower error rate than the baseline.
Here are a couple of the candidates we found:
<pre><code class="hljs language-json">[
 {
 "from": "PRON",
 "to": "SCONJ",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 0,
 "word": "that"
 }
 },
 "b": {
 "WordIsTaggedWith": {
 "relative": -1,
 "is_tagged": "VERB"
 }
 }
 }
 }
 },
 {
 "from": "PART",
 "to": "ADP",
 "criteria": {
 "Combined": {
 "a": {
 "WordIs": {
 "relative": 1,
 "word": "there"
 }
 },
 "b": {
 "AnyWordIsTaggedWith": {
 "max_relative": -4,
 "is_tagged": "NOUN"
 }
 }
 }
 }
 }
]
</code></pre>
That’s the whole process! With it, I was able to bring our previous accuracy all the way up to 95% (from 40%) without a meaningful change in linting latency or compiled binary size.

Harper is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible.

Harper is currently undergoing some pretty radical changes when it comes to its language analysis. These improvements will improve the output of our existing rule engine, in addition to making entirely new corrections possible.

Transformation Based Learning

<h1>Why Rust Might Attract More Developers Than Java and JavaScript</h1>
Rust is undeniably a greatly appreciated language, after all, it has placed as "#1 Most Loved Programming Language" on the StackOverflow Developer Survey for 6th year running.
Reading this, you are probably very aware of why so many people like it, but I will be going over those reasons anyway, at least before I reveal its secret weapon.
It’s fast, very fast. While it may not be a great benchmark, The Computer Language Benchmarks Game, measures that it falls just a hair behind the fastest language, C.
That makes sense, given that Rust was meant to replace C/C++. Even better than the benchmark, it has been shown that in many situations, the Rust compiler is able to optimize code significantly better than C or C++, just because it has more information about the developers intent.
It’s consistent. For more than pretty much any other language, Rust developers can expect that, if their code compiles, it works. When it doesn’t compile, the compiler gives better feedback and advice than any other language. It can even check if your arithmetic is wrong.
It’s complicated, but only where it needs to be. The language itself, as well as its standard library are built very consciously, by its users. Most places where you might find you have to write tedious boilerplate, Rust makes it easy. It also gives you the freedom to write code that does exactly what you want, exactly how you want. We will get back to this later.
<h2>The Problem</h2>
When starting a new project what is the first thing you are going to do? It depends on what stack you are using.
If you want to use NodeJS, it might look something like:
<pre><code class="hljs language-bash">mkdir new_project
touch index.js
npm install
</code></pre>
That is just the bare minimum. If you want to add any dependencies, you have to go through an unreasonable number of hoops, and the standard package repository, npm, is known for being quite insecure, not to mention it being controlled by a private corporation.
If you want to use Java, you have to use some kind of template, otherwise it’s impossible to create a project from scratch. Even then, you have to decide between two or three package managers, and deal with it when libraries don’t support your choice.
If you want to use C or C++ forget it. As a beginner, you are going to be stuck figuring out how to use the local linker and compiler to get any library to work.
<h2>The Rusty Solution</h2>
Cargo is one of the simplest, easiest to use build tools out there. It does everything you might need, and it may be why Rust may soon become a major player. 99% of the time, you can just copy and paste a given library’s string directly from <a href="https://crates.io">crates.io</a>, which is a repository owned and operated by the Rust Foundation, and you are off to the races.
Testing is easy too, just run <code>cargo test</code>, and you're done. It covers both unit tests, and integration tests if you need it.
`cargo doc`` provides a consistent, and easy way to document code. It’s also great for users, allowing developers access to a straightforward representation of what a library looks like.
It’s extensible, some of the greatest binary crates out there exist to extend the functionality. So, in the few areas where <code>cargo</code> on its own doesn’t cut it, there are additional crates to fill the gaps.
<h2>Closing</h2>
At least for the moment, of all the great reasons to use Rust, <code>cargo</code> will be the most significant reason it will increase in popularity.
<img src="/images/most_loved_stackoverflow_2021.png" alt="Most loved programming languages, according to the 2021 StackOverflow Developer Survey">
Rust can be tough to learn, especially for people who are used to dynamic languages like JavaScript. But what Rust has beat for most languages is the build system. <code>cargo</code> alone makes Rust more approachable than C/C++, Java and many others.
More and more new developers will be going to Rust instead of those other languages, because it is easier to start with, easier to learn, and has a great community to request help from.

A key part of Rust is far better than what JavaScript has to offer.

A key part of Rust is far better than what JavaScript has to offer.

Why Rust May Be More Attractive than JavaScript

<h1>A Case for Procrastination</h1>
<img src="/images/model_t.jpg" alt="A linup of Model T Fords">
The most valuable, unique aspect of software development is the speed at which we can iterate.
Software projects that don't iterate quickly and frequently usually stagnate and fail, while projects that work with user feedback on an almost real-time basis are the most successful.
I've noticed the difference personally, with my work at <a href="https://archytasinc.com/">Archytas Automation</a>.
Most recently, I've read The Lean Startup, Zen and the Art of Motorcycle Maintenance and The Pragmatic Programmer.
While each uses their own terminology, both go into depth on the depth on the topic of fast iteration cycles.
<h2>Lean Manufacturing and Lean Thinking</h2>
Lean manufacturing, as explained in The Lean Startup is a production methodology focused on avoiding waste.
The book's primary example is the Toyota Production System.
In the mid 1930s, the world's largest car manufacturer was Ford.
Ford's factories produced thousands of unique parts, which together could form a car.
They produced these parts in bulk, and each distinct part had an entire machine dedicated to producing it.
There was a machine that made springs, a machine that made nuts, and a machine that made bolts.
No machine could make any part other than the one it was designed for.
This was great: it meant that cars would be cheaper than ever, and more people could experience car ownership.
Around this time, Toyota entered the automotive industry.
However, they did not have the time, nor the money to design and build the same kind of hyper-specific manufacturing machines that Ford used.
Instead, they chose to purchase comparatively few hyper-generalized machines.
Their strategy was to build one car at time, and sell it as soon as possible (ideally the moment it rolled off of the factory floor).
This way, if there is an issue with the vehicle, the design can be modified before too many faulty cars are assembled.
The Toyota Production System emphasized flexibility over efficiency.
By staying flexible, Toyota gave itself the ability to optimize all aspects of the business in the middle of a production run.
On one hand, the Ford process looked like this:
<ul>
<li>Get materials for one hundred cars.</li>
<li>Fabricate over a thousand distinct parts required for each car (likely more than a hundred thousand total parts).</li>
<li>Assemble one hundred cars .</li>
<li>Sell one hundred cars.</li>
<li>Receive feedback on vehicle and production process.</li>
<li>Perform all possible changes to the vehicle's design and production process (which would be very little).</li>
</ul>
With this many steps, each taking place over a comparatively long time, the iteration cycles are few and far between.
On the other hand, the Toyota Production System looked like this:
<ul>
<li>Get materials for one car</li>
<li>Fabricate the relatively few distinct parts required for the car</li>
<li>If there is an issue with the production process (a machine breaks, someone gets hurt, etc.).
Stop the whole process and fix the issue, so it will not affect future cars.</li>
<li>Assemble one car.</li>
<li>Sell one car.</li>
<li>Receive feedback on vehicle and production process.</li>
<li>Perform all possible changes to the vehicle's design and production process (which could be quite a lot, given the flexibility of both).</li>
</ul>
Counterintuitively, performing each step with a batch size of one actually ended up being more efficient.
Even though Ford experienced a higher level of efficient within each batch, their process led to many more wasted parts being manufactured,
as well as a lack of communication between steps, which dramatically increased the amount of wasted effort.
Additionally, because the Toyota Production System allowed comparatively rapid iteration, Toyota vehicles were simply better: more durable, and more useful to their customers.
<h3>Lean Thinking In Software</h3>
In software development, it can be tempting to go the Ford route:
create a detailed plan of all the features you wish to add to your service,
then, develop the features, and deliver them all at once.
Only then do you accept feedback and evaluate the usefulness of each feature.
In The Lean Startup however, Eric Ries suggests an alternative methodology, inspired by the Toyota Production System.
Before deciding on which features to include in your service in an all-at-once, shot-in-the-dark manner, perform experiments.
Create an absolutely minimal prototype of the feature, give to a customer, and observe how they use it.
It is possible they find the feature unhelpful, or they use it in a way you didn't expect.
By running the experiment early on, there are far fewer unhelpful features introduced, and the customer gets a product more closely tied to their needs.
Further, it is possible that your understanding of the problem domain is misinformed.
Planning out the entire product release months in advance is problematic when the requirements of the project change (and they will, often).
<h2>Why Interpreted Languages Can Be So Productive</h2>
Interpreted programming languages are almost laughably pervasive.
Of the top ten most popular programming languages on the 2023 StackOverflow Developer survey, six were interpreted.
That count does not include TypeScript, which can be used in a semi-interpreted manner.
Interpreted language popularity is often chalked up to their ease-of-use, abstraction, and simple, English-like syntax.
However, I believe there is an additional reason: they allow one to iterate much, much faster.
<h2>Micro-iterations</h2>
Iterations can happen on any timescale.
Micro-iterations occur happen over the shortest time scale.
These are small, incremental changes that occur (usually) when debugging or optimizing code.
It is the amount of time (and by extension, mental overhead) that results from a change.
The longer it takes for tests to compile and run, the longer it takes to debug.
I believe you can model it like so:
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>time to debug</mtext><mo>=</mo><mo stretchy="false">(</mo><mtext>time to test</mtext><msup><mo stretchy="false">)</mo><mn>2</mn></msup><mo>∗</mo><mtext>text editing</mtext></mrow><annotation encoding="application/x-tex">\text{time to debug} = (\text{time to test})^2 * \text{text editing}</annotation></semantics></math>time to debug=(time to test)2∗text editing
<blockquote>
This is not an exact, representative equation, merely an abstraction to illustrate my point, based on personal experience.
</blockquote>
In other words, the total debugging time increases super-linearly as the time it takes to test a change to the program increases.
Notice how, as the <code>time to test</code> increases, the actual amount of time <code>text editing</code> falls to a negligible ratio of the total.
I believe this comes from the fact that as compilation time increases, the amount of information gleaned per micro-iteration tends to decrease.
<h3>JIT Compilation and Hot Reload</h3>
This is where we get to the title of the article.
The concepts of lean thinking extend into compiler technology.
The Toyota Production System includes the concept of "Just-in-time", meaning,
"to only produce exactly what is needed for the product, exactly when in the process it is required."
Avoid planning resource requirements ahead of time and build systems that only produce what is actually needed.
While the meaning of "Just-in-time" is different when we refer to compilers, the line of thinking is ultimately the same.
Instead of wasting valuable, human-scale time on compilation and optimization of irrelevant code, just compile the code you need, when you need it.
This drastically reduces the amount of time to test, and ultimately, the time to debug.
Further, technologies like React and Flutter's Hot Reload make it possible to complete these micro-iterations without having to restart your program.
I believe this is partly why interpreted languages are so beginner-friendly (other than the high level of abstraction they typically provide).
Newcomers to programming spend most of their time on syntactic issues and simple runtime errors.
By making micro-iterations happen as fast as possible, you make these issues less annoying to deal with.
<h3>Procrastination</h3>
If you are in university (like I currently am), or any other level of education, it can actually be beneficial to hold off on doing your homework.
It is not uncommon (at least in my experience) for the project or paper requirements to change at the last minute.
Some of your peers may choose to complete the project or paper as soon as possible.
If this is the case, you may benefit from their experience.
If a certain chain of logic or analysis is confusing or misleading, they can warn you to avoid losing time.
Assuming they are willing, they can also provide insightful feedback on your work mid-way through your writing or study process.
<h2>Conclusion</h2>
Iteration is incredibly important to any design and development cycle, especially in software.
Since I have embraced iteration, I've seen a dramatic improvement in my productivity and effectiveness.
So I encourage you: if your build process takes 30 minutes, spend some time on improving it.
If you are working on a project that hasn't been put in front of a customer, try showing it to a family member or non-technical friend.
Ask them for brutal honesty, and your project will be much better off.

Or: why doing things at the last minute actually saves time. I talk about the importance of shockingly fast iteration cycles and lean manufacturing.

Or: why doing things at the last minute actually saves time. I talk about the importance of shockingly fast iteration cycles and lean manufacturing.

A Case for Procrastination

photography_as_meditation

<h1>Photography as Meditation</h1>
The goal of meditation is to become more mindful and aware.
For me, it is often an attempt to get an appreciation for my surroundings—to see how utterly complex the world is.
<img src="/images/pink_flower_tree.webp" alt="Spring has sprung in Colorado. That means pink flowers everwhere.">
Photography does something similar for me.
The heft of the camera in my hands reminds me to slow down and take in the view.
My mother is a photographer—a great one at that.
She's been in the business for at least as long as I've been alive.
She tells me something similar: the act of framing a picture is itself a way to understand your subject better.
<img src="/images/puddle.webp" alt="Spring brings rain and rain brings puddles.">

Reflecting on the clarity photography brings.

Reflecting on the clarity photography brings.

Elijah Potter

Photography as Meditation

Other Stuff

Transformation Based Learning

Why Rust May Be More Attractive than JavaScript

A Case for Procrastination