Training a Chunker with Burn

In a pre­vi­ous post, I de­tailed how I im­ple­mented a ba­sic nom­i­nal phrase chun­ker us­ing Transformation-based learn­ing (not to be con­fused with the trans­former class of neural net­work mod­els). Since then, I’ve taken an­other crack at the prob­lem. My main goal: im­prove the ac­cu­racy. The end re­sult is a portable neural net­work model that achieves ~95% ac­cu­racy on gram­mat­i­cally cor­rect text when tasked with iden­ti­fy­ing the nom­i­nal phrases in text.

Our Goal

We want Harper to be able to match against sub­jects and ob­jects in sen­tences. This is a pre­req­ui­site for check­ing a di­verse ar­ray of gram­mat­i­cal rules. For ex­am­ple, to catch the er­ror in this sen­tence, we need to cor­rectly iden­tify which to­kens rep­re­sent our sub­ject.

Neither of the big blue bottle would be broken by the fall.

In this case, our user has ac­ci­den­tally made the sub­ject sin­gu­lar, while the verb “neither” im­plies that the sub­ject should be plural. We call this an agree­ment er­ror. Be­cause our sub­ject, “big blue bot­tle” con­tains mul­ti­ple to­kens, we need a way to iden­tify sub­jects at a higher level than per-to­ken. That is what a chun­ker does.

The Failures of the Brill Chunker

The Brill chun­ker was by many ac­counts a suc­cess. It was­n’t, how­ever, a suc­cess in the main way that mat­tered: It was­n’t re­li­able enough to be used in Harper’s gram­mat­i­cal rule sys­tem. While fast and small, it failed to catch most edge-cases in English text. In some senses, it over­fit its train­ing dataset.

Why Train Our Own?

As our needs con­tinue to ex­pand along­side our user-base, I need the chun­ker to be flex­i­ble. If its needed ca­pa­bil­i­ties ex­pand, I need to be able to re­train the model to meet them. That would not be pos­si­ble with­out hav­ing a deep un­der­stand­ing of how the sys­tem works.

Building a Neural Net

To build a new chun­ker, I just needed to im­ple­ment the Harper Chunker trait. Easy enough.

/// An implementer of this trait is capable of identifying the noun phrases in a provided sentence.
pub trait Chunker {
    /// Iterate over the sentence, identifying the noun phrases contained within.
    /// A token marked `true` is a component of a noun phrase.
    /// A token marked `false` is not.
    fn chunk_sentence(&self, sentence: &[String], tags: &[Option<UPOS>]) -> Vec<bool>;
}

For the nerds in the crowd, I de­cided to use a Word + POS embedding -> BiLSTM -> Linear ar­chi­tec­ture. To keep things portable and con­sis­tent with the rest of the Harper code­base, I used Burn, a Rust-native ma­chine learn­ing toolkit. While I be­lieve the BiLSTM to be good enough for this ap­pli­ca­tion, one ad­van­tage of Burn is the abil­ity to eas­ily swap it out for a trans­former if the need arises. It also makes it un­be­liev­ably easy to quan­tize mod­els.

This ar­chi­tec­ture gives us some hy­per­pa­ra­me­ters to tune against. Af­ter dozens of train­ing runs of ex­per­i­men­ta­tion, these worked best:

Dropout prob­a­bil­ity	Embedding di­men­sions	Learning rate (I used Adam)	Dataset
30%	16 Word Embeddings + 8 UPOS Embeddings	0.003	GUM + EWT + LINES

What’s Next?

Similar to the Brill Chunker, I’ll be try­ing to use this new sys­tem in our gram­mar checker. From there, I’ll know what ad­di­tional in­for­ma­tion we’d like for it to in­fer. Once I’ve got­ten it to re­li­ably work for >= 3 rules, I’ll de­clare it ready to merge.

Update as of December 2025

I to­tally for­got to up­date this post. I sup­pose it is bet­ter late than never. Back in August I ac­tu­ally ended up giv­ing a talk about our chun­ker at WordCamp U.S. It was a great ex­pe­ri­ence, and I highly sug­gest you go watch it if you find this stuff in­ter­est­ing.