Bytes Are Not Big Enough

I apol­o­gize for us­ing some rel­a­tively tech­ni­cal ter­mi­nol­ogy in to­day’s post. In the in­ter­est of sav­ing space on the page, I will not re­it­er­ate the de­f­i­n­i­tions of “tokenization” nor the var­i­ous meth­ods of do­ing so. If you are al­ready well-versed in the field, great! If not, I be­lieve this page gives a de­cent ex­pla­na­tion.

I first set out to build a small lan­guage model sev­eral months ago. While the orig­i­nal goal — to emit leg­i­ble English by pass­ing bare bytes into a trans­former — was a suc­cess, I was left un­sat­is­fied with the ap­plic­a­bil­ity of the model it­self. To put it plainly: The model worked, but it was not use­ful for any­thing. My long-term vi­sion is to of­fer a “Pro” ver­sion of Harper that al­lows pro­fes­sion­als in ser­vice in­dus­tries to rewrite their text quickly us­ing an on-de­vice small lan­guage model.

Since then, I have been dis­tracted with sev­eral other im­por­tant mis­sions, in­clud­ing a Harper Desktop app for ma­cOS and Windows.

This week, how­ever, I al­lowed my­self some more time to work on it.

The Problem

I am try­ing to cre­ate a strong and small base model. A model that can se­man­ti­cally un­der­stand small pas­sages of text, which we can then re­train to do a va­ri­ety of dif­fer­ent things. If a small base model like this can be made, we could do a lot of other things us­ing its in­ter­me­di­ate rep­re­sen­ta­tions:

• Rewrite sen­tences (from pas­sive voice to ac­tive voice or from ca­sual tone to for­mal tone).
• Infer where com­mas should be placed.
• Prioritize Harper’s sug­ges­tions to bet­ter sur­face them for users.

The key here is that the model must be aware of both the in­tri­ca­cies of sub-word com­po­nents (including in­di­vid­ual af­fixes), but also be aware of higher-level de­tails, like word or­der and tone. My pre­vi­ous at­tempt was well-aware of the for­mer (better than many larger mod­els), but failed in tasks that re­quired higher level gram­mat­i­cal rea­son­ing.

However, we could not ex­pand the to­kens to be much larger with­out dra­mat­i­cally in­creas­ing the in­fer­ence cost for the end user. To ac­com­plish what we aim to, this fi­nal model must be able to achieve triple-digit to­kens per sec­ond on con­sumer-grade CPUs. It must be very small and very fast.

The Solution

In my ex­per­i­men­ta­tion this week, I found that I was right about one thing and wrong about an­other.

I was right that there is an ideal av­er­age to­ken size for our work­load. I was wrong that it was a sin­gle byte.

As it turns out, the ideal to­ken size for our tasks seem to be about 3.47 char­ac­ters per to­ken. Big enough to al­low the model to fo­cus on the big pic­ture tasks, but small enough for the model to have enough aware­ness of the struc­ture of in­di­vid­ual words.

“Elijah, how did you cal­cu­late that av­er­age?” I ran a task us­ing the model and av­er­aged the size of the to­kens emit­ted. Ob­vi­ously, it prob­a­bly is not a su­per ac­cu­rate fig­ure, but I think it of­fers enough salience and pre­ci­sion for this post.

To scale up the to­ken size, I used riptoken and I tested out a few off-the-shelf pre­trained to­k­eniz­ers. I ended up us­ing one with a vo­cab­u­lary of about 8192 to­kens.

Moving for­ward, I be­lieve I need to up the model size just a tad, then work on bet­ter post-train­ing.