<h1>Local-First Software is Easier to Scale</h1>
<img src="/images/flatirons.webp" alt="The Flatirons">
The ti­tle of this post is some­what mis­lead­ing. Local-ﬁrst soft­ware rarely needs to be scaled at all.
<a href="https://writewithharper.com">Harper</a> re­cently re­ceived a mas­sive in­crease in both traf­ﬁc and user count.
How?
By mak­ing it to the front page of Hacker News.
If you could­n’t tell by the ex­tent to which I talk about this, I’m thrilled.
This amounted to a tremen­dous amount of free ad­ver­tis­ing, if noth­ing else.
I’ve been think­ing a lot about the won­drous ben­e­ﬁts of work­ing at the edge.
That’s just one term of many that peo­ple use to de­scribe Harper: “edge-ﬁrst”.
Others in­clude “local-ﬁrst” or sim­ply “on-device”.
They are all just jar­gon for a sim­ple idea: We run all the rel­e­vant gram­mar check­ing code as phys­i­cally close to the users as pos­si­ble.
That way, we can serve up sug­ges­tions faster than any­one else.
To achieve our goal, we make sure that every­thing is well-op­ti­mized and lean.
Harper barely uses any CPU or mem­ory at all.
What does all this mean in the con­text of the re­cent uptick in user ac­qui­si­tion? Imagine, for a mo­ment, we are LanguageTool.
Our soft­ware is writ­ten in Java, with ques­tion­able at­ten­dance to the ef­ﬁ­ciency of our code.
Since our soft­ware re­quires a big server to run, we have a cer­tain num­ber of servers on standby to han­dle a cer­tain amount of ex­pected load.
If sud­denly the num­ber of re­quests we re­ceived were to dou­ble (as it did for the Chrome ex­ten­sion), we would be big trou­ble.
To avoid in­creas­ing the la­tency on re­quests (or drop­ping re­quests all to­gether), we would need to scale up the num­ber of run­ning servers.
This not only takes hir­ing an ex­pert in cloud ar­chi­tec­ture, but also ad­di­tional fund­ing to pay the AWS (or what­ever) bill at the end of the month.
Because Harper runs at the edge (no server re­quired), we don’t have to worry about that.
In fact, I only no­ticed that our user count had spiked af­ter I vis­ited Hacker News the morn­ing af­ter.
No hic­cups at all.
Lots of cloud providers like to brag about be­ing able to scale with their users. I like to brag about not hav­ing to scale at all.

The title of this post is somewhat misleading. Local-first software rarely needs to be scaled at all.

The ti­tle of this post is some­what mis­lead­ing. Local-ﬁrst soft­ware rarely needs to be scaled at all.

Local-First Software Is Easier to Scale

Back in my day, we used math for autocomplete.

Back in my day, we used math for au­to­com­plete.

Markov Chains Are the Original Language Models

<h1>Slop Paralysis</h1>
<blockquote>
slop paral­y­sis - noun
A com­plete or par­tial loss of func­tion while re­view­ing the out­put of a cod­ing agent.
</blockquote>
Let me paint you a pic­ture.
You have an idea for prod­uct.
It could be any­thing: A mo­bile app, a dash­board, or a script to au­to­mate your work.
So, you sit down with your fa­vorite LLM and de­scribe your idea.
Maybe you have a pretty good un­der­stand­ing of how it should be im­ple­mented.
You might even know the over­all struc­ture of the pro­ject in your mind.
You tell your cod­ing agent all of this.
Then, you let it loose.
It cooks and cleans and im­ple­ments your prod­uct for you.
In the end, you have some idea of how the end re­sult works, but it is not com­plete.
Since this is a pro­ject that is im­por­tant, which you in­tend to main­tain, you go to read the code.
That’s when it hits you.
Slop paral­y­sis.
<h2>The Problem</h2>
Slop paral­y­sis is the com­plete lack of de­sire to re­view the out­put of an LLM.
It could show up for any num­ber of rea­sons.
The most com­mon, for me, are:
<ul>
<li>The sheer amount of code de­mand­ing con­sump­tion.</li>
<li>Missing con­text (that the agent was aware of, but not you).</li>
<li>A fear of break­ing some­thing.</li>
</ul>
In the end, slop paral­y­sis slows you down and makes your life more stress­ful.
So, what can we do about it?
<h2>Some Solutions</h2>
I will de­scribe some so­lu­tions that have worked for me.
As al­ways, your mileage may vary.
The ob­vi­ous so­lu­tion is to sim­ply not use an agent.
Some­times cod­ing agents make your job eas­ier.
Some­times they do not.
<a href="./when_to_use_an_agent">Knowing which is which is dif­ﬁ­cult, but ex­tremely valu­able.</a>
When you do de­cide to use an agent, you can have the agent sim­ply pro­duce less code.
I have heard of <a href="https://github.com/DietrichGebert/ponytail">pro­jects</a> that help you do this, but I have not ac­tu­ally tried them.
In­stead, I ask the agent to make a plan ﬁrst, then re­work that plan my­self to min­i­mize the num­ber of changes.
In the end, a plan which was orig­i­nally writ­ten by the agent be­comes one that has been mer­ci­lessly edited by my­self.
In this way, I min­i­mize the amount of code which de­mands con­sump­tion.
The side-ef­fect of this is that I then have an ex­cel­lent un­der­stand­ing of what the agent ac­tu­ally does, which helps with the con­text prob­lem.
The ﬁ­nal strat­egy, which I use when all else fails, is to refac­tor the code by hand.
When the agent has proven the tech­ni­cal fea­si­bil­ity of the pro­ject, but the code is sim­ply too ex­pan­sive to prop­erly grok all at once, I sit down with a ﬁne-tooth comb and refac­tor it, mod­ule by mod­ule.
I do not need to care­fully read each func­tion and eval­u­ate its mer­its, but I do force my­self to, at min­i­mum, lay eyes on them all.
<h2>Stay Steady Out There</h2>
I ﬁnd my­self be­com­ing more over­whelmed by the dis­course every sin­gle day.
It’s ex­haust­ing how much is hap­pen­ing in the world, and even worst, how much you are ex­pected to care about it all.
Take some time for your­self.
Save your­self from slop paral­y­sis.

A complete or partial loss of function while reviewing the output of a coding agent.

A com­plete or par­tial loss of func­tion while re­view­ing the out­put of a cod­ing agent.

Slop Paralysis

<h1>Prompting Large Language Models In Bash Scripts</h1>
I’ve been ex­per­i­ment­ing with us­ing LLMs lo­cally for gen­er­at­ing datasets to test <a href="https://writewithharper.com">Harper</a> against.
I might write a blog post about the tech­nique (which I am grandiosely call­ing “LLM-assisted fuzzing”), but I’m go­ing to make you wait.
I’ve writ­ten a lit­tle tool called <a href="https://github.com/elijah-potter/ofc"><code>ofc</code> that lets you in­sert Ollama into your bash scripts</a>.
I think it’s pretty neat, since it (very eas­ily) lets you do some pretty cool things.
For ex­am­ple, you can swap out the sys­tem prompt, so if you want to com­pare be­hav­ior across prompts, you can just toss it in a loop:
<pre><code class="hljs language-bash">#!/bin/bash

subreddits=("r/vscode" "r/neovim" "r/wallstreetbets")

# Loop over each item in the list
for subreddit in "${subreddits[@]}"; do
 echo "++++++++ BEGIN $subreddit ++++++++"
 ofc --system-prompt "Assume the persona of a commenter of $subreddit" "What is your opinion on pepperjack cheese?"
 cat
done
</code></pre>
Or, you can in­struct a model to prompt it­self:
<pre><code class="hljs language-bash">ofc --system-prompt "$(ofc "Write a prompt for a large language model that makes it think harder. ")" "What is a while loop?"
</code></pre>
<h1>Installation</h1>
<code>ofc</code> is in­stal­lable from ei­ther crates.io or its repos­i­tory.
<pre><code class="hljs language-bash">cargo install ofc --locked

# Or...
cargo install --git https://github.com/elijah-potter/ofc --locked
</code></pre>

Elijah Potter

Prompting Large Language Models In Bash Scripts

Installation

Comments

Other Stuff

Local-First Software Is Easier to Scale

Markov Chains Are the Original Language Models

Slop Paralysis