<h1>The Easiest Way to Run LLMs Locally</h1>
<img src="/images/llama.webp" alt="A Goofy Lookin&#x27; Llama">
<h2>LLMs</h2>
Unless you’ve been liv­ing un­der a rock for the past year, you al­ready know what LLMs are.
If you do hap­pen to be one of the lucky few un­aware of the cur­rent hype around these things, I’ll go through it real quick.
A large lan­guage model (or LLM) is a sta­tis­ti­cal model ca­pa­ble of “predicting” a sub­se­quent word or let­ter, given a body of text.
Es­sen­tially, it is a com­puter pro­gram ca­pa­ble of ﬁll­ing in the blank.
If you let it pre­dict the next word, then feed the re­sult back in, you can get some pretty hu­man-look­ing text.
<h2>Let’s Be Clear</h2>
I hold a lot of skep­ti­cism on the prac­ti­cal ap­pli­ca­tions of LLMs as a tool.
As a blan­ket rule, I never use LLMs or any sim­i­lar tech­nol­ogy in my ed­u­ca­tion.
I know some peo­ple ask LLMs ques­tions like “explain the fun­da­men­tal the­o­rem of cal­cu­lus to me like I’m ﬁve.”
While they may get good re­sults for ques­tions, I do not want to lean on them as a crutch.
Col­lege is not only an op­por­tu­nity to learn the raw ma­te­r­ial, but also an op­por­tu­nity to learn how to learn.
If we know any­thing about LLMs, it’s that its abil­ity to an­swer com­plex ques­tions break down as you move to more spe­cial­ized classes.
Which is all to say: I did not in­ves­ti­gate this with the in­ten­tion of us­ing it as a tool, I just wanted to play around.
<h2>My Circumstance</h2>
I use <a href="https://archlinux.org/">arch, btw.</a>
While I en­joy the level of con­trol it pro­vides, I don’t think it’s for every­body.
This is partly be­cause some things are quite dif­ﬁ­cult to set up.
For ex­am­ple, GPU sup­port is lim­ited and ﬁnicky, es­pe­cially if you run an Intel Arc card, like I do.
While it works per­fectly for some apps, like <a href="https://www.blender.org/">blender</a>, it does­n’t work so well for other things.
My card only has 3 GB of VRAM, so it would­n’t be able to ﬁt most mod­els any­way.
So when I took on the task of run­ning an LLM on my lo­cal ma­chine, I started at look­ing at CPU-only so­lu­tions.
Initially, I tried to raw-dog <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a>.
That worked but only so.
The com­mand-line in­ter­face left a lot to be de­sired, and the process of down­load­ing and load­ing var­i­ous mod­els was te­dious and con­fus­ing.
<h2>Ollama</h2>
That’s when I dis­cov­ered <a href="https://ollama.ai/">Ollama</a>.
Installing it was as easy as run­ning:
<pre><code class="hljs language-bash">sudo pacman -S ollama
</code></pre>
To avoid wast­ing re­sources on mul­ti­ple in­stances of each model, Ollama uses a server ar­chi­tec­ture.
You can start the server by run­ning
<pre><code class="hljs language-bash">ollama serve
</code></pre>
Then, you can down­load an start chat­ting with a model with:
<pre><code class="hljs language-bash">ollama run llama2
# Or:
ollama run mistral
</code></pre>
<h2>That’s It</h2>
That’s it!
It re­ally is that sim­ple.
Again, you might have no rea­son to do any of this.
Es­pe­cially if you are happy with the pri­vacy night­mare that is OpenAI, Google or Anthropic, or if you al­ready have a sys­tem that works for you.

It saved me enough time, I had some to share about it.

It saved me enough time, I had some to share about it.

The Easiest Way to Run Llms Locally

Back in my day, we used math for autocomplete.

Back in my day, we used math for au­to­com­plete.

Markov Chains Are the Original Language Models

The Optimal Workspace

a_harper_record

<h1>A Harper Record</h1>
When we started work on <a href="https://writewithharper.com/docs/harperjs/introduction"><code>harper.js</code></a>, our goal was sim­ple.
We wanted it to take less than 24 hours for a de­vel­oper to em­bed high-qual­ity gram­mar check­ing in their app.
A friend of mine, <a href="https://lukaswerner.com/">Lukas Werner</a> re­cently set a record for this.
In just over 15 min­utes, he was able to em­bed Harper into Raycast, a com­mand palate for MacOS.
This is fan­tas­tic news.
It’s al­ways great to see when your goals are hav­ing a real im­pact on some­one’s abil­ity.
I hope you take the time to look at <a href="https://github.com/lukasmwerner/harper-raycast">the pro­ject on GitHub</a>.
In other news, we have some other small pro­jects us­ing Harper:
<ul>
<li><a href="https://github.com/lockbook/lockbook">Lockbook</a></li>
<li><a href="https://stubby.io/">Stubby CMS</a></li>
</ul>

🎉

Elijah Potter

A Harper Record

Other Stuff

The Easiest Way to Run Llms Locally

Markov Chains Are the Original Language Models

The Optimal Workspace