The latest FOSS Weekly from It's FOSS ends its AI rundown with a push: it is time to get moving towards local AI, where models live on your own machine instead of a cloud service. The newsletter admits that small local models won't match Claude or other frontier models, but says smaller tasks are still within reach. Its proof is Myna, Ubuntu's AI-based voice input tool (source).
This guide takes that advice literally. You will check your hardware, install Ollama, pull one small model, confirm it runs without a network connection, and learn how to remove everything if it isn't for you.
Commands status: all commands below are untested. Compare them with the official Ollama documentation before you run them. Install steps change, and the docs are the authority.
What you need before you start
The newsletter says Ubuntu 26.10 was due for release on the day it was published, so ISOs and upgrade paths may still be settling. The same steps should apply to other recent Ubuntu releases, but confirm current Linux support in the Ollama documentation. That is an inference, not a tested claim.
Here is the checklist:
- A 64-bit Ubuntu install with sudo access and an internet connection for the one-time download.
- At least 8 GB of RAM, with a few gigabytes free once your browser and desktop are running.
- Free disk space. The model page lists the exact download size before you pull, and small quantized models are usually a few gigabytes at most.
- Optional: a supported GPU. A GPU speeds things up, but small models run on a CPU, just more slowly.
Step 1: Check your hardware
Open a terminal and look at memory, disk, and graphics hardware.
free -h
df -h ~
lscpu | grep 'Model name'
lspci | grep -Ei 'vga|3d'
Read the "available" column in the free -h output, not "total". If it shows under 4 GB, close heavy apps first or stick to the smallest models.
Which model size fits your RAM
This table is a rule of thumb, not a benchmark. A quantized model needs roughly its file size in memory, plus overhead for context. Results vary by model and by how much your desktop uses.
| Total RAM | Model size to try | What to expect | |---|---|---| | 8 GB | 1B to 3B parameters | Usable on CPU for short tasks; close browser tabs | | 16 GB | 7B to 8B parameters | Noticeably better writing and shell help, still modest | | 32 GB | 13B to 14B parameters, possibly larger | More headroom and longer context; speed depends on CPU or GPU |
Start small even if you have 32 GB. A model that answers in seconds teaches you more than one that crawls.
Step 2: Install Ollama
Ollama is a local runner that downloads models and serves them from your machine. The common install route pipes a one-line script straight into a shell. Don't do that blindly. Check the official Ollama documentation for the current Linux instructions, then download the script and read it before you run it. This pairs well with see also: fix ea app games on linux: switch proton versions in steam.
## UNTESTED: verify against the official Ollama docs first
curl -fsSL https://ollama.com/install.sh -o ollama-install.sh
less ollama-install.sh
sh ollama-install.sh
If you want to control exactly where files land, the docs also describe a manual install from a downloaded archive. Either way, confirm the install worked:
ollama --version
systemctl status ollama
You should see a version string and, if the installer set up a systemd service, an "active (running)" status. If there is no service, run ollama serve in a second terminal.
Step 3: Pull one small model
Browse the model library on the Ollama website. Pick something in the 1B to 3B range for an 8 GB machine, or 7B to 8B for 16 GB. Model names and popularity shift quickly, so treat the example below as a placeholder and check it against the library.
## UNTESTED: example model name; confirm it exists in the library
ollama pull llama3.2:3b
ollama list
Expect progress bars during the download, a line such as "pulling manifest" at the start, and "success" at the end. ollama list then shows the model name, its size on disk, and when it was modified. The exact wording may differ by version.
Step 4: Run it and give it a small job
ollama run llama3.2:3b
At the prompt, try tasks with answers you can check:
Summarize this in two sentences: [paste a paragraph]
Write a polite three-line reply declining a meeting invite.
What does 'tar -xzf archive.tar.gz -C /tmp' do?
Type /bye to exit. The first reply is slower because the model loads into memory. Later replies in the same session are faster.
Step 5: Verify it runs locally
Don't take it on faith. Run three checks.
Check where the server listens
ss -tlnp | grep 11434
The address should be 127.0.0.1 (localhost only), not 0.0.0.0. Port 11434 is Ollama's default, but confirm that in the docs for your version. This pairs well with fex vs box64 on arm64 linux: which to use for games? in depth.
Cut the network and try again
Turn off Wi-Fi or unplug Ethernet, then run the model again. If you are working over SSH, skip the nmcli lines, because they will cut your session.
## UNTESTED
nmcli networking off
ollama run llama3.2:3b "Say hello in five words."
nmcli networking on
If the model answers with the network off, inference is happening on your machine. A cloud service can't answer without a connection.
Watch the load
Run ollama ps while a prompt is working to see which model is loaded and whether it uses CPU or GPU. Run htop alongside and watch CPU or memory climb, which a remote call wouldn't cause.
This proves the model runs offline. It doesn't audit every other feature of the software. Before you assume an add-on interface never touches the network, read its documentation and settings.
What local models do well, and where cloud AI still wins
The newsletter's caveat holds: small local models won't match frontier models. Use them for small, low-stakes jobs and keep your expectations modest.
Also read: enable nvidia reflex in proton on linux: setup & checks
| Task | Local small model | Cloud frontier model | |---|---|---| | Summarizing a document you don't want to upload | Good fit. Private, decent quality | Better quality, but your text leaves your machine | | Drafting short emails or notes | Good fit for first drafts | Stronger tone and nuance | | Explaining a shell command or writing a one-liner | Useful, but check the output before running it | More reliable on unusual tools | | Long reasoning, large codebases, research | Weak. Small models lose the thread | Clearly better | | Speed and cost | Depends on your CPU or GPU; no per-query fee | Fast, but needs a connection and may cost money |
A sensible habit: ask the local model first for anything private or routine. Move to cloud AI only when the local answer isn't good enough and the content is safe to share. Never paste a command from any model into a root shell without reading it.
Roll back: uninstall Ollama cleanly
If local AI isn't for you, remove it. The steps below follow the pattern the Ollama docs describe for a service install. Confirm them there, because paths can differ, especially for a manual install.
## UNTESTED: compare with the official uninstall steps
ollama rm llama3.2:3b
sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -r /usr/share/ollama
sudo userdel ollama
sudo groupdel ollama
Run ollama rm for each model you pulled, or delete the model directory, to reclaim the disk space. Afterward, df -h ~ should show the space back and which ollama should print nothing. If you ran the server manually and never created a service, skip the systemctl lines.
What to try next
Once the first model works, pull a second one at a similar size and compare answers on the same three prompts. That shows you quickly whether model choice matters more than hardware. After that, look at a graphical front end or editor integration, and check its network behavior the same way you checked Ollama's. The newsletter also recommends its Local AI Weekly newsletter if you want to keep following this space.
