Find The Loop

Pain Point: Finding The Right Combination Of Parameters

This is a story about using a cloud LLM agent to tune a local LLM setup. If you do not care about the story, jump to the general pattern.

I had a local large language model (LLM) setup that worked well enough to use: llama.cpp running on Windows, pi running from WSL¹, and a Qwen3.5 35B model served through llama-server, llama.cpp’s local API server. I had already found a 49k context configuration² through trial and error, but long-running loops or sessions would eventually fill the context window. Compaction would then take time and break the flow.

Researching potential flags and trying different combinations or hypotheses is time consuming, and I probably will not do it unless I can delegate it. In some cases, I am not even guaranteed to find better performance than I currently have. This made it a good candidate for letting an agent try alternatives in a systematic manner, benchmark which combinations met the desired goals, and keep the measurements.

The important requirement was that the agent had to control every part that could affect the result. One part was the Windows llama-server process. Another was pi’s compaction behaviour in WSL.

Simple Starting Prompt

This was the prompt I gave the Windows Codex App³. Even if it is lazy and inaccurate, it’s good enough to get the agent’s loop going.

I wonder if we can iterate and test on what the best combination of powershell llama-server config is and wsl pi settings is so that a long running task compacts less often and I can use a larger context window without losing much quality/speed. e.g. run on powershell something like:

.\llama-server.exe `
  --host 0.0.0.0 `
  --port 8080 `
  --models-dir "C:\source\ai\models" `
  --models-max 1 `
  -c 49152 `
  --jinja `
  -ngl 99 `
  -fa on `
  -ctk q8_0 `
  -ctv q8_0 `
  -n 4096 `
  --metrics

and for the wsl run pi with the model Qwen3.5-35B-A3B-GGUF [llamacpp]. You’d have to control both things and hypothesize stuff and confrim. you can run the llama benchmark firstand store the different runs and performance. You should never run more than one pi agent at a time targeting the server.

Things of note here: I gave the current command for the baseline (llama-server and its parameters), the parameters it could modify (WSL + pi, and llama-server), the way to measure (llama-bench, llama.cpp’s benchmark tool), and a small constraint of ensuring pi is run once at a time.

Conversation Shape

Codex:

Started by establishing a baseline. It inspected the machine, found the model and pi configuration, checked the one-agent constraint, and benchmarked the existing setup, including the KV cache⁴ precision:

llama-server context: 49,152
KV cache:              Q8/Q8
pi reserve:            8,192
effective threshold:   40,960

Codex:

Ran a real pi session and found a failure mode: a long tool-heavy turn could grow past the effective threshold and hit the server limit before compaction protected it.

Tried a 65k context with a larger pi reserve, which fixed the overflow.

Alex:

Also, you should serach in reddit.com/r/locallama for posts about llama-server configs for hardware similar to mine, recent ones, for consideration.

Alex:

I’ve already demonstrated that 49k works on my own works, the config I shared was the one I use. I was hoping to see if I could get more than that in terms of context window because it consumes too fast and then it takes too long to compact and all that jazz

Codex:

Tested 96k and 128k. The 96k configuration looked like the better option for regular use; 128k gave more room and measured worse for regular use.

Alex:

remember you can and probably should experiment with the ~/.pi/agent/ configs (models.json settings.json) or equivalent flags because those are what govern the behaviour of compaction or some optimizations of how it works, if you have to reserach pi.dev or the github repo do so

Codex:

Found reserveTokens and keepRecentTokens, which control when compaction triggers and how much recent context survives it.

Identified pi-continue as another possible lever for reducing compaction disruption.

Alex:

yes, pi-continue would be great to test, keep annotating your hypothesis and tests so that we have a good data of benchmarks and final deciisons.

Codex:

Tested pi-continue. The successful controlled run compacted 48,863 tokens to an estimated 24,407 while preserving the task, constraints, evidence anchors, open questions, and next actions.

Outcome

The recommended configuration for regular use was:

llama-server context: 98,304
KV cache:              Q4/Q4
fit target:            512 MiB
pi reserve:            16,384
pi threshold:          81,920

The KV cache setting was Q4/Q4.

The fit target was 512 MiB.⁵

The selected setup combined a larger server context, lower KV cache precision, a larger pi reserve, adjusted recent-context retention, and pi-continue. The point of the loop was that I did not need to know upfront which one of those would matter.

The practical compaction point is context window - reserve tokens, so the threshold roughly doubled:

old threshold: 49,152 - 8,192  = 40,960
new threshold: 98,304 - 16,384 = 81,920

The short warm API measurement moved from 56.66 tok/s to 53.17 tok/s, or tokens per second.

old: 56.66 tok/s
new: 53.17 tok/s

The 128k candidate gave more room and measured worse for regular use. The preserved notes in the sample artifact packet explain that decision, which is more useful than only keeping the final command.

General Pattern

1. You state the pain point.
2. You identify a specific goal to aim for.
3. You identify the parameters the agent can change to try different combinations.
4. You tell the agent to measure the current starting point as the baseline.
5. The agent tries alternatives systematically.
6. The agent compares each alternative against the baseline and goal.
7. You steer if the agent optimizes for the wrong thing or misses part of the system.
8. You test-drive the chosen result manually before adopting it.
9. You keep the measurements and final config.

This applies beyond local LLMs. If you can define the first three points, you can apply it anywhere. Be mindful of step 3 and whether your agent can really reach all the places it needs to. In my case, if the bot could not open a WSL instance and run pi, this would not work.

A final note is that the config, process, and measurements are reusable. I can apply the same loop to an incoming model, llama-server upgrade, or hypothesis that I have. It’s one of those flows that is good to turn into a personal Agent Skill.

Footnotes

1. Windows Subsystem for Linux lets you run a Linux environment directly on Windows.

2. The context window is the amount of conversation/history the model can work with. In this workflow, compaction means summarizing earlier context so the session can keep going without exceeding that window. The practical threshold is lower than the full context window because pi reserves some tokens before compacting.

3. In the prompt, -c 49152 was the context size, or how much conversation/history the server can hold. -ctk q8_0 and -ctv q8_0 controlled KV cache precision, a memory/speed/quality tradeoff for long context. -ngl 99 asked llama.cpp to put as much of the model as possible on the GPU. -fa on enabled flash attention, which helps long-context performance. The pi settings mattered because compaction was controlled outside llama-server. The one-agent rule kept tests from overlapping.

4. The KV cache stores intermediate attention data so the model can continue from previous tokens without recalculating everything. Quantizing it uses fewer bits per value to save memory. Q4/Q4 uses less memory than Q8/Q8, which helped fit a larger context window, but I did not compare Q4/Q4 versus Q8/Q8 with my own task-specific evals. The comparison here used llama-server, generic speed measurements, VRAM behaviour, and the observed pi workflow.

5. fit target refers to llama.cpp’s -fitt / --fit-target, the target memory margin per device for --fit. In this run, 512 MiB meant asking llama.cpp to leave about that much GPU memory margin while fitting the model and context.