Simple Starting Prompt

This was the prompt I gave the Windows Codex App³. Even if it is lazy and inaccurate, it’s good enough to get the agent’s loop going.

I wonder if we can iterate and test on what the best combination of powershell llama-server config is and wsl pi settings is so that a long running task compacts less often and I can use a larger context window without losing much quality/speed. e.g. run on powershell something like:

.\llama-server.exe `
  --host 0.0.0.0 `
  --port 8080 `
  --models-dir "C:\source\ai\models" `
  --models-max 1 `
  -c 49152 `
  --jinja `
  -ngl 99 `
  -fa on `
  -ctk q8_0 `
  -ctv q8_0 `
  -n 4096 `
  --metrics

and for the wsl run pi with the model Qwen3.5-35B-A3B-GGUF [llamacpp]. You’d have to control both things and hypothesize stuff and confrim. you can run the llama benchmark firstand store the different runs and performance. You should never run more than one pi agent at a time targeting the server.

Things of note here: I gave the current command for the baseline (llama-server and its parameters), the parameters it could modify (WSL + pi, and llama-server), the way to measure (llama-bench, llama.cpp’s benchmark tool), and a small constraint of ensuring pi is run once at a time.

Conversation Shape

Codex:

Started by establishing a baseline. It inspected the machine, found the model and pi configuration, checked the one-agent constraint, and benchmarked the existing setup, including the KV cache⁴ precision:

llama-server context: 49,152
KV cache:              Q8/Q8
pi reserve:            8,192
effective threshold:   40,960

Codex:

Ran a real pi session and found a failure mode: a long tool-heavy turn could grow past the effective threshold and hit the server limit before compaction protected it.

Tried a 65k context with a larger pi reserve, which fixed the overflow.

Alex:

Also, you should serach in reddit.com/r/locallama for posts about llama-server configs for hardware similar to mine, recent ones, for consideration.

Alex:

I’ve already demonstrated that 49k works on my own works, the config I shared was the one I use. I was hoping to see if I could get more than that in terms of context window because it consumes too fast and then it takes too long to compact and all that jazz

Codex:

Tested 96k and 128k. The 96k configuration looked like the better option for regular use; 128k gave more room and measured worse for regular use.

Alex:

remember you can and probably should experiment with the ~/.pi/agent/ configs (models.json settings.json) or equivalent flags because those are what govern the behaviour of compaction or some optimizations of how it works, if you have to reserach pi.dev or the github repo do so

Codex:

Found reserveTokens and keepRecentTokens, which control when compaction triggers and how much recent context survives it.

Identified pi-continue as another possible lever for reducing compaction disruption.

Alex:

yes, pi-continue would be great to test, keep annotating your hypothesis and tests so that we have a good data of benchmarks and final deciisons.

Codex:

Tested pi-continue. The successful controlled run compacted 48,863 tokens to an estimated 24,407 while preserving the task, constraints, evidence anchors, open questions, and next actions.

Outcome

The recommended configuration for regular use was:

llama-server context: 98,304
KV cache:              Q4/Q4
fit target:            512 MiB
pi reserve:            16,384
pi threshold:          81,920

The KV cache setting was Q4/Q4.

The fit target was 512 MiB.⁵

The selected setup combined a larger server context, lower KV cache precision, a larger pi reserve, adjusted recent-context retention, and pi-continue. The point of the loop was that I did not need to know upfront which one of those would matter.

The practical compaction point is context window - reserve tokens, so the threshold roughly doubled:

old threshold: 49,152 - 8,192  = 40,960
new threshold: 98,304 - 16,384 = 81,920

The short warm API measurement moved from 56.66 tok/s to 53.17 tok/s, or tokens per second.

old: 56.66 tok/s
new: 53.17 tok/s

The 128k candidate gave more room and measured worse for regular use. The preserved notes in the sample artifact packet explain that decision, which is more useful than only keeping the final command.

General Pattern

1. You state the pain point.
2. You identify a specific goal to aim for.
3. You identify the parameters the agent can change to try different combinations.
4. You tell the agent to measure the current starting point as the baseline.
5. The agent tries alternatives systematically.
6. The agent compares each alternative against the baseline and goal.
7. You steer if the agent optimizes for the wrong thing or misses part of the system.
8. You test-drive the chosen result manually before adopting it.
9. You keep the measurements and final config.

This applies beyond local LLMs. If you can define the first three points, you can apply it anywhere. Be mindful of step 3 and whether your agent can really reach all the places it needs to. In my case, if the bot could not open a WSL instance and run pi, this would not work.

A final note is that the config, process, and measurements are reusable. I can apply the same loop to an incoming model, llama-server upgrade, or hypothesis that I have. It’s one of those flows that is good to turn into a personal Agent Skill.

Some context on my economics team

I lead engineering in a cross-functional team called Economic Decision Science at Amazon, the central economics team in Europe. It’s mostly PhD economists, with a small embedded engineering function. The engineering vision predates AI agents entirely: economists should be able to deliver without engineering bottlenecks. We defined the things economists should be able to do (we called them capabilities) and built reusable building blocks ¹ with good UX that they can mix and match to tackle ambiguous problems fast. We called the framework “Factory of Economics” where work goes straight to production, without the sometimes common practice of “producing scrappy code and handing over to devs to scale”.

When we started adopting coding agents in 2025, the fit was immediate. The ways of working we already had (capabilities, building blocks, fast iteration) mapped directly onto how agent skills work. Late 2025, models got noticeably better². It became easier to advocate internally because the broader community was visibly succeeding with these tools, and our own economists were seeing more success in more tasks, which increased trust and ambition in what to delegate.

Everyone uses them, not just for code

Today, the entire cross-functional team uses coding agents. They help across roughly everything: analysis and research design, data pipelines, operations, writing and communication, troubleshooting. The “coding” in “coding agents” undersells it; a lot of the value is in reasoning through problems conversationally, without directly manipulating the implementation yourself. The agent is effective because it works where the economist works: it has access to their tools, data, and machine.

The talk showed three illustrative patterns:

“Explain this model to me.” You have SQL or a model you wrote months ago. You ask the agent to explain it. It doesn’t matter how long it is. During the explanation you use your judgment and spot something wrong, like “counting the wrong window.” You know what correct looks like and conversationally get it fixed. In fifteen minutes it can be in production.

“Help me analyze this policy.” You start from a policy question with no design yet. You’re basically rubber ducking³ with the agent, externalizing your own thoughts: what methodology? what assumptions? what could go wrong? (e.g. Power analysis, clustering, spillovers). The nice thing is that as you go along, you’re actually proving and validating things. You might try twenty variations in an afternoon and land on a research design you trust.

“Help me respond to this challenge.” Your methodology gets challenged in a meeting. It’s very cheap for stakeholders to challenge and it used to be very hard to produce the right data, come back and respond effectively. Now you can open the agent, pull the data, produce the chart, draft an appropriate response immediately after the meeting, knowing it’s backed by real numbers.

What economists never delegate

With all that delegation, here’s what stays with the economist:

1. Methodology selection: “which estimator, which design”
2. Causal identification: “how to isolate the effect”
3. Economic interpretation: “whether results make sense”
4. Stakeholder judgment: “tone, framing, politics”
5. Research taste: “what questions matter”

The economists may discuss these things with the agent, but delegation means handing over the decision, and that stays with the economist.

I’ve seen on occasion well-meaning software engineers build tools that pick methodologies for economists: “define your problem and the AI will choose the right approach.” Economists reject these immediately. If you don’t know why you framed the problem a certain way, you can’t validate whether the output is taking you where you need to go, which takes us back to the issue of solving the right problem.

In a follow-up post I’ll cover Part 2: the pain points of working with AI agents, and how to keep control while being bolder on delegation.

Abstract

How a cross-functional economics team at Amazon uses AI coding agents operationally - across analysis, data pipelines, operations, writing, and troubleshooting. The talk covers what economists delegate (implementation, iteration, exploration) and what they never delegate (methodology selection, causal identification, economic interpretation, stakeholder judgment, research taste). It introduces exam-first iteration as a feedback loop for maintaining control as delegation scales: encode intent, let the agent iterate against it, and use failure as an informative signal.

Slides

• AI Coding Agents in an Economics Team
• Download PDF

Related

• Solving the Right Problem: Economist Edition, Part 1 — blog post expanding on the core idea

The Skill

sketch-to-text takes a handwritten PDF or image and converts it into a Quarto .qmd file with Mermaid diagrams. The output renders, links, and lives with the rest of my writing.

Here’s one of my sketches converted (original PDF):

flowchart TD
    start("I want to share a game to teach")
    describe{"Can I describe the game? Intent"}
    tell_ai("Tell the AI to create game")
    explain("Explain it to the AI and play up scenarios that you had")
    iterate["Iterate asking for changes"]
    client_only{"Only client side?"}
    firebase[/Firebase/]
    publish("Publish to easy app")

    start --> describe
    describe -- yes --> tell_ai
    describe -- no --> explain
    explain --> describe
    tell_ai --> iterate
    iterate --> iterate
    iterate --> client_only
    client_only -- yes --> publish
    client_only -- no --> firebase
    firebase --> publish

That came from telling Claude: “convert diagram-1.pdf to quarto”.

How I built it

flowchart TD
    bus(("Bus data"))
    tram(("Tram data"))
    prices(("Prices"))
    pt["Public Transport"]
    sim["Simulation"]
    qt["Quality Table"]
    dash((("exposure: Dashboard")))
    result((("exposure: result_table")))

    bus --> pt
    tram --> pt
    prices --> sim
    pt --> sim
    sim --> qt
    sim --> dash
    sim --> result

flowchart TD
    banner["Design, De-Risk & Jumpstart"]
    intent["Intent"]
    goals["`Reduce effort
      Increase ROI
      Fail fast`"]
    guardrails["Guardrails"]

    banner -- Design --> intent
    banner -- De-Risk --> goals
    banner -- Jumpstart --> guardrails

Why the evals matter more than the skill

The skill is useful immediatly but the evals are what make it safe and easy to change weeks later when you don’t remember any of the details.

sketch-to-text handles flowcharts well for what I tested and is harmless enough to use whenever. If I notice inaccuracies, I can figure out whether the problem is in the source file, the skill logic, or the model. Fix it, add the new case to the evals, and catch future regressions. If I want to expand it, I’ll know I haven’t broken anything for my existing cases.

The investment is a bit of ground truth up front, in exchange for confidence on every future change.

Links

• Skill: blog-samples/skills/sketch-to-text
• Evals: github-eval-ception/exams/sketch-to-text

Mental Model

You can think of Agents as assistants that can take the load from you and Agent Skills as the high level instructions you might leave to them, in a standardized format.

You need to know what you want from them, and the tradeoffs between micromanagement and agency - Intent. You want to offload mechanical work so they’re not reasoning about things that a tool like a calculator or a spreadsheet could handle - Determinism and you want the whole thing to hold up even if you swap one assistant for another - Stability.

Looking at the shape of an Agent Skill, you can mentally map it as:

• Intent -> Markdown instructions
• Determinism -> tools and scripts
• Stability -> Tests and AI Evals

If your intent is clear enough, the built-in skill-builder in many agent CLIs may be sufficient especially if you still validate outputs manually or stay in the loop for approvals. The more you want to change the skill without manually validating every run, or the more you worry about undesired/rogue behavior, the more determinism helps: scripts reduce error compounding, and unit tests plus AI evals reduce drift risk and make iteration safer and faster, especially when sharing or collaborating.

Decision Tree

flowchart TD
  A([Start: I have or am developing an Agent Skill]) --> Q1{Only for you<br/>and you're happy manually reviewing outputs?}

  Q1 -- Yes --> L0["Level 0: Intent (Markdown only might be enough for you)<br/>- Clear inputs/outputs help<br/>- Examples help<br/>- Define good enough"]
  Q1 -- No --> Q2{Need repeatable structure<br/>or mechanical consistency?}

  Q2 -- No --> L0
  Q2 -- Yes --> L1["Level 1: Determinism (Tools) <br/>- Move mechanical steps into tools/scripts<br/>- Use structured output<br/>- If possible, log tool inputs/outputs"]

  L1 --> Q3{Will others use it<br/>or will you modify it often<br/>without manual re-checking?}

  Q3 -- No --> L1
  Q3 -- Yes --> L2["Level 2: Stability (Tests + Evals)<br/>- Unit tests for tools/scripts<br/>- AI evals for behavior/user stories<br/>- Minimal Golden cases + some edge cases"]

  L2 --> Q4{Can it access sensitive data<br/>or take impactful actions<br/>or run unattended?}

  Q4 -- No --> L2
  Q4 -- Yes --> L3["Level 3: Safety/Scale<br/>- Guardrails + least privilege<br/>- Human approval for high impact<br/>- Security-focused evals"]

Go deeper on each level:

• Level 0 - Intent: Skills Spec
• Level 1 - Determinism: Error compounding + determinism
• Level 2 - Stability: Evals primer
• Level 3 - Security: If you’re here, least privilege and human approval for high-impact actions are usually a good baseline. For each integration point, ask what the worst-case outcome is. It’s also worth understanding Prompt Injection.

Start with pain, not tooling

My eval approach is pain-point driven:

• I can’t compare what I can’t measure.
• I can’t trust an AI system to run on its own if I can’t quantify failure.

If those two are unresolved, I am not “doing evals.” I am guessing.

Treat it like automation engineering

I frame agent work the same way as any automation effort. Unlike traditional ML experimentation, automation engineering demands you define acceptable behaviour upfront. Not after deployment:

• Can I describe exactly what I want? (intentionality)
• What is the worst-case blast radius¹ if this fails?

That framing forces clarity early and keeps risk visible.

Build the smallest useful test loop

I treat early eval work like integration testing plus TDD² habits:

• Skip big infra at the beginning.
• Put the agent where users already work so it behaves like an extra set of hands.
• Recreate real user stories and questions.
• Use deterministic checks wherever possible; don’t default to LLM-as-a-judge for everything.

The goal is not “more tests.” The goal is tests that maximize iteration speed and control.

Optimize late

This space moves fast. Over-optimizing too early is usually waste.

I prefer to keep things:

• Minimal
• Easy to Change (ETC)

In practice that means:

• Parameterized experiments
• Easy comparison across runs, configs, and components

Once benchmarks are stable, then optimize cost and latency:

• Pick the cheapest model/config that still meets the bar.

That’s it: measure first, constrain risk, iterate fast, optimize last.

The idea

Instead of running audits occasionally, automate the audit itself.

An audit is just a set of questions.

If you make those questions explicit, and make them answerable repeatedly, the audit stops being a one-off activity and becomes something you can run continuously.

The method

Start from pain points you already feel, or decisions you struggle to make.

From each pain point, write the questions that would help you address it.

Those questions imply dimensions (what you want coverage over) and, over time, a set of entities that describe your world.

Don’t try to be complete. Just describe enough of your world to support the questions you care about.

For example, “we don’t know if we can reproduce the data of our projects” is a pain point that may prompt the questions:

• Which projects have code packages?
• Which code packages have a README with reproducibility steps?
• Which README file instructions actually work?

The first two should be relatively trivial to check (in this GenAI world) and you can decide how much value the third one gets but the important point is that you’ve introduced what you know and surfaced what you currently can answer in case it’s worth it later.

The dimensions in our example are Code Packages and Projects.

These are actionable and describe different states of knowledge and reproducibility readiness.

The entities in your world

After writing a few questions, you can jump to a whiteboard and try to describe a lot of the entities in your world to brainstorm about what you own, what you interact with and how things cluster together. The entities are the targets of the questions, and dimensions allow you to define the coverage. Example: “We have a total of 12 projects: 5 have READMEs with reproducibility steps, 2 don’t have READMEs, 5 have no packages.”

It’s up to you whether you want to immediately define actions to take or treat it as a helpful data point for others to decide. The point is you stop spending time re-answering ad hoc questions.

The forcing function

Once you have questions, you write evals, checks that verify you can answer them. (See promptfoo’s documentation for one way to implement them.)

An eval is simply:

If I ask this question, I expect to get this kind of answer.

Writing the eval is where the value appears.

The moment you write it, you’re forced to confront whether you can actually answer the question at all.

• If you can, the eval is straightforward.
• If you can’t, you’ve discovered something you thought you knew but didn’t.
• If the answer is subjective, that gap becomes explicit.

You don’t need perfect answers. You need to know which evals pass and which ones reveal that you never had a real answer.

That alone is valuable.

What you get

Once questions have evals, answering them becomes an implementation detail. With MCPs (Model Context Protocol servers), Playwright, etc. Programmatic access is easier than ever, the hard part isn’t answering questions, it’s knowing which ones to ask and systematizing what “good” looks like.

Over time, you start to see where your questions apply, which are too vague, which are easier than expected, and where your description of the world is still thin.

You don’t need to go all in. Start with a few questions and a small report that grows over time.

At that point, you’re no longer redoing one-off audits.

You’re running a continuous audit whose scope is defined by the questions you care about.

Why this is low-risk

• You don’t need to know everything upfront.
• You don’t need to define “good” everywhere.
• You don’t need a complete model of your world.

You just need to acknowledge you may not know the answer for questions you’ve not explicitly tried to answer (quite similar to feeling pain when doing unit testing while being unsure of what you want)

Even unanswered questions are useful. They tell you what’s unclear, subjective, or not worth investing in.

Closing

You don’t need to audit everything.

You just need to stop rediscovering the same answers, and the same unknowns, over and over again.

Start with a few questions. Let the audit grow.

If it doesn’t help, stop.

The cost is small. The insight compounds.

How bad does it get?

Consider a simple case with just 4 steps (note: this is illustrative, consider how your system might have 10, 20, or more LLM calls):

• Step 1: 95% chance of correct
• Step 2: 95% chance of correct
  
• Step 3: 95% chance of correct
• Step 4: 95% chance of correct

End-to-end: ~81% chance everything is correct.

Now compare:

• Step 1 (LLM): user intent → structured call (95%)
• Step 2 (deterministic tool): execute (98%)
• Step 3 (deterministic validation): parse + check (97%)
• Step 4 (LLM): result → response (96%)

End-to-end: ~87%

Same model. Different architecture. 6+ point improvement.

Two high-leverage approaches

1. Remove 1 or many GenAI steps entirely, fewer chances to fail
2. Replace GenAI steps with deterministic ones, lower error rate per step

What makes deterministic steps different

Deterministic steps still fail, but the failure characteristics differ from LLM failures:

• Bounded: failures come from a finite set of causes (parse error, timeout, missing field), not open-ended misinterpretation
• Repeatable: same input, same failure: you can reproduce and fix it
• Non-semantic: a crashed process doesn’t convince the next step that “actually the user meant X”

The design pattern

LLMs do many hard parts (interpreting intent, choosing tools, dealing with syntax, reasoning through results, deciding what comes next).

In a simplified flow, the model might:

1. Receive user intent (natural language)
2. Decide which tool to call and with what parameters
3. Receive structured output from the tool
4. Decide: done, or call another tool?
5. Repeat until ready to respond
6. Translate the final result back to the user

A lot of reasoning and orchestration happens there and the point isn’t to limit that but to give it good building blocks.

A human user is more effective with better building blocks (e.g. well designed libraries or cli tools) and so is an LLM.

What you want are building blocks that are:

• reusable
• composable
• well tested
• easy to change and maintain
• cost effective

And a model that acts as a translation layer, not the tool running all the logic.

Practical recommendations

Closing

This isn’t about distrusting models, it’s about giving them good building blocks to use.

Use GenAI to translate intent. Use the building blocks to execute. Keep errors where you can measure them.

That is how automation becomes something you can trust instead of manually testing once and being hopeful.

Solution

Put HTML onto the clipboard the same way a browser would, so Slack pastes it as rich content instead of plain text.

This requires a recent xclip build that supports advertising text/html correctly.

1. Build xclip from source to get the latest features around html
2. pip install beautifulsoup4 lxml
3. Run pandoc sample.md -t html
4. Optionally modify the HTML to fix things like lists.
5. and pipe it to xclip -selection clipboard -t text/html

What it looks like:

pandoc -f gfm -t html sample.md \
| python -c '
import sys
from bs4 import BeautifulSoup as BS, Tag

s = BS(sys.stdin.read(), "lxml")

def inline_html(tag: Tag) -> str:
    return "".join(str(x) for x in tag.contents).strip()

out_lines = []

def emit_block(tag: Tag):
    name = tag.name.lower()

    if name in ("h1","h2","h3","h4","h5","h6"):
        txt = tag.get_text(strip=True)
        if txt:
            out_lines.append(f"<strong>{txt}</strong>")
            out_lines.append("<br/>")
        return

    if name in ("p",):
        txt = inline_html(tag)
        if txt:
            out_lines.append(txt)
            out_lines.append("<br/>")
        return

    if name in ("ul","ol"):
        walk_list(tag, 0)
        out_lines.append("<br/>")
        return

def walk_list(lst: Tag, level: int):
    for li in lst.find_all("li", recursive=False):
        # separate nested lists
        nested = [c for c in li.find_all(["ul","ol"], recursive=False)]
        for n in nested:
            n.extract()

        text = inline_html(li)
        if text:
            indent = "&nbsp;" * (4 * level)  # 4 = indent width per level
            bullet = "&#8226;"               # •
            out_lines.append(f"{indent}{bullet} {text}<br/>")

        for n in nested:
            walk_list(n, level + 1)

body = s.body if s.body else s
for child in list(body.children):
    if isinstance(child, Tag):
        emit_block(child)

print("".join(out_lines), end="")
' \
| xclip -sel clipboard -t text/html -alt-text "Updates"

Note: The Python step is only needed if you want to fix Slack’s broken handling of nested lists. For simple formatting (bold, links, headings), Pandoc -> xclip alone is enough.

Inspiration

Authoring Markdown externally and pasting the ‘pretty’ output into Slack (on Linux) does the same thing without the extra formatting to fix the nested lists.

Annex

sudo apt install autoconf automake libtool libxmu-dev
git clone https://github.com/astrand/xclip
cd xclip
autoreconf -fi
./configure --prefix=/usr/local
make
sudo make install

Get-Content out.html -Raw | Set-Clipboard -AsHtml

The Rule

If DNS/connection to files.pythonhosted.org hangs but pypi.org works, suspect IPv6 preference + broken IPv6 routing.

Minimal Diagnosis

python -c "import urllib.request; print(urllib.request.urlopen('https://pypi.org/simple/').status)"
# expected: 200

getent hosts pypi.org
# returns quickly

getent hosts files.pythonhosted.org
# may hang

If files.pythonhosted.org hangs, pip will hang. That host is where wheels and sdists are served from.

Fix

Prefer IPv4 for address selection using gai.conf:

sudo tee /etc/gai.conf >/dev/null <<'EOF'
precedence ::ffff:0:0/96  100
EOF

This does not disable IPv6. It changes the precedence so IPv4 is tried first.

Verify

getent hosts files.pythonhosted.org
# should return immediately

Then retry:

pip install ipython

Revert

sudo tee /etc/gai.conf >/dev/null <<'EOF'
# empty override: use glibc defaults
EOF

Notes

If you see multiple stuck installs, clear them before retrying:

pkill -f "python -u -m pip install" || true

Other sources

• https://askubuntu.com/questions/958876/how-to-disable-ipv6-on-windows-subsystem-for-linux
• gai.conf(5) — Linux manual page

Slides and Transcript

• Toward a Shared Vision for LLM Evaluation in the Airflow Ecosystem
• Transcript (WebVTT)

What problem are we solving?

“When can we launch this?” is a recurring question in cross-functional teams, and the answer is often “ask the engineers”. But is that really necessary? Do scientists need to build and then hand off to engineers to rewrite code for scalability or reliability?

I challenged this pattern since I wanted to scale without having to grow engineering headcount, and empower our scientists to deliver more impact independently. The original handover approach introduced delays, estimation misses, integration surprises, and iteration overhead.

How dbt solves these problems

Data Build Tool (dbt) enables engineers and scientists alike to transform data using software engineering best practices. Crucially, there are no tradeoffs between scrappy exploration and production-ready code, the same code serves both purposes:

• Production-ready from day one: The code scientists write IS the production code. No handovers, no rewrites, no “let me translate this for production.” Your development SQL becomes the scheduled pipeline automatically.
• Collaboration and early integration: Since both engineers and scientists can run the same dbt code, collaboration happens naturally from day one, fostering cross-domain learning and surfacing integration or reproducibility issues early, reducing project risk.
• Simple workflows that scale: A simple dbt run -s "model_name+" runs your model and all dependencies. The same code that works for individual data exploration works for production scheduling.
• Modularity without orchestration headaches: dbt forces you to break apart monolithic SQL into focused models, but handles all the dependency management automatically, so you get the benefits of clean, debuggable code without the cognitive overhead of managing execution order.
• Automatic lineage and documentation: dbt generates interactive dependency graphs showing how your models connect. Schema documentation automatically appears in the warehouse tables.
• Built-in quality controls: Define data tests that run automatically.
• Built for integration and extensibility: dbt integrates seamlessly with our existing AWS stack (Athena, Glue, Iceberg), internal services and datalakes and industry standard tools.
• Compliance and governance: Data policies can be built into packages, ensuring compliance and empowering your users to make the right tradeoffs around data handling.

Impact

Our approach enabled delivery of multiple high-impact scientist-led projects that would have otherwise been delayed or blocked due to engineering constraints. Peer teams adopted or expressed desires to adopt it, when they had a chance to work with us and experiment the productivity speed ups, in different dimensions.

Local LLMs

1. Make sure you have ollama installed and running.
2. Note down a wich model(s) you have installed and plan to use. We’ll use deepseek-r1 and qwen2.5-coder as example models. Deepseek is general purpose and a good candidate for reasoning while qwen2.5-coder is specialized for coding tasks.

$ ollama list

NAME                                        ID              SIZE      MODIFIED
deepseek-r1:14b                             ea35dfe18182    9.0 GB    2 hours ago
qwen2.5-coder:14b                           9ec8897f747e    9.0 GB    2 hours ago

I’m using the 14-B distilled models based on my hardware. You can experiment with different ones and find what speed vs quality tradeoff you’re comfortable with. The Ollama models site is very handy to get information about models and their distilled versions.

3. follow the guide which tells you to run:

aider --model ollama_chat/<model>

So in our case, that becomes:

aider --model "ollama_chat/deepseek-r1:14b" --editor-model "ollama_chat/qwen2.5-coder:14b"

We could simply use one model for everything but this “plan vs execution” pattern works really well both locally and remotely.

Use aider --help or visit the options page on aider’s site to understand the differences between --model (main model), --editor-model (editor tasks), and --weak-model (commit messages and history summarization).

Cheaply with OpenRouter

If you’re not satisfied with using your hardware for everything and are ok with sending data to an LLM in the cloud, you can use OpenRouter.

The advantage of using OpenRouter over a specific LLM service like Claude, ChatGPT API or others is that you can have a cloud independent approach and mix and match APIs paying in only one place, while also setting specific budgets that you can’t go over.

user u/Baldur-Norddahl Reddit LocalLLama shared a snippet of what it looks like. You’ll notice it’s very similar to our local example with the addition of the OpenRouter API Key as an environment variable and that we use Claude 3.7 and the full version of Deepseek r1:

export OPENROUTER_API_KEY=sk-or-v1-xxxx
aider --architect --model openrouter/deepseek/deepseek-r1 --editor-model openrouter/anthropic/claude-3.7-sonnet --watch-files

You can easily monitor your activity an estimate what your coding sessions are actually like. This may lead you to switch from Claude 3.7 to something cheaper. Again, it’s all about personal experience and quality tradeoffs.

In Closing

Both patterns are very useful and allow you a great degree of flexibility. There’s a lot of power in customization and avoiding vendor lock-in. You’ll be able to experiement with cline/aider or whatever the next tool is. As hardware becomes more powerful, you could have a very productive experience on a plane, even without internet access.

Shoutout to Georgi Gerganov’s llama.cpp which is the core that allows ollama to work.

The Rule

After your change is approved and enters the delivery pipeline, you should be able to forget it.

No following it through pipelines. No watching for when it lands. No manual checking for failure states.

Pain Point

Tracking deployments “just in case” creates unnecessary cognitive load.

It turns delivery into a background worry: tabs left open, dashboards checked, attention fragmented.

If something requires attention, you should be told.

Do

Treat delivery as a system property:

• Push “surprises” left: run fast, automated cross-system checks (e.g. integration tests) before merge, at code-review time, to minimise post-merge failures.
• You should get a signal if something is blocked or broken.
• You should not need manual reassurance.
• When the system is healthy, silence is expected.

(see No News Is Good News).

Do Not

• Follow a deploy through the pipeline to feel safe.
• Keep checking “did it land yet?”
• Watch logs/dashboards after merge for reassurance.

Scope

This is for routine, continuous delivery of typical changes.

This does not cover:

• major migrations
• one-way / high-blast-radius changes
• communicating expected delivery times (ETAs)

Pain Point

Manually checking systems to confirm they are “okay” creates unnecessary cognitive load.

Dashboards and queues become reassurance rituals: they consume time and attention without changing outcomes.

If something is broken, you should be told.

Do

For anything that might require action, ensure there is an automated signal.

• The signal should reach the people who can meaningfully act on it.
• It does not need to prescribe the action.
• Over time, actions may be formalised (runbooks, automation), but that is secondary.

If something requires attention, it should create noise. If it does not, it should remain silent.

Do Not

• Create mechanisms to confirm system health.
• Regularly inspect dashboards “just to be sure”.
• Rely on manual checks for reassurance.

Silence is expected when coverage is adequate.

Scope

This applies to operational, actionable failures: things that are broken now and require attention.

This does not cover:

• slow degradation
• trend monitoring
• preemptive or exploratory analysis

Analogy

Think of a perfect assistant: they interrupt you only when there is something you can act on. If they do not interrupt you, you can assume everything is fine.

The Rule

• Set a hard budget for fixed meetings.
• Example: 40 h week, 6 h meeting budget.

Total - Budget = Free -> 40 - 6 = 34

• If you go over budget, cut or shrink the least important meetings.
• You can adjust the budget, but do so rarely, otherwise it loses meaning.
• Ad-hoc syncs are fine. It’s the recurring ones that eat up your time.
  • Consider doing a similar thing for ad-hoc meetings, if they become a problem.
• Like code, less is better. Always look for ways to reduce, even if you’re under budget.

Analogy

This is similar to the U.S. Senate’s PAYGO rule:

if Congress wants to add $N to a program, they must “find room” by reducing $N somewhere else or by increasing taxes to cover it.

Intro

Often, when working in the terminal, you’ll find yourself running a command that takes a non-trivial amount of time and you don’t want to just stare at the screen until it finishes.

So you switch tabs/windows and do something else in the meantime. Problem is, when is the other task finished? You don’t want to waste time checking too often nor too late…

So what you want is a notification. One that lets you carry on merrily until the original command is actually finished.

It turns out that many .bashrc files come with an alias called alert and, some SO answers even improve upon it.

Desktop notifications with notify-send

Here’s the one I’m using lately and has served me well:

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)"  
"$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

As the comment says, using it is just a matter of writing the command you want, a semi-colon and the alias alert (Remember that semi-colon ; means execute after the previous command is finished, no matter the return code, unlike && which only executes the next command if return code is 0 (success).

So if you’re compiling and running tests in a project you could just do:

make test; alert

and you’ll get notified whenever make test ends.

But what if you decided running that lengthy task is a good moment to step away from your computer and take a coffee break or talk with a coworker? How will you know when it’s done if you’re not in front of the computer to see the desktop notification?

Email notifications

That’s when email comes in handy. You just gotta take your phone with you and have access to an SMTP server.

$ sudo apt install mailutils

The logic is the same as before, once the command is done, execute the “alert”.

If you want to do it in python, here’s a simple way to go about it.

Just make sure it doesn’t go to SPAM.

Cheers

Was this helpful? Do you do it another way? All comments are welcome!

Intro

Some time ago I came into an issue where people served git repositories in a local network using apache but used a self-signed certificate for the server.

Everyone was already trained to add the exception in their browsers to access HTML content but what happened when it came to source code control?

The Problem

It turns out Subversion (SVN) presented no issue since it prompted the user to accept the new server key just once and then didn’t pester them again but git was another story. Git tried to verify that the cert was signed by a proper authority and couldn’t.

user@user-linux:git$ git clone https://user@dev-server-01/git/repo_name.git 
Cloning into 'repo_name'...
fatal: unable to access 'https://user@dev-server-01/git/repo_name.git/': server certificate verification failed. 
CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

The Solution

After some googling I came across suggestions to disable SSL verification with git config http.sslVerify "false" but that looked like it could induce some bad habits and it actually wouldn’t prevent tampering if, for instance, the user was pointed elsewhere instead of the proper original server.

That’s when Stack Overflow came into play and I found about this neat solution where you can associate a hostname with a given certificate that you store locally.

Steps:

1- Download the self signed certificate from the server and store it somewhere like /etc/ssl/certs

/etc/ssl/certs/ssl-cert-dev-01.pem
/etc/ssl/certs/ssl-cert-dev-02.pem

2- Modify your git config (globally or per-repository) to associate hosts with certs:

(From git config --help)

http.sslCAInfo
    File containing the certificates to verify the peer with when fetching or pushing over HTTPS. 
    Can be overridden by the GIT_SSL_CAINFO environment variable.

In this case we’re going to do it globally by modifying ~/.gitconfig

[http "https://dev-server-01:/"]
    sslCAInfo = /etc/ssl/certs/ssl-cert-dev-01.pem

[http "https://dev-server-02"]
    sslCAInfo = /etc/ssl/certs/ssl-cert-dev-02.pem

Or you can do it with the command line:

$ git config --global http."https://dev-server-01/".sslCAInfo /etc/ssl/certs/ssl-cert-dev-01.pem
$ git config --global http."https://dev-server-02/".sslCAInfo /etc/ssl/certs/ssl-cert-dev-02.pem

Of course, this breaks the flow of those who were using HTTP and the IP address directly since you need the same name that appears in the certificate. That’s the one con I can think of and, if your users where not in the habit of doing so, you’ll better start getting them used to it.

Cheers

Was this helpful? Do you do it another way? All comments are welcome!