ai-tooling-philosophy

AI tooling philosophy

I set up my AI tooling process-first: one general agent follows one written work process for every task, and I decide at fixed points. Skills, connectors, and context are pieces that this process uses, each for its own job. This page explains why I prefer this setup, how the work process runs, and which principles any AI tooling of mine follows, whatever the harness. The motivation comes from my talk Using AI for Reproducible Research.

The problem

Large language models (LLMs) are probabilistic: the same task can return different outputs, and an LLM can report the same steps differently. However, I need work that I can verify and repeat. Thus, AI tooling should do two things.

  1. Clear objectives and processes. The tooling should help me set well-defined goals and structured processes, so that the same task follows the same steps each time.
  2. Clear roles for humans and agents. The tooling should separate the steps in which human judgment is needed from the routine steps that an agent can do (i.e., steps in which the inputs fix the result).

Basic terms

Term Description
Harness The software around an LLM that lets it read files, run code, and search the web (e.g., Claude Code or Codex).
Agent An LLM in a harness that works on a task independently and chooses its own steps.
Skill A folder with an instruction file, and at times templates, context, or scripts, that tells any agent or model how to do a specific task.
Connector A link that gives an agent access to an outside system or data source (e.g., an MCP server).
Context Input for a task that a model reads when the task needs it (e.g., standards, templates, data, or a body of texts).
AGENTS.md A text file with our rules and the work process that the agent reads at the start of every session (CLAUDE.md in Claude Code).

Process-first and agent-first

Setups tend to be either process-first or agent-first.

I prefer a process-first setup because it can keep the researcher in the loop. The process is written down before the work runs, so the same task follows the same steps each time, and the work leaves a trace that I can inspect. In an agent-first setup, I receive a result, but I find it challenging to say how the agents produced it. The two setups are not mutually exclusive. My process uses further agents, but the process calls them: helper models for routine steps, and further sessions of the main model for a review with fresh eyes or for a task of its own.

The work process

The work process fixes the steps of a task, the model that does each step, and the points at which I decide. Its rules are written once, in AGENTS.md, and hold for every task and every harness; this page explains them. The process first sorts each task as routine or non-routine. A routine task (i.e., a task in which the inputs fix the result) needs no plan: the main model states how it reads the request, does the work, and shows the task’s own check (e.g., a command, a test, or a read of the change). Every other task is non-routine and follows the extended version below. The simple version is the most basic form of the process: one model plans, executes, and reviews, and I decide once, at the end. It runs only when I ask for it.

The simple version of the work process

The extended version is the form that every non-routine task follows. It adds a human gate before the plan, a review of the plan by a peer model, and an audit.

The extended version of the work process

Step Responsibility Description
Intent Main model The main model writes down how it reads my request.
Human gate Human I correct or approve that reading.
Plan Main model The main model turns the approved intent into a plan.
Review Peer model A peer model judges with fresh eyes whether the plan is good, and the main model revises the plan. When no peer model can start, the main model reviews the plan in a separate pass instead, and the final gate says so.
Execute Main model, with helper models The main model does the work and hands routine parts to helper models.
Review Main model The main model judges whether its finished work is good and revises it.
Audit Helper model A helper model checks the work against something fixed (e.g., the approved intent and the sources), and the main model revises. This loop runs at most three times.
Human gate Human I check the result and decide whether it is done.

Three kinds of models do the work. The main model runs the session and owns the job from the intent to the result. A helper model does one bounded, routine task (i.e., a task in which the inputs fix the result). A peer model is a fresh session of the main model, with the same strength and no memory of how the plan was made. The main model can also run in several sessions at once: when one session takes several tasks, it gives each non-routine task to a worker session of the main model, which runs that task through the same process.

A review and an audit differ. A review judges whether the plan or the work is good, which takes taste, so a peer model (for the plan) or the main model itself (for the work) does it. An audit compares the work with something fixed that already exists, so a helper model can do it. The peer model gets the plan and the helper model gets the work, each with the approved intent only (i.e., my request and the reading of it that I approved), never the main model’s own view of it, because a reviewer who is told that the work is fine tends to agree. An audit reports only errors it can show, such as a broken function, a wrong number or quote, or a missing part; it judges no quality or reasoning, because the review already does that.

I decide twice: on the reading of my request before the plan, and on the result at the end. Between the two gates, the models work without stopping to ask, unless the work cannot go on.

In my setup, as in any harness, a skill is an instruction file for how a task is done, for any agent or model that takes the task (e.g., the main model, a helper model, or a peer model). It holds what the task needs: instructions, templates, context, scripts, and my rules for that task (e.g., the standards of a referee report and the template it follows). The big change is that skills are much lighter than before. In my earlier setup, each skill also organized the work process of its task, with its own intent, plan, review, audit, loops, and choice of models. That process now lives once, at the harness level, in AGENTS.md. Thus, a skill does not repeat or change these steps, and it may add a gate of its own only when it states the reason for it.

Which form for which need

A harness offers several forms of tooling, and each form does a different job in the work process.

Principles for the setup

Repository

ai-tooling-philosophy/
├── README.md                       this page
├── work-process/
│   ├── work_process_simple.tex     code of the simple version
│   ├── build_simple.sh             builds work_process_simple.pdf and .png
│   ├── work_process_simple.pdf     the simple version, for slides
│   ├── work_process_simple.png     the simple version, for everything else
│   ├── work_process.tex            code of the extended version
│   ├── build.sh                    builds work_process.pdf and .png
│   ├── work_process.pdf            the extended version, for slides
│   └── work_process.png            the extended version, for everything else
└── .gitignore                      keeps build files out of git

Each figure is one TikZ file. To change a figure, edit its .tex file and run its script (./work-process/build.sh or ./work-process/build_simple.sh). The build needs a LaTeX distribution with XeLaTeX and TikZ, pdftocairo (part of Poppler), and the Lato font.


Maintained by Victor van Pelt.