ai-tooling-philosophy

AI tooling philosophy

I set up my AI tooling skill-first: one general agent gets its way of working from written workflows that I design, and I decide at fixed points. This page explains why I prefer this setup, how a skill workflow runs, and which principles any AI tooling of mine follows, whatever the harness. The motivation comes from my talk Using AI for Reproducible Research.

The problem

Large language models (LLMs) are probabilistic: the same task can return different outputs, and an LLM can report the same steps differently. However, I need work that I can verify and repeat. Thus, AI tooling should do two things.

  1. Clear objectives and processes. The tooling should help me set well-defined goals and structured processes, so that the same task follows the same steps each time.
  2. Clear roles for humans and agents. The tooling should separate the steps in which human judgment is needed from the routine steps that an agent can do (i.e., steps in which the inputs fix the result).

Basic terms

Term Description
Harness The software around an LLM that lets it read files, run code, and search the web (e.g., Claude Code or Codex).
Agent An LLM in a harness that works on a task independently and chooses its own steps.
Skill A folder with written instructions and scripts that tell an agent how to do a specific task.
AGENTS.md A text file with our rules that the agent reads at the start of every session (CLAUDE.md in Claude Code).

Skill-first and agent-first

Setups tend to be either skill-first or agent-first.

I prefer a skill-first setup because it can keep the researcher in the loop. The workflow is written down before the work runs, so the same task follows the same steps each time, and the work leaves a trace that I can inspect. In an agent-first setup, I receive a result, but I find it challenging to say how the agents produced it. The two setups are not mutually exclusive. My skills use further agents, but a skill calls them, and only for steps whose result I do not need to follow.

The skill workflow

A skill fixes the steps of a task, the model that does each step, and the points at which I decide. The simple version is the most basic process that a skill can follow: one model plans, executes, and reviews, and I decide once, at the end.

The simple version of the skill workflow

The extended version is my current skill template. It adds a human gate before the plan, a review of the plan by a peer model, and an audit.

The extended version of the skill workflow

Step Responsibility Description
Intent Main model The main model writes down how it reads my request.
Human gate Human I correct or approve that reading.
Plan Main model The main model turns the approved intent into a plan.
Review Peer model A peer model judges with fresh eyes whether the plan is good, and the main model revises the plan. When no peer model can start, the main model reviews the plan in a separate pass instead, and the final gate says so.
Execute Main model, with helper models The main model does the work and hands routine parts to helper models.
Review Main model The main model judges whether its finished work is good and revises it.
Audit Helper model A helper model checks the work against something fixed (e.g., the approved intent and the sources), and the main model revises. This loop runs at most three times.
Human gate Human I check the result and decide whether it is done.

Three kinds of models do the work. The main model runs the session and owns the job from the intent to the result. A helper model does one bounded, routine task (i.e., a task in which the inputs fix the result). A peer model is a fresh session of the main model, with the same strength and no memory of how the plan was made.

A review and an audit differ. A review judges whether the plan or the work is good, which takes taste, so a peer model (for the plan) or the main model itself (for the work) does it. An audit compares the work with something fixed that already exists, so a helper model can do it. The peer model gets the plan and the helper model gets the work, each with the approved intent only (i.e., my request and the reading of it that I approved), never the main model’s own view of it, because a reviewer who is told that the work is fine tends to agree. An audit reports only errors it can show, such as a broken function, a wrong number or quote, or a missing part; it judges no quality or reasoning, because the review already does that.

I decide twice: on the reading of my request before the plan, and on the result at the end. Between the two gates, the models work without stopping to ask, unless the work cannot go on.

Which form for which need

A harness offers several forms of tooling, and each form answers a different question.

Principles for the setup

Repository

ai-tooling-philosophy/
├── README.md                       this page
├── skill-workflow/
│   ├── skill_workflow_simple.tex   code of the simple version
│   ├── build_simple.sh             builds skill_workflow_simple.pdf and .png
│   ├── skill_workflow_simple.pdf   the simple version, for slides
│   ├── skill_workflow_simple.png   the simple version, for everything else
│   ├── skill_workflow.tex          code of the extended version
│   ├── build.sh                    builds skill_workflow.pdf and .png
│   ├── skill_workflow.pdf          the extended version, for slides
│   └── skill_workflow.png          the extended version, for everything else
└── .gitignore                      keeps build files out of git

Each figure is one TikZ file. To change a figure, edit its .tex file and run its script (./skill-workflow/build.sh or ./skill-workflow/build_simple.sh). The build needs a LaTeX distribution with XeLaTeX and TikZ, pdftocairo (part of Poppler), and the Lato font.


Maintained by Victor van Pelt.