MatIntellect

The easiest way to check a running AI system is to ask the model to check its own work. Usually that's the shittiest option: the model defends its own mistakes. The second option is to bring in an outside model. I develop with Claude Code and usually call in Codex to review the work. I have other models connected too, but Codex does a good job reviewing what Claude Code built. The third option is to run my audit script. It asks the questions the model doesn't normally ask itself. Did we overcomplicate this? Is there a simpler way? What do the docs say? I wrote that script for myself and use it often

Circuit board under inspection
Pexels

What to give an outside reviewer

Give the reviewer the original task, the changes and the acceptance criteria. If all you send is the author's explanation, the review starts from the same assumptions. Replace a claim like "retries work" with a concrete requirement: running the job again must not duplicate its output. Ask the reviewer to back up each finding with a file and line, a log entry or an exact section of the official docs. If it proposes a different approach, have it explain what you can remove and what has to stay

A self-check is still useful for a quick pass against the brief. Ask the model what it actually opened or ran and what it expected to happen. That separates observations from a confident recap of its own plan. Save the question the author might never have thought to ask for the outside review: did we draw the boundaries of the task in the right place?

Workers on a production line
Pexels

Check what the system did

My auto-subtitles workflow takes about 2 minutes to turn a raw clip into a finished video. One word in Telegram tells it to pick up the file from Drive, proofread the transcript and save the result. If proofreading fails, it still renders the video with the original transcript. A useful check is to open the video and match the subtitles to a few points in the audio, then test what happens when proofreading is unavailable. A green status on its own won't tell you which path ran

The content factory takes about 5 minutes from idea to publication. Here the boundary is different: the system drafts the post, a person authorizes publication. Check that the post goes out after approval and that nothing goes out until someone presses the button. Use test material for those runs and check the destination itself rather than relying on a message that says "done"

Before fixing anything, distinguish an actual failure from an idea for making the code nicer. For a failure, record the input, what you expected and what happened. Run the same case again after the fix. If it's still a reviewer hypothesis, label it that way in the report

If the system works but you can't tell why the business should automate the process at all, that's a separate question for a business AI audit

Pick a test you can repeat

Take a task the system handles every day and find a real input with a known result. For a scheduling assistant, try asking it to move an appointment. On a test calendar, check what happened to the old event and what time the new one shows. The reply in chat comes second: "rescheduled" doesn't prove the calendar changed. For the next run, keep the same request but make the requested slot busy. Decide in advance whether the assistant should suggest another slot or ask the person what to do

To compare versions, keep the same test inputs and record what changed between runs: the model, an instruction or a tool. If you change everything at once, it's hard to tell which edit brought a bug back. Watch usage too. If a newer version achieves the same result but calls a model at every step, see which steps straightforward code can handle. Checking whether a slot is free can be a calendar lookup

If you are measuring latency, record the time to the first reply separately from the time to completion. A quick reply tells you nothing about how long the action took

The checklist behind /audit

I run /audit when I want to review an answer, a file or a plan. These checks come from its configuration. You can hand them to your reviewer without installing /audit

1. **Correctness** — are the conclusions and facts right?
2. **Completeness** — is everything covered? Is anything missing?
3. **Hidden assumptions** — what was taken on faith and never checked?
4. **Edge cases** — what happens in boundary cases or rare scenarios?
5. **Simplicity/overengineering** — what's unnecessary? Is there a simpler, more reliable, or more elegant way?
6. **Alternatives** — is there a better solution or approach?
7. **Official docs/GitHub** — does it match the official recommendation?
8. **Safety** — secrets, risks, dangerous operations?
9. **Cost/quotas** — pricing, limits, usage?
10. **Maintainability/fragility** — how easy is it to break or maintain?
11. **Reversibility/rollback** — can you roll it back? How?
12. **Tested vs. claimed** — what was actually checked, and what was just asserted?

Bonus: check your Claude instructions

/claude-api prompt-audit from Anthropic's official skill reviews prompts, tool descriptions, and instructions such as CLAUDE.md. It can spot conflicting rules or a leftover instruction from an older model that makes Claude check the same thing several times. You get a report with file and line references and a set of proposed edits. It doesn't apply them without your approval

Share:

No comments yet

Leave a Comment

Fields marked with an asterisk (*) are required