Problem
Security teams get review requests in chat all the time: a Terraform file, a SaaS posture export, a vendor SOC 2 packet, an architecture diagram, a folder of audit evidence. One general-purpose prompt handles these badly. It mixes finding problems, mapping them to compliance frameworks and proposing fixes, so it can force irrelevant controls onto an artifact or fix problems nobody found.
We wanted first-pass reviews that are consistent, bounded and traceable to evidence in the artifact, with no person choosing the reviewer each time.
Approach
Hermes runs on Hermes Agent, an open-source, self-hosted agent framework. The review team is 11 skills with strictly separated duties:
- One entry point.
security-leadis the only skill a user talks to in Slack. It classifies the request and does no technical analysis itself. - One orchestrator. A single background orchestrator runs the whole review in one lifecycle.
- Specialist reviewers. Seven domain reviewers (SaaS posture, AWS, Snowflake PCI, application security, vendor risk, audit evidence, architecture) plus three core roles: vulnerability finder, compliance reviewer and remediation engineer.
Each reviewer emits one fixed table and nothing else. None proposes fixes. The remediation engineer fixes only the findings it is handed and may not run a fresh review. The compliance reviewer checks applicability first and answers “Cannot Assess” when the artifact gives no scope evidence.
How it works
- Intake.
security-leadreads the artifact and picks a route from a routing matrix. An AWS review goes to the AWS and compliance reviewers, a vendor review to the vendor and audit-evidence reviewers, an unclear request to architecture, security and compliance. It never dispatches every reviewer. - Phase A. The orchestrator runs the selected reviewers in parallel and treats their tables as canonical.
- Phase B. Only concrete technical findings reach the remediation engineer, and only when the review type allows fixes. Vendor and audit-evidence reviews return follow-up questions and risk decision points instead.
- Report. The orchestrator saves a report with five fixed sections and replies in Slack with a summary and the full report attached, so it opens on a phone. Prose is Traditional Chinese; code and identifiers stay in English.
Two runtime details matter. Delegation depth must be at least 2, or the orchestrator is downgraded to a leaf worker and cannot launch reviewers. And because top-level delegation runs in the background, the phases live inside the orchestrator, not in separate chat turns.
Results
- Eleven skills, a design document and a set of Python contract tests are in place. The tests check that every skill has valid front matter, that
security-leadholds the full routing matrix and the single-orchestrator contract, that domain reviewers refuse direct requests and point back tosecurity-lead, and that remediation depends only on canonical findings. - The agent itself was vetted before use: the installer was reviewed instead of piped to a shell, commands ran in a container, the API key sat in the macOS keychain, and the agent passed 7 of 7 behaviour checks there, including read-only tasks, no commit without approval and no reading of real secrets.
- End to end: a full run through Slack, from request to saved report, now works. The contract tests still show only that the files agree with each other; review quality on real artifacts is judged case by case.
Lessons
- Separate finding, mapping and fixing. When each role has one narrow output and a rule against doing the others’ work, invented findings have fewer places to hide.
- Check the plumbing, not only the prompts. A runtime limit such as delegation depth changes behaviour without any error in the skill text, so the contract tests and the deploy guide record it.
- Choose isolation over approval prompts. In the version first installed, container isolation and manual command approval were mutually exclusive, so we chose real isolation over a prompt people learn to click through.