Problem
An AWS account that passes the CIS benchmark on Monday can fail it on Tuesday because someone opened a security group. A periodic scan only says a control now fails, not what changed, who changed it, or whether the finding is new or accepted noise. An LLM in the loop adds a risk: a model that decides what counts as an alert can be talked out of one by text inside the resource it reads.
CloudSheriff was a time-boxed build for the Daytona HackSprint, with two hard limits: read-only access to the account, and a model that never decides.
Approach
Three sources of evidence, each with one job:
- Prowler runs the CIS AWS benchmark and says which controls pass or fail.
- A canonical resource snapshot says exactly what changed between scans.
- CloudTrail Event history says who changed it, when and through which API. It enriches an alert but is not the source of the before state.
Alerting is plain code over structured fields. The LLM receives an alert that is already decided and writes a short explanation for the on-call engineer.
We tested it on a deliberately misconfigured lab in a dedicated sandbox account, built with OpenTofu: a compliant security group as the drift target, a legacy group with RDP open from day one (known noise that must stay silent, and carries text written to talk the model into downgrading it), and negative controls that must never alert.
How it works
- Least privilege. The tool assumes an audit role (AWS SecurityAudit and ViewOnlyAccess, plus an explicit deny on reading object contents, secret values and decryption) with an external ID and short-lived credentials.
- Ephemeral scan room. Prowler, at a pinned version, runs in a fresh Daytona sandbox that holds the credentials only for that scan and is deleted afterwards.
- Stable evidence. Only structured JSON-OCSF fields are normalized. Status text is never hashed, because scanner wording changes would create false drift.
- Three-axis diff. Each finding is compared with saved state by evaluation (pass, fail, not evaluated), lifecycle (active, resolved, resource gone) and change (new, regression, changed, unchanged). A targeted re-scan never marks unscanned controls as fixed.
- Deterministic gate. An alert needs a failed check, a new, regressed or changed finding, and medium or higher severity. The baseline run never alerts.
- Attribution. For a security-group alert the tool looks up matching CloudTrail events. If none has arrived yet, it reports “attribution pending”, not “nobody”.
- Explanation. The model gets only the decided facts, with resource tags in a block marked as untrusted data, and no AWS credentials. If the API fails or refuses, a template takes over. The call is traced in Arize.
Results
On 2026-09-30 the full loop ran live: baseline, SSH opened to the internet on the demo group, exactly one regression alert with CloudTrail attribution and an LLM explanation, then RESOLVED once the rule was closed. CloudTrail delivered the event in about two minutes.
Offline, 70 unit tests passed (one skipped) and 6 OpenTofu tests passed against a mock provider. A script that simulates the audit role’s policy checks that 14 write or payload-read actions are denied and 6 read actions are allowed. It is the evidence behind the read-only claim.
The project is decommissioned: the lab was destroyed on 2026-10-06 and the scan image deleted.
Lessons
- Give each data source one question. Mixing scanner status, configuration history and audit logs invites confident but wrong attribution.
- Keep the model out of the decision path. Gating on structured fields makes the injection bait harmless by construction, not by prompt wording.
- After a destroy, check the cloud account, not only the state. OpenTofu reported 28 of 29 resources destroyed, and one bucket stayed in AWS until it was deleted by hand.
- This is a proof of concept: snapshots and attribution cover security groups only.