I have been playing with building a modular DFIR platform called Reaper for a while now. The idea is that you deploy a Core (datastore, ingest, SSO, certificates, a reverse proxy) and then bolt on whatever components you actually need for the job.
As part of this I have built in presets like analysis and incident-response that bundle sensible groups together.
Which is great, until you realise that it currently has seven optional components. Which is 2^7 which = 128 possible deployment combinations. I generally only test a handful of these, but I wanted to be sure that any combination would work, but I didn’t want to have to deploy it 128 times.
The bit where I did the maths and regretted it
Each component isn’t independent, it shares the Core’s databases, connects to Authentik for SSO, use NGINX to present their interface. I was concerned that any two of them could interfere in ways that neither one does alone. And for a DFIR platform, finding this out when the pressure is on wouldn’t be ideal.
Not all tests need a real deployment
A deployment can vary in its deployment time, but on average takes roughly fifteen minutes. at this time deploying each combination would take about thirty-two hours of machine time. I dont have a straight 32 hours of time so this would actually be a lot longer as it would be waiting for me to come back and finish tasks. After giving it some thought I had a brain wave: maybe I could get AI to do this for me, and I could spend my time doing other things (Isn’t this the promise of AI? I do the thinking, it does the boring work).
So I asked AI to do this. The first step it did was validation. When creating a new stack it creates its own that gets merged with Core’s. The risk was introduction on conflicts. This is an easy check and only takes about a second per combination, so all 128 were done in a few minutes.
The second layer was the deployment planner. Reaper works out which components you asked for, pulls in anything they depend on, and orders them. That runs in about three seconds with a dry run flag, so again all 128 went through in about 10 minutes or so.
Next dependency checks. The AI component needs IRIS for note taking. Fortunately, both halves of that worked. IRIS was added automatically when you ask for AI, and asking for AI while explicitly excluding IRIS was refused with a sensible error.
Two layers done, this left the most expensive of the three.
| Layer | What it catches | Per combination |
|---|---|---|
| Compose merge | Duplicate services, port clashes, paths that resolve differently when merged | ~1 second |
| Deployment planner | Missing dependencies, bad ordering, unknown components | ~3 seconds |
| Real clean install | Everything else — boot order, provisioning, certificates, SSO registration | 5 to 31 minutes |
Full teardown, full deployment × 128.
What “from scratch” actually has to mean
When deploying the stack, if you leave the .env files in place between tests, you never exercise key parts of the initialisation including the secret generation parts. The init only fills in values that are still placeholders. By leaving the .env files behind the test becomes a test of redeploying, not a fresh install.
I learned that the hard way a month or two back. I had been wiping the data directories between test builds but keeping .env. This meant that tools like Velociraptor was reading stale data into its config before parts of the stack were even created resulting in unexpected failures and errors.
So for these tests, it was vital that we made sure that a tear down deleted everything:
- all the containers,
- their volumes
- networks,
- every locally built image,
- all the stack data directories,
- every
.envfile, - the resolved deployment state,
- and all the generated nginx and OpenSearch configs.
This only left the base images pulled from registries, and the Ollama model cache (it is 4.5 GB, which would have meant nearly 300 gigabytes of redundant downloading across the runs that include the AI component).
What the loop actually looked like
Next AI wrote three scripts:
- A teardown,
- a runner that did one combination,
- and a driver that walked all 128 subsets in order of how many components they had
This would allow the simplest failures to surface before the combinations that contained them. The driver ran in the background and appended a line per result to a file.
AI:
So the deploying was unattended, but the judging was not. Every failure got looked at while the evidence still existed, which matters more than it sounds like it should — the next combination’s teardown destroys everything the last one left behind. Twice it stopped the run outright rather than letting it continue. Once when it worked out that a bug in its own argument handling was about to fail every remaining multi-component combination, and once to go and investigate a failure properly before the thirty-two combinations containing that same pair of components went past.
For every single run it recorded the exit code, the elapsed time, how many containers were up, and how many of those were not healthy. On a failure it also grabbed the tail of stderr, plus
docker logs, the exit code and the health check output for every container — captured before the next teardown wiped them. That last part only got added after the first real failure, where the evidence was gone before anyone thought to look at it. Lesson learned, slightly too late.
Is it just marking its own homework?
This is where we really worked together, I had to define what a “pass” looks like (and I would regularly check in and try out different parts that were under test). And AI did the testing.
The tests performed two independent things:
- The launcher itself had to exit zero,
- and the launcher will not record a deployment as successful until its own readiness check has fetched every enabled component’s web route through the reverse proxy and got an acceptable response back.
Then separately, the harness counted the containers and checked their health status.
This wasn’t completely smooth (Which is why we did this) At one point Arkime got marked as unhealthy as it had no health check defined. The harness was treating the absence of the word healthy as a failure (Docker reporting it as “Up” rather then “Up (healthy)”). A quick inspection of the container allowed this to be identifed.
The test harness was buggier than the thing it was testing
This process wasn’t perfect. The first few hours were almost entirely AI finding bugs in scripts AI had written an hour earlier:
- The teardown stripped control characters to tidy up Docker’s output, using a character class that also strips newlines. Every list of containers collapsed into one enormous line resulting in garbage data being passed to
docker rmduring teardown. - PowerShell, when handing a list of components to a child process, passed one string with commas in it instead of an actual list. So the planner was being asked to deploy a component literally named “Dashboards,Arkime” and quite correctly refused.
This was all done by the AI itself, it wrote all of the testing scripts, and through testing identified and fixed them without intervention.
During all this, I was mostly just annoying to the process. I would log into a half deployed platform and looked around at what it looked like and provided feedback on things that were broken but in reality were just not finished deploying.
In the end, none of these were faults with the code base.
Catching bugs
Despite a few false starts, the process ended up being extremely valuable. Three real bugs came out of it which were all patched improving future deployments.
The numbers
- 128 combinations.
- 125 passed.
- 32.7 hours of deployment time.
- 2,272 containers started and torn down.
The largest single deployment was 25 containers, and Core on its own took five minutes while everything at once took thirty one.
Broken down by how many optional components were in the mix:
| Components | Combinations | Passed | Failed | Typical time |
|---|---|---|---|---|
| 0 (Core alone) | 1 | 1 | 0 | 5 min |
| 1 | 7 | 7 | 0 | 8 min |
| 2 | 21 | 20 | 1 | 9 min |
| 3 | 35 | 34 | 1 | 12 min |
| 4 | 35 | 35 | 0 | 18 min |
| 5 | 21 | 20 | 1 | 21 min |
| 6 | 7 | 7 | 0 | 24 min |
| 7 (everything) | 1 | 1 | 0 | 31 min |
That shape in the combinations column is just Pascal’s triangle, which is a nice way of confirming every subset really was tested once and not some of them twice. It also shows where the time goes — over half the total runtime went on the four and five component rows, simply because that is where most of the combinations live.
So what?
As everyone continues to look for reasons to use and implement AI, I have found this to be a genuinely useful case.
The machine spent 32.7 hours deploying. This is time I didnt spend sitting at my computer watching it run. I spent a few hours up front while the harness was being debugged into something trustworthy, and then short bursts whenever a failure landed and needed a decision.
This was a pretty good trade off, however it isn’t just press a button and walk away. (Hats off to people who have that level of trust).
I also now have the suite rather than just the result. The teardown, the runner and the driver all still exist, and the driver skips any combination that already has a result, so it can be stopped and restarted without losing work. They live in the repo alongside everything else now, and the list of components is read from the stack manifests rather than hard-coded, so adding an eighth component means the sweep just quietly starts covering 256 combinations instead. Re-running all 128 before a release is now a thing I can choose to do rather than a thing I would have to build.