How we test OwlSOC, including the bar we currently fail
Published
OwlSOC is evaluated against a fixed set of 30 labelled synthetic scenarios before every release: 15 true positives, 10 false positives and 5 genuinely ambiguous cases. On the latest run the deterministic triage tier detected 15 of 15 true positives with no missed criticals, but escalated 2 of 10 false positives, a 20% false-positive rate against our own bar of 15%. We publish the failure because a benchmark you only publish when you pass is marketing, not evidence.
Why publish a test set at all?
Every AI SOC vendor, including us, can tell you it is accurate. None of those claims are verifiable from the outside. There is no shared benchmark for security alert triage in the way there is for, say, image classification, and the numbers in vendor marketing are almost never accompanied by a methodology, a sample, or a definition of what was being counted.
That leaves a buyer with two options: believe the number, or test it themselves. We would rather make the second option easier, and go first by showing what we test against and what happened.
There is also a self-interested reason to be honest here, and it is worth stating plainly. A vendor who publishes only wins has told you nothing, because you cannot tell a strong result from a selected one. A vendor who publishes a failure has given you a way to calibrate everything else they say.
What is in the evaluation set?
Thirty labelled cases, deliberately split across three buckets so the set measures more than one thing. The scenarios are synthetic but realistic in shape. They contain no real client telemetry, and we would not use customer data for this.
- 15 true positives spanning the common attack categories: credential stuffing that succeeds, impossible travel on a privileged account, OAuth consent abuse, encoded PowerShell, lateral movement, anomalous egress, persistence via scheduled task and via IAM key, privilege escalation, MFA fatigue followed by an accept, ransomware precursors, and a compromised service principal.
- 10 false positives that a naive rule gets wrong: the genuine business traveller, an admin doing an approved bulk operation, a scheduled scanner, vendor SSO, dev and test traffic, a routine password change, an expected OAuth renewal, routine admin sign-in from a baselined country, shared inbox polling, and scheduled backup traffic.
- 5 genuinely ambiguous cases where the correct answer is "needs review" rather than a forced guess: a cold-start first sign-in with no baseline, gradual geographic drift, a partial indicator with an unknown hash, a weak anomaly signal, and novel application usage.
What bars do we hold ourselves to?
The bars matter more than the score, because a score without a threshold set in advance is just a number chosen after the fact. Ours are set before the run and they are not all equally weighted.
Missing a real attack is the failure that matters most, so the bar for recall on true positives is 95% and the bar for missed criticals is zero. Escalating a false positive wastes an analyst's time, which is bad but recoverable, so that bar is 15%. The ambiguous cases must not be escalated as though they were confirmed, because that is how an alert queue becomes noise again.
We also check that the system degrades honestly. If the AI tier is unavailable, the deterministic result must still be produced with nothing silently dropped and nothing auto-closed as benign. An outage that quietly stops investigating is worse than one that fails loudly.
The results, including the one we fail
These figures are from the deterministic triage tier, which is the layer that runs on every alert. We are not publishing AI-investigation-tier results yet, because that evaluation is not complete, and a partial number would be worse than none.
- Every alert produced a case: 30 of 30. Nothing was dropped.
- Recall on confirmed true positives: 15 of 15, against a bar of 95%. Passed.
- Missed criticals: 0, against a bar of 0. Passed.
- False-positive rate on confirmed false positives: 2 of 10, or 20%, against a bar of 15%. Failed.
- Ambiguous cases correctly not escalated: 5 of 5. Passed.
- MITRE ATT&CK family recall on true positives: 15 of 15, against a bar of 90%. Passed.
- Time to a deterministic verdict: 43ms median, 56ms at the 95th percentile. The AI investigation is a separate, slower step that adds provider latency and typically completes in under two minutes.
- Degraded-mode parity: 30 of 30, with no drops and nothing auto-closed as benign.
Why does it fail, and what does that tell you?
The two cases it gets wrong are named in the results, because "we have some false positives" is not a useful disclosure. They are a domain admin performing an approved bulk user-provisioning operation, and an admin signing in normally from a country already in their baseline.
In both, the same rule fires: a privileged identity is involved, which raises the blast radius and escalates the priority. The rule is correct in general. Privileged identities genuinely do warrant more scrutiny. It is wrong in these two specific cases because it has no way to know that this particular bulk operation was scheduled and approved, or that this particular admin logs in from that country every week.
That is the honest limitation of deterministic rules without organisational context, and it is not fixable by writing a cleverer rule. It is fixable by context: an investigation tier that can weigh the surrounding evidence, and a feedback loop where confirming a pattern as benign once changes how it is triaged next time. That is exactly why those layers exist in the product, and it is why we would rather show you the gap than hide it.
What this proves, and what it does not
It proves that on a defined, labelled set, the system detects the attacks in it, maps techniques correctly, keeps ambiguous cases out of the escalation path, and fails in a specific, explainable, reproducible way.
It does not prove anything about your environment. Your alert mix, your detection rules, your baseline and your business context are different from the test set, and a synthetic scenario is a simplification of a real one by construction. It also does not prove the verdicts would be right on an attack pattern nobody has thought of, which is the nature of any fixed test set.
This is why the pilot exists and why it is refundable. A test set is how we catch regressions before they reach anyone; it is not evidence about your estate. Thirty days on your own alerts is.
How to run the same test on any vendor
You do not need our set to do this, and you should not take our word for the results. The method is straightforward and works against any AI triage tool, including ours.
- Take a sample of alerts you have already dispositioned, so you have a ground truth to compare against. Two weeks of real alerts is usually enough.
- Record, per alert: what your team concluded, what the tool concluded, and whether they agree. Agreement rate against your own analysts is a far more meaningful number than any vendor accuracy claim.
- Count the disagreements in both directions separately. A tool that never misses an attack but escalates everything is a different problem from one that is quiet but misses things, and a single accuracy figure hides which one you have.
- Check the ambiguous cases specifically. A tool that is never unsure is either working on unusually clean data or is concealing uncertainty, and concealed uncertainty produces confident wrong closures.
- For each verdict, try to trace one claim back to the underlying log. If you cannot check the reasoning, you are being asked to trust a score.
- Ask the vendor for their own failure cases. If they have none to show you, either they have not looked, or they are not telling you.
Frequently asked
How do you know an AI SOC works before you connect it to your environment?
You cannot know how it will perform on your estate, but you can check three things beforehand: whether the vendor publishes a test set and methodology rather than a bare accuracy figure, whether they can show you a case where the tool was wrong and explain why, and whether every verdict can be traced back to the underlying log so your team can check the reasoning rather than trust a score. Anything beyond that requires a trial on your own alerts, which is why a short refundable pilot is a more useful signal than a benchmark.
What is in the OwlSOC evaluation set?
Thirty labelled synthetic scenarios: 15 true positives covering common attack categories such as credential stuffing, impossible travel on a privileged account, OAuth consent abuse, encoded PowerShell, lateral movement, exfiltration and MFA fatigue; 10 false positives that naive rules get wrong, such as a genuine business traveller or an approved admin bulk operation; and 5 genuinely ambiguous cases where the correct answer is "needs review". The scenarios are synthetic in shape and contain no real client telemetry.
Does OwlSOC pass its own evaluation?
Not entirely. On the latest run the deterministic triage tier passed the bars for true-positive recall (15 of 15), missed criticals (zero), ATT&CK mapping (15 of 15), handling of ambiguous cases (5 of 5) and degraded-mode behaviour, but failed the false-positive bar: it escalated 2 of 10 confirmed false positives, a 20% rate against our own target of 15%. Both failing cases involve a privileged identity performing legitimate activity, and both are named in the published results.
How fast is a verdict?
The deterministic triage verdict is effectively instant: 43ms median and 56ms at the 95th percentile on the evaluation set. That is the layer that runs on every alert. The AI investigation, which produces the calibrated confidence and the written narrative, is a separate and slower step that adds provider latency and typically completes in under two minutes. The two should not be conflated.
Do these results predict how OwlSOC will perform in my environment?
No, and we would be wary of any vendor who claims otherwise. A fixed test set catches regressions and demonstrates behaviour on known scenarios, but your alert mix, detection rules, baselines and business context differ from any synthetic set. It is a defined test set, not a guarantee about a specific environment, which is why OwlSOC starts with a 30-day fully refundable pilot on your own alerts.
Start with a 30-day refundable pilot. £495, one environment, every alert investigated, a full report at week four. Read-only, live within 48 hours of access.