Blog Evaluating external datasets
Blog · Security AI
Evaluating external security datasets without leaking your eval set
Public datasets of attack payloads, threat reports and vulnerability descriptions are a tempting shortcut for security AI. Used carelessly, they make a model look better than it is. Used with discipline, they can genuinely help.
Why external data is tempting
Security teams that build or tune models never have enough labelled data. Real incidents are rare, labels are expensive, and much of what exists is sensitive. Public collections fill the gap: web attack payloads, malware descriptions, threat intelligence reports mapped to MITRE ATT&CK, vulnerability records, question sets used as benchmarks. They are large, easy to download and often already labelled.
The risk is not that this data is bad. The risk is that it changes what your evaluation measures. A model can appear to improve because it has seen the test questions, because the training and test sets share near-identical samples, or because the external data matches the benchmark’s style rather than your environment. None of these improvements survive contact with real alerts.
Contamination: the quiet failure
Contamination happens when information from the evaluation set leaks into training. With public data it is easy to cause by accident. Popular benchmarks get copied into other collections. Threat reports are quoted in several datasets with slightly different formatting. Payload lists are merged, re-encoded and redistributed. A team can download two datasets from different sources and end up training on the exact items it later tests on.
Your own evaluation data can leak too. If a held-out set built from your environment, such as rule names or analyst-settled cases, overlaps with public text, a model trained on that text has effectively seen part of the answer. The defence is the same in both directions: decide what is held out first, fingerprint it, and keep it out of every training path.
Hygiene checklist before training
- Fix the evaluation set first, from your own data, and never score a model on the external dataset’s own test split.
- Normalise before comparing: decode, lowercase and replace random tokens, numbers and identifiers, so trivial variants are recognised as duplicates.
- De-duplicate across all sources and against every held-out item, and refuse to train if overlap remains.
- Split by group, not by row: by report, by attacker group, by host or by publication year, so near-copies cannot fall on both sides.
- Check the licence of every dataset and keep the attribution it requires. Exclude anything whose terms do not allow your use.
- Record provenance on every record: source, licence, whether it is synthetic, and how much weight it carries.
- Treat downloads as untrusted: fetch only expected data formats, never executable or serialised model files, and pin the exact revision.
Common pitfalls and their fixes
| Pitfall | What it looks like | Fix |
|---|---|---|
| Benchmark leakage | Scores jump on a public benchmark after adding a public dataset | Fingerprint benchmark items and remove matches from training |
| Row-level random split | Near-identical samples land in train and test | Split by report, group, host or time |
| Style over substance | Model learns the dataset’s phrasing, not the security concept | Evaluate on your own held-out data from real operations |
| Synthetic overload | Generated samples dominate and narrow the model | Mark synthetic data and cap its weight |
| Licence surprise | A dataset’s terms forbid commercial or derivative use | Review licences before download and record them |
| Unverifiable origin | Nobody can say where a training record came from | Require provenance fields and reject records without them |
Measure on your own ground
The most reliable question is simple: does adding the external data improve results on held-out data from your own operations, compared with the same model trained without it? Answer it with the same rows, the same metrics and a fair comparison. If the result is a tie or a loss, the external data is not helping, however impressive it looked on a public leaderboard.
Time-based and environment-based hold-outs make this sharper. Testing on cases that happened after the training data was frozen shows whether the model generalises forward. Leaving out a whole site or tenant shows whether it generalises across environments. Both matter more in security than raw accuracy on a static set, because attackers and environments both change.
It also helps to treat external data as a supplement that has to prove itself, not as a replacement for your own. Run experiments alongside the existing training pipeline rather than inside it, so a bad external source cannot quietly degrade a model already in use.
Governance is part of quality
Licence checks and provenance records are sometimes treated as paperwork. In security AI they are quality controls. A record whose origin is known can be removed when a problem is found. A dataset whose licence is understood will not have to be pulled from a model later. And a training pipeline that can explain every input is far easier to defend in an audit, which matters when the model’s conclusions drive security decisions.
How ApexAI approaches it
ApexAI treats external data as an optional supplement, never as the yardstick. Models are judged on held-out data from real operations, external sources are checked for licence and provenance, screened for overlap with evaluation data, and kept away from the main training path until they prove their value. Combined with local-first models that learn from your own environment, this keeps reported improvements meaningful for the SOC that relies on them.
Questions
What is data contamination in machine learning?
It is when information from the evaluation set ends up in training data, so the model appears better than it is. With public datasets it often happens through duplicated or lightly reformatted items.
Why not split public datasets randomly into train and test?
Security data has many near-duplicates: the same payload re-encoded, the same report quoted twice. A random row split puts these on both sides. Splitting by report, group, host or time avoids that.
Do licences really matter for internal models?
Yes. Some datasets forbid commercial or derivative use, and models trained on them may have to be withdrawn. Recording licence and attribution per record keeps the pipeline defensible.
See ApexAI on your own alerts.
A walkthrough with your team, in observe mode first: ApexAI scores and proposes, nothing acts on its own.