THREATBYTESTM research
Limitations
Limitations include concept drift, label noise, incomplete sandbox coverage, and the risk of collateral disruption from autonomous mitigation. Each documented to maintain research integrity.
Research integrity01
Acknowledged limitations prevent over-claiming detection performance or autonomous readiness; evaluation reports separate predictive gains from operational deployment requirements.
Known limitations02
- Concept drift: malware changes, so performance measured today decays over time.
- Label noise: antivirus-derived labels disagree and change.
- Sandbox coverage: evasive malware may not reveal its behaviour during analysis.
- Corpus bias: historical collections over-represent some platforms and families.
- Synthetic realism: simulated attacks may not match real adversaries.
- Live testing: ethics and safety rule out testing autonomous mitigation on real production networks.
How they are addressed03
- Time-aware and family-aware evaluation splits.
- Consensus labelling with recorded label sources.
- Combining static, behavioural, and reasoning signals so no single view is relied on.
- Cyber-range and red-team evaluation in place of live deployment.
- Explicit threats-to-validity sections in every published result.