Knowledge
Explainable SI for security
Techniques that show why a model or agent reached a decision, so analysts can check it, trust it, and challenge it.
A verdict alone cannot be audited. Analysts need to know which evidence drove a decision to confirm it, spot errors, and defend actions taken on it. Explanations are also a research instrument: they reveal whether a model learned real malicious behaviour or an accidental shortcut in the data.
- Feature attribution (SHAP, LIME): how much each feature pushed a prediction up or down.
- Counterfactuals: the smallest change that would flip the verdict.
- Attention and saliency: which bytes, instructions, or events a neural model focused on.
- Reasoning traces: the chain of observations and hypotheses an agent followed to a proposed action.
Attribution methods can disagree and can be unstable on correlated features. Explanations should be checked for consistency across similar samples and validated against known malicious behaviour, and should never be presented as proof on their own.
The evaluation plan applies SHAP and LIME to the learned models and records reasoning traces for agentic decisions. Explanations will appear next to verdicts in analysis reports and next to mitigation proposals in the approval queue.