TriageLens

01 - Context

Context

A SOC watches an organisation's security tools and reacts to what they report. Detection products (endpoint, email, identity, cloud) raise alerts. Related alerts are grouped into incidents, and each incident carries pieces of evidence: the devices, accounts, files, IP addresses and URLs involved.

A tier-1 analyst reads incidents one by one and closes or escalates each.

Analysts get more alerts than they can read. In Vectra's 2024 survey of 2,000 practitioners, SOC teams received 3,832 alerts per day and ignored 62%. They could handle only 38% and called 16% real attacks.

In Splunk's State of Security 2025, 59% of respondents have too many alerts, 55% too many false positives, 52% an overworked team, and 52% have thought about quitting cybersecurity over stress.

Missed attacks are expensive. IBM's Cost of a Data Breach Report 2026 puts the average breach at USD 4.99 million, taking 247 days on average to identify and contain. Breaches lasting over 200 days cost USD 5.65 million, against USD 4.32 million for shorter ones.

GUIDE adds measured data. Microsoft released it in 2024 from Defender XDR telemetry, graded by each customer's analysts: over 13 million pieces of evidence, 1.6 million alerts and 1 million incidents from 6.1k organisations, covering 441 MITRE ATT&CK techniques and 33 entity types. Kaggle version 2 (July 2026) adds expert ranks for 9,980 test incidents across 499 organisation queues.

In the files we downloaded, 23,670 of the 1,033,998 incidents have no grade. The remaining 1,010,328 split into 49.0% benign positive, 30.4% false positive and 20.6% true positive.

02 - Problem

The business problem

A tier-1 analyst faces a queue that is too long to read in full. Most incidents are false alarms or harmless, and about one in five is a real attack. Three questions follow:

1. Is it real? (true positive, benign positive or false positive)
2. Which incidents should be read first?
3. Which can be closed unread, and at what risk?

A new customer has no history in the training data, yet in GUIDE's official split 4,054 of the 4,400 test organisations also appear in training. Scores there mostly reflect known organisations.

A per-detector lookup shows the gap: macro-F1 0.68 on the official split, but 0.43 on organisations it has never seen. Closing that gap is the problem TriageLens addresses. Both scores are reproduced in the Part 2 notebook (0.6769 and 0.4324).

03 - Objectives

Objectives

TriageLens supports a tier-1 SOC analyst. The analyst decides; the tool proposes.

1. Grade each incident as true positive (TP), benign positive (BP) or false positive (FP), with a confidence and the reasons behind it (detectors, categories, ATT&CK techniques, entity types).
2. Rank each organisation's queue so the incidents worth reading come first.
3. Propose auto-closes only where the chance of closing a real attack is held below a fixed limit, and send everything else to the analyst.
4. Measure honestly on unseen organisations and on newer data, next to the official split.
5. Describe the groups: cluster incidents and check whether outliers are more often attacks.

The grade-then-gate design is inspired by decision models, new in 2026, which return one choice from a fixed list with a calibrated confidence instead of free text. We build our own with course methods, with no paid API.

04 - Problem types

Problem types

05 - Questions

Questions the model answers

On a single incident:

• Is it TP, BP or FP, and how confident is the model?
• Which features and ATT&CK techniques drove that grade?
• Can it be closed without review?

On the queue:

• Which incidents should I read first, and how many of the top 10 match the experts' top 10?

On the evaluation:

• How well does it grade unseen organisations versus known ones?
• Does a model trained before 10 June 2024 work after it?
• Which organisation-detector pairs raise the most false alarms?
• Do incidents form groups, and are outliers more often real attacks?

06 - Success criteria

Success criteria

How the splits are built:

• Final test: 20% of organisations held out from training and tuning (1,144 organisations with 188,378 graded incidents, seed 42).
• Tuning: 5-fold GroupKFold on OrgId.
• Time split: incidents run from 20 November 2023 to 17 June 2024 (UTC); 534,581 start before 10 June 2024 and 499,417 on or after (ungraded included).

Macro-F1 is the main metric because the classes are unbalanced (the GUIDE authors recommend it too); accuracy would reward always predicting benign positive.

07 - Error costs

Error-cost table

A missed attack is worse than a false alarm. The GUIDE authors agree: they call misclassifying a TP as BP or FP "the more critical incident misclassification".

Alahmadi et al. (USENIX Security 2022) argue the other side: false positives are equally critical, because they bury real alerts and wear analysts down. Auto-close exists to cut false-alarm load, and its threshold exists to limit missed attacks.

Severity is qualitative: we have no measured cost per incident, so we invent none.

1. !!Critical True positive - Auto-closed (not read)

   The attack goes unseen and keeps running. This is the error the 1% limit controls.

   "The TP-or-not threshold is set from this table, not at 0.5."

2. !High True positive - Sent to analyst, low in queue

   The attack is seen late, after other work.

   "A missed attack is worse than a false alarm."

3. ✓Correct True positive - Sent to analyst, top of queue

   "Rank each organisation's queue so the incidents worth reading come first."

4. ✓Correct Benign positive - Auto-closed (not read)

   "Auto-close exists to cut false-alarm load, and its threshold exists to limit missed attacks."

5. -Low Benign positive - Sent to analyst, low in queue

   Some analyst time spent.

   "Confusing BP with FP is the mildest error, because both lead to closing the incident."

6. ~Medium Benign positive - Sent to analyst, top of queue

   Analyst time spent before real attacks lower in the queue.

   "Alahmadi et al. (USENIX Security 2022) argue the other side: false positives are equally critical, because they bury real alerts and wear analysts down."

7. ✓Correct False positive - Auto-closed (not read)

   "Auto-close exists to cut false-alarm load, and its threshold exists to limit missed attacks."

8. -Low False positive - Sent to analyst, low in queue

   Some analyst time spent.

   "Confusing BP with FP is the mildest error, because both lead to closing the incident."

9. ~Medium False positive - Sent to analyst, top of queue

   Analyst time spent and trust in the tool drops.

   "Alahmadi et al. (USENIX Security 2022) argue the other side: false positives are equally critical, because they bury real alerts and wear analysts down."

• The TP-or-not threshold is set from this table, not at 0.5.
• Confusing BP with FP is the mildest error, because both lead to closing the incident. It is also the most common error in the GUIDE paper.

08 - Flow

TriageLens flow

1. Step 1 Evidence rows
2. Step 2 One row per incident
3. Step 3 Graded
4. Step 4 Ranked in the queue
5. Step 5 Auto-close gate
6. Step 6 The analyst decides

Follow one incident's journey ->

09 - Data source

Data source

All row and incident counts in this part were counted from the Kaggle files.

From the repository root:

python scripts/download_guide.py

The script saves the files to data/raw/, skips ones already there, counts rows and prints PASS or FAIL. It uses only the standard library; our run passed in under a minute.

Notes on the data:

• The paper says "a two-week period", but timestamps span 20 November 2023 to 17 June 2024. Part 2 confirmed the span. The release perturbs timestamps, so the dates are only relative.
• The paper counts 441 ATT&CK techniques overall; GUIDE_Train.csv alone has 432.
• The paper and Kaggle card describe a Rank and a split column; the file has QueueRank and no split.
• GUIDE_Test.csv has an extra Usage column (2,915,437 rows Public, 1,232,555 Private). What it marks is unverified (probably a leaderboard split); it is not a feature.
• The ranks are about investigation priority, not about the grade: a top-ranked incident is not always a true positive.

10 - Ethics

Ethics

• Real telemetry. The data is anonymised customer data; we do not try to re-identify anyone.
• No raw data in the repository. Only a download script and aggregated results.
• The analyst decides. TriageLens proposes grades, an order and auto-closes. It does not close or escalate anything by itself, and every auto-close proposal shows its confidence and reasons.
• Known limits are shown. Unseen-organisation scores sit next to official-split ones.
• Label quality. "Benign positive" varies between SOCs, and the model learns those inconsistencies too.

11 - Risks

Risks