Getting different alert counts for the same target rule on the same dataset
I'm getting a different number of the *same alert type* depending on how many *other* rules I use in the same run:
When running Zircolite against a certain dataset using just a singular rule:
```powershell
python .\zircolite.py `
--evtx ..\cats\tools\normalizer\conversion_docs\dedale\test\D15_H10_2025-01-06T10_winlogbeat_F347.jsonl `
--json-input --fileext jsonl `
--ruleset .\raw_sigma\rules\windows\process_creation\proc_creation_win_susp_web_request_cmd_and_cmdlets.yml `
--save-ruleset `
--pipeline windows-logsources `
--rulefilter "Browser Started with Remote Debugging" `
--template ..\cats\tools\normalizer\conversion_docs\dedale\flat_alerts.jsonl.tmpl `
--templateOutput raw_alerts.jsonl `
```
it results in 74 hits
```
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Severity ┃ Rule ┃ Events ┃ ATT&CK ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
│ MEDIUM │ Usage Of Web Request Commands And Cmdlets │ 74 │ T1059.001 │
└────────────────┴───────────────────────────────────────────┴──────────┴───────────┘
```
The converted rule looks like this
```json
[
{
"title": "Usage Of Web Request Commands And Cmdlets",
"id": "9fc51a3c-81b3-4fa7-b35f-7c02cf10fd2d",
"status": "test",
"description": "Detects the use of various web request commands with commandline tools and Windows PowerShell cmdlets (including aliases) via CommandLine",
"author": "James Pemberton / @4A616D6573, Endgame, JHasenbusch, oscd.community, Austin Songer @austinsonger",
"tags": [
"attack.execution",
"attack.t1059.001"
],
"falsepositives": [
"Use of Get-Command and Get-Help modules to reference Invoke-WebRequest and Start-BitsTransfer."
],
"level": "medium",
"rule": [
"SELECT * FROM logs WHERE CommandLine LIKE '%[System.Net.WebRequest]::create%' ESCAPE '\\' OR CommandLine LIKE '%curl %' ESCAPE '\\' OR CommandLine LIKE '%Invoke-RestMethod%' ESCAPE '\\' OR CommandLine LIKE '%Invoke-WebRequest%' ESCAPE '\\' OR CommandLine LIKE '% irm %' ESCAPE '\\' OR CommandLine LIKE '%iwr %' ESCAPE '\\' OR CommandLine LIKE '%Resume-BitsTransfer%' ESCAPE '\\' OR CommandLine LIKE '%Start-BitsTransfer%' ESCAPE '\\' OR CommandLine LIKE '%wget %' ESCAPE '\\' OR CommandLine LIKE '%WinHttp.WinHttpRequest%' ESCAPE '\\'"
],
"filename": "",
"channel": [],
"eventid": []
}
]
```
But when I am using a superset instead, i.e., the entire Sigma Windows ruleset (all other parameters are the same), something strange happens
```powershell
python .\zircolite.py `
--evtx ..\cats\tools\normalizer\conversion_docs\dedale\test\D15_H10_2025-01-06T10_winlogbeat_F347.jsonl `
--json-input --fileext jsonl `
--ruleset .\raw_sigma\rules\windows\ `
--save-ruleset `
--pipeline windows-logsources `
--rulefilter "Browser Started with Remote Debugging" `
--template ..\cats\tools\normalizer\conversion_docs\dedale\flat_alerts.jsonl.tmpl `
--templateOutput raw_alerts.jsonl
```
While we of course now have other alerts as well, there are somehow *fewer* `Usage Of Web Request Commands And Cmdlets` alerts than before
```
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity ┃ Rule ┃ Events ┃ ATT&CK ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ HIGH │ External Remote SMB Logon from Public IP │ 30 │ T1133, T1078, T1110 │
│ HIGH │ Windows Binaries Write Suspicious Extensions │ 4 │ T1036 │
│ MEDIUM │ Usage Of Web Request Commands And Cmdlets │ 40 │ T1059.001 │
│ MEDIUM │ Potential Binary Or Script Dropper Via PowerShell │ 1 │ │
└────────────────┴───────────────────────────────────────────────────┴──────────┴─────────────────────┘
```
The converted rule looks like this - exactly the same
```
[
...
{
"title": "Usage Of Web Request Commands And Cmdlets",
"id": "9fc51a3c-81b3-4fa7-b35f-7c02cf10fd2d",
"status": "test",
"description": "Detects the use of various web request commands with commandline tools and Windows PowerShell cmdlets (including aliases) via CommandLine",
"author": "James Pemberton / @4A616D6573, Endgame, JHasenbusch, oscd.community, Austin Songer @austinsonger",
"tags": [
"attack.execution",
"attack.t1059.001"
],
"falsepositives": [
"Use of Get-Command and Get-Help modules to reference Invoke-WebRequest and Start-BitsTransfer."
],
"level": "medium",
"rule": [
"SELECT * FROM logs WHERE CommandLine LIKE '%[System.Net.WebRequest]::create%' ESCAPE '\\' OR CommandLine LIKE '%curl %' ESCAPE '\\' OR CommandLine LIKE '%Invoke-RestMethod%' ESCAPE '\\' OR CommandLine LIKE '%Invoke-WebRequest%' ESCAPE '\\' OR CommandLine LIKE '% irm %' ESCAPE '\\' OR CommandLine LIKE '%iwr %' ESCAPE '\\' OR CommandLine LIKE '%Resume-BitsTransfer%' ESCAPE '\\' OR CommandLine LIKE '%Start-BitsTransfer%' ESCAPE '\\' OR CommandLine LIKE '%wget %' ESCAPE '\\' OR CommandLine LIKE '%WinHttp.WinHttpRequest%' ESCAPE '\\'"
],
"filename": "",
"channel": [],
"eventid": []
},
...
]
```
What could possibly cause this? Are they using different mappings?
The only file that was modified is `fieldMappings.yaml` (I'm using `v3.1`), but that should apply to both runs in the same manner, right?
I've attached the alerts resulting from running all rules (`ALL_RULES.jsonl`, removed all other alerts) and just the single rule (`SINGLE_RULE.jsonl`), respectively. I'm a bit puzzled here.
[alerts.zip](https://github.com/user-attachments/files/25363912/alerts.zip)
When looking only at timestamps, `ALL_RULES` seems to be a subset of `SINGLE_RULE`
```sh
comm -23 <(cat ALL_RULES.jsonl | jq -r '.log.timestamp' | sort -u) <(cat SINGLE_RULE.jsonl | jq -r '.log.timestamp' | sort -u)
# no output - every timestamp in ALL_RULES can be found in SINGLE_RULES
# if we reverse this, we get 23 unique timestamps that cannot be found in ALL_RULES
comm -23 <(cat SINGLE_RULE.jsonl | jq -r '.log.timestamp' | sort -u) <(cat ALL_RULES.jsonl | jq -r '.log.timestamp' | sort -u)
```
The additional alerts still make sense in that they fulfill the condition of the rule. I just don't understand why the total alert count differs here.
1 条评论