Home
Cognium Labs
Published snapshot: cognium-dev 3.19.4 · April 22, 2026  ·  Go + C#/.NET expansion: cognium-dev 4.9.13 · September 11, 2026

cognium-dev SAST Benchmarks

Static analysis only. No LLM. Published snapshot.

cognium-dev is an MIT-licensed semantic static analysis engine. These benchmarks measure the static analysis engine only—no LLM verification layer is involved. The dated summaries, machine-readable result artifacts, raw runner logs and every dataset's source and ground truth are in the cogniumhq/sast-benchmarks repository for independent review (results/2026-04-22, results/2026-09-11, datasets). Read the plain-language OWASP results analysis for the methodology and tradeoffs behind the comparison.

Results

Benchmarks by Language

Java (6 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
OWASP Benchmark * 1,415 708 707 0 0 100% 0% 100%
Juliet Test Suite 243 122 121 0 0 100% 0% 100%
SecuriBench Micro 123 60 60 1 2 96.8% 1.6% 97.7%
CWE-Bench-Java 120 61 — — 59 50.8% — 50.8%
WebGoat 29 26 — — 3 89.7% — 89.3%
DVJA 7 7 — — 0 100% — 100%

Node.js / TypeScript (3 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
NodeGoat 14 14 — — 0 100% — 100%
Juice Shop 14 14 — — 0 100% — 100%
NodeJS Synthetic 25 23 — — 2 92.0% — 92.9%

Python (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
PyGoat 26 23 — — 3 88.5% — 90.0%
DVPWA 6 6 — — 0 100% — 100%

Rust (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
Rust Synthetic 50 46 — — 4 92.0% — 92.3%
CWE-Bench-Rust 30 28 — — 2 93.3% — 94.4%

Other Languages (3 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
Bash Synthetic 31 31 — — 0 100% — 100%
HTML/JS Synthetic 30 30 — — 0 100% — 100%
Firing Range 40 35 — 2 3 92.1% — 92.1%

Legend: TP = True Positives · TN = True Negatives · FP = False Positives · FN = False Negatives · TPR = True Positive Rate · FPR = False Positive Rate

Summary

Results by Language

Language Perfect (100%) Near-perfect (90%+) Total Benchmarks
Java 3 4 6
Node.js / TypeScript 2 3 3
Python 1 2 2
Rust 0 2 2
Bash 1 1 1
HTML/JS 1 1 1
Other (Firing Range) 0 1 1
Total 8 14 16
Deep Dive

CWE-Bench-Java by Category

CWE Category Detected Missed Rate
CWE-022 Path Traversal 37 / 55 18 67.3%
CWE-078 Command Injection 6 / 13 7 46.2%
CWE-079 XSS 13 / 31 18 41.9%
CWE-094 Code Injection 5 / 21 16 23.8%
Language expansion · cognium-dev 4.9.13 · September 11, 2026

Go and C#/.NET

First scored results for the two engine-supported languages missing from the April snapshot. This is a separate dated result set on a newer engine version; it does not re-score the 16 benchmarks above, and the two tables are not merged. C#/.NET support is preview: these are its first public numbers and the Juliet recall is the baseline to improve from, not a coverage claim. Artifacts, raw runner logs and dataset ground truth: sast-benchmarks/results/2026-09-11/.

Go (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
Go Synthetic 29 15 10 0 4 78.9% 0.0% 78.9%
Vulnerability-goapp 13 3 6 1 3 50.0% 14.3% 45.0%

C#/.NET preview (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
C# Synthetic 15 10 3 1 1 90.9% 25.0% 65.9%
Juliet C# (NIST, baseline _01) 123 17 — — 106 13.8% — 13.8%

Juliet C# by CWE (baseline _01, recall)

CWE Category Detected Missed Rate
CWE-023 Relative Path Traversal 0 / 10 10 0.0%
CWE-036 Absolute Path Traversal 0 / 10 10 0.0%
CWE-078 Command Injection 0 / 10 10 0.0%
CWE-080 XSS (basic) 0 / 18 18 0.0%
CWE-081 XSS (error message) 0 / 9 9 0.0%
CWE-083 XSS (attribute) 0 / 9 9 0.0%
CWE-089 SQL Injection 8 / 27 19 29.6%
CWE-090 LDAP Injection 2 / 10 8 20.0%
CWE-094 Code Injection 0 / 10 10 0.0%
CWE-643 XPath Injection 7 / 10 3 70.0%
Total 17 / 123 106 13.8%
  • Score is defined per runner: TPR − FPR for the synthetic sets (same rule as the Bash, HTML/JS, NodeJS and Rust rows above); a per-category mean of TPR − FPR for Vulnerability-goapp; recall only for Juliet C#, whose _01 baseline files are all vulnerable, so no false-positive rate is measurable on it
  • Vulnerability-goapp is scored engine-only. The harness runner's regex fallback (which would lift it to 70.0%) was disabled for the published row; both logs are in the repository
  • Juliet C#: NIST SARD suite 110 (Juliet C# 1.3), archive SHA-256 pinned in datasets/juliet-csharp; 123 baseline files across the 10 CWE directories the engine models for C#. Vulnerability-goapp is pinned to commit 6e51a89
Tool comparison · OWASP Benchmark Java v1.2 · full suite

OWASP Benchmark, all 2,740 cases, one rule for every tool

The comparison lane. Every tool is scored on the full OWASP Benchmark v1.2 (2,740 cases: 1,415 vulnerable, 1,325 safe; BenchmarkJava 20cbf3d) with the official scorecard rule — a case is flagged when a finding lands in its file with exactly the expected CWE — by one open scorer, score-owasp-benchmark.mjs, with each tool's reported CWEs normalized onto the Benchmark categories exactly as OWASP's own BenchmarkUtils does. Youden = TPR − FPR is the OWASP "score". Every tool is shown at its best-scoring standard configuration; the other configurations tried are published alongside. Numbers from papers or vendor pages never enter this table.

Results · September 11, 2026

Tool Version TP FP FN TN TPR FPR Precision Youden
cognium-dev 4.9.13 1,287 231 128 1,094 91.0% 17.4% 84.8% 73.5%
CodeQL (java-security-extended) 2.27.0 1,415 531 0 794 100.0% 40.1% 72.7% 59.9%
Semgrep OSS (p/java + p/security-audit) 1.177.0 1,224 512 191 813 86.5% 38.6% 70.5% 47.9%
SonarQube Community planned — added only when run on the same revision with the same scorer
SpotBugs + Find-Sec-Bugs planned — added only when run on the same revision with the same scorer

cognium-dev 4.9.13 by category

Category CWE Cases Vuln TP FP FN TN TPR FPR Precision Youden
cmdi CWE-78 251 126 113 39 13 86 89.7% 31.2% 74.3% 58.5%
crypto CWE-327 246 130 130 0 0 116 100.0% 0.0% 100.0% 100.0%
hash CWE-328 236 129 89 0 40 107 69.0% 0.0% 100.0% 69.0%
ldapi CWE-90 59 27 24 0 3 32 88.9% 0.0% 100.0% 88.9%
pathtraver CWE-22 268 133 116 0 17 135 87.2% 0.0% 100.0% 87.2%
securecookie CWE-614 67 36 33 0 3 31 91.7% 0.0% 100.0% 91.7%
sqli CWE-89 504 272 255 0 17 232 93.8% 0.0% 100.0% 93.8%
trustbound CWE-501 126 83 58 0 25 43 69.9% 0.0% 100.0% 69.9%
weakrand CWE-330 493 218 218 0 0 275 100.0% 0.0% 100.0% 100.0%
xpathi CWE-643 35 15 14 0 1 20 93.3% 0.0% 100.0% 93.3%
xss CWE-79 455 246 237 192 9 17 96.3% 91.9% 55.2% 4.5%
  • * About the 1,415-case OWASP row above. The April snapshot's row (100% / 0% FP) was produced by the historical harness on a case selection that cannot be reconstructed from whole OWASP categories, and that harness is not public. It is kept as published, but it is not comparable with any other tool's number. The full-suite, official-rule result — 91.0% TPR / 17.4% FPR / Youden 73.5 — is the OWASP figure to use in comparisons.
  • Nine of eleven categories have zero false positives. XSS is the weakness: 192 of 209 safe XSS cases are flagged (FPR 91.9%) because the engine does not credit OWASP's output-encoding controls — 192 of the 231 false positives come from this one category. cmdi FPR is 31.2%. Both are engine work items, and the FP case lists are published.
  • CodeQL java-security-extended finds every vulnerable case (100% recall) while flagging 40% of the safe ones (sqli FPR 89.2%); its default java-code-scanning suite scores 82.0% / 24.1% (Youden 57.9). A rule declaring several CWEs is credited for each — under OWASP's literal first-tag rule CodeQL would score 53.3, and that number is published too. Semgrep OSS is pattern-based: 100 / 0 on crypto, weakrand and securecookie, but FPR above 75% wherever data flow decides (cmdi, ldapi, pathtraver); p/default scores 46.5, p/java alone 29.9. SonarQube Community and SpotBugs + Find-Sec-Bugs are next.
  • Run: cognium-dev scan <testcode> -l java -f json, default configuration, no LLM, 13 s wall-clock. Raw scan, normalized findings, scorecards and the per-case FN / FP lists: sast-benchmarks/results/2026-09-11-owasp-java-comparison/
Methodology

How we measured

  • cognium-dev is a semantic static analyzer with inter-procedural taint tracking
  • All results are from static analysis only—no LLM involvement in detection or verification
  • Each benchmark's source dataset is linked to its origin: OWASP Benchmark, NIST Juliet Test Suite, CWE-Bench-Java, and others
  • CWE-Bench-Java uses per-project binary detection: each project contains one CVE, scored as detected or not
  • The published summary, CSV, JSON, and CWE breakdown are in cogniumhq/sast-benchmarks/results/2026-04-22/ (April snapshot) and results/2026-09-11/ (Go and C#/.NET)
  • Every benchmark row maps to a folder under datasets/ recording its source, revision, acquisition command and ground truth; the label files for the real-application suites and the full source of every synthetic set are committed there
  • Score, "perfect" (100%) and "near-perfect" (90%+) are defined per benchmark kind in benchmarks/static-analysis-suite/README.md; a dash means the dataset has no scored negatives, not zero
Limitations

Known gaps

  • SSTI (Server-Side Template Injection) is not currently in cognium-dev's CWE coverage—this causes the PyGoat false negative
  • Firing Range has 2 false positives in the escape/ category (escaped output flagged) and 3 false negatives in cors/ (CORS misconfigurations not detected)
  • CWE-Bench-Java uses per-project detection, not per-CVE counts—a single missed sink in a complex project counts as a full miss
  • These benchmarks test static analysis only—the full cognium-dev + LLM verification pipeline (SAST+LLM) produces different results, published separately
  • Go: XSS through fmt.Fprintf / io.WriteString and SQL executed through a shell client are not detected—these account for most Go misses
  • C#/.NET is preview: Juliet C# recall is 13.8% on the _01 baseline (0% on path traversal, command injection, XSS and code injection), only the baseline variant was scored, and no C# false-positive rate has been published yet
  • The historical one-command benchmark harness for the April snapshot is not present in the current source tree, so that result set is auditable but not currently reproducible with a single published command. The Go and C#/.NET runs used per-suite runner scripts against the published [email protected] npm package; the steps are in the repository, the runner scripts are not yet public
Audit

Inspect the published snapshot

Each result set includes a human-readable summary plus JSON and CSV artifacts; the September 11, 2026 set also carries the raw runner logs. A fresh full rerun of the April set remains blocked until the historical benchmark harness is restored and published.

git clone https://github.com/cogniumhq/sast-benchmarks
cd sast-benchmarks
ls results/2026-04-22        # summary.md results.csv results.json cwe-bench-java-breakdown.csv
ls results/2026-09-11        # summary.md results.csv results.json juliet-csharp-breakdown.csv
ls raw/2026-09-11 datasets   # runner logs; per-benchmark dataset source + ground truth

Review the published result artifacts, the dataset records, and the current rerun status.