Short version: TPR tells you how many known vulnerabilities a scanner catches. FPR tells you how often it flags safe cases. Precision tells you how many reported findings are actually vulnerable. You need all three—and the underlying counts—to understand a SAST result.
Start with four counts
- True positive (TP): vulnerable code correctly reported.
- False negative (FN): vulnerable code the scanner missed.
- False positive (FP): safe code incorrectly reported.
- True negative (TN): safe code correctly left unflagged.
A test suite needs both vulnerable and safe examples. Without safe cases, false-positive rate cannot be measured and a scanner can look good simply by reporting everything.
True-positive rate measures recall
True-positive rate answers: “Of the known vulnerable cases, what percentage did the scanner detect?” A high TPR reduces missed vulnerabilities, but it says nothing about how much safe code was also flagged.
False-positive rate measures noise on safe cases
False-positive rate answers: “Of the known safe cases, what percentage did the scanner flag?” This denominator is important. FPR is not the percentage of all alerts that are false.
Precision measures alert quality
Precision answers: “Of everything reported by the scanner, what percentage is a known vulnerable case?” In a controlled benchmark it helps estimate alert quality. In production, precision also depends on the prevalence and mix of vulnerabilities in the codebase.
Youden score balances TPR and FPR
OWASP Benchmark scorecards use the difference between true-positive rate and false-positive rate. A tool improves the score by finding more vulnerable cases without indiscriminately flagging safe ones.
A same-suite example
The following dated results use the same 2,740-case OWASP Benchmark Java revision and the same open scorer. They are configuration-specific measurements, not permanent product rankings.
| Tool and configuration | TPR | FPR | Precision | Youden |
|---|---|---|---|---|
| cognium-dev 4.9.13 | 91.0% | 17.4% | 84.8% | 73.5% |
| CodeQL 2.27.0, java-security-extended | 100.0% | 40.1% | 72.7% | 59.9% |
| Semgrep OSS 1.177.0, p/java + p/security-audit | 86.5% | 38.6% | 70.5% | 47.9% |
CodeQL's tested extended suite found every vulnerable case, but it also flagged 40.1% of safe cases. Cognium missed more vulnerable cases but produced fewer false positives in this run. Semgrep's tested rules had lower TPR and a similar FPR to the CodeQL configuration. A team may value those tradeoffs differently depending on whether it is exploring a repository, gating pull requests, or auditing a release.
Why false positives cost more than one review
A noisy finding consumes triage time, interrupts feature work, and can weaken confidence in future alerts. But eliminating all false positives by reporting almost nothing is not useful either. The goal is an operating point that fits the workflow.
For pull-request gates
Favor high-confidence, high-severity rules. A small number of actionable findings is easier to enforce consistently than a broad stream of uncertain warnings.
For scheduled audits
A broader ruleset may be acceptable because reviewers have more time to investigate. Track confirmed findings, dismissed findings, and recurring noisy patterns so configuration improves over time.
For benchmark evaluation
Inspect per-category results. An acceptable overall number can hide a severe weakness in one category. In Cognium's dated run, XSS produced 192 of the 231 false positives even though nine other categories produced none.
A practical SAST evaluation checklist
- Choose datasets that contain both vulnerable and safe cases.
- Record exact versions and configurations for the dated run.
- Measure TP, FP, FN, and TN—not only total findings.
- Break results down by vulnerability category and language.
- Review a sample of raw paths and remediation guidance.
- Test the intended workflow: local feedback, CI gate, or audit.
- Re-run after meaningful engine or rule changes.
See the counts behind the score.
Open the full comparison and category tables, then try the same engine on your own source code.
Source note: Version numbers are retained here only to identify the dated benchmark configurations. Installation and CI documentation continue to follow the latest public cognium-dev release.