Thirty testing conditions. Circles are Medium effort, triangles are XHigh, lines join the same condition, and ellipses are joint 50 percent paired-bootstrap uncertainty. Out-of-memory runs are excluded.
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
Cost
Perfect (%)
Default
Default
built-in
built-in
Proptest
Proptest
Hegel
Hegel
TDD
TDD
PBT
PBT
Fuzzing
Fuzzing
Audit
Audit
Differential
Differential
Metamorphic
Metamorphic
Mutation
Mutation
Judgement
Judgement
QuickCheck
QuickCheck
rstest
rstest
Insta
Insta
Fuzz risky
Fuzz risky
Make no mistakes
Make no mistakes
Kani
Kani
Verus
Verus
Creusot
Creusot
Lean 4
Lean 4
ACL2
ACL2
Alloy
Alloy
TLA+
TLA+
Spin
Spin
SMT
SMT
Hegel Skill
Hegel Skill
Skill
Skill
ECC
ECC
ToB
ToB
Medium
XHigh
overall
Thirty testing conditions. Circles are Medium effort, triangles are XHigh, lines join the same condition, and ellipses are joint 50 percent paired-bootstrap uncertainty. Out-of-memory runs are excluded.
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
Cost
Perfect (%)
Default
Default
built-in
built-in
Proptest
Proptest
Hegel
Hegel
TDD
TDD
PBT
PBT
Fuzzing
Fuzzing
Audit
Audit
Differential
Differential
Metamorphic
Metamorphic
Mutation
Mutation
Judgement
Judgement
QuickCheck
QuickCheck
rstest
rstest
Insta
Insta
Fuzz risky
Fuzz risky
Make no mistakes
Make no mistakes
Kani
Kani
Verus
Verus
Creusot
Creusot
Lean 4
Lean 4
ACL2
ACL2
Alloy
Alloy
TLA+
TLA+
Spin
Spin
SMT
SMT
Hegel Skill
Hegel Skill
Skill
Skill
ECC
ECC
ToB
ToB
Medium
XHigh
overall
Thirty testing conditions. Circles are Medium effort, triangles are XHigh, lines join the same condition, and ellipses are joint 50 percent paired-bootstrap uncertainty. Out-of-memory runs are excluded.
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
Cost
Perfect (%)
Default
Default
built-in
built-in
Proptest
Proptest
Hegel
Hegel
TDD
TDD
PBT
PBT
Fuzzing
Fuzzing
Audit
Audit
Differential
Differential
Metamorphic
Metamorphic
Mutation
Mutation
Judgement
Judgement
QuickCheck
QuickCheck
rstest
rstest
Insta
Insta
Fuzz risky
Fuzz risky
Make no mistakes
Make no mistakes
Kani
Kani
Verus
Verus
Creusot
Creusot
Lean 4
Lean 4
ACL2
ACL2
Alloy
Alloy
TLA+
TLA+
Spin
Spin
SMT
SMT
Hegel Skill
Hegel Skill
Skill
Skill
ECC
ECC
ToB
ToB
Medium
XHigh
overall
Thirty testing conditions. Circles are Medium effort, triangles are XHigh, lines join the same condition, and ellipses are joint 50 percent paired-bootstrap uncertainty. Out-of-memory runs are excluded.
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
Cost
Perfect (%)
Default
Default
built-in
built-in
Proptest
Proptest
Hegel
Hegel
TDD
TDD
PBT
PBT
Fuzzing
Fuzzing
Audit
Audit
Differential
Differential
Metamorphic
Metamorphic
Mutation
Mutation
Judgement
Judgement
QuickCheck
QuickCheck
rstest
rstest
Insta
Insta
Fuzz risky
Fuzz risky
Make no mistakes
Make no mistakes
Kani
Kani
Verus
Verus
Creusot
Creusot
Lean 4
Lean 4
ACL2
ACL2
Alloy
Alloy
TLA+
TLA+
Spin
Spin
SMT
SMT
Hegel Skill
Hegel Skill
Skill
Skill
ECC
ECC
ToB
ToB
Medium
XHigh
overall