Read a verdict
A task shows one of three labels. Clean comes with its flags, Flawed comes with a finding and its proof, and Needs from you is no verdict.
Each task comes back with one of three labels: "Clean", "Flawed" or "Needs from you". A clean result holds for the exact files that were audited, and it lists its flags. A flawed result holds a finding with its proof. "Needs from you" is not a verdict.
The three labels
| Label | What it means |
|---|---|
| Clean | Every promise of a clean verdict held. The verdict holds for the exact files that were audited. |
| Flawed | A flaw that can change a realistic score was proved. You get the flaw, where it is, and the proof. |
| Needs from you | The task cannot be audited until you supply something. This is not a verdict, and it is never counted as clean. |
What a clean verdict proves
A clean verdict makes five promises about the task.
- No reward without the work. An agent that does nothing, or only looks busy, earns nothing.
- The task can be solved as written. The task's own solution earns full reward in the task's own environment.
- A wrong answer is refused. An answer that is close but wrong does not pass.
- A right answer is accepted. A correct answer passes, also when it does not look like the author's.
- The answer cannot be found without the work. The answer is not there to be read from the environment.
Anything else worth a look is listed beside the verdict as a flag. A flag never decides a verdict.
A clean result
A clean task shows "Clean". The result contains the verdict and the flags.
The files it holds for
The verdict holds for the exact files that were audited. Each task is pinned to a digest of its files. If the task's files change in any way, the digest changes, and the verdict does not cover the changed files.
The flags
The flags are listed beside the verdict. They are marked as worth a look, and as not deciding the verdict.
Clean is a claim about what was tested
A clean verdict does not say the task has no flaw. It says each promise above held when the task was audited.
A flawed result
A flawed result contains a finding for each flaw. An engineer confirms every flawed verdict before you see it.
Under its title, a finding gives the task digest. Then it has three parts.
| Part | What it holds |
|---|---|
| Flaw location | Where the flaw is: the file and the line, with the lines quoted |
| How we established it | Numbered steps you can repeat, each linked to its saved output |
| What this is not | What the finding does not claim |
The saved outputs are the proof. Every step of a published finding opens the output it rests on. A finding does not suggest a fix.
What a flag is
A flag is something an engineer or you may want to look at. A flag never decides a verdict.
A proved flaw that cannot change a realistic score is also listed as a flag. It comes with its finding and with the reason it does not change the verdict. Flaw families says when a flaw can change a realistic score.
Not checked is never clean
A task that could not be audited is never shown as clean. If the task needs something from you, it shows "Needs from you" and names what is needed. A task that has no verdict yet stays "In review" until it has a verdict that can be published.
Fix and resubmit says what to do after a request or a flawed verdict.