Checkers

A checker decides whether what a participant's program printed is a correct answer for one test. It runs after that program has finished, with access to the test's input, the program's output and the stored answer, and its verdict is what turns a run into a passed or failed test.

Every problem needs one. A problem with neither a checker nor an interactor cannot grade submissions at all.

The checker is set on the problem's Testing tab, in the Checker row, which shows the current checker type — or the runtime name, when the checker is a program. You need permission to write problems.

Fig 1. The Checker dialog, with the built-in comparison types and their options.

Built-in checkers

Picking one of these saves it straight from the dialog; there is nothing to compile.

Line by line

Compares the output and the answer line by line. Trailing whitespace on a line and empty lines at the end of the file are ignored; everything else has to match exactly.

Use it when there is exactly one acceptable output and you want it produced exactly.

Tokens

Compares token by token, so whitespace and line breaks stop mattering. Numbers are compared as numbers, to a Precision you set, and strings are compared with Case sensitive on or off as you choose.

Use it when the answer is a sequence of numbers or words and you do not want to fail a correct solution over a stray space. On very large outputs it is the slower of the two built-ins.

Query results

Compares JSON-encoded query results. Order sensitive controls whether the order of rows counts: with it off, the rows of the output and the answer are sorted before comparison.

This is the checker for SQL problems.

None

No verification at all. Available in the full checker editor rather than in the quick dialog, and only sensible for a problem whose grading is done entirely by its interactor.

Custom program checkers

When correctness cannot be decided by comparing text — several valid answers, an answer that has to be verified rather than matched, partial credit — write the checker as a program. Two options in the dialog do this, and they differ in one respect only: the order of the arguments they are called with.

Custom program is the current form and is compatible with testlib, so a checker written for testlib works unchanged. Custom program (legacy) exists for checkers written against the argument order used by older E-Olymp checkers — the output and answer arguments are the other way round. Pick the wrong one and the checker will compare the participant's output against itself.

Both are also given EOLYMP=1, INPUT_FILE, OUTPUT_FILE and ANSWER_FILE in the environment, so a checker can find its files without relying on argument positions at all.

Choosing either type continues to the full-page checker editor, where you pick the runtime, add additional files and write the source.

When the checker runs

A program checker runs only for submissions that finished without a runtime error, a timeout or a memory violation. If the participant's program crashed or ran out of time or memory, that is already the verdict and the checker is never consulted — so a checker never has to cope with a truncated or missing output file.

For interactive problems the interactor runs first; the checker grades the test only if the interaction ended successfully.

Replacing a checker

The checker editor's Reset returns the problem to line-by-line comparison, after a confirmation — it does not leave the problem without a checker.

A checker marked secret, as imported problems often are, cannot be displayed. The editor offers Override, which replaces the stored checker with whatever you write; the original is gone from the console afterwards.