Testsets and scoring

A testset is a group of tests that share limits, a scoring rule and a feedback rule. Testsets are how a problem is broken into subtasks: a first testset for the small cases, a second for the full constraints, each worth its own points and each judged on its own terms.

Every test belongs to a testset, so even a problem with a single flat set of tests has one. Testsets are edited from Testing → Tests: Add testset creates one, and clicking a testset's header row opens it. You need permission to write problems.

Fig 1. The testset editor: index, scoring mode, feedback policy, dependencies and the limit override.

Index

The Index is the testset's position within the problem, and it is also its handle: dependencies refer to a testset by index, not by name. Changing indices therefore changes what other testsets depend on.

Scoring mode

The scoring mode decides how the testset turns its tests' scores into a score for the testset:

Mode	Score
Sum of all tests	The sum of the scores of the tests that passed.
Sum if all pass	The same sum, but only if every test passed — otherwise nothing.
Lowest score	The lowest score awarded across the tests.
Highest score	The highest score awarded across the tests.

Sum if all pass is the usual subtask rule: the subtask is worth its points only when it is solved completely. Sum of all tests gives credit test by test. The tests card shows each testset's computed score on its group header, so you can see what the mode adds up to.

Feedback policy

The feedback policy decides what a participant is told about the tests in this testset:

Policy	Shown
Show each test individually	Every test's result.
ICPC format	Only the first test that was not accepted.
ICPC with test number	The same, plus the number of the failing test.

The policy is per testset, so a problem can show everything about its sample testset and almost nothing about the rest. What participants see while solving covers the consequences in full.

Dependencies

A testset can be made to wait on others. Dependencies lists the testsets it depends on, by index — its own index is excluded from the picker — and Dependency mode says how strictly:

All dependencies must pass — every listed testset has to be fully accepted.
Any dependency with at least one point — one listed testset scoring anything is enough.

If the condition is not met, the testset is not run. This is how you avoid spending judge time running the hard subtask for a solution that already failed the easy one.

Limits

Override limits for this testset reveals CPU limit, Time limit and Memory limit for these tests. When a testset overrides the limits, it is the testset's values the judge applies — the problem-level limits in Testing → Configuration are the defaults for testsets that do not override them.

Unchecking the box clears all three, so the problem-level defaults apply again.

A new testset starts with a CPU limit of 1000 ms, 64 MB of memory, All dependencies must pass, Show each test individually and Sum if all pass.

Saving and deleting

The editor writes the whole testset on save, so a field you clear is cleared rather than left as it was.

Deleting a testset deletes every test inside it — the confirmation says so. It also leaves any other testset that depended on this index pointing at nothing, so check the rest of the problem afterwards. Problem versions that were already published keep the testset as it was.

Whether a testset ran, and what it scored, appears in the per-testset breakdown of each submission; see Statuses and verdicts for what the results mean.