Designing a testset

A generator produces one test; this page is about the layer above — how the tests are grouped,scored and proved to discriminate. Writing the generator itself isWrite a test generator; what data to put in it isTest shapes.

A testset is the unit of scoring. Every configuration choice below changes what asubmission is worth, so none of them are cosmetic.

1. Layout

Testsets are indexed from 0 and the layout is fixed:

Examples belong in their own testset because they are the only tests participants can see, sotheir feedback policy and score necessarily differ from every other testset. Mixing them intoa scored one either leaks scored data or pays marks for samples.

Subtask numbering is shared across the problem: "Subtask 2" in the statement is testset 2 andis covered under that number in the editorial.

2. Scoring mode

scoringMode on the testset takes one of these values — the names are the API's, and aguessed min or sum is rejected:

(NO_SCORE is the enum's zero value and means no score is awarded at all. The examples arenot scored NO_SCORE: they are an ordinary EACH testset whose tests carry 0 points, withCOMPLETE feedback, which is what every problem in this archive does.)

A subtask is WORST, not EACH. A subtask means "your solution handles this class ofinput"; passing 9 of 10 tests in it has not demonstrated that, and EACH would pay 90% for asolution that is wrong.

ALL and WORST are not interchangeable, and the difference is what a partial score isworth. ALL pays only when every run was ACCEPTED, so a partially scored run collapses thewhole testset to zero. WORST takes the minimum of the scores, so a partial score survives asa partial score. Any subtask whose checker or interactor can award a fraction must beWORST — see §3 for the value each test must then carry.

Use BEST only when the problem genuinely rewards the best of several attempts — anoptimisation task scored by quality rather than correctness.

The EACH + ICPC-feedback trap

Under ICPC feedback the judge stops a testset at its first zero-scoring run. An EACHtestset then silently loses everything after the first failure: the remaining tests never run,so they score nothing, and the total is wrong rather than merely low.

This is invisible — the submission gets a plausible number and no error. Audit for it ratherthan trusting it: it was found on 40 testsets across this archive in one sweep. Give an EACHtestset COMPLETE feedback. Under WORST the interaction is worse than harmless in one wayworth knowing: a skipped run is not counted, so the minimum is taken over the tests that ran.

3. The arithmetic: everything adds to 100

A problem is worth 100 points. Three separate sums have to be right, and only the first isusually checked.

Across testsets — the scored testsets total 100:

testset 0 (examples)   0
testset 1 (subtask 1)  20
testset 2 (subtask 2)  25
testset 3 (subtask 3)  25
testset 4 (subtask 4)  30
                      ---
                      100

The examples score 0. They are shown to participants, so paying for them gives marks forcopying the statement.

Within a testset, the arithmetic depends on the scoring mode, and this is where it goeswrong:

A subtask worth 20 with 10 tests scored WORST needs each test set to 20, not 2. Set themto 2 and the subtask pays 2 — a solution that passes everything scores a tenth of what thestatement promised, and nothing reports an error.

The zero-cost free pass

A run that earns points equal to its test's cost is recorded ACCEPTED — and on a test worth0, any partial score does that. min(cost, points) is 0, which is that test's full cost.

So the layout "the subtask's whole value on test 1, 0 on every other test", which problemsimported from other judges often carry, is a trap the moment the checker or interactor canaward a fraction: every test after the first passes whatever it scored, and under ALL thesubtask pays in full for a solution that was only partially correct. It was worth 60 of 100points on one IOI problem in this archive.

Two rules keep it from firing, and a partially scored subtask needs both:

• the mode is WORST, not ALL;

• every test carries the subtask's full value, so no test costs 0.

A pass/fail checker — one that never awards a fraction — is unaffected, because it neverproduces a partial score to be rounded up.

A scoring program never hard-codes any of this. With eolymp.h, eo::score takes afraction of the test and reads the cost itself (Write a checker);a testlib checker reads TEST_COST from the environment. Either way the weights stay in thetestset configuration, so the same program is correct under every mode and at whatever valuethe testset ends up carrying.

The statement must agree. Its \Scoring block lists the same point values; if theydisagree, the testsets are what the judge pays and the statement is what the participantplanned around.

4. Feedback policy

Decides whether a participant sees per-test results or only the testset's total. Set itdeliberately per testset:

• Examples (testset 0) — full feedback; participants already have this data.

• Scored subtasks during a live contest — usually the total only. Per-test feedback on asubtask tells a participant exactly which shape broke them, which is a hint the problem didnot intend to give.

• After the contest / in an archive — per-test feedback is more useful than protective.

5. Dependencies and nesting

A testset can depend on others, and is then evaluated only if those were fully accepted. Tworules follow:

• Declare dependencies when the subtasks nest. If subtask 3's constraints are a strictsubset of subtask 5's, a solution that fails 3 cannot legitimately pass 5, and thedependency saves the judge the work of proving it.

• Nesting must be real. If the statement implies subtask 3 ⊂ subtask 5, the validatorhas to enforce it — a test that is valid for 3 must be valid for 5. Check the two groupbound-sets against each other, not just each against the statement.

Dependencies are a list of testset indices, not ids.

6. Limits

Do not ship the placeholder limits the problem was created with.

Run the reference solution against the real tests, read its actual timeUsage, cpuUsage andmemoryUsage from the resulting submission, and set the limits from that with headroom —conventionally 2–3× the reference solution's time. A limit derived from a guess either lets awrong-complexity solution through or fails a correct one on the judge's hardware.

Limits are per testset, so a subtask with smaller inputs can carry a smaller limit. That isworth doing when a subtask exists specifically to separate complexities.

7. How many tests, and where

There is no correct count, but there is a shape:

• Examples: 1–3. Enough to make the format unambiguous, few enough to read.

• A subtask: enough to cover its constraint's edge cases and its maximum — typically10–30. A subtask with two tests is not testing a class of input, it is testing two inputs.

• The main testset of a problem without subtasks: 20–50.

Distribute by shape, not by count: every subtask should contain its minimum case, itsmaximum case, and each degenerate structure the constraint still admits(Test shapes). Ten tests covering ten shapes beat fifty covering three.

Order tests so the cheap and discriminating ones come first. Under a stop-at-first-failurefeedback policy, a failing submission then reports the smallest case that breaks it, which isthe one a participant can actually debug.

8. Prove the suite discriminates

A test suite that every wrong solution passes is worth nothing, and this is the step mostoften skipped.

1. The reference solution scores 100. check_solutions after generation; anything lessmeans the tests, the checker or the solution disagree.

2. Each wrong approach fails, and fails where intended. Write a solution that embodies aplausible mistake — the wrong complexity, the missing edge case, the greedy that almostworks — attach it, and confirm it scores what you expect. A brute force should pass thesmall subtasks and time out on the large ones; if it passes everything, the large subtasksare not large enough.

3. Every subtask is reachable and losable. Each should have at least one attached solutionthat earns it and one that does not. A subtask no solution has ever failed is not testinganything.

4. The maximum tests are actually maximal. Compare the generated file against thestatement's stated bound rather than assuming the generator honoured it.

9. Review checklist

BLOCKER scores wrongly · MAJOR weakens the problem · MINOR tidiness.

• [ ] BLOCKER examples in testset 0, flagged example: true, worth no points

• [ ] BLOCKER subtasks scored WORST, not EACH; any subtask that can be partially scored is WORST, never ALL

• [ ] BLOCKER testset indices match the statement's subtask numbers

• [ ] BLOCKER scored testsets total 100; examples score 0

• [ ] BLOCKER under WORST, every test carries the testset's full value — no test in a partially scored subtask costs 0

• [ ] BLOCKER under EACH or ALL, the tests' scores total the testset's value

• [ ] BLOCKER the statement's \Scoring values match the testsets'

• [ ] BLOCKER no EACH testset under ICPC feedback

• [ ] BLOCKER the reference solution scores 100

• [ ] MAJOR limits set from a measured run, not a placeholder

• [ ] MAJOR declared nesting is enforced by the validator

• [ ] MAJOR every subtask contains its minimum, its maximum and its degenerate shapes

• [ ] MAJOR at least one attached solution fails each subtask

• [ ] MINOR feedback policy chosen deliberately per testset

• [ ] MINOR cheap, discriminating tests ordered first