The Non-Determinism Testing Prompt

How do I test a system that gives different answers each run?

System: [DESCRIBE]
Observed variance: [DESCRIBE WHAT CHANGES BETWEEN RUNS]
What must be consistent: [DESCRIBE]

Produce:
1. What properties should hold on every run, regardless of path
2. How to assert those properties rather than exact outputs
3. Where to test deterministically instead — the parts of the
   system that are or could be deterministic
4. Statistical approaches: run N times, assert on the distribution
5. What N needs to be to detect a real change
6. How to distinguish a regression from normal variance
7. Snapshot testing: what is stable enough to snapshot
8. Where to inject determinism for testing purposes without
   changing production behaviour

Then produce a concrete test strategy: what runs on every commit,
what runs nightly, what runs before a release.

Be realistic about cost. A 200-case eval run against a large model
is not something you run on every commit.