Testing¶
The local-first test system: taxonomy, commands, fixtures, and coverage.
PySymGym uses a local-first pytest setup. The fast unit tier — no network,
GPU, .NET, or built binary fixtures — runs across every component in one
command (make test-unit or poetry run pytest -m "not integration and
not e2e"). Slower tiers are selected explicitly and the full end-to-end
pipeline stays in CI.
This page is the single source of truth for the testing system; the developer guide and the agent skills point here.
Test pyramid¶
The suite is layered so most checks run fast and locally:
Unit (the base): pure, deterministic tests of isolated logic. No network, GPU, subprocess, or binary fixtures. This is the default tier and must run in seconds.
Integration (the middle): in-process tests that wire real components together with fakes and golden fixtures (for example the ONNX export, the pc-remover graph transform, and the compstrat pipeline).
E2E (the top): the full pipelines that need built servers, maps and the
.NETtoolchain; they run in the existing self-hosted CI workflows.
A change should add the lowest tier that can catch its regression; the higher tiers stay in CI where the required infrastructure is available.
Test taxonomy¶
Tests are classified with pytest markers:
unitPure, deterministic tests with no network, GPU, subprocess, or binary fixtures.
integrationIn-process tests that use fakes and golden fixtures.
e2eFull pipeline tests that need built servers, maps, or the
.NETtoolchain.gpuRequires a CUDA device. Excluded from the gated tier (see “Coverage”), so the tier runs identically with or without a GPU; keep a runtime guard (for example
@pytest.mark.skipif(not torch.cuda.is_available())) for when the test is selected explicitly on a CUDA-less machine (make test-all,pytest -m integration).TestForwardOnCUDAis the example.networkRequires network access. Excluded from the gated tier (see “Coverage”).
slowTakes noticeably long to run.
serialMust not run in parallel with other tests. The suite runs serially in a single process (pytest-xdist is deliberately not a dependency), so the marker is a documented convention, not a runtime lock: it names the modules that must not be parallelized (they spawn processes, bind ports, or mutate global state) and constrains any future parallelization. The guard test in
tests/test_pytest_infra.pyfails if xdist is ever enabled, forcing a scheduling decision for the serial modules first.
Tier selection is explicit at the invocation: the Makefile targets and the CI
steps pass -m, and addopts carries only --import-mode=importlib —
so a node-ID invocation (pytest path::test) is never filtered by tier.
Commands¶
poetry run pytest -m "not integration and not e2e" # fast unit tier
poetry run pytest -m integration
poetry run pytest -m e2e
poetry run pytest # every tier (e2e needs the built toolchain)
The same tiers are wrapped in a root Makefile so nobody has to memorize the
invocations:
make test-unit # fast unit tier
make test-integration # -m integration
make test-all # everything (e2e needs the built toolchain)
make test-cov # gated tier (unit + integration) with the coverage ratchet
The configuration lives in [tool.pytest.ini_options] in the root
pyproject.toml.
To run a single component or a single test (a node-ID invocation is not filtered by tier — the named test runs whatever tier it carries):
poetry run pytest AIAgent/tests/unit/test_fixtures.py
poetry run pytest AIAgent/tests/integration/test_onnx.py::TestONNXConversion -sv
Local e2e runs¶
The e2e tier is reproduced locally as part of the quality gate. The game
servers must run on the same .NET runtime the dataset was generated
with: the runtime version changes the game server’s path-condition
encoding (the model’s input), so a model validated against a different
runtime fails. The prerequisite is a .NET 8 runtime on the machine
with dotnet on the PATH resolving to it.
The broker pins the runtime at launch: it sets
DOTNET_ROLL_FORWARD=LatestPatch in the game server’s environment, so
the server runs on the major.minor its runtimeconfig.json declares
(the highest installed patch of it) or fails to start at all — never on a
newer major version. A mismatched ambient runtime therefore fails loudly
at server start — the port never opens and the broker refuses to issue
the instance — instead of silently changing the encoding. The pin
covers only the broker-launched servers. Two .NET invocations in the
local e2e flow are not broker-launched and still run under the ambient
environment: the dataset-generation step (a plain dotnet invocation in
the shell) and the model-validation pipeline’s game server (svms_model
launches it directly from the training process, not through the broker).
Run both under env -u DOTNET_ROLL_FORWARD.
Layout¶
Each component keeps its own test tree, split by tier into a directory named after the marker its tests carry:
unit/— the fast, isolated tests (AIAgent/tests/unit/,tools/*/tests/unit/).integration/— the in-process tests that need golden fixtures (AIAgent/tests/integration/,tools/compstrat/tests/integration/).e2e/— the full pipelines (tools/runstrat/tests/e2e/).
A component only creates the tier directories it needs. The tests/ tree at
the repository root holds the cross-component infrastructure tests, including
the static guards: besides the marker taxonomy (test_pytest_infra.py),
test_import_boundaries.py pins the import boundary of the shared layer —
no module under AIAgent/common may import ml.* or the top-level
config module — and test_config_base_guard.py pins the shared
pydantic config — every @pydantic_dataclass under AIAgent/common
takes its config from common.config.base.CONFIG. Both guards are AST
walks that never import the guarded code.
The marker is the source of truth, not the directory: CI selects markers
explicitly (-m "not e2e and not gpu and not network" for the gated-tier
ratchet step, -m e2e for the runstrat job). A module may declare its tier
once at module level (pytestmark = pytest.mark.unit) or per function;
every test_* function must resolve to exactly one tier, which
tests/test_pytest_infra.py enforces statically together with the tier
directory (unit/, integration/ or e2e/) it lives in.
Fixtures¶
Shared fixtures live in a conftest.py next to the component they serve: the
AIAgent/conftest.py fixtures are visible to every test under AIAgent/,
and similarly for tools/compstrat/ and tools/runstrat/.
AIAgent/conftest.pycpu_device(autouse) forcesml.device.get_device()to CPU,seeded_rng(autouse) seedsrandom/numpy/torch,gamestate_factorybuilds a syntheticGameState,hetero_factoryconverts it through the productionconvert_input_to_tensor(with feature-width overrides),step_factorybuilds a processed step (thehetero_factorygraph with a giveny_true),serial_poolreplacesmultiprocessing.Poolwith an in-process serial stand-in (forked workers are invisible to pytest-cov),tmp_datasetyields an emptyTrainingDatasetin a temporary directory,fake_websocketreturns a recording, no-socket websocket for the connection tests, andfake_namespace/fake_procprovide the game managers’ multiprocessing namespace and asubprocess.Popenstand-in.tools/compstrat/conftest.pyResource-directory and mock-run DataFrame fixtures.
tools/runstrat/conftest.pyTemporary artifacts directory and the built
ForTestsmap paths.AIAgent/tests/integration/conftest.pyresources_direxposes the absoluteintegration/resources/directory, and itspytest_generate_testshook parametrizes the config-driven tests (the ONNX model configs and the SVMs validation configs) from that directory, so no test computes a resource path itself.
tools/dataset_tools and tools/util now have tests/unit/ roots as
well. They need no conftest of their own: the root conftest.py puts every
tool directory on sys.path, so clean, generate_episodes and
parse_pretty import as top-level modules.
AIAgent/tests/unit/ml/ is the unit root for the ML layer (dataset
transforms, training and experiment helpers, model-forward smoke tests) and
AIAgent/tests/unit/ml/validation/ its orchestration subtree (game managers
and the coverage-validation flow). They reuse the AIAgent/conftest.py
fixtures and build every graph input through
ml.dataset.convert_input_to_tensor so the tests cannot drift from the
production tensor schema. The dataset’s domain types (Result,
TrainingDatasetMode) live in common.dataset and the graph tensor
feature widths in ml.schema; ml.dataset re-exports both, and the
lifecycle tests pin that the re-exports are the same objects. The dataset
code itself is the ml/dataset/ package (processing: the
TrainingDataset class; tensors: domain→tensor conversion;
maintenance: the online-maintenance mixin), and the lifecycle tests pin
that TrainingDataset is defined in ml.dataset.processing. The
transforms tests (test_dataset_transforms.py) pin the malformed-input
branches of convert_input_to_tensor by mutating a gamestate_factory
copy (exact error type and message) rather than hand-rolled gamestates: a
missing state child raises the KeyError keyed by the missing child Id.
The get_hetero_data single-row contract (the model’s one output tensor
flattened to one row of per-state values) is pinned both ways: the one-row
input becomes the [S, 1] y_true column and a multi-row input raises
ValueError. filter_map_steps enforces the same [S, 1] shape on
the steps it receives (one logit per state, which is what makes the global
argmax the winning row): a wide y_true raises ValueError, while
single-row and all-NaN steps are dropped and the remaining steps are
one-hot-encoded. The TrainingDataset lifecycle tests
(test_dataset_lifecycle.py) cover the in-memory (load_to_cpu) and
on-disk loading paths, the train/validation index switching, and the
metadata refresh; they pin the stale len() before the first
switch_to and drive the auto-processing run at the end of
__init__ through the serial_pool fixture. The processing and
maintenance tests (test_dataset_maintenance.py) build the raw dataset
layout (per map a result file plus <id>_gameState /
<id>_movedState step files) and drive process() through the
serial_pool fixture as well; a double-process() pin verifies that a
re-run over an already-processed map is idempotent (the result file and
the saved step are unchanged). The degenerate-step filter (#619) is pinned
there too: a step whose converted graph has zero states or zero game
vertices is skipped (reusing ml.inference.degenerate_counts) with a
warning naming the map and step, while a zero-path-condition step is saved
as usual. They also walk the update_map branch
matrix (new map, below-threshold result-only save, merge, reproduced,
replace, and the two no-op cases) with step_factory steps, pinning the
exact logging.info messages via caplog (including the trailing space of
the merge/reproduce messages) and the load_to_cpu cache bookkeeping.
They also exercise _get_map_steps (sorted .pt load, result
excluded, default device) and _merge_steps directly (graph-match merge
math, unique-step append order, new vertex-count groups). The
merge_distributions device-mismatch branch (3 statements) is not
coverable on a CPU-only machine — meta tensors cannot be copied out — so
ml/dataset/maintenance.py misses exactly those 3 statements. The
protocol tests
(test_protocols.py) pin the ml.protocols abstract interfaces:
abstractness, the abstract-method set, a concrete subclass satisfying the
interface, the abstract bodies raising NotImplementedError when invoked
unbound, and the Model forward contract (the exact 11-parameter
forward signature infer calls). The registry tests
(test_registry.py) pin ml.models.registry: name lookup, the
unknown-name ValueError listing the known models, reverse lookup by
instance and class, and the forward-contract pin (every registered model’s
forward signature equals Model’s). The training-loop tests
(test_train.py) pin
ml.training.train with a real small model, optimizer and criterion: the
PYTORCH_CUDA_ALLOC_CONF expandable-segments switch, the y_true
clamping to {0, 1} before the criterion sees it, the per-batch optimizer
step, and the skip of backward/step when the criterion returns zero. The
degenerate-graph regression tests (test_degenerate_graphs.py, #619)
mutate a gamestate_factory copy into zero-path-condition, zero-states
and zero-vertices variants and pin: convert_input_to_tensor emitting
2-D (0, F) node tensors for the empty node types, degenerate_counts
reporting both counts for a degenerate graph and accepting a zero-PC graph
(a state with no path condition is valid), infer returning (0, 1)
or the neutral zeros without calling the model and logging a warning with
both counts, the zero-PC graph forwarding through the real small model to
a finite output (the exact #619 crash input), and train skipping a
degenerate step as a no-op (bit-identical weights) while still training a
normal step in a mixed dataloader. The inference
tests (test_predict_wrapper.py) pin ml.predict.predict_state_with_dict
with a fixed-output fake model (the exact max-row-sum selection, the
state-map remapping, the no_grad output) plus one real-model run, and
ml.training.wrapper.TrainingModelWrapper (the name/model accessors, the
initial model_copy alias, the make_copy deep copy and rename, the
predict delegation to predict_state_with_dict with the production
converted graph and state map, and a real-model predict run). The
loss-validation tests
(AIAgent/tests/unit/ml/validation/test_validate_loss.py) pin
ml.validation.loss.validate_loss with a real TrainingDataset (steps
written through step_factory) and a fixed-output fake model: the exact
averaged MSELoss over the {0, 1}-thresholded y_true (the same run
would average 0.135 without the thresholding), the per-batch loss sequence,
and the DataLoader (batch_size, shuffle=False) and tqdm
(desc/ncols/colour) kwargs. The ModelGameManager play-flow
tests (test_process_game_manager.py) pin _play_game_map with
instance-level fakes for _run_game_process / _get_result and the
fake_proc fixture: the GameResult / GameFailed wrapping into
GameResultDetails / GameFailedDetails (with the
SVMConnectionInfo), the exact
"Failed to instantiate process for <map>" message, the timeout and
exception branches killing the process and storing the exception CLASS as
the GameFailed reason, and the no-process warning; the preparator is
pinned (clean-then-export order, the existing-dir delete, the
no-op-when-missing) and the full-coverage-without-oracle warning of
_get_result is pinned via caplog. The shared game-manager utilities
(test_game_managers_utils.py) pin play_with_timeout against the
real worker-thread wrapper: the result pass-through, the direct
propagation of an inner FunctionTimedOut (timedOutAfter intact —
the func_timeout swallow quirk, #672, is gone), the
None-return-as-timeout-signal translation, the plain-exception
pass-through, the disabled pass-through, and the wall-clock timeout
(raised FunctionTimedOut plus the worker thread registered in the
tracker, pruned once it finishes). The EachStepGameManager tests
(test_each_step_game_manager.py) pin the remaining branches: the
FunctionTimedOut branch (the model is saved, the reason is the
structured timeout message, the exact warning) driven by a
play_with_timeout that raises on a simulated wall-clock timeout, the
inner-timeout case (a FunctionTimedOut raised from inside the game
loop propagates directly through the real play_with_timeout into the
same branch, #672), the
GameInterruptedError branch (no save, the exact
"<name> failed on map <map> with <class>: <desc>" message), the
hardcoded 42.0 predicted usefulness, delete_game_artifacts
(drop and no-op), the notify_steps_requirement no-op, and the
preparator’s prepare.
The entrypoint seam tests (AIAgent/tests/unit/test_run_training.py) pin
run_training.py without end-to-end runs: run_training() is driven once
with recording fakes for TrainingDataset, optuna, mlflow,
joblib and objective, and the objective partial it wires up is
inspected to extract the model_init / validate / criterion_init
closures, which are then driven directly. The study branch pins the
study-level seed, the dataset construction from the module path constants,
the TPE sampler and study creation, and the per-trial callback (joblib dumps
of the study and trial, re-entering the active run, logging both artifacts,
tagging the best trial number); the trial-replay branch (trial_uri set)
pins the artifact download to REPORT_PATH, the joblib load, and the
replay of the loaded trial n_trials times without creating a fresh
study. The validate closures are pinned per mode: the criterion mode
switches the dataset to the configured split, drives validate_loss with
a fresh BCEWithLogitsLoss(reduction="sum"), and names the metric
BCEWithLogitsLoss__<DATASET MODE>; the SVM mode writes the per-trial
svms_result_table.csv (sorted map-name header, written once), drives
ValidationCoverage and get_svms_statistics, logs the table, and
raises RuntimeError("Validation failed: ...") naming each failed map
and its reason when fail_immediately is set; the custom mode runs the
sequence in order,
merges the metrics, and returns the first stage’s result. Each mode also
pins its PYTORCH_CUDA_ALLOC_CONF expandable-segments switch. The
per-trial objective loop is pinned with a scripted trial and a recording
model: the per-trial seed derivation, the exact optuna search space, the
EarlyStopping construction, the model_init kwargs drawn from the
trial settings, the trial artifact layout (model_kwargs.yaml, per-epoch
checkpoints, mlflow params/metrics), the per-epoch train/validate sequence,
the early-stop break (with its print), and the dynamic_dataset metadata
refresh. main() is pinned with a production-shaped config YAML (alias
keys): the parsed config forwarded to run_training, the spawn start
method, the MLflow setup (tracking uri only when set, experiment, tags,
system metrics), the gpu-monitor logger silenced to ERROR, the printed
device, and the Config validator rejecting weights_uri without
trial_uri. The model_init checkpoint loading is pinned through that
seam, including the map_location fix for CUDA-saved checkpoints on
CPU-only machines (#667, regression test requires CUDA and skips otherwise).
The companion entrypoint seam tests
(AIAgent/tests/unit/test_launch_servers.py) pin launch_servers.py
the same way: wait_for_port with a scripted socket (the probe socket
is closed on success and timeout alike), the four broker handlers with
fake requests and queues (the dequeue success / empty / failure-requeue
paths — the success also pins the lease recording in ISSUED —, the
live-instance InstanceIssuanceError with the re-queue, the enqueue
kill-and-rewrap versus keep-pid paths — the return also pins the lease
ending —, the results append / recv), reap_expired_leases with a
faked clock and a faked loop sleep (the expired entry is reaped — the
game server is killed and a placeholder re-queued —, the fresh entry is
left alone, and an entry returned between the snapshot and the kill is
skipped, no double-kill; a kill failure is logged without stopping the
reaper, the slot re-queued anyway), run_server_instance with a fake Popen (the
DEVNULL stdout, the pinned DOTNET_ROLL_FORWARD environment, the
pid-keyed handle registration in PROCS, the fail-fast group kill
+ reap with InstanceIssuanceError when the port never opens, and
the collision retry — a port that is open while the server is dead
reaps the corpse and retries the next port, bounded by
MAX_PORT_ATTEMPTS with an InstanceIssuanceError on
exhaustion), a pinned port and a fresh port lock, the
run_servers placeholders,
kill_server / kill_process with fake os.killpg /
os.getpgid and the fake_proc stand-in (the untracked-pid warning,
the process-group kill with the proc.wait() reap, the vanished-group
suppression, the never-dies RuntimeError), server_manager (queue
on enter, kill-all on exit that survives a per-pid failure), the reaper
lifecycle hooks (_on_startup creates the reaper task with the live
globals and the configured TTL, _on_cleanup cancels and awaits it),
and
main() (the parsed process_count handed to server_manager
with a fresh queue, the four declared broker routes on the application,
the broker port, the reaper hooks registered on the application, and the
module-global reset). The
derive_dataset_improvement_config.py pins are split by tier: the unit tests
(AIAgent/tests/unit/test_derive_dataset_improvement_config.py) pin the
main() CLI seam (the parsed --base-config / --output-config
Paths forwarded to derive_config) and the missing-experiment_name
SystemExit (which fails before an MlflowClient is created, so no
MLflow store is needed); the integration tests
(AIAgent/tests/integration/test_derive_dataset_improvement_config.py)
drive the real sqlite-backed MLflow store with the log_tuning_structure
helper and pin the best-trial lookup (best trial not last finished, single
trial, no tagged runs, unended tag, a tag pointing at a missing run), the
last-model-epoch lookup (highest epoch, no model artifacts), the missing
experiment, and derive_config (the derived URIs, the preserved base
config with its path values anchored against the base config’s directory,
and the missing-experiment / missing-OptunaConfig
SystemExit paths that write nothing).
Model smoke tests (test_models_forward.py) are registry-driven: every
model in ml.models.registry is instantiated with its small init kwargs
from a test-local table (a coverage assertion pins the table against
model_names(), so a registered model without a test entry — or a stale
entry — fails) and forwards a synthetic hetero_factory graph through
the production infer entrypoint, asserting output dtype float32,
shape (num_states, 1) and finiteness — the smoke test is the forward
contract’s enforcement. modelop.filemanager.save_model is exercised
with tmp_path and a patched datetime.
The ONNX export tests pin onyx.py. The unit tests
(AIAgent/tests/unit/test_onyx.py) pin the pure helpers: the tensor-input
builders (the exact 11 ONNX wire names
as literals — the V# contract and the drift net for #597’s single-source input
names — each mapped to its hetero_factory tensor, the modifier
application, the .numpy() conversion, and the input_names order),
the input_names()/dynamic_axes() derivations from the single-source
MODEL_INPUTS (the literal wire names and the legacy dim names as the V#
contract), the dynamo exporter’s dynamic_shapes derivation (one Dim
instance per symbolic dimension name, shared across the axes that carry the
name — #646; a monkeypatched torch.onnx.Dim stand-in, since the locked
torch predates it) and the torch-version guard that selects the exporter
API, and load_gamestate (a reference gamestate JSON into the exact
GameState); model selection goes through the registry (pinned in
test_registry.py). The integration
tests (AIAgent/tests/integration/test_onnx.py) are config-driven from
integration/resources/model_configurations/ and read the reference
gamestates through AIAgent/paths.py: TestONNXConversion runs the full
entrypoint (export + verification) with a randomized
RealStateModelEncoder; TestONNXExportWarnings pins that
save_in_onnx emits no warnings from torch.onnx, PyG conv, or this repo
(#617 regression guard; the export is version-aware per #646 — legacy
dynamic_axes below torch 2.13, dynamo dynamic_shapes from 2.13 — so
the guard covers whichever exporter the locked torch uses); TestForwardOnCUDA pins the CPU/CUDA forward
equivalence with and without v-v edges (#617 regression guard, skipped
without CUDA); TestModelGamePreparatorKwargs pins the preparator’s
explicit-kwarg forwarding to onyx.entrypoint (including the
registry-resolved model class). The TestMain CLI seam tests drive
main() with a patched sys.argv and a recording onyx.entrypoint
fake (no export run): the exact dispatch kwargs (the loaded sample
GameState, the verification GameState``s in argv order, the
registry-resolved model class, the ``yaml.safe_load``ed model kwargs, the
forwarded paths), the empty ``--verify-on default, the argparse
SystemExit for each missing required flag, and the opset version
(the default and --opset-version forwarded). The TestONNXVerification
tests pin the verification failure path with real exports and a
monkeypatched onyx.onnx_run: the verification compares the full torch
and ONNX Runtime outputs element-wise with an explicit tolerance and raises
ValueError on divergence (#597) — a numerically divergent conversion
must raise that proper (non-AssertionError) error, and a divergence
beyond the first 10 elements (a synthesized 12-state verification
gamestate) must be rejected.
The runstrat CLI tests (tools/runstrat/tests/unit/) pin the benchmarking
tool without end-to-end strategy runs. The parsing tests (test_parsing.py)
pin parse_runner_output (the exact (total_time_sec, tests, errors,
coverage) tuple from a runner transcript, the single-digit and three-digit
time fields, the integer and decimal coverage, and the AttributeError with
its __notes__ on a malformed transcript), parse_prebuilt (class-qualified
versus top-level method expansion), and the file-reading seams load_config
(a dll,method CSV into LaunchInfo rows, the Path-typed dll
field) and load_prebuilt_config (a JSON config structured into
PrebuiltConfig and expanded). The subprocess_calls tests
(test_subprocess_calls.py) pin the exact dotnet argv (the base flags, the
--model suffix for the AI strategy, the cwd), the
start_new_session group-leader launch, the exception-propagation branches
(CalledProcessError with its output/cmd, FunctionTimedOut with its
timedOutKwargs), and the timeout child cleanup (the process group is
killed with SIGKILL and reaped; an already-dead group is tolerated).
The main() seam tests (test_runstrat_main.py) drive main() with
a patched sys.argv and a
recording entrypoint fake: the parsed Args (strategy dispatch through
BasePSStrategy.parse, the assembly_infos list-of-pairs shape, the
timestamp default savedir), the argparse exits for missing
-s/-t/-ps/-as, the unknown-strategy ValueError, the
AI-without-model-path AssertionError, and the nonzero exit for
AllMethodsFailedError. The entrypoint() seam tests
(test_runstrat_entrypoint.py) drive entrypoint() with a method-keyed
call_test_runner fake and a recording logging.basicConfig: the
per-method loop, the RunResult → asdict → CSV column order
(method,tests,errors,coverage,total_time_sec), the
{strategy.name}.csv / {strategy.name}.log artifact names, the
empty-maps empty-CSV case, the per-method incremental CSV persistence (a
mid-run crash keeps the completed rows), the AllMethodsFailedError when
every method fails, and the setup_logging basicConfig call. The
error-handling tests pin the BENCHMARKS skip-versus-DEBUG re-raise for
CalledProcessError / FunctionTimedOut / AttributeError /
ValueError and the up-front FileNotFoundError for a missing V# runner.
The compstrat CLI tests (tools/compstrat/tests/unit/test_compstrat_main.py)
pin main() with a patched sys.argv and a recording entrypoint fake
(no matplotlib run); read_configs runs for real against the
mock_compare_confs fixture. They pin the parsed Args value (the
-r1/-r2 action="extend" + nargs="+" accumulation across
repeated flags, the --savedir default of report) and the argparse
SystemExit for each missing required flag. The entrypoint() pipeline
itself is pinned by the integration-tier test_pipeline.py.
The dataset_tools tests (tools/dataset_tools/tests/unit/) drive the
dataset maintenance CLIs over temporary datasets decoded through
GameMap.schema(). The clean tests (test_clean.py) pin
get_bad_episodes (the immediate GameOver on <name> extraction with
deduplication, the empty result for a missing log, the named log-format
constants GAME_OVER_MARKER/GAME_OVER_MAP_DELIMITER that are the
single explicit coupling to the free-form log phrasing) and clean (bad
episodes removed, the (method, step, searcher) sort order, the atomic
rewrite); the main() seam tests pin the CLI wiring end-to-end (argv
Path-typed arguments into the real clean) and the exact sys.exit
messages for a missing dataset (checked before the log) and a missing log.
The generate_episodes tests (test_generate_episodes.py) pin the
helpers (generate_steps bucket sizes, the create_episode map-name
rules, is_duplicate) and their composition in generate_episodes():
the per-method grouping, the grid from each method’s FIRST episode’s
StepsToPlay crossed with BFS/DFS plus the step-0 BFS
episode, the deduplication that matches on (method, step, searcher) and
ignores the MapName, and the exact sorted MapName list read back
from the rewritten file (including the empty-dataset case); the main()
seam tests pin the CLI wiring and the missing-dataset behaviour (the error
printed to stdout, then sys.exit(1)). The atomic_io tests
(test_atomic_io.py) pin the shared atomic rewrite: the content
round-trip, the temp file living in the target’s directory, and the
crash-safety property (the original intact and the temp removed on a
write-phase failure, a replace-phase failure, a concurrent temp cleanup,
and a KeyboardInterrupt).
The util tests (tools/util/tests/unit/test_parse_pretty.py) pin the
parse_pretty results-table parser: parse_stats (the exact
(steps, tests, errors, coverage) tuple from a stats cell, the
AttributeError on a malformed cell, the \d+ regexes rejecting a
non-digit count), the single-model transpose (one row per method, the
_Method_0 map-name suffix stripped), the multi-model transpose (one row
per (model, method) pair, model-major, with a leading model column —
the former #592 strict xfail for the transpose crash, flipped to a
passing test by #592), the ValueError on a model row whose stats-cell
count mismatches the header, the empty and header-only tables (an empty
frame with the standard columns), the configurable map_name_suffix (the
--map-name-suffix CLI flag), the Epoch# and +|- separator line
skips, and main() (the parsed CSV written to -o, the ./parsed.csv
default in the current working directory, the stderr message and exit code
1 for a missing input, an unparseable table, and an unwritable output).
- Adding a fixture
Put it in the
conftest.pyof the component it serves, resolve resources relative to__file__(never the current working directory), and prefer building real project objects through production code over hand-rolled copies.
Integration fixtures live under <component>/tests/integration/resources/ and
reach the tests only through the component conftest (an absolute path resolved
from conftest.py), never through a path relative to the current working
directory. Production inputs a test also consumes — such as the ONNX reference
gamestates under resources/onnx/ — are read through AIAgent/paths.py
instead of being copied into the test tree.
Fakes and sockets¶
Orchestration seams are tested with fakes, never real I/O:
Sockets. Replace
socket.socket,httplib2.Http,psutilandwebsocket.WebSocketwith recording fakes and drive the real client code. Thefake_websocketfixture records outgoing frames insentand replays queuedincomingframes onrecv(a queued exception is raised instead of returned), soConnector’s start/step/reward loop can be asserted without a server.game_server_socket_managermust callreturn_instanceon both the success and failure paths — assert it explicitly rather than trusting thefinally— and areturn_instancefailure must not mask the game body’s exception (the body error propagates and the return failure is logged only, #599).process_runningis pinned for a live pid (yields), a dead pid (ProcessStoppedError) and aNonepid (the restart-disabled placeholder: the check is skipped instead of lettingpsutil.pid_exists(None)raiseTypeError). The free-port helpers (next_free_port,look_for_free_port_locked) are driven by a fake socket factory whose sockets failbindfor a chosen set of occupied ports: the selection loop, the range bounds, the lock usage, the retry/exhaustion behaviour, the probe-socket close on the success and error paths alike, and the probe address —next_free_portprobes the caller’saddress(the interface the real bind will use), andlook_for_free_port_lockedis pinned to probe and bind the same address.Connection layer. The message deserialization helper in
unsafe_jsonis pinned as a pure unit test:obj_from_dictbuilds the nested attribute objects (BareObject``s) that the unchecked ``from_json_handlereturns when itsdisable_message_checksparameter is set (the flag is threaded from the run’sFeatureConfig, not a process global). Theerrors_connectionhierarchy is pinned as pure unit tests:GameInterruptedErroris abstract with an abstractdescproperty (its no-op stub getter is pinned by callingfgetdirectly — the body is unreachable in production), and the concrete subclasses (ProcessStoppedError,ConnectionLostError,ProtocolVersionMismatchError) expose their exactdescstrings. The broker HTTP calls are pinned with thehttplib2.Httprecording fake:acquire_instanceissues aGETto the broker’sget_ws_url(BrokerConfig.get_ws_url) with the urlencodedSVMInfofields and raisesRuntimeErroron a non-200 (the request is issued before the decode, so the shape is pinned without touching the decode); the single-encoded 200 body returns the parsedServerInstanceInfo.return_instanceposts the instance’sto_jsontopost_ws_urland raisesRuntimeErroron a non-200. The merged values are pinned on the pydantic models (test_config_models.py) as the drift net for #585 (merge the constant classes into the pydantic YAML system), which carries the current values over as the pydantic defaults: theBrokerConfigport and url properties (replacingWebsocketSourceLinks), theFeatureConfigdefaults and theGameServerConnectorConfigvalues — the CWD-relativesave_pathvalue is pinned (the #584 target is the CWD-dependence, not the value) and theGameServerConnectorConfigbudgets are pinned as the sane defaults (#599): the connect-retry budget (wait_for_socket_reconnection_max_retries60 → 2 min, not the old ~20 min) and theresponse_timeout_secvalue 60, an explicit field smaller than the per-map timeout (without the old cross-feature derivation, the #585 target). The save-on-failure behavior is pinned as a pure unit test:ml.model_io.save_state_dictcreates the save directory and writes the model’sstate_dictto a{with_name}_{timestamp}.ptfile (thestr(datetime)pattern, not the exact wall-clock time). Strict validation is the default: theFeatureConfigpin above assertsdisable_message_checks is False(the flag is kept for compatibility; the tests that pin the uncheckedBareObjectpath set it explicitly, #579). The connection-loss translation is pinned with real exception types from the pinned websocket-client: theWebSocketExceptionfamily (WebSocketConnectionClosedExceptionandWebSocketTimeoutException) andOSError(the syntheticConnectionResetErrorpin) all translate toConnectionLostError, chained as the cause (#579). The protocol-version check (check_protocol_version, #582) is pinned as a pure unit test: the currentPROTOCOL_VERSIONis accepted, a foreign version raisesProtocolVersionMismatchError(named in the message), and a legacy versionless envelope returnsNonewith a loud warning logged once per process (the module-level throttle flag is reset per test viamonkeypatch; non-dict envelopes are ignored by the check and leave no warning).from_json_handleruns the check under bothdisable_message_checkssettings (a foreignVersionraises before the flag branch) and wrapsjson.JSONDecodeErrorinDeserializationExceptionunder both settings (#582). The reward channel’s strict-mode decoding is pinned for both wire message types it carries:MoveReward(aRewardbody) andIncorrectPredictedStateId(aStateIdbody, decoded through the union body type ofRewardServerMessage) — the latter used to raiseDeserializationExceptionunder strict validation, a defect discovered during #582 (untracked; recorded on the issue). The start body’s explicit mapping ofGameMap(#582 replaced the multiple inheritance that fused the wire body with the domain model) is pinned by a field identity sync test (names and types, in order) and a wire-JSON equivalence test (from_game_map(m).to_json() == m.to_json()), so a domain-model change cannot silently change the wire format. The wire-format pins evolved to the v2 envelope (#582): the server→client test payloads carry"Version": PROTOCOL_VERSION(derived from the constant, not a literal), and a dedicated legacy test queues a versionless payload to pin the v1 tolerance; the client→server pins are unchanged — the client does not emitVersionyet (the coordinated server rollout).Subprocesses and files. For the model game manager, patch
subprocess.Popen(use thefake_procfixture), the port helper (common.network_utils.look_for_free_port_locked),delete_dirand the module path constants; pointsvms_output_pathattmp_pathand write the{MapName}resultfile directly. The manager is built withfake_namespace(a real lock, nomultiprocessing.Manager).Multiprocessing. The each-step manager and
ValidationCoverageare tested by patchingConnector(the fake must exposeGameOver) andgame_server_socket_manager; for the publicvalidate_coverageloop, patchmultiprocessing.Manager/Poolandtqdmand mark the testserial. A fakeBaseGameManagersubclass plus a fake dataset cover the manager-unset, GameFailed, missing-steps and exception branches without spawning a worker.Cleanup invariants. When a function acquires a resource (a socket, an instance, a process), test the success path and an exception in the body, and assert the release call in each.
Writing a test¶
Prefer the cheapest tier that can catch the regression: cover pure logic with plain unit tests (no I/O beyond
tmp_path) before reaching for a fixture or an integration test.Put the test under the component it exercises (
AIAgent/tests,tools/*/tests) and in the directory named after its tier (unit/,integration/ore2e/).Mark it with exactly one of
unit/integration/e2e(plus any orthogonalgpu/network/slow/serialmarker — but nevergpu/networkon a unit test; the guard test rejects the combination). The directory must match the tier marker; an unmarked test still runs in the fast tier, so an omitted marker silently changes the tier it appears in.Depend on the fixtures from your component’s
conftest.pyand build project objects through the production code paths so they cannot drift from the real schema.For graph/dataset transforms, build the input through
ml.dataset.convert_input_to_tensor(viagamestate_factory/hetero_factory) and assert exact tensor contents; thetmp_datasetfixture yields an emptyTrainingDatasetfor the filtering helpers.Never open sockets in a unit test: drive the pure parsing helpers directly (the
disable_message_checksflag is a plain parameter offrom_json_handle, threaded from the run’sFeatureConfig) and patch process-global switches withmonkeypatch.Write anything that touches disk under the
tmp_pathfixture.Keep it deterministic: the autouse
seeded_rngandcpu_devicefixtures make theAIAgentsuite reproducible, but a test must still not rely on wall-clock time or on iteration order.
Testability rules¶
No filesystem side effects at import. Importing a module must never create, move or delete files. Logging configuration, directory creation and similar work belongs in
main()(or theif __name__ == "__main__"guard). This keeps imports safe for tests, tooling and documentation builds;AIAgent/tests/unit/test_import_side_effects.pyenforces it for the runnable AIAgent scripts.Device selection is lazy. The torch device comes from
ml.device.get_device()at call time, never from a module-level constant, so importing AIAgent code never probes CUDA and tests can force CPU by patchingtorch.cuda.is_available.Never depend on the current working directory. Resolve resource paths from
__file__(orAIAgent/paths.py), and make filesystem seams injectable so tests can redirect them.
Coverage¶
Coverage is measured over the production packages (--cov=AIAgent
--cov=tools) with pytest-cov. The tests/ trees, conftest.py
files and __main__.py entrypoints are excluded from the measurement
([tool.coverage.run] omit in pyproject.toml is the source of truth),
so the number reflects the code under test rather than the tests themselves.
The gated tier is unit + integration, minus the gpu/network-marked
tests, so it runs on any machine (no GPU, no network guarantees) and its
coverage number is identical with or without either:
poetry run pytest -m "not e2e and not gpu and not network" --cov=AIAgent --cov=tools --cov-report=term-missing
make test-cov # same tier, plus XML and JSON reports
The threshold is a ratchet: [tool.coverage.report] fail_under in
pyproject.toml is set to the measured baseline, floored to a whole percent.
A pull request that drops coverage below it fails CI
(.github/workflows/python_tests.yaml). Raise the value in the same pull
request that improves coverage; never lower it without a deliberate decision
recorded in the pull request.