Developer Guide

Contribution model, tests, style checks, docs build, and CI.

Contribution guidelines

Branches. All work targets an integration branch. By default this is main; a developer redirects it to a personal, long-lived branch (for example gsv) with a local, uncommitted git config key:

git config pysymgym.integrationBranch gsv

When the key is unset the integration branch is main. Each task is developed on its own feature branch created from the integration branch; never commit directly to the integration branch. Integration always uses rebase and merge (fast-forward) so history stays linear and the per-subtask commit structure is preserved. There are two modes:

  • Direct mode (integration branch is main): each task is integrated by a pull request targeting main.

  • Stacked mode (integration branch is a personal branch): each task is integrated by rebasing its feature branch onto the integration branch and fast-forward merging it; no pull request is opened. Several tasks thus accumulate on the integration branch. A single pull request from the integration branch to main is opened only on the user’s explicit request.

A personal integration branch is long-lived: it is never deleted and is periodically rebased onto main (at task boundaries and immediately before the final pull request) to limit divergence. Before the final pull request, the whole main...integration diff passes an aggregated code review and the quality gate.

Commit messages. Conventional Commits with exactly one subtask identifier:

<type>(<task-issue>-S<n>): <summary>

where <type> is one of feat, fix, refactor, docs, test, chore. Ranges, lists, or comma-separated identifiers are forbidden: one commit per completed atomic subtask. The body must explain why the change was required.

Issue closing. The last subtask’s commit carries the closing keyword for the task’s own issue (Closes #<N>) plus one per linked issue it fully resolves (Fixes #N for defects, Closes #N otherwise), each on its own standalone line. A partially addressed linked issue uses a bare #N. The closing keyword takes effect when the commit reaches main: in stacked mode the issue stays open until the final pull request to main is merged.

A task is done when its subtask commits are on the integration branch, even if the linked issue has not yet closed.

Task tracking. Each task is a GitHub issue labeled task. A set of related tasks is grouped under a hub issue labeled hub: the task issues are attached to the hub as GitHub sub-issues (through the REST API, not a link in the body), the batch’s global plan is mirrored onto the hub as a comment, and its progress is updated as tasks integrate. The hub closes with the last task of the batch (Closes #<hub> in that task’s final commit). The operational procedure lives in the workflow-management and planning skills.

Quality gate. No pull request is opened until the quality gate passes. See the quality-gates skill and the sections below.

Pull requests. Every change updates its documentation: user-visible behavior is documented under docs/ (with the :description: and :group: metadata the navigation map is generated from), and README.md links must resolve. The pull request template lists these checks, and the docs build (.github/workflows/docs.yaml) enforces them. See CONTRIBUTING.md for the pointers.

Submodules

The repository pins third-party code as submodules; .gitmodules is the source of truth for their URLs and paths.

Pointer bumps. Moving a submodule to a new commit is an explicit task, not an incidental change: the bump must pass the matching build target (for maps, make build_maps) and CI before it lands.

Pin stability. A submodule pointer is stable when the commit it names is reachable from a stable ref of the submodule’s remote — a tag, or the default branch. A pointer to a bare feature-branch commit is unstable: if the branch is force-pushed or deleted, git submodule update breaks on every clone and the build stops being reproducible, while CI silently tracks moving WIP code instead of a stable baseline. Every pointer must therefore be stable, and CI enforces it: the submodule-pins job of .github/workflows/python_tests.yaml runs make check_submodule_pins, which fails when a pointer is not reachable from any tag or from the default branch of its remote. No branch = entries are used in .gitmodules: tracking a moving branch with git submodule update --remote is the opposite of pinning.

How to bump a pin. The procedure depends on who owns the remote:

  • Engine forks (GameServers/VSharp, GameServers/usvm — both in the PySymGym org): pin to a tag. When new fork work must be pinned, cut an annotated tag on the fork at that commit and point the submodule at the tag’s commit. The tag series starts at v0.2.0 for VSharp (after the inherited 2019 upstream v0.1.0) and at v0.1.0 for usvm; bump the patch level for fixes and the minor level for new functionality.

  • .NET map forks (maps/DotNet/Maps/*): pin to a commit on the fork’s default branch. Land fork-local changes on the default branch first (a fast-forward when the branch has no unique commits), then point the submodule at the landed commit.

  • Upstream repositories (maps/Java/Maps/LogicNG, maps/Java/Maps/jflex): no refs can be created there, so pin to a commit that is already reachable from the upstream default branch or a tag.

.NET forks. The .NET repositories — the maps and the V# game server (GameServers/VSharp) — are forks in the PySymGym org: they are pinned, not dependencies to track. Syncing a fork to newer upstream content is an explicit task and must keep the fork’s TargetFramework compatible with maps/DotNet/Maps/Root (net8.0).

Health. Submodule worktrees can arrive wiped on a fresh clone; detection and repair are documented on the Setup page (make check_submodules).

Path conventions

Nothing in the entrypoints resolves a path against the current working directory; two anchors cover the two kinds of paths:

  • Runtime artifact paths (AIAgent/paths.py: REPORT_PATH, LOG_PATH, BROKER_LOG_PATH and everything derived from them, plus the SaveIfFailOrTimeout.save_path default) are anchored to the AIAgent directory, located via __file__ the same way RESOURCES_PATH locates the repo root. The entrypoints therefore run from any CWD and always write their generated artifacts (models, trials, datasets, study/trial pickles, logs) under AIAgent/. AIAgent — rather than the repo root — is the anchor because the e2e pipeline stages the dataset at AIAgent/report/SerializedEpisodes and collects AIAgent/report/study.pkl and AIAgent/*.log.

  • Config-file paths (dataset_base_path, dataset_description, server_working_dir) are anchored to the config file’s own directory: a relative value in the config is resolved against the directory the config file lives in, never against the CWD. The config models themselves require absolute paths — a relative value is ambiguous without an anchor and is rejected at construction. The entrypoints load their config through common.config.loader.load_config, which does the anchoring and then validates that every path names an existing filesystem object of the required kind (dataset_base_path and server_working_dir: directory, dataset_description: file), failing with an error that names every offending key and its location in the config. derive_dataset_improvement_config.py writes its derived config with the path values anchored against the base config’s directory (absolute): the derived config is written next to the tuning artifacts, not next to the base, so the base’s relative values would anchor against the wrong directory there.

Config models

Every pydantic config model shares one config (common.config.base.CONFIG): unknown keys are rejected (a typo or stray key in a YAML config fails fast at load time instead of being silently ignored), and fields may be populated by their Python name as well as their alias (the code constructs the models with snake_case kwargs, the YAML configs use the PascalCase aliases). A model that needs a different behaviour overrides the base dict ({**CONFIG, ...}) rather than redefining the contract.

Test pipeline

The testing system — the unit/integration/e2e taxonomy, the exact commands, the fixtures, and the coverage policy — is documented in Testing, which is the single source of truth. The fast tier is one command from the repository root:

poetry run pytest        # or: make test-unit

CI (.github/workflows/python_tests.yaml) runs each component suite and then a repository-root coverage step that enforces the ratchet documented in Testing.

Quality checks

Linting and formatting use ruff:

poetry run ruff check
poetry run ruff format --check

The ruff configuration (target version, enabled rule sets, included paths) lives in pyproject.toml. CI runs the same two commands in .github/workflows/python_linting.yaml.

The ruff editor integrations (for example the VSCode extension) run the same checks on save.

Docs build

The Sphinx sources live in docs/. Build them locally with:

poetry install --only docs
poetry run sphinx-build -W --keep-going -b html docs docs/_build/html

The build runs under the no-warnings policy (-W --keep-going): any warning fails the build. CI runs the same command in .github/workflows/docs.yaml.

Published docs

The built HTML is published to GitHub Pages at pysymgym.github.io/PySymGym. The deploy job in .github/workflows/docs.yaml runs only for pushes to main; pull requests build without deploying. Read the Docs is planned as a secondary mirror.

Runner requirements

Four workflow files contain jobs that run on a self-hosted runner (jbLabSelfHostedCI) rather than on GitHub-hosted compute: e2e_build_and_run.yml (the shared pipeline called by build_and_run.yaml and build_and_run_model_val.yaml), build_and_test_usvm.yaml, publish_image.yaml and runstrat_tool.yaml. All of those jobs also declare a container: — except the ensure-image gate jobs in e2e_build_and_run.yml and runstrat_tool.yaml, which run container-less on GitHub-hosted runners — so the work happens inside a container while the workflow itself is still driven by the runner.

Every action referenced by .github/workflows/ runs on Node.js 24, which requires Actions Runner v2.327.1 or newer. The GitHub-hosted runners satisfy this by construction, but the self-hosted runner does not: on one older than v2.327.1 every action in those four workflow files fails to start, taking the whole training, image-publishing and tool-testing pipeline down with it. Keep that runner current before bumping an action to a release that raises its runtime requirement.

The runner’s installed version is not visible from the repository, so this is the one CI constraint that cannot be checked by reading the workflow files.

Self-hosted end-to-end workflows

The policies below keep the long self-hosted pipelines deterministic and stop them from piling up on the single runner. This section explains the non-obvious decisions behind the pipelines; the workflow files are the source of truth for the exact steps.

The two e2e workflows share one pipeline. build_and_run.yaml and build_and_run_model_val.yaml differ only in their name, concurrency group, and training inputs, so their common pipeline (Dockerfile hash, checkout, toolchain setup, V# and maps build, data generation, training with MLflow, artifact upload, sanity check) lives once in the reusable workflow .github/workflows/e2e_build_and_run.yml (workflow_call only — it never runs on its own). Each e2e workflow file is a thin caller that selects the pipeline mode through two inputs: training-config (the config of the main training run, relative to AIAgent/) and the optional improvement-base-config (see below). To add another e2e variant, create such a thin caller with its own name and concurrency group — do not copy the pipeline.

Shared setup and build steps. The pipeline’s environment preparation (checkout with submodules, Python 3.11.8, pinned Poetry, the self-healing dependency install, the cached-venv integrity check) lives once in the composite action .github/actions/checkout-poetry, and the .NET builds (V# game server plus the caller-selected maps project) in .github/actions/dotnet-build. The e2e pipeline and the runstrat pipeline (runstrat_tool.yaml) are thin configurations of those actions; the Poetry version they install is the one documented in README.md.

Container jobs bootstrap the workspace before a local action. The runner reads a local composite action (uses: ./.github/actions/...) from the job’s workspace, and the self-hosted runner’s workspace is shared across jobs of any branch or PR: a previous job may have left a checkout whose tree lacks .github/actions/ (e.g. main, which predates the actions), in which case the container job dies before its first step with “Can’t find ‘action.yml’”. Every container job that uses a local action therefore starts with a plain actions/checkout step that re-checks-out the exact commit, guaranteeing the action is loadable; the composite action’s own checkout (with submodules) remains the authoritative one.

Image publishing. publish_image.yaml runs on pushes and pull requests that touch .github/docker/Dockerfile. The pull-request trigger is load-bearing: a PR that changes the Dockerfile must publish the image for the new Dockerfile hash, or the build workflows — which compute the same hash from the same merge-commit checkout — cannot pull it and die with an opaque container-pull failure. The publish job first checks whether the image for the computed hash is already on GHCR (the same .github/scripts/check_ci_image.sh the gates use) and skips the build and push when the manifest exists: the tag is a content hash, so a published image is immutable and re-pushing it is wasted work.

The base image is digest-pinned. The tag is the sha1 of the Dockerfile, so the tag must address immutable content: a floating ubuntu:24.04 tag would let the base update underneath an unchanged tag, and consumers would pull different content under the same name. The Dockerfile therefore pins the base by digest. Refreshing the pin is a deliberate act: inspect the current digest of ubuntu:24.04 (for example with docker buildx imagetools inspect ubuntu:24.04) and update the FROM line. The Dockerfile content changes, so does the tag, and the publish workflow republishes under the new tag — the ensure-image gates of the build workflows pick it up as usual.

The build jobs gate on the published image. Publishing the image is a separate workflow from building with it, and on pull requests both start at the same time — so a build job can start before the image exists and die with an opaque container-pull failure. Both pipelines therefore start with an ensure-image job that computes the same Dockerfile hash and polls GHCR (.github/scripts/check_ci_image.sh) until the image’s manifest appears, bounded at 60 × 30 s. The gate runs on a GitHub-hosted runner on purpose: it must not occupy the single self-hosted runner the publish job needs, or the gate could starve the very job it waits for.

Servers live in the step that uses them. MLflow and the game-server broker are started inside the same step as the training command that talks to them, gated on readiness polling (an HTTP /health check for MLflow, a TCP connect to the broker port) instead of fixed sleeps, and cleaned up when the step exits. A background process does not survive a step boundary, so a server started in an earlier step is gone by the time a later step needs it; a fixed sleep additionally races with the variable startup time. The polling itself is AIAgent/wait_for_service.py (stdlib only, http and tcp subcommands), called from the workflow step.

Triggers and concurrency. The five self-hosted workflows (build_and_run.yaml, build_and_run_model_val.yaml, runstrat_tool.yaml, build_and_test_usvm.yaml, publish_image.yaml) run only on pushes to main and on pull requests — feature-branch pushes do not trigger them, so a dependabot bump does not trigger the long pipelines twice (push and pull request) and stale runs do not queue up. Each uses a per-workflow concurrency group keyed by ref that cancels in-progress runs for pull requests only: a new push to the same branch supersedes the stale run, while a run started by a push to main is never cancelled.

Dataset improvement continues from tuning. In build_and_run.yaml the tuning run and the dataset-improvement run share one MLflow experiment. The improvement is an optional mode of the shared pipeline, selected by that workflow’s improvement-base-config input: after the main (tuning) run, derive_dataset_improvement_config.py queries the same-step server for the best trial’s model.pth and trial.pkl artifact URIs and writes them into a copy of the base config that the improvement step consumes. The base config remains the single source of truth — the pipeline derives the URIs from the tuning run instead of hard-coding them. build_and_run_model_val.yaml leaves the input empty and runs only the main training run.

Unexhausted steps fail CI. Both e2e validation workflows run with fail_immediately enabled (flag semantics in Usage, “SVM validation failure flag”). The symbolic engine ending a map before all planned steps are played without 100% coverage is an engine defect; in CI it must fail the run instead of being swallowed by a warning, while local runs keep the default warning-only behavior.

Host GPU in e2e CI (#567). The real-world pipeline runs on a GPU, so the self-hosted e2e run exercises the GPU path as its primary mode and falls back to CPU when the runner host has no usable GPU. (The original #567 spike measured no speedup and concluded DROP; the decision was overridden because the value of the GPU path in CI is correctness coverage of the real-world device, not speed — measurement history in issue #567.)

Two properties make this a workflow-only change:

  • The CI environment already ships a CUDA-enabled torch. The Docker image (.github/docker/Dockerfile) provides only ubuntu and the dotnet SDKs; Python dependencies are installed at runtime by poetry install from the lock, and the locked torch resolves on Linux x86_64 to the manylinux wheel with all nvidia-*-cu12 dependencies (cuDNN, cuBLAS, NCCL, …).

  • Device selection needs no code change: ml.device.get_device picks cuda:0 when available and falls back to CPU, and the training step prints the selected device at startup — every CI run logs which path it took.

The only missing piece was device passthrough to the job container, and the right docker flag is host-dependent (--gpus all vs --runtime nvidia, two configurations of nvidia-container-toolkit). The shared pipeline’s detect-gpu job probes the runner host before the build-and-launch job: it tries each flag with a disposable ubuntu:24.04 container and checks that /dev/nvidia0 was actually injected, then passes the first working flag to the build-and-launch job as container.options. When no flag works (no GPU, broken driver, missing toolkit) it passes nothing: the container starts device-less and the Python side runs on CPU — the e2e run degrades instead of failing.

The locked torch must be able to run the GPU (#662). Passthrough is necessary but not sufficient: a GPU whose compute capability the locked torch build has no kernels for would pass the passthrough probe and then crash the first CUDA op mid-training (“no kernel image is available for execution on the device”). The pipeline therefore verifies compatibility where torch is actually available — a step in the build-and-launch job, after the Python environment is prepared: the image itself ships no Python (the venv with the locked torch is created at runtime by the checkout-poetry action), so the probe cannot run earlier. AIAgent/probe_gpu_torch.py runs a small CUDA op on cuda:0 with the exact torch the pipeline trains with. The verdict is the op, not the arch-list membership: a capability absent from torch.cuda.get_arch_list() can still work through driver JIT of bundled PTX (the locked torch 2.7.1 runs on the runner’s sm_61 GPU this way), and an arch-list hit proves nothing if the op fails. On failure the probe logs the device, its compute capability, the torch version and arch list, and hides the device from torch for the rest of the run (an empty CUDA_VISIBLE_DEVICES via $GITHUB_ENV), so ml.device.get_device falls back to CPU and the e2e run degrades instead of failing.

CI as source of truth

The CI workflows are the source of truth for the exact commands they run. If a command in this guide and a workflow disagree, the workflow wins; fix this guide.