Developer Guide¶
Contribution model, tests, style checks, docs build, and CI.
Contribution guidelines¶
Branches. All work targets an integration branch. By default this is
main; a developer redirects it to a personal, long-lived branch (for
example gsv) with a local, uncommitted git config key:
git config pysymgym.integrationBranch gsv
When the key is unset the integration branch is main. Each task is
developed on its own feature branch created from the integration branch; never
commit directly to the integration branch. Integration always uses rebase and
merge (fast-forward) so history stays linear and the per-subtask commit
structure is preserved. There are two modes:
Direct mode (integration branch is
main): each task is integrated by a pull request targetingmain.Stacked mode (integration branch is a personal branch): each task is integrated by rebasing its feature branch onto the integration branch and fast-forward merging it; no pull request is opened. Several tasks thus accumulate on the integration branch. A single pull request from the integration branch to
mainis opened only on the user’s explicit request.
A personal integration branch is long-lived: it is never deleted and is
periodically rebased onto main (at task boundaries and immediately before
the final pull request) to limit divergence. Before the final pull request,
the whole main...integration diff passes an aggregated code review and the
quality gate.
Commit messages. Conventional Commits with exactly one subtask identifier:
<type>(<task-issue>-S<n>): <summary>
where <type> is one of feat, fix, refactor, docs, test,
chore. Ranges, lists, or comma-separated identifiers are forbidden: one
commit per completed atomic subtask. The body must explain why the change was
required.
Issue closing. The last subtask’s commit carries the closing keyword for
the task’s own issue (Closes #<N>) plus one per linked issue it fully
resolves (Fixes #N for defects, Closes #N otherwise), each on its own
standalone line. A partially addressed linked issue uses a bare #N. The
closing keyword takes effect when the commit reaches main: in stacked mode
the issue stays open until the final pull request to main is merged.
A task is done when its subtask commits are on the integration branch, even if the linked issue has not yet closed.
Task tracking. Each task is a GitHub issue labeled task. A set of
related tasks is grouped under a hub issue labeled hub: the task issues
are attached to the hub as GitHub sub-issues (through the REST API, not a link
in the body), the batch’s global plan is mirrored onto the hub as a comment, and
its progress is updated as tasks integrate. The hub closes with the last task of
the batch (Closes #<hub> in that task’s final commit). The operational
procedure lives in the workflow-management and planning skills.
Quality gate. No pull request is opened until the quality gate passes. See the quality-gates skill and the sections below.
Pull requests. Every change updates its documentation: user-visible
behavior is documented under docs/ (with the :description: and
:group: metadata the navigation map is generated from), and README.md
links must resolve. The pull request template lists these checks, and the docs
build (.github/workflows/docs.yaml) enforces them. See CONTRIBUTING.md
for the pointers.
Submodules¶
The repository pins third-party code as submodules; .gitmodules is the
source of truth for their URLs and paths.
Pointer bumps. Moving a submodule to a new commit is an explicit task, not
an incidental change: the bump must pass the matching build target (for maps,
make build_maps) and CI before it lands.
Pin stability. A submodule pointer is stable when the commit it names
is reachable from a stable ref of the submodule’s remote — a tag, or the
default branch. A pointer to a bare feature-branch commit is unstable: if the
branch is force-pushed or deleted, git submodule update breaks on every
clone and the build stops being reproducible, while CI silently tracks moving
WIP code instead of a stable baseline. Every pointer must therefore be
stable, and CI enforces it: the submodule-pins job of
.github/workflows/python_tests.yaml runs make check_submodule_pins,
which fails when a pointer is not reachable from any tag or from the default
branch of its remote. No branch = entries are used in .gitmodules:
tracking a moving branch with git submodule update --remote is the
opposite of pinning.
How to bump a pin. The procedure depends on who owns the remote:
Engine forks (
GameServers/VSharp,GameServers/usvm— both in the PySymGym org): pin to a tag. When new fork work must be pinned, cut an annotated tag on the fork at that commit and point the submodule at the tag’s commit. The tag series starts atv0.2.0for VSharp (after the inherited 2019 upstreamv0.1.0) and atv0.1.0for usvm; bump the patch level for fixes and the minor level for new functionality..NET map forks (
maps/DotNet/Maps/*): pin to a commit on the fork’s default branch. Land fork-local changes on the default branch first (a fast-forward when the branch has no unique commits), then point the submodule at the landed commit.Upstream repositories (
maps/Java/Maps/LogicNG,maps/Java/Maps/jflex): no refs can be created there, so pin to a commit that is already reachable from the upstream default branch or a tag.
.NET forks. The .NET repositories — the maps and the V# game server
(GameServers/VSharp) — are forks in the PySymGym org: they are pinned,
not dependencies to track. Syncing a fork to newer upstream content is an
explicit task and must keep the fork’s TargetFramework compatible with
maps/DotNet/Maps/Root (net8.0).
Health. Submodule worktrees can arrive wiped on a fresh clone; detection
and repair are documented on the Setup page
(make check_submodules).
Path conventions¶
Nothing in the entrypoints resolves a path against the current working directory; two anchors cover the two kinds of paths:
Runtime artifact paths (
AIAgent/paths.py:REPORT_PATH,LOG_PATH,BROKER_LOG_PATHand everything derived from them, plus theSaveIfFailOrTimeout.save_pathdefault) are anchored to the AIAgent directory, located via__file__the same wayRESOURCES_PATHlocates the repo root. The entrypoints therefore run from any CWD and always write their generated artifacts (models, trials, datasets, study/trial pickles, logs) underAIAgent/. AIAgent — rather than the repo root — is the anchor because the e2e pipeline stages the dataset atAIAgent/report/SerializedEpisodesand collectsAIAgent/report/study.pklandAIAgent/*.log.Config-file paths (
dataset_base_path,dataset_description,server_working_dir) are anchored to the config file’s own directory: a relative value in the config is resolved against the directory the config file lives in, never against the CWD. The config models themselves require absolute paths — a relative value is ambiguous without an anchor and is rejected at construction. The entrypoints load their config throughcommon.config.loader.load_config, which does the anchoring and then validates that every path names an existing filesystem object of the required kind (dataset_base_pathandserver_working_dir: directory,dataset_description: file), failing with an error that names every offending key and its location in the config.derive_dataset_improvement_config.pywrites its derived config with the path values anchored against the base config’s directory (absolute): the derived config is written next to the tuning artifacts, not next to the base, so the base’s relative values would anchor against the wrong directory there.
Config models¶
Every pydantic config model shares one config
(common.config.base.CONFIG): unknown keys are rejected (a typo or stray
key in a YAML config fails fast at load time instead of being silently
ignored), and fields may be populated by their Python name as well as their
alias (the code constructs the models with snake_case kwargs, the YAML
configs use the PascalCase aliases). A model that needs a different behaviour
overrides the base dict ({**CONFIG, ...}) rather than redefining the
contract.
Test pipeline¶
The testing system — the unit/integration/e2e taxonomy, the exact
commands, the fixtures, and the coverage policy — is documented in
Testing, which is the single source of truth. The fast tier is one
command from the repository root:
poetry run pytest # or: make test-unit
CI (.github/workflows/python_tests.yaml) runs each component suite and then
a repository-root coverage step that enforces the ratchet documented in
Testing.
Quality checks¶
Linting and formatting use ruff:
poetry run ruff check
poetry run ruff format --check
The ruff configuration (target version, enabled rule sets, included paths)
lives in pyproject.toml. CI runs the same two commands in
.github/workflows/python_linting.yaml.
The ruff editor integrations (for example the VSCode extension) run the same checks on save.
Docs build¶
The Sphinx sources live in docs/. Build them locally with:
poetry install --only docs
poetry run sphinx-build -W --keep-going -b html docs docs/_build/html
The build runs under the no-warnings policy (-W --keep-going): any warning
fails the build. CI runs the same command in .github/workflows/docs.yaml.
Published docs¶
The built HTML is published to GitHub Pages at
pysymgym.github.io/PySymGym. The
deploy job in .github/workflows/docs.yaml runs only for pushes to
main; pull requests build without deploying. Read the Docs is planned as a
secondary mirror.
Runner requirements¶
Four workflow files contain jobs that run on a self-hosted runner
(jbLabSelfHostedCI) rather than on GitHub-hosted compute:
e2e_build_and_run.yml (the shared pipeline called by
build_and_run.yaml and build_and_run_model_val.yaml),
build_and_test_usvm.yaml, publish_image.yaml and
runstrat_tool.yaml. All of those jobs also declare a container: —
except the ensure-image gate jobs in e2e_build_and_run.yml and
runstrat_tool.yaml, which run container-less on GitHub-hosted runners —
so the work happens inside a container while the workflow itself is still
driven by the runner.
Every action referenced by .github/workflows/ runs on Node.js 24, which
requires Actions Runner v2.327.1 or newer. The GitHub-hosted runners
satisfy this by construction, but the self-hosted runner does not: on one older
than v2.327.1 every action in those four workflow files fails to start,
taking the whole training, image-publishing and tool-testing pipeline down with
it.
Keep that runner current before bumping an action to a release that raises its
runtime requirement.
The runner’s installed version is not visible from the repository, so this is the one CI constraint that cannot be checked by reading the workflow files.
Self-hosted end-to-end workflows¶
The policies below keep the long self-hosted pipelines deterministic and stop them from piling up on the single runner. This section explains the non-obvious decisions behind the pipelines; the workflow files are the source of truth for the exact steps.
The two e2e workflows share one pipeline. build_and_run.yaml and
build_and_run_model_val.yaml differ only in their name, concurrency group,
and training inputs, so their common pipeline (Dockerfile hash, checkout,
toolchain setup, V# and maps build, data generation, training with MLflow,
artifact upload, sanity check) lives once in the reusable workflow
.github/workflows/e2e_build_and_run.yml (workflow_call only — it never
runs on its own). Each e2e workflow file is a thin caller that selects the
pipeline mode through two inputs: training-config (the config of the main
training run, relative to AIAgent/) and the optional
improvement-base-config (see below). To add another e2e variant, create
such a thin caller with its own name and concurrency group — do not copy the
pipeline.
Shared setup and build steps. The pipeline’s environment preparation
(checkout with submodules, Python 3.11.8, pinned Poetry, the self-healing
dependency install, the cached-venv integrity check) lives once in the
composite action .github/actions/checkout-poetry, and the .NET builds
(V# game server plus the caller-selected maps project) in
.github/actions/dotnet-build. The e2e pipeline and the runstrat pipeline
(runstrat_tool.yaml) are thin configurations of those actions; the
Poetry version they install is the one documented in README.md.
Container jobs bootstrap the workspace before a local action. The
runner reads a local composite action (uses: ./.github/actions/...) from
the job’s workspace, and the self-hosted runner’s workspace is shared across
jobs of any branch or PR: a previous job may have left a checkout whose
tree lacks .github/actions/ (e.g. main, which predates the actions),
in which case the container job dies before its first step with
“Can’t find ‘action.yml’”. Every container job that uses a local action
therefore starts with a plain actions/checkout step that re-checks-out
the exact commit, guaranteeing the action is loadable; the composite
action’s own checkout (with submodules) remains the authoritative one.
Image publishing. publish_image.yaml runs on pushes and pull
requests that touch .github/docker/Dockerfile. The pull-request trigger
is load-bearing: a PR that changes the Dockerfile must publish the image for
the new Dockerfile hash, or the build workflows — which compute the same
hash from the same merge-commit checkout — cannot pull it and die with an
opaque container-pull failure. The publish job first checks whether the
image for the computed hash is already on GHCR (the same
.github/scripts/check_ci_image.sh the gates use) and skips the build and
push when the manifest exists: the tag is a content hash, so a published
image is immutable and re-pushing it is wasted work.
The base image is digest-pinned. The tag is the sha1 of the Dockerfile,
so the tag must address immutable content: a floating ubuntu:24.04 tag
would let the base update underneath an unchanged tag, and consumers would
pull different content under the same name. The Dockerfile therefore pins
the base by digest. Refreshing the pin is a deliberate act: inspect the
current digest of ubuntu:24.04 (for example with
docker buildx imagetools inspect ubuntu:24.04) and update the FROM
line. The Dockerfile content changes, so does the tag, and the publish
workflow republishes under the new tag — the ensure-image gates of the
build workflows pick it up as usual.
The build jobs gate on the published image. Publishing the image is a
separate workflow from building with it, and on pull requests both start at
the same time — so a build job can start before the image exists and die
with an opaque container-pull failure. Both pipelines therefore start
with an ensure-image job that computes the same Dockerfile hash and polls
GHCR (.github/scripts/check_ci_image.sh) until the image’s manifest
appears, bounded at 60 × 30 s. The gate runs on a GitHub-hosted runner on
purpose: it must not occupy the single self-hosted runner the publish job
needs, or the gate could starve the very job it waits for.
Servers live in the step that uses them. MLflow and the game-server
broker are started inside the same step as the training command that talks to
them, gated on readiness polling (an HTTP /health check for MLflow, a TCP
connect to the broker port) instead of fixed sleeps, and cleaned up when the
step exits. A background process does not survive a step boundary, so a
server started in an earlier step is gone by the time a later step needs it;
a fixed sleep additionally races with the variable startup time. The polling
itself is AIAgent/wait_for_service.py (stdlib only, http and tcp
subcommands), called from the workflow step.
Triggers and concurrency. The five self-hosted workflows
(build_and_run.yaml, build_and_run_model_val.yaml,
runstrat_tool.yaml, build_and_test_usvm.yaml,
publish_image.yaml) run only on pushes to main and on pull requests —
feature-branch pushes do not trigger them, so a dependabot bump does not
trigger the long pipelines twice (push and pull request) and stale runs do
not queue up. Each uses a per-workflow concurrency group keyed by ref that
cancels in-progress runs for pull requests only: a new push to the same
branch supersedes the stale run, while a run started by a push to main is
never cancelled.
Dataset improvement continues from tuning. In build_and_run.yaml the
tuning run and the dataset-improvement run share one MLflow experiment. The
improvement is an optional mode of the shared pipeline, selected by that
workflow’s improvement-base-config input: after the main (tuning) run,
derive_dataset_improvement_config.py queries the same-step server for the
best trial’s model.pth and trial.pkl artifact URIs and writes them
into a copy of the base config that the improvement step consumes. The base
config remains the single source of truth — the pipeline derives the URIs from
the tuning run instead of hard-coding them. build_and_run_model_val.yaml
leaves the input empty and runs only the main training run.
Unexhausted steps fail CI. Both e2e validation workflows run with
fail_immediately enabled (flag semantics in Usage, “SVM validation
failure flag”). The symbolic engine ending a map before all planned steps are
played without 100% coverage is an engine defect; in CI it must fail the run
instead of being swallowed by a warning, while local runs keep the default
warning-only behavior.
Host GPU in e2e CI (#567). The real-world pipeline runs on a GPU, so the self-hosted e2e run exercises the GPU path as its primary mode and falls back to CPU when the runner host has no usable GPU. (The original #567 spike measured no speedup and concluded DROP; the decision was overridden because the value of the GPU path in CI is correctness coverage of the real-world device, not speed — measurement history in issue #567.)
Two properties make this a workflow-only change:
The CI environment already ships a CUDA-enabled torch. The Docker image (
.github/docker/Dockerfile) provides only ubuntu and the dotnet SDKs; Python dependencies are installed at runtime bypoetry installfrom the lock, and the locked torch resolves on Linux x86_64 to the manylinux wheel with allnvidia-*-cu12dependencies (cuDNN, cuBLAS, NCCL, …).Device selection needs no code change:
ml.device.get_devicepickscuda:0when available and falls back to CPU, and the training step prints the selected device at startup — every CI run logs which path it took.
The only missing piece was device passthrough to the job container, and the
right docker flag is host-dependent (--gpus all vs --runtime nvidia,
two configurations of nvidia-container-toolkit). The shared pipeline’s
detect-gpu job probes the runner host before the build-and-launch job:
it tries each flag with a disposable ubuntu:24.04 container and checks
that /dev/nvidia0 was actually injected, then passes the first working
flag to the build-and-launch job as container.options. When no flag
works (no GPU, broken driver, missing toolkit) it passes nothing: the
container starts device-less and the Python side runs on CPU — the e2e run
degrades instead of failing.
The locked torch must be able to run the GPU (#662). Passthrough is
necessary but not sufficient: a GPU whose compute capability the locked
torch build has no kernels for would pass the passthrough probe and then
crash the first CUDA op mid-training (“no kernel image is available for
execution on the device”). The pipeline therefore verifies compatibility
where torch is actually available — a step in the build-and-launch job,
after the Python environment is prepared: the image itself ships no Python
(the venv with the locked torch is created at runtime by the
checkout-poetry action), so the probe cannot run earlier.
AIAgent/probe_gpu_torch.py runs a small CUDA op on cuda:0 with the
exact torch the pipeline trains with. The verdict is the op, not the
arch-list membership: a capability absent from
torch.cuda.get_arch_list() can still work through driver JIT of bundled
PTX (the locked torch 2.7.1 runs on the runner’s sm_61 GPU this way), and
an arch-list hit proves nothing if the op fails. On failure the probe logs
the device, its compute capability, the torch version and arch list, and
hides the device from torch for the rest of the run (an empty
CUDA_VISIBLE_DEVICES via $GITHUB_ENV), so ml.device.get_device
falls back to CPU and the e2e run degrades instead of failing.
CI as source of truth¶
The CI workflows are the source of truth for the exact commands they run. If a command in this guide and a workflow disagree, the workflow wins; fix this guide.