:description: Contribution model, tests, style checks, docs build, and CI. :group: Development Developer Guide =============== Contribution guidelines ----------------------- **Branches.** All work targets an *integration branch*. By default this is ``main``; a developer redirects it to a personal, long-lived branch (for example ``gsv``) with a local, uncommitted git config key: .. code-block:: console git config pysymgym.integrationBranch gsv When the key is unset the integration branch is ``main``. Each task is developed on its own feature branch created from the integration branch; never commit directly to the integration branch. Integration always uses rebase and merge (fast-forward) so history stays linear and the per-subtask commit structure is preserved. There are two modes: - **Direct mode** (integration branch is ``main``): each task is integrated by a pull request targeting ``main``. - **Stacked mode** (integration branch is a personal branch): each task is integrated by rebasing its feature branch onto the integration branch and fast-forward merging it; **no pull request is opened**. Several tasks thus accumulate on the integration branch. A single pull request from the integration branch to ``main`` is opened only on the user's explicit request. A personal integration branch is long-lived: it is never deleted and is periodically rebased onto ``main`` (at task boundaries and immediately before the final pull request) to limit divergence. Before the final pull request, the whole ``main...integration`` diff passes an aggregated code review and the quality gate. **Commit messages.** Conventional Commits with exactly one subtask identifier:: (-S): where ```` is one of ``feat``, ``fix``, ``refactor``, ``docs``, ``test``, ``chore``. Ranges, lists, or comma-separated identifiers are forbidden: one commit per completed atomic subtask. The body must explain why the change was required. **Issue closing.** The last subtask's commit carries the closing keyword for the task's own issue (``Closes #``) plus one per linked issue it fully resolves (``Fixes #N`` for defects, ``Closes #N`` otherwise), each on its own standalone line. A partially addressed linked issue uses a bare ``#N``. The closing keyword takes effect when the commit reaches ``main``: in stacked mode the issue stays open until the final pull request to ``main`` is merged. A task is done when its subtask commits are on the integration branch, even if the linked issue has not yet closed. **Task tracking.** Each task is a GitHub issue labeled ``task``. A set of related tasks is grouped under a **hub** issue labeled ``hub``: the task issues are attached to the hub as GitHub sub-issues (through the REST API, not a link in the body), the batch's global plan is mirrored onto the hub as a comment, and its progress is updated as tasks integrate. The hub closes with the last task of the batch (``Closes #`` in that task's final commit). The operational procedure lives in the ``workflow-management`` and ``planning`` skills. **Quality gate.** No pull request is opened until the quality gate passes. See the `quality-gates` skill and the sections below. **Pull requests.** Every change updates its documentation: user-visible behavior is documented under ``docs/`` (with the ``:description:`` and ``:group:`` metadata the navigation map is generated from), and ``README.md`` links must resolve. The pull request template lists these checks, and the docs build (``.github/workflows/docs.yaml``) enforces them. See ``CONTRIBUTING.md`` for the pointers. Submodules ---------- The repository pins third-party code as submodules; ``.gitmodules`` is the source of truth for their URLs and paths. **Pointer bumps.** Moving a submodule to a new commit is an explicit task, not an incidental change: the bump must pass the matching build target (for maps, ``make build_maps``) and CI before it lands. **Pin stability.** A submodule pointer is *stable* when the commit it names is reachable from a stable ref of the submodule's remote — a tag, or the default branch. A pointer to a bare feature-branch commit is unstable: if the branch is force-pushed or deleted, ``git submodule update`` breaks on every clone and the build stops being reproducible, while CI silently tracks moving WIP code instead of a stable baseline. Every pointer must therefore be stable, and CI enforces it: the ``submodule-pins`` job of ``.github/workflows/python_tests.yaml`` runs ``make check_submodule_pins``, which fails when a pointer is not reachable from any tag or from the default branch of its remote. No ``branch =`` entries are used in ``.gitmodules``: tracking a moving branch with ``git submodule update --remote`` is the opposite of pinning. **How to bump a pin.** The procedure depends on who owns the remote: - **Engine forks** (``GameServers/VSharp``, ``GameServers/usvm`` — both in the PySymGym org): pin to a tag. When new fork work must be pinned, cut an annotated tag on the fork at that commit and point the submodule at the tag's commit. The tag series starts at ``v0.2.0`` for VSharp (after the inherited 2019 upstream ``v0.1.0``) and at ``v0.1.0`` for usvm; bump the patch level for fixes and the minor level for new functionality. - **.NET map forks** (``maps/DotNet/Maps/*``): pin to a commit on the fork's default branch. Land fork-local changes on the default branch first (a fast-forward when the branch has no unique commits), then point the submodule at the landed commit. - **Upstream repositories** (``maps/Java/Maps/LogicNG``, ``maps/Java/Maps/jflex``): no refs can be created there, so pin to a commit that is already reachable from the upstream default branch or a tag. **.NET forks.** The .NET repositories — the maps and the V# game server (``GameServers/VSharp``) — are forks in the PySymGym org: they are pinned, not dependencies to track. Syncing a fork to newer upstream content is an explicit task and must keep the fork's ``TargetFramework`` compatible with ``maps/DotNet/Maps/Root`` (net8.0). **Health.** Submodule worktrees can arrive wiped on a fresh clone; detection and repair are documented on the :doc:`setup` page (``make check_submodules``). Path conventions ---------------- Nothing in the entrypoints resolves a path against the current working directory; two anchors cover the two kinds of paths: - **Runtime artifact paths** (``AIAgent/paths.py``: ``REPORT_PATH``, ``LOG_PATH``, ``BROKER_LOG_PATH`` and everything derived from them, plus the ``SaveIfFailOrTimeout.save_path`` default) are anchored to the **AIAgent directory**, located via ``__file__`` the same way ``RESOURCES_PATH`` locates the repo root. The entrypoints therefore run from any CWD and always write their generated artifacts (models, trials, datasets, study/trial pickles, logs) under ``AIAgent/``. AIAgent — rather than the repo root — is the anchor because the e2e pipeline stages the dataset at ``AIAgent/report/SerializedEpisodes`` and collects ``AIAgent/report/study.pkl`` and ``AIAgent/*.log``. - **Config-file paths** (``dataset_base_path``, ``dataset_description``, ``server_working_dir``) are anchored to the **config file's own directory**: a relative value in the config is resolved against the directory the config file lives in, never against the CWD. The config models themselves require absolute paths — a relative value is ambiguous without an anchor and is rejected at construction. The entrypoints load their config through ``common.config.loader.load_config``, which does the anchoring and then validates that every path names an existing filesystem object of the required kind (``dataset_base_path`` and ``server_working_dir``: directory, ``dataset_description``: file), failing with an error that names every offending key and its location in the config. ``derive_dataset_improvement_config.py`` writes its derived config with the path values anchored against the **base** config's directory (absolute): the derived config is written next to the tuning artifacts, not next to the base, so the base's relative values would anchor against the wrong directory there. Config models ------------- Every pydantic config model shares one config (``common.config.base.CONFIG``): unknown keys are rejected (a typo or stray key in a YAML config fails fast at load time instead of being silently ignored), and fields may be populated by their Python name as well as their alias (the code constructs the models with snake_case kwargs, the YAML configs use the PascalCase aliases). A model that needs a different behaviour overrides the base dict (``{**CONFIG, ...}``) rather than redefining the contract. Test pipeline ------------- The testing system — the ``unit``/``integration``/``e2e`` taxonomy, the exact commands, the fixtures, and the coverage policy — is documented in :doc:`testing`, which is the single source of truth. The fast tier is one command from the repository root: .. code-block:: console poetry run pytest # or: make test-unit CI (``.github/workflows/python_tests.yaml``) runs each component suite and then a repository-root coverage step that enforces the ratchet documented in :doc:`testing`. Quality checks -------------- Linting and formatting use `ruff `_: .. code-block:: console poetry run ruff check poetry run ruff format --check The ruff configuration (target version, enabled rule sets, included paths) lives in ``pyproject.toml``. CI runs the same two commands in ``.github/workflows/python_linting.yaml``. The `ruff editor integrations `_ (for example the `VSCode `__ extension) run the same checks on save. Docs build ---------- The Sphinx sources live in ``docs/``. Build them locally with: .. code-block:: console poetry install --only docs poetry run sphinx-build -W --keep-going -b html docs docs/_build/html The build runs under the no-warnings policy (``-W --keep-going``): any warning fails the build. CI runs the same command in ``.github/workflows/docs.yaml``. Published docs -------------- The built HTML is published to GitHub Pages at `pysymgym.github.io/PySymGym `_. The ``deploy`` job in ``.github/workflows/docs.yaml`` runs only for pushes to ``main``; pull requests build without deploying. Read the Docs is planned as a secondary mirror. Runner requirements ------------------- Four workflow files contain jobs that run on a self-hosted runner (``jbLabSelfHostedCI``) rather than on GitHub-hosted compute: ``e2e_build_and_run.yml`` (the shared pipeline called by ``build_and_run.yaml`` and ``build_and_run_model_val.yaml``), ``build_and_test_usvm.yaml``, ``publish_image.yaml`` and ``runstrat_tool.yaml``. All of those jobs also declare a ``container:`` — except the ``ensure-image`` gate jobs in ``e2e_build_and_run.yml`` and ``runstrat_tool.yaml``, which run container-less on GitHub-hosted runners — so the work happens inside a container while the workflow itself is still driven by the runner. Every action referenced by ``.github/workflows/`` runs on Node.js 24, which requires Actions Runner ``v2.327.1`` or newer. The GitHub-hosted runners satisfy this by construction, but the self-hosted runner does not: on one older than ``v2.327.1`` every action in those four workflow files fails to start, taking the whole training, image-publishing and tool-testing pipeline down with it. Keep that runner current before bumping an action to a release that raises its runtime requirement. The runner's installed version is not visible from the repository, so this is the one CI constraint that cannot be checked by reading the workflow files. Self-hosted end-to-end workflows -------------------------------- The policies below keep the long self-hosted pipelines deterministic and stop them from piling up on the single runner. This section explains the non-obvious decisions behind the pipelines; the workflow files are the source of truth for the exact steps. **The two e2e workflows share one pipeline.** ``build_and_run.yaml`` and ``build_and_run_model_val.yaml`` differ only in their name, concurrency group, and training inputs, so their common pipeline (Dockerfile hash, checkout, toolchain setup, V# and maps build, data generation, training with MLflow, artifact upload, sanity check) lives once in the reusable workflow ``.github/workflows/e2e_build_and_run.yml`` (``workflow_call`` only — it never runs on its own). Each e2e workflow file is a thin caller that selects the pipeline mode through two inputs: ``training-config`` (the config of the main training run, relative to ``AIAgent/``) and the optional ``improvement-base-config`` (see below). To add another e2e variant, create such a thin caller with its own name and concurrency group — do not copy the pipeline. **Shared setup and build steps.** The pipeline's environment preparation (checkout with submodules, Python 3.11.8, pinned Poetry, the self-healing dependency install, the cached-venv integrity check) lives once in the composite action ``.github/actions/checkout-poetry``, and the .NET builds (V# game server plus the caller-selected maps project) in ``.github/actions/dotnet-build``. The e2e pipeline and the runstrat pipeline (``runstrat_tool.yaml``) are thin configurations of those actions; the Poetry version they install is the one documented in ``README.md``. **Container jobs bootstrap the workspace before a local action.** The runner reads a local composite action (``uses: ./.github/actions/...``) from the job's workspace, and the self-hosted runner's workspace is shared across jobs of *any* branch or PR: a previous job may have left a checkout whose tree lacks ``.github/actions/`` (e.g. ``main``, which predates the actions), in which case the container job dies before its first step with "Can't find 'action.yml'". Every container job that uses a local action therefore starts with a plain ``actions/checkout`` step that re-checks-out the exact commit, guaranteeing the action is loadable; the composite action's own checkout (with submodules) remains the authoritative one. **Image publishing.** ``publish_image.yaml`` runs on pushes **and** pull requests that touch ``.github/docker/Dockerfile``. The pull-request trigger is load-bearing: a PR that changes the Dockerfile must publish the image for the new Dockerfile hash, or the build workflows — which compute the same hash from the same merge-commit checkout — cannot pull it and die with an opaque container-pull failure. The publish job first checks whether the image for the computed hash is already on GHCR (the same ``.github/scripts/check_ci_image.sh`` the gates use) and skips the build and push when the manifest exists: the tag is a content hash, so a published image is immutable and re-pushing it is wasted work. **The base image is digest-pinned.** The tag is the sha1 of the Dockerfile, so the tag must address immutable content: a floating ``ubuntu:24.04`` tag would let the base update underneath an unchanged tag, and consumers would pull different content under the same name. The Dockerfile therefore pins the base by digest. Refreshing the pin is a deliberate act: inspect the current digest of ``ubuntu:24.04`` (for example with ``docker buildx imagetools inspect ubuntu:24.04``) and update the ``FROM`` line. The Dockerfile content changes, so does the tag, and the publish workflow republishes under the new tag — the ``ensure-image`` gates of the build workflows pick it up as usual. **The build jobs gate on the published image.** Publishing the image is a separate workflow from building with it, and on pull requests both start at the same time — so a build job can start before the image exists and die with an opaque container-pull failure. Both pipelines therefore start with an ``ensure-image`` job that computes the same Dockerfile hash and polls GHCR (``.github/scripts/check_ci_image.sh``) until the image's manifest appears, bounded at 60 × 30 s. The gate runs on a GitHub-hosted runner on purpose: it must not occupy the single self-hosted runner the publish job needs, or the gate could starve the very job it waits for. **Servers live in the step that uses them.** MLflow and the game-server broker are started inside the same step as the training command that talks to them, gated on readiness polling (an HTTP ``/health`` check for MLflow, a TCP connect to the broker port) instead of fixed sleeps, and cleaned up when the step exits. A background process does not survive a step boundary, so a server started in an earlier step is gone by the time a later step needs it; a fixed sleep additionally races with the variable startup time. The polling itself is ``AIAgent/wait_for_service.py`` (stdlib only, ``http`` and ``tcp`` subcommands), called from the workflow step. **Triggers and concurrency.** The five self-hosted workflows (``build_and_run.yaml``, ``build_and_run_model_val.yaml``, ``runstrat_tool.yaml``, ``build_and_test_usvm.yaml``, ``publish_image.yaml``) run only on pushes to ``main`` and on pull requests — feature-branch pushes do not trigger them, so a dependabot bump does not trigger the long pipelines twice (push and pull request) and stale runs do not queue up. Each uses a per-workflow concurrency group keyed by ref that cancels in-progress runs for pull requests only: a new push to the same branch supersedes the stale run, while a run started by a push to ``main`` is never cancelled. **Dataset improvement continues from tuning.** In ``build_and_run.yaml`` the tuning run and the dataset-improvement run share one MLflow experiment. The improvement is an optional mode of the shared pipeline, selected by that workflow's ``improvement-base-config`` input: after the main (tuning) run, ``derive_dataset_improvement_config.py`` queries the same-step server for the best trial's ``model.pth`` and ``trial.pkl`` artifact URIs and writes them into a copy of the base config that the improvement step consumes. The base config remains the single source of truth — the pipeline derives the URIs from the tuning run instead of hard-coding them. ``build_and_run_model_val.yaml`` leaves the input empty and runs only the main training run. **Unexhausted steps fail CI.** Both e2e validation workflows run with ``fail_immediately`` enabled (flag semantics in :doc:`usage`, "SVM validation failure flag"). The symbolic engine ending a map before all planned steps are played without 100% coverage is an engine defect; in CI it must fail the run instead of being swallowed by a warning, while local runs keep the default warning-only behavior. **Host GPU in e2e CI (#567).** The real-world pipeline runs on a GPU, so the self-hosted e2e run exercises the GPU path as its primary mode and falls back to CPU when the runner host has no usable GPU. (The original #567 spike measured no speedup and concluded DROP; the decision was overridden because the value of the GPU path in CI is correctness coverage of the real-world device, not speed — measurement history in issue #567.) Two properties make this a workflow-only change: - The CI environment already ships a CUDA-enabled torch. The Docker image (``.github/docker/Dockerfile``) provides only ubuntu and the dotnet SDKs; Python dependencies are installed at runtime by ``poetry install`` from the lock, and the locked torch resolves on Linux x86_64 to the manylinux wheel with all ``nvidia-*-cu12`` dependencies (cuDNN, cuBLAS, NCCL, ...). - Device selection needs no code change: ``ml.device.get_device`` picks ``cuda:0`` when available and falls back to CPU, and the training step prints the selected device at startup — every CI run logs which path it took. The only missing piece was device passthrough to the job container, and the right docker flag is host-dependent (``--gpus all`` vs ``--runtime nvidia``, two configurations of nvidia-container-toolkit). The shared pipeline's ``detect-gpu`` job probes the runner host before the build-and-launch job: it tries each flag with a disposable ``ubuntu:24.04`` container and checks that ``/dev/nvidia0`` was actually injected, then passes the first working flag to the build-and-launch job as ``container.options``. When no flag works (no GPU, broken driver, missing toolkit) it passes nothing: the container starts device-less and the Python side runs on CPU — the e2e run degrades instead of failing. **The locked torch must be able to run the GPU (#662).** Passthrough is necessary but not sufficient: a GPU whose compute capability the locked torch build has no kernels for would pass the passthrough probe and then crash the first CUDA op mid-training ("no kernel image is available for execution on the device"). The pipeline therefore verifies compatibility where torch is actually available — a step in the build-and-launch job, after the Python environment is prepared: the image itself ships no Python (the venv with the locked torch is created at runtime by the checkout-poetry action), so the probe cannot run earlier. ``AIAgent/probe_gpu_torch.py`` runs a small CUDA op on ``cuda:0`` with the exact torch the pipeline trains with. The verdict is the op, not the arch-list membership: a capability absent from ``torch.cuda.get_arch_list()`` can still work through driver JIT of bundled PTX (the locked torch 2.7.1 runs on the runner's sm_61 GPU this way), and an arch-list hit proves nothing if the op fails. On failure the probe logs the device, its compute capability, the torch version and arch list, and hides the device from torch for the rest of the run (an empty ``CUDA_VISIBLE_DEVICES`` via ``$GITHUB_ENV``), so ``ml.device.get_device`` falls back to CPU and the e2e run degrades instead of failing. CI as source of truth --------------------- The CI workflows are the source of truth for the exact commands they run. If a command in this guide and a workflow disagree, the workflow wins; fix this guide.