:description: Build the engines, run training and tuning, improve the dataset, and execute with a trained model.
:group: Guides
Usage
=====
This guide assumes the environment is already set up as described in the
:doc:`setup` page.
Main loop
---------
Hyper-parameters and training artifacts are logged with
`MLflow `_. Start its server first:
.. code-block:: console
cd AIAgent
poetry run mlflow server -h 127.0.0.1 -p 8080 --serve-artifacts
The training process is a *main loop* that alternates two kinds of runs:
*hyper-parameters tuning* and *dataset improvement with symbolic virtual
machines*. Before the loop, generate an initial dataset. Then alternate tuning
and dataset improvement until the desired result is reached, and finally run
symbolic execution with the trained model.
The point of alternating is mutual improvement: tuning yields the best network
for the current dataset, and that network improves the dataset; a better
dataset then has better hyper-parameters, so tune again.
.. image:: https://raw.githubusercontent.com/PySymGym/PySymGym/main/resources/game_process_illustration.png
:width: 50%
:alt: Illustration of the game process
Build symbolic virtual machines and maps
----------------------------------------
To build the symbolic virtual machines (`V# `_
and `usvm `_) and the methods used for
training, set up the toolchain as described in the :doc:`setup` page, then run:
.. code-block:: console
make build_SVMs build_maps
Optionally add new maps under ``maps/`` and integrate another engine (see
:ref:`integrate-a-new-symbolic-machine`).
Generate initial dataset
------------------------
Supervised learning needs some initial data, which any path-selection strategy
can produce. This project generates it with one of the strategies from V#:
.. code-block:: console
make init_data STEPS_TO_SERIALIZE=
``STEPS_TO_SERIALIZE`` is optional and defaults to 200. The initial dataset is
saved under ``./AIAgent/report/SerializedEpisodes`` and is later updated by the
neural network when it finds a better solution.
Generation is deterministic: two runs with the same seed produce
byte-identical serialized path conditions.
Hyper-parameters tuning
-----------------------
Hyper-parameters are tuned with `Optuna `_. Tuning yields
the best network for the current dataset, which gives a better chance of
improving the dataset in the other kind of run.
1. Create a training configuration. ``./workflow/config_for_tests.yml`` is a
template. To use the training loss as the tuning objective, set the
validation config as follows:
.. code-block:: yaml
ValidationConfig:
validation_mode:
val_type: loss
batch_size:
Configure Optuna, for example:
.. code-block:: yaml
OptunaConfig:
n_startup_trials: 10
n_trials: 30
n_jobs: 1
study_direction: "minimize"
The ``TrainingConfig`` section requires a ``seed``. It makes the run
reproducible: the dataset split, step sampling, validation map order, and
model weight initialization are seeded with it, each trial re-seeds from a
seed derived from (study seed, trial number), and the Optuna sampler is
seeded with the study seed — so two runs of the same configuration produce
identical splits, samples, and hyper-parameter suggestions.
Each trial writes its artifacts to ``report/trials//``
(model weights, model kwargs, and the SVM statistics table for SVM
validation), so parallel trials cannot overwrite each other; the study
snapshots ``study.pkl`` and ``trial.pkl`` stay at the top of ``report/``.
Each trial also runs in its own MLflow run named after the trial number.
After each trial finishes, the ``study.pkl`` and ``trial.pkl`` snapshots
are logged to that run together with a ``best_trial_number`` tag holding
the best trial among those completed so far;
``derive_dataset_improvement_config.py`` locates the best trial's
artifacts through that tag.
A trial stops early when its validation result stops improving. On early
stop (or after the final epoch), the best checkpoint — the epoch with the
best validation result — is restored into the model and exported, so the
retained ``model.pth`` and the highest-epoch ``/model.pth`` MLflow
artifact hold the best weights, and the trial's value is that best
result, not the last epoch's.
2. Move to the ``AIAgent`` directory and run training:
.. code-block:: console
cd AIAgent
poetry run python3 run_training.py --config path/to/config.yml
The entrypoints are CWD-independent: the artifacts always land under
``AIAgent/`` and the config's relative paths resolve against the config
file's own directory, so the command works from any directory (the
``cd`` only locates the script).
Dataset improvement with symbolic virtual machines
---------------------------------------------------
The optimal sequence of symbolic execution steps is unknown, so the best models
from the previous step are used to obtain relatively good ones:
1. Build the SVMs if you have not yet (see `Build symbolic virtual machines
and maps`_).
2. Using the `workflow example
`_
as a template, create a configuration with the maps to use, or reuse an
existing ``maps/*/Maps/dataset.json``.
3. Create a configuration (server and training parameters), again using
``./workflow/config_for_tests.yml`` as a template. The improvement run must
start from the tuning run's best model, so the config needs two URIs: the
best weights and the trial that produced them. Derive both from MLflow
instead of copying them by hand — the script reads the server and experiment
from the config's ``MLFlowConfig`` section and writes the config plus:
.. code-block:: console
cd AIAgent
poetry run python3 derive_dataset_improvement_config.py \
--base-config path/to/config.yml --output-config path/to/derived.yml
.. code-block:: yaml
weights_uri: mlflow-artifacts:///artifacts//model.pth
OptunaConfig:
...
trial_uri: mlflow-artifacts:///artifacts/trial.pkl
The derived config's path values are absolute (anchored against the base
config's directory), so it loads from any directory.
4. Move to the ``AIAgent`` directory, launch the server manager, and run
training with the derived config:
.. code-block:: console
cd AIAgent
poetry run python3 launch_servers.py --config path/to/derived.yml
poetry run python3 run_training.py --config path/to/derived.yml
The improvement run replays the frozen trial through a fresh study,
enqueuing its hyper-parameters once per replay. It persists
``study.pkl``/``trial.pkl`` and the per-trial MLflow artifacts exactly
like a tuning run, so the next round of the loop derives its config the
same way.
SVM validation failure flag
---------------------------
The SVM validation section of the config (``val_type: svms_model`` or
``svms_each_step``) has an opt-in ``fail_immediately`` flag, off by default.
When enabled:
- a game that ends before all planned steps are played without reaching 100%
coverage is marked as a failed map — the symbolic engine stopping that early
is an engine defect, not a model quality issue; by default it only produces
a warning; and
- any failed map (timeout, engine error, ...) fails the run with a non-zero
exit.
.. _svm-validation-step-metric:
SVM validation step metric
--------------------------
The ``steps_count`` reported per map (in the per-trial statistics table and in
the dataset ``Result``) is the number of **steps the model/agent played**: the
server-reported total trace steps minus ``StepsToStart``. The first
``StepsToStart`` steps of a map are played by the default searcher (BFS/DFS)
before the model takes over — in ``svms_each_step`` mode the agent's connection
is only used after that start phase, and in ``svms_model`` mode the server plays
the start steps itself before handing control to the ONNX model. Subtracting
them makes the metric identical across both validation modes for an identical
game outcome, so per-map statistics and early-stopping signals are comparable
regardless of the mode used.
.. _guide-symbolic-execution:
Guide symbolic execution with a trained model
---------------------------------------------
After training, choose the best MLflow-logged model and run symbolic execution:
1. Convert the PyTorch model to ONNX with ``onyx.py`` (see
:ref:`onnx-conversion`).
2. Use the ONNX model to guide execution with your SVM (see
:ref:`integrate-a-new-symbolic-machine`),
or use an existing engine extension in this repository:
- Place the model at
``GameServers/VSharp/VSharp.Explorer/models/model.onnx``.
- Run:
.. code-block:: console
dotnet GameServers/VSharp/VSharp.Runner/bin/Release/net8.0/VSharp.Runner.dll \
--method BinarySearch \
maps/DotNet/Maps/Root/bin/Release/net8.0/ManuallyCollected.dll \
--timeout 120 --strat AI --check-coverage