Configuration scenario

The tuning module can generate the scenario files for several AC tools keeping the same code for the configurable functions defined by the user.

class optilog.tuning.configurators.TuningEntrypointType

NOTE: This is not a class, it is a Type Alias

This alias can be:

  • A Python function. If this python function is configurable it must be annotated with @ac

  • A PythonBlackBox (object or class)

  • A SystemBlackBox (object or class)

TYPES

alias of Union[BlackBox, Callable, Type[BlackBox]]

class optilog.tuning.configurators.TuningGlobalConfigurableType

NOTE: This is not a class, it is a Type Alias

This alias can be:

  • A Python function annotated with @ac

  • A PythonBlackBox (class)

  • A SystemBlackBox (class)

TYPES

alias of Union[Callable, Type[BlackBox]]

class optilog.tuning.configurators.GGAScenario(entrypoint, constraints, run_obj, seed, cost_min, cost_max, global_cfg=[], eval_time_limit=None, data_kwarg=None, seed_kwarg=None, input_data=None, quality_regex=None, default_as_elite=False, **scenario_kwargs)

Handles the creation of all the needed files to use the GGA AC tool over a configurable function.

Parameters

  • entrypoint (TuningEntrypointType) – Entrypoint function to be configured. You may set as an entrypoint functions with the @ac decorator, blackbox objects or blackbox classes.

  • global_cfg (List[TuningGlobalConfigurableType]) – List of global configurables. In case the entrypoint calls some configurable it must be present in this list. You may set as global configurables functions with the @ac decorator or blackbox classes.

  • input_data (Optional[str]) – Input data that the AC tool will use to configure the entrypoint function. This parameter can be either a string or a list of inputs. If this parameter is a list of inputs, the type of its elements must be the same that the entrypoint function will receive on the data_kwarg parameter. The accepted types are int, str or float. If this parameter is a string, it will get expanded recursively with glob. See examples.

  • constraints (ExecutionConstraints) – Defines the execution constraints for the Configuration Process

  • run_obj (str) – Run objective (runtime or quality). NOTE: When run_onj='quality', the configurator will minimize the result reported by the entrypoint function.

  • data_kwarg (Optional[str]) – Name of the parameter inside the entrypoint function that the AC tool will use to provide the data specified in input_data. If None no data will be passed.

  • seed_kwarg (Optional[str]) – Name of the parameter inside the entrypoint function that the AC tool will use to provide the RNG seed. If None no seed will be passed.

  • quality_regex (Optional[str]) – Python regular expression used to capture the quality of an execution of the entrypoint function. The quality must be the first group within the regular expression. If run_obj='runtime' this parameter is not needed.

  • seed (int) – Seed used by GGA on every instance

  • cost_min (int) – Minimum possible cost of the algorithm

  • cost_max (int) – Maximum possible cost of the algorithm

  • eval_time_limit (Optional[int]) – Time limit for each evaluation in seconds (defaults to the wall time limit constraint)

  • **scenario_kwargs – GGA scenario parameters as they appear in GGA’s official documentation. These parameters will be directly written to the GGA scenario file. GGA documentation can be accessed through the official website of LOG.

generate_scenario(out_dir, log=True)

Generates all the files required by the Automatic Configurator.

Parameters

  • out_dir (str) – Output directory where files will be generated.

  • log (bool) – If True, will print a log to the console

static get_command(scenario_path, config, instance, seed=None)

Generates the execution command of the scenario. Only usable if the entrypoint is a BlackBox.

Parameters

  • scenario_path (str) – Path to the scenario directory

  • config (Dict[str, Any]) – Configuration extracted from a parser

  • instance (str) – Instance to substitute in the blackbox

  • seed (Optional[int]) – Seed to substitute in the blackbox

class optilog.tuning.configurators.SMACScenario(entrypoint, constraints, run_obj, global_cfg=[], data_kwarg=None, seed_kwarg=None, input_data=None, quality_regex=None, cost_for_crash=2147483647.0, **scenario_kwargs)

Handles the creation of all the needed files to use the SMAC AC tool over a configurable function.

Warning

Version 2.0 of SMAC3 dropped support for CLI. At the moment you must use version 1.4 of SMAC to run the scenarios generated by OptiLog. See https://github.com/automl/SMAC3#important-changes-in-v20.

Parameters

  • entrypoint (TuningEntrypointType) – Entrypoint function to be configured. You may set as an entrypoint functions with the @ac decorator, blackbox objects or blackbox classes.

  • global_cfg (List[TuningGlobalConfigurableType]) – List of global configurables. In case the entrypoint calls some configurable it must be present in this list. You may set as global configurables functions with the @ac decorator or blackbox classes.

  • input_data (Optional[str]) – Input data that the AC tool will use to configure the entrypoint function. This parameter can be either a string or a list of inputs. If this parameter is a list of inputs, the type of its elements must be the same that the entrypoint function will receive on the data_kwarg parameter. The accepted types are int, str or float. If this parameter is a string, it will get expanded recursively with glob. See examples.

  • constraints (ExecutionConstraints) – Defines the execution constraints for the Configuration Process

  • run_obj (str) – Run objective (runtime or quality). NOTE: When run_onj='quality', the configurator will minimize the result reported by the entrypoint function.

  • data_kwarg (Optional[str]) – Name of the parameter inside the entrypoint function that the AC tool will use to provide the data specified in input_data. If None no data will be passed.

  • seed_kwarg (Optional[str]) – Name of the parameter inside the entrypoint function that the AC tool will use to provide the RNG seed. If None no seed will be passed.

  • quality_regex (Optional[str]) – Python regular expression used to capture the quality of an execution of the entrypoint function. The quality must be the first group within the regular expression. If run_obj='runtime' this parameter is not needed.

  • cost_crash – Cost assigned in case of crash or timeout.

  • **scenario_kwargs (dict) – SMAC scenario parameters as they appear in SMAC’s official documentation. Will be directly written to the SMAC scenario file.

generate_scenario(out_dir, log=True)

Generates all the files required by the Automatic Configurator.

Parameters

  • out_dir (str) – Output directory where files will be generated.

  • log (bool) – If True, will print a log to the console

static get_command(scenario_path, config, instance, seed=None)

Generates the execution command of the scenario. Only usable if the entrypoint is a BlackBox.

Parameters

  • scenario_path (str) – Path to the scenario directory

  • config (Dict[str, Any]) – Configuration extracted from a parser

  • instance (str) – Instance to substitute in the blackbox

  • seed (Optional[int]) – Seed to substitute in the blackbox

Here you can see a few examples with GGA and SMAC. Starting with a SMAC example:

 1from optilog.tuning import ac
 2from optilog.tuning.configurators import SMACScenario
 3from optilog.blackbox import ExecutionConstraints, RunSolver
 4
 5@ac
 6def entrypoint(data, seed, ...):
 7    ...
 8
 9if __name__ == '__main__':
10
11    configurator = SMACScenario(
12        entrypoint,
13        input_data=['path1', 'path2', 'path3'],
14        run_obj='quality',
15        data_kwarg="data",
16        seed_kwarg="seed",
17        quality_regex=r"^Result: (\d+)$",
18        constraints=ExecutionConstraints(
19            s_wall_time=30,
20            s_real_memory="8GB",
21            enforcer=RunSolver()
22        ),
23        wallclock_limit=43200,  # Item inside **scenario_kwargs
24    )
25
26    configurator.generate_scenario('./smac-scenario')

In this example we specify:

  • The entrypoint function as entrypoint

  • The path to runsolver is omitted because runsolver is in PATH

  • input_data is the list instances_list

  • data_kwarg as instance and seed_kwarg as seed

  • Other GGA arguments

Warning

The generation of the scenario needs to be inside a __main__ block because the file may be dynamically reimported by the execution scenario

 1from optilog.tuning import ac
 2from optilog.tuning.configurators import GGAScenario
 3from optilog.blackbox import ExecutionConstraints, RunSolver
 4
 5@ac
 6def entrypoint(instance, seed, ...):
 7    ...
 8
 9if __name__ == '__main__':
10    configurator = GGAScenario(
11        entrypoint,
12        input_data='path/to/instances/*.txt',
13        constraints=ExecutionConstraints(
14            s_wall_time=120,
15            s_real_memory="18G",
16            enforcer=RunSolver()
17        ),
18        run_obj="quality",
19        data_kwarg="instance",
20        seed_kwarg="seed",
21        quality_regex="^o (\d+)$",
22        seed=1,
23        cost_min=0,
24        cost_max=(2 << 64) - 1,
25    )
26    configurator.generate_scenario('./gga-scenario')

The following example presents a few differences:

  • The path to runsolver is omitted because runsolver is in PATH

  • input_data, data_kwarg and seed_kwarg are omitted because they are not required to configure func1

 1from optilog.tuning import ac, Bool, Int, Real, Categorical
 2from optilog.tuning.configurators import GGAScenario
 3from optilog.blackbox import ExecutionConstraints, RunSolver
 4
 5@ac
 6def func1(
 7    p1: Bool() = True, p2: Real(-1.3, 2) = 0,
 8    p3: Int(-5, 5) = 0, p4: Categorical("A", "B", "C") = "A"):
 9    ...
10
11if __name__ == '__main__':
12    configurator = GGAScenario(
13        func1,
14        constraints=ExecutionConstraints(
15            s_wall_time=25,
16            s_real_memory="8GB",
17            s_cpu_time=30,
18            enforcer=RunSolver()
19        ),
20        run_obj="quality",
21        quality_regex="^o (\d+)$",
22        seed=1,
23        cost_min=0,
24        cost_max=(2 << 64) - 1,
25    )
26    configurator.generate_scenario('./gga-scenario')

In case we want to configure over a set of *.cnf.gz instances in a directory we could specify input data as follows:

 1from optilog.tuning import ac, Bool, Int, Real, Categorical
 2from optilog.tuning.configurators import GGAScenario
 3from optilog.blackbox import ExecutionConstraints, RunSolver
 4
 5@ac
 6def func1(
 7    data,
 8    p1: Bool() = True, p2: Real(-1.3, 2) = 0,
 9    p3: Int(-5, 5) = 0, p4: Categorical("A", "B", "C") = "A"):
10    ...
11
12if __name__ == '__main__':
13    configurator = GGAScenario(
14        func1,
15        constraints=ExecutionConstraints(
16            s_wall_time=25,
17            s_real_memory="8GB",
18            s_cpu_time=30,
19            enforcer=RunSolver()
20        ),
21        input_data='path/to/instances/*.txt',
22        data_kwarg="data",
23        run_obj="quality",
24        quality_regex="^o (\d+)$",
25        seed=1,
26        cost_min=0,
27        cost_max=(2 << 64) - 1,
28    )
29    configurator.generate_scenario('./gga-scenario')