CLI Reference
In this section we provide a quick reference for the Command Line Interface (CLI) of CTLog.
SAT-based Combinatorial Testing tools and algorithms in Python
usage: ctlog [-h] [--version] {IPOG,PRBOT-its,maxsat-its,CALOT,maxsat-mcac,SUT-G} ...
Positional Arguments
- command
Possible choices: IPOG, PRBOT-its, maxsat-its, CALOT, maxsat-mcac, SUT-G
Named Arguments
- --version
show program’s version number and exit
Sub-commands
IPOG
In-Parameter Order General algorithm
ctlog IPOG [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
[--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--seed SEED] [-bpt] [-bvt]
in_file
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
Input SUT
Input System Under Test (SUT) parameters
- in_file
Input SUT in one of the supported formats: acts, extacts.
- --in-format
Possible choices: acts, extacts
Format of the input SUT. If None, inferred from the extension of the input file.
- --strength, -t
Desired strength for the MCAC
Default: 2
Output MCAC
Output Mixed Covering Array with Constraints (MCAC) parameters
- --out-file
Output file where the MCAC will be written. If None -> stdout
- --out-format
Possible choices: csv, acts
Output MCAC format
Default: “csv”
- --skip-validation
Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)
Default: False
- --all-missing-tuples
Print the number of missing tuples during the validation
Default: False
Algorithm
Parameters of the algorithm
- --seed, -s
Default: 1
- -bpt, --break-param-ties
Default: False
- -bvt, --break-val-ties
Default: False
PRBOT-its
Pool-Refine Build-One-Test iterative test suite construction algorithm
ctlog PRBOT-its [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
[--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--no-complete-tests] [--seed SEED] [--pv-heuristic {exact,approx,ipog}]
[--time-limit TIME_LIMIT] [--pool-impl {py,nim}] [--max-pool-size MAX_POOL_SIZE | --max-mem-GB MAX_MEM_GB] [--maxsat-timeout MAXSAT_TIMEOUT]
[--intermediate-maxsat-timeout INTERMEDIATE_MAXSAT_TIMEOUT] [--max-window-size MAX_WINDOW_SIZE] [--min-free-tests MIN_FREE_TESTS]
[--parallel-num-pools PARALLEL_NUM_POOLS] [--max-parallel-jobs MAX_PARALLEL_JOBS] [--out-folder OUT_FOLDER]
[--submitter-script SUBMITTER_SCRIPT] [--filter-tuples] [--progressive] [--start-tupl START_TUPL] [--end-tupl END_TUPL]
[--ws-csv-folder WS_CSV_FOLDER]
in_file
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
- --no-complete-tests
Do not complete the empty values for the tests in the final test suite
Default: True
Input SUT
Input System Under Test (SUT) parameters
- in_file
Input SUT in one of the supported formats: acts, extacts.
- --in-format
Possible choices: acts, extacts
Format of the input SUT. If None, inferred from the extension of the input file.
- --strength, -t
Desired strength for the MCAC
Default: 2
Output MCAC
Output Mixed Covering Array with Constraints (MCAC) parameters
- --out-file
Output file where the MCAC will be written. If None -> stdout
- --out-format
Possible choices: csv, acts
Output MCAC format
Default: “csv”
- --skip-validation
Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)
Default: False
- --all-missing-tuples
Print the number of missing tuples during the validation
Default: False
BOT-its
Arguments for the BOT-its algorithm. By default this will be the algorithm that will be executed.
- --seed
RNG seed
Default: 1
- --pv-heuristic
Possible choices: exact, approx, ipog
Heuristic for choosing the best param-value
Default: “exact”
- --time-limit
Time limit in seconds after which the solution will be printed
- --pool-impl
Possible choices: py, nim
Implementation of the pool
Default: “py”
P(R)BOT-its
Arguments for the pool version of BOT-its and RBOT-its algorithms. Pool limit will be enforced when one of these parameters is set by the user.
- --max-pool-size
Maximum pool size of tuples
- --max-mem-GB
Maximum memory in GB consumed by the pool of tuples
(P)RBOT-its
Arguments for the refine version of BOT-its and PBOT-its. Will be activated when at least one of the timeouts is provided by the user
- --maxsat-timeout
Overall timeout in seconds for each call to MaxSAT in refine
- --intermediate-maxsat-timeout
Timeout for MaxSAT between solutions in seconds
- --max-window-size
Maximum window size in literals for MaxSAT after which the refine procedure is triggered
Default: 7500000
- --min-free-tests
Minimum number of ‘free’ tests to start refining
Default: 10
Parallel P(R)BOT-its
Arguments for the parallel pool version of P(R)BOT-its. Activated when ‘parallel_num_pools’ is not None.
- --parallel-num-pools
Number of pools per parallel process. If != None, activate parallel P(R)BOT-its
- --max-parallel-jobs
Maximum number of parallel jobs
Default: 48
- --out-folder
Output folder where partial test suites will be stored
Default: .
- --submitter-script
Submitter script (required if using the parallel mode)
- --filter-tuples
Each worker will try to filter covered tuples by previous workers
Default: False
- --progressive
Master will submit jobs progressively until reaching ‘max_parallel_jobs’
Default: False
- --start-tupl
Start tuple (only used by parallel master)
- --end-tupl
End tuple (only used by parallel master)
- --ws-csv-folder
Folder with csv files to warmstart the current worker (only used by parallel master)
maxsat-its
MaxSAT-its algorithm
ctlog maxsat-its [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
[--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--soft-param-tupls] [--filter-allowed]
[--maxsat-timeout MAXSAT_TIMEOUT] [--intermediate-maxsat-timeout INTERMEDIATE_MAXSAT_TIMEOUT] [--seed SEED]
in_file
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
Input SUT
Input System Under Test (SUT) parameters
- in_file
Input SUT in one of the supported formats: acts, extacts.
- --in-format
Possible choices: acts, extacts
Format of the input SUT. If None, inferred from the extension of the input file.
- --strength, -t
Desired strength for the MCAC
Default: 2
Output MCAC
Output Mixed Covering Array with Constraints (MCAC) parameters
- --out-file
Output file where the MCAC will be written. If None -> stdout
- --out-format
Possible choices: csv, acts
Output MCAC format
Default: “csv”
- --skip-validation
Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)
Default: False
- --all-missing-tuples
Print the number of missing tuples during the validation
Default: False
Algorithm
Parameters of the algorithm
- --soft-param-tupls
Use a variable for each parameter tuple for soft clauses (i.e. less soft constraints)
Default: False
- --filter-allowed
Filter forbidden tuples in the encoding
Default: False
- --maxsat-timeout
Overall timeout in seconds for each call to MaxSAT in refine
- --intermediate-maxsat-timeout
Timeout for MaxSAT between solutions in seconds
Default: 10
- --seed, -s
Default: 1
CALOT
CALOT algorithm with the CCX encoding
ctlog CALOT [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
[--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] --ub UB [--seed SEED] [--timeout TIMEOUT]
[--solver {Glucose41,Cadical,Cadical152}]
in_file
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
Input SUT
Input System Under Test (SUT) parameters
- in_file
Input SUT in one of the supported formats: acts, extacts.
- --in-format
Possible choices: acts, extacts
Format of the input SUT. If None, inferred from the extension of the input file.
- --strength, -t
Desired strength for the MCAC
Default: 2
Output MCAC
Output Mixed Covering Array with Constraints (MCAC) parameters
- --out-file
Output file where the MCAC will be written. If None -> stdout
- --out-format
Possible choices: csv, acts
Output MCAC format
Default: “csv”
- --skip-validation
Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)
Default: False
- --all-missing-tuples
Print the number of missing tuples during the validation
Default: False
Algorithm
Parameters of the algorithm
- --ub
Upper bound on the number of tests of the MCAC
- --seed
Random Number Generator seed
Default: 1
- --timeout
Timeout after which the MaxSAT solver will be interrupted
Default: 10000
- --solver
Possible choices: Glucose41, Cadical, Cadical152
SAT solver in OptiLog used by CALOT
Default: “Glucose41”
maxsat-mcac
MCAC generation through MaxSAT
ctlog maxsat-mcac [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
[--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] --ub UB [--msat-path MSAT_PATH] [--seed SEED] [--timeout TIMEOUT]
[--enc {ccx,cx,ccx-a1}]
in_file
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
Input SUT
Input System Under Test (SUT) parameters
- in_file
Input SUT in one of the supported formats: acts, extacts.
- --in-format
Possible choices: acts, extacts
Format of the input SUT. If None, inferred from the extension of the input file.
- --strength, -t
Desired strength for the MCAC
Default: 2
Output MCAC
Output Mixed Covering Array with Constraints (MCAC) parameters
- --out-file
Output file where the MCAC will be written. If None -> stdout
- --out-format
Possible choices: csv, acts
Output MCAC format
Default: “csv”
- --skip-validation
Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)
Default: False
- --all-missing-tuples
Print the number of missing tuples during the validation
Default: False
Algorithm
Parameters of the algorithm
- --ub
Upper bound on the number of tests of the MCAC
- --msat-path
Path to the MaxSAT solver that will be used to solve the encoding. If None use our Python Linear MaxSAT solver
- --seed
Random Number Generator seed
Default: 1
- --timeout
Timeout after which the MaxSAT solver will be interrupted
Default: 10000
- --enc
Possible choices: ccx, cx, ccx-a1
Encoding to be used. Described in ‘Incomplete MaxSAT approaches for Combinatorial Testing’
Default: “ccx”
SUT-G
Implementation of the SUT-G Combinatorial Testing Generator
ctlog SUT-G [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--ext-glob EXT_GLOB] [--timeout TIMEOUT] [--max-confl MAX_CONFL] [--min-confl MIN_CONFL]
[--n-params N_PARAMS] [--n-seeds N_SEEDS] [--inc-assumps INC_ASSUMPS] [--dec-assumps DEC_ASSUMPS] [--max-tries MAX_TRIES] [--n-jobs N_JOBS] --out
OUT [--seed SEED]
input
Positional Arguments
- input
Path to the input cnf instance. If a folder is given, all the instances with ‘ext’ are considered.
Named Arguments
- --verbose
Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG
CTLOG verbosity level
Default: “INFO”
- --ext-glob
Extension glob for the instances if ‘input’ is a folder.
- --timeout
Timeout for the SAT solver in seconds
Default: 30
- --max-confl
Maximum number of conflicts for the final instance
Default: 10000
- --min-confl
Minimum number of conflicts for the final instance
Default: 5000
- --n-params
Number of parameters of the generated SUT
Default: 30
- --n-seeds
Number of seeds to compute an average on the number of conflicts
Default: 5
- --inc-assumps
Number of variables to increse the assumptions
Default: 10
- --dec-assumps
Number of variables to decrease the assumptions
Default: 5
- --max-tries
Maximum number of tries to reach the desired conflicts’ range
Default: 100
- --n-jobs
Number of parallel jobs (only valid when processing a folder)
Default: 1
- --out
- --seed
Random seed
Default: 1