Hands-on CTLog

In this section we will present a full example on how to use ctlog to build CAs for the example in the Introduction to Combinatorial Testing section. To start, ctlog should be already installed in your system.

Defining the System Under Test

ctlog supports different representations for Systems Under Test (SUTs). See the Input SUT formats section for more details.

In the following we will represent the SUT of the Introduction to Combinatorial Testing section using the ACTS format:

[System]
Name: DefaultName

[Parameter]
OS (enum) : L,W,M,i,A
Pl (enum) : F,S,C,A
Re (enum) : K,F,H,W
Or (enum) : P,L

[Constraint]
C1: (OS != L || Or = L)
C2: (OS != L || Pl != A)
C3: (OS != W || Or = L)
C4: (OS != W || Pl != A)
C5: (OS != M || Or = L)
C6: (OS != M || Pl != A)
C7: (Pl != S || OS = M || OS = i)
C8: (OS != i || Re != K)
C9: (OS != A || Re != K)

You can check the ACTS format to find a full description of the ACTS format. For our example, we will define the parameters as enum. You can use the full expressiveness of the ACTS format to define non-CNF constraints (we handle them by using OptiLog’s Modelling module).

We will assume this SUT is stored under the sut.acts file. With this file we can now apply any of the algorithms implemented in ctlog to generate MCACs.

Generating an MCAC using CTLog

Once we have defined our SUT, we can generate an MCAC using any of the MCAC Algorithms implemented in CTLog. For this example, we will use the IPOG algorithm for strength 2:

ctlog IPOG sut.acts -t 2

This command would produce the following output:

c Params domains: [5, 4, 4, 2]
c Fixing first parameters: [0, 1]
c Num tests: 14
c ***** Extending parameter '2' (3 of 4 params; domain: 4) *****
c Num tuples: 36
c Horizontal extension. Num covered: 25
c Vertical growth. Num fit: 0; New tests: 8; Num fit in new: 0; Num forbidden: 3
c ***** Extending parameter '3' (4 of 4 params; domain: 2) *****
c Num tuples: 26
c Horizontal extension. Num covered: 22
c Vertical growth. Num fit: 0; New tests: 0; Num fit in new: 0; Num forbidden: 4
c T: OS,Pl,Re,Or
c T: L,F,K,L
c T: L,C,F,L
c T: W,F,F,L
c T: W,C,K,L
c T: M,F,H,L
c T: M,S,K,L
c T: M,C,W,L
c T: i,F,W,P
c T: i,S,F,P
c T: i,C,H,P
c T: i,A,F,L
c T: A,F,F,P
c T: A,C,H,L
c T: A,A,W,P
c T: L,F,H,L
c T: L,F,W,L
c T: W,F,H,L
c T: W,F,W,L
c T: M,F,F,L
c T: M,S,H,L
c T: M,S,W,L
c T: i,A,H,L
Test suite size: 22
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

As we can see, the output MCAC is printed through standard output by default. However, there is no need to parse this output if we want to extract the MCAC, as we can directly write the MCAC to a file by using the --out-file parameter:

ctlog IPOG sut.acts -t 2 --out-file mcac.csv

In this case the generated output would be:

c Params domains: [5, 4, 4, 2]
c Fixing first parameters: [0, 1]
c Num tests: 14
c ***** Extending parameter '2' (3 of 4 params; domain: 4) *****
c Num tuples: 36
c Horizontal extension. Num covered: 25
c Vertical growth. Num fit: 0; New tests: 8; Num fit in new: 0; Num forbidden: 3
c ***** Extending parameter '3' (4 of 4 params; domain: 2) *****
c Num tuples: 26
c Horizontal extension. Num covered: 22
c Vertical growth. Num fit: 0; New tests: 0; Num fit in new: 0; Num forbidden: 4
Test suite size: 22
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

And the mcac.csv file:

OS,Pl,Re,Or
L,F,K,L
L,C,F,L
W,F,F,L
W,C,K,L
M,F,H,L
M,S,K,L
M,C,W,L
i,F,W,P
i,S,F,P
i,C,H,P
i,A,F,L
A,F,F,P
A,C,H,L
A,A,W,P
L,F,H,L
L,F,W,L
W,F,H,L
W,F,W,L
M,F,F,L
M,S,H,L
M,S,W,L
i,A,H,L

Using other MCAC algorithms

One of the main benefits of CTLog is the common interface that we provide for using the implemented MCAC algorithms. This way we will be able to try different algorithms over the same SUT without much hassle. For this example, we will use the RBOT-its algorithm instead of IPOG:

ctlog PRBOT-its sut.acts -t 2 --maxsat-timeout 30

The output of this comand is the following:

c Initializing PRBOT-its with start tuple 'None' and end tuple 'None'
c Finish TuplsMngr initialization
c Memory consumption (MB): 80.453125
c [PoolManager] No more unseen tuples
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 1
o T:1;TCov:6;AccCov:6
c Num forbidden: 2
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 2
o T:2;TCov:6;AccCov:12
c Num forbidden: 2
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 3
o T:3;TCov:6;AccCov:18
c Num forbidden: 4
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 4
...
o T:20;TCov:3;AccCov:67
c Num forbidden: 11
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 21
o T:21;TCov:1;AccCov:68
c Num forbidden: 11
c Memory consumption (MB): 80.453125
c [REFINE] Expanded sliding window. Size: 22
o T:22;TCov:1;AccCov:69
c Num forbidden: 13
c [REFINE] Starting refine process with 69 tuples and 22 tests
c Memory consumption (MB): 80.453125
c [REFINE] LB 18
c [REFINE] Finished maxsat encoding
c Memory consumption (MB): 80.96875
c Not deciding over [] vars
c Cost: 3; Best Cost: inf
o 3
s OPTIMUM FOUND
c [REFINE] Improved UB (before: 22; now: 21; start index: 0)
c Updated test suite. New size: 21 Num new covered: 0
c [REFINE] Update sliding window. Current size: 22
c [REFINE] Refinement END
c Memory consumption (MB): 81.33203125
o T:21;TCov:0;AccCov:69
c Memory consumption (MB): 81.33203125
c Num forbidden: 13
c T: OS,Pl,Re,Or
c T: L,C,K,L
c T: i,F,F,P
c T: M,F,H,L
c T: A,F,F,P
c T: M,S,K,L
c T: W,C,H,L
c T: A,C,W,L
c T: W,F,W,L
c T: L,C,F,L
c T: i,S,W,P
c T: M,C,W,L
c T: L,F,H,L
c T: A,A,H,P
c T: L,C,W,L
c T: W,F,F,L
c T: W,F,K,L
c T: M,S,F,L
c T: i,C,H,P
c T: M,S,H,L
c T: i,A,W,L
c T: i,A,F,L
Test suite size: 21
Validating real test suite with 21 tests...
VALIDATION OK
Num covered: 69
Num forbidden: 13