.. _handson-ctlog:

Hands-on CTLog
==============

In this section we will present a full example on how to use ``ctlog`` to build CAs for the example in the :ref:`ct_intro` section. To start, ``ctlog`` should be :ref:`already installed <installation>` in your system.

Defining the System Under Test
------------------------------

``ctlog`` supports different representations for Systems Under Test (SUTs). See the :ref:`input_sut_formats` section for more details.

In the following we will represent the SUT of the :ref:`Introduction to Combinatorial Testing section<ctlog/ct-intro:System Under Test (SUT)>` using the `ACTS <https://csrc.nist.gov/CSRC/media/Projects/Automated-Combinatorial-Testing-for-Software/documents/acts_user_guide_2_92.pdf>`_ format:

.. code::

    [System]
    Name: DefaultName

    [Parameter]
    OS (enum) : L,W,M,i,A
    Pl (enum) : F,S,C,A
    Re (enum) : K,F,H,W
    Or (enum) : P,L

    [Constraint]
    C1: (OS != L || Or = L)
    C2: (OS != L || Pl != A)
    C3: (OS != W || Or = L)
    C4: (OS != W || Pl != A)
    C5: (OS != M || Or = L)
    C6: (OS != M || Pl != A)
    C7: (Pl != S || OS = M || OS = i)
    C8: (OS != i || Re != K)
    C9: (OS != A || Re != K)

You can check the :ref:`acts_format` to find a full description of the ACTS format. For our example, we will define the parameters as *enum*. You can use the full expressiveness of the ACTS format to define non-CNF constraints (we handle them by using `OptiLog's Modelling module <http://hardlog.udl.cat/static/doc/optilog/html/optilog/modelling.html>`_).

We will assume this SUT is stored under the ``sut.acts`` file. With this file we can now apply any of the algorithms implemented in ``ctlog`` to generate MCACs.

Generating an MCAC using CTLog
------------------------------

Once we have defined our SUT, we can generate an MCAC using any of the :ref:`mcac_algorithms` implemented in CTLog. For this example, we will use the :ref:`ipog` algorithm for strength 2:

.. code-block:: console

    ctlog IPOG sut.acts -t 2

This command would produce the following output:

.. code-block::

    c Params domains: [5, 4, 4, 2]
    c Fixing first parameters: [0, 1]
    c Num tests: 14
    c ***** Extending parameter '2' (3 of 4 params; domain: 4) *****
    c Num tuples: 36
    c Horizontal extension. Num covered: 25
    c Vertical growth. Num fit: 0; New tests: 8; Num fit in new: 0; Num forbidden: 3
    c ***** Extending parameter '3' (4 of 4 params; domain: 2) *****
    c Num tuples: 26
    c Horizontal extension. Num covered: 22
    c Vertical growth. Num fit: 0; New tests: 0; Num fit in new: 0; Num forbidden: 4
    c T: OS,Pl,Re,Or
    c T: L,F,K,L
    c T: L,C,F,L
    c T: W,F,F,L
    c T: W,C,K,L
    c T: M,F,H,L
    c T: M,S,K,L
    c T: M,C,W,L
    c T: i,F,W,P
    c T: i,S,F,P
    c T: i,C,H,P
    c T: i,A,F,L
    c T: A,F,F,P
    c T: A,C,H,L
    c T: A,A,W,P
    c T: L,F,H,L
    c T: L,F,W,L
    c T: W,F,H,L
    c T: W,F,W,L
    c T: M,F,F,L
    c T: M,S,H,L
    c T: M,S,W,L
    c T: i,A,H,L
    Test suite size: 22
    Validating MCAC...
    VALIDATION OK
    Num covered: 69
    Num forbidden: 13

As we can see, the output MCAC is printed through standard output by default.
However, there is no need to parse this output if we want to extract the MCAC, as we can directly write the MCAC to a file by using the ``--out-file`` parameter:

.. code-block:: console

    ctlog IPOG sut.acts -t 2 --out-file mcac.csv

In this case the generated output would be:

.. code-block::

    c Params domains: [5, 4, 4, 2]
    c Fixing first parameters: [0, 1]
    c Num tests: 14
    c ***** Extending parameter '2' (3 of 4 params; domain: 4) *****
    c Num tuples: 36
    c Horizontal extension. Num covered: 25
    c Vertical growth. Num fit: 0; New tests: 8; Num fit in new: 0; Num forbidden: 3
    c ***** Extending parameter '3' (4 of 4 params; domain: 2) *****
    c Num tuples: 26
    c Horizontal extension. Num covered: 22
    c Vertical growth. Num fit: 0; New tests: 0; Num fit in new: 0; Num forbidden: 4
    Test suite size: 22
    Validating MCAC...
    VALIDATION OK
    Num covered: 69
    Num forbidden: 13

And the ``mcac.csv`` file:

.. code-block::

    OS,Pl,Re,Or
    L,F,K,L
    L,C,F,L
    W,F,F,L
    W,C,K,L
    M,F,H,L
    M,S,K,L
    M,C,W,L
    i,F,W,P
    i,S,F,P
    i,C,H,P
    i,A,F,L
    A,F,F,P
    A,C,H,L
    A,A,W,P
    L,F,H,L
    L,F,W,L
    W,F,H,L
    W,F,W,L
    M,F,F,L
    M,S,H,L
    M,S,W,L
    i,A,H,L

Using other MCAC algorithms
---------------------------

One of the main benefits of CTLog is the common interface that we provide for using the implemented MCAC algorithms.
This way we will be able to try different algorithms over the same SUT without much hassle.
For this example, we will use the :ref:`rbot_its` algorithm instead of IPOG:

.. code-block:: console

    ctlog PRBOT-its sut.acts -t 2 --maxsat-timeout 30

The output of this comand is the following:

.. code-block::

    c Initializing PRBOT-its with start tuple 'None' and end tuple 'None'
    c Finish TuplsMngr initialization
    c Memory consumption (MB): 80.453125
    c [PoolManager] No more unseen tuples
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 1
    o T:1;TCov:6;AccCov:6
    c Num forbidden: 2
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 2
    o T:2;TCov:6;AccCov:12
    c Num forbidden: 2
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 3
    o T:3;TCov:6;AccCov:18
    c Num forbidden: 4
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 4
    ...
    o T:20;TCov:3;AccCov:67
    c Num forbidden: 11
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 21
    o T:21;TCov:1;AccCov:68
    c Num forbidden: 11
    c Memory consumption (MB): 80.453125
    c [REFINE] Expanded sliding window. Size: 22
    o T:22;TCov:1;AccCov:69
    c Num forbidden: 13
    c [REFINE] Starting refine process with 69 tuples and 22 tests
    c Memory consumption (MB): 80.453125
    c [REFINE] LB 18
    c [REFINE] Finished maxsat encoding
    c Memory consumption (MB): 80.96875
    c Not deciding over [] vars
    c Cost: 3; Best Cost: inf
    o 3
    s OPTIMUM FOUND
    c [REFINE] Improved UB (before: 22; now: 21; start index: 0)
    c Updated test suite. New size: 21 Num new covered: 0
    c [REFINE] Update sliding window. Current size: 22
    c [REFINE] Refinement END
    c Memory consumption (MB): 81.33203125
    o T:21;TCov:0;AccCov:69
    c Memory consumption (MB): 81.33203125
    c Num forbidden: 13
    c T: OS,Pl,Re,Or
    c T: L,C,K,L
    c T: i,F,F,P
    c T: M,F,H,L
    c T: A,F,F,P
    c T: M,S,K,L
    c T: W,C,H,L
    c T: A,C,W,L
    c T: W,F,W,L
    c T: L,C,F,L
    c T: i,S,W,P
    c T: M,C,W,L
    c T: L,F,H,L
    c T: A,A,H,P
    c T: L,C,W,L
    c T: W,F,F,L
    c T: W,F,K,L
    c T: M,S,F,L
    c T: i,C,H,P
    c T: M,S,H,L
    c T: i,A,W,L
    c T: i,A,F,L
    Test suite size: 21
    Validating real test suite with 21 tests...
    VALIDATION OK
    Num covered: 69
    Num forbidden: 13