diff --git a/course/MutationTesting.md b/course/MutationTesting.md index 7715ca7acb8ac9bb0833a4e1f46e215902b2d133..2efe48f071d5991e5910cc875d2212b48c8c0a51 100644 --- a/course/MutationTesting.md +++ b/course/MutationTesting.md @@ -20,9 +20,6 @@ However, in this lab, we will introduce [Dextool](http://joakim-brannstrom.githu to develop and maintain high-quality tests. ## Preparation -First of all, make sure that you read the README.md for the example project [game tutorial](https://github.com/joakim-brannstrom/dextool/blob/master/plugin/mutate/examples/game_tutorial/README.md) thoroughly. -Instructions described and presented in the README assumes that you are in the correct directory before executing any commands/scripts, *plugin/mutate/examples/game_tutorial/*. - The lab will involve writing a few lines of C++. It is not required that you understand all the caveats of C and C++. However, we encourage you to get acquainted with the documentation for [Google test](https://github.com/google/googletest/blob/master/googletest/docs/primer.md). @@ -31,43 +28,72 @@ However, we encourage you to get acquainted with the documentation for [Google t Dextool is a framework for writing plugins using libclang. The main focus is on tools for testing and static analysis. In a standard installation of Dextool, six different plugins are included. The mutation testing plugin, [Dextool Mutate](https://github.com/joakim-brannstrom/dextool/tree/master/plugin/mutate), functions in such a way that the user provides a configuration-file where scripts, settings, and different paths are specified. The picture above shows the flow for the plugin, where the test part of mutation testing, is depicted in a more detailed manner. As shown in the image, the plugin is divided into different parts (executable commandos) - *analyze*, *test* and *report*. There are two more executable commandos for Dextool Mutate, *admin* and *generate*, but these can be ignored in this lab. See [Dextool Mutate README](https://github.com/joakim-brannstrom/dextool/tree/master/plugin/mutate) for more information. ### Setup -In order to set up and use Dextool, it needs to build first. See the [README](https://github.com/joakim-brannstrom/dextool/blob/master/README.md) for a more detailed walkthrough. Below are the same steps listed, but in a shorter version. - -1. Open your command line terminal and add the module files for the course: - - module add courses/TDDD04 - -2. Clone the repository by executing the following in your command line terminal: - - git clone https://github.com/joakim-brannstrom/dextool.git --branch v1.3.2 --single-branch - -3. Create a build directory for the installation: - - pushd dextool - mkdir build - popd - -4. Setup [cmake](https://cmake.org/): - - pushd dextool/build - cmake -DCMAKE_INSTALL_PREFIX=. -DD_COMPILER=/path/to/d/compiler .. - popd -**NOTE**: Change *\#path/to/d/compiler* to correct path to your D-compiler. If you are unsure of the path, execute the following in your terminal and use the output: - - which dmd - - -**NOTE**: If you change the *-DCMAKE_INSTALL_PREFIX* to something else, you have to remember that path for later (specifying a certain path for your binaries could be highly individual for developers, hence the reason for the flag) - -5. Make the project and install the binaries: - - pushd dextool/build - make install - popd - - -The previous steps **should** provide you with a freshly built version of Dextool and also the executable binaries for the different plugins in a standard installation. If you encountered any problem during the setup/installation of Dextool, ask the lab-assistant to guide you (don't hesitate to create issues on the [Dextool github-page](https://github.com/joakim-brannstrom/dextool) if anything needs to be improved or clarified in the README-guide). +**NOTE** +If you use **thinlinc** follow the tutorial below: + +### Using dextool on thinlinc +If you wish to use dextool on thinlinc there exists binaries that allows you to do so. +open the thinlinc terminal and run the following commands: + +``` +module add courses/TDDD04 +alias dextool=/courses/TDDD04/dextool/dextool +``` +In the given terminal execute: +``` +dextool --help +``` +The expected output should be: +``` +usage: + dextool <command> [options] [<args>...] + +options: + -h, --help show this global help + -d, --debug turn on debug output for detailed tracing + --version print the version of dextool + --plugin-list print a list of plugins + +commands: + help + uml generate PlantUML and GraphViz diagrams + graphml static code analyse information as GraphML + mutate mutation testing plugin + ctestdouble generate a C test double. Language is set to C + cpptestdouble generate a C++ test double. Language is set to C++ + fuzzer generate a wrapper for a block of code to fuzz with AFL + analyze static code analysis of c/c++ source code + example print all AST nodes of some c/c++ source code + + +See 'dextool <command> -h' to read about a specific subcommand. +``` +Now let's see if we have all the plugins for dextool. +Run the following command: + +``` +dextool --plugin-list +``` + +The output should be: +``` +uml +graphml +mutate +ctestdouble +cpptestdouble +fuzzer +analyze +example +``` + +#### Using the latest version of Dextool (If you wish to conduct the lab on your own machine) +In order to set up and use Dextool, it needs to build first. See the [README](https://github.com/joakim-brannstrom/dextool/blob/master/README.md) for a more detailed walkthrough if you wish to use the latest version of Dextool. + +The steps **should** provide you with a freshly built version of Dextool and also the executable binaries for the different plugins in a standard installation. (**don't hesitate** to create issues on the [Dextool github-page](https://github.com/joakim-brannstrom/dextool) if anything needs to be improved or clarified in the README-guide). + +**NOTE: This option is might be harder to setup and install compared to the thinlinc alternative.** ## Part 1 - Manual mutation testing Consider the following code snippet written in C: @@ -109,21 +135,26 @@ Present the findings of this small investigation and in your report, include a t For this small exercise we would like that you apart from providing a structured answer to your investigation provide us with an answer to the following questions: -* What is the practical use case of introducing mutants. Say that the five pairs of input and output printed to the terminal was our tests? +* **What is the practical use case of introducing mutants. Say that the five pairs of input and output printed to the terminal was our tests?** -* Why would we in an industrial context want to automate the process of introducing mutants? +* **Why would we in an industrial context want to automate the process of introducing mutants?** ## Part 2 - Automatic mutation testing First of all, make sure that you have done the **Preparation**. In this section, we would like you to use mutation testing on a small roguelike game. +Instructions described and presented in the READMEs assumes that you are in the correct directory before executing any commands/scripts, *plugin/mutate/examples/game_tutorial/*. + +### Running on your own machine via a custom installation Read the **instructions** for [game tutorial](https://github.com/joakim-brannstrom/dextool/blob/master/plugin/mutate/examples/game_tutorial/README.md). +### Running on thinlinc +Read the **instructions** for [game tutorial](https://github.com/joakim-brannstrom/dextool/blob/v1.3.2/plugin/mutate/examples/game_tutorial/README.md). There exist a few tests for this game already. After you have read the instructions, we would like you to run dextool in the way that it is described in the README. Once you have done so, you should analyze the results. -**Note that the game tutorial is located in dextool/plugin/mutate/examples/game_tutorial** +**NOTE: that the game tutorial is located in dextool/plugin/mutate/examples/game_tutorial** -What do the results of running dextool tell us about test quality? Relate the findings from doing this experiment to the first lab. Would we make similar observations for tests for highly coupled code as in the Colony example? +- **What do the results of running dextool tell us about test quality? Relate the findings from doing this experiment to the first lab. Would we make similar observations for tests for highly coupled code as in the Colony example?** *Hint:* Consider the test case similarities! @@ -131,7 +162,7 @@ What do the results of running dextool tell us about test quality? Relate the fi In this part, we would like you to extend the test suite. Provide an extension to the test suite with the goal of improving the mutation score and optionally coverage. For each new test, you add to provide a rationale for why this particular new test is being introduced. -For passing, it is enough that the mutation score is somewhat improved. +For passing (Grade 3), it is enough that the mutation score is somewhat improved. *Hint:* Use the results from the previous part, i.e., the generated mutation and coverage reports. @@ -141,13 +172,12 @@ For passing, it is enough that the mutation score is somewhat improved. [Equivalent mutants](http://madeyski.e-informatyka.pl/download/Madeyski13TSE.pdf) and how to deal with these is an active research topic. We would like you to read up on what an Equivalent mutant is and then do the following tasks: -* Find at least one equivalent mutant in the **fizz_buzz** example. You might have already found one! +* **Find at least one equivalent mutant in the fizz_buzz example. You might have already found one!** -* If we consider mutation score when doing automated mutation testing. Why is the existence of equivalent mutants a practical problem? +* **If we consider mutation score when doing automated mutation testing. Why is the existence of equivalent mutants a practical problem?** ## Reporting your results See the general instructions for successfully completing the lab assignment. -Note that you will have to upload the results on Gitlab before demonstrating to your assistant. Write a lab report that answers the questions stated above. For **Part 2** and **Part 3** we would like to see **screenshots** of the tool in action that **motivate** the answers to the questions raised