Like some of our colleagues before him (see e.g. here), Aykut also wrote his bachelor thesis at Faktor Zehn following his working student activities. Here he worked on the development of a prototype for calculating the threshold coverage for JVM languages. We are therefore very happy to present you an article by Aykut about his bachelor thesis today:
“More than 2 years ago I (Aykut Yilmaz) started as an intern at Faktor Zehn and then continued working as a working student, which is why I decided to write my bachelor thesis at Faktor Zehn. In my bachelor thesis I worked on the development of a prototype to calculate the Boundary Value Coverage for JVM languages.
In software development there are now a lot of metrics to measure the quality of source code. They help the developers to verify the functionality of the software and facilitate future maintenance and enhancement work. Thus they keep the economic effort low in the long run. One of these metrics is test coverage. However, over time, many different test methods have been developed that check the quality of the code from a specific point of view. Some of these test methods have even become automated tools, helping developers around the world to maintain the quality of their software and detect errors.
With the motivation to further eradicate quality defects in software, my bachelor thesis explores another method to measure test coverage and tries to implement it as an automated tool. The method to measure this test coverage is called limit coverage.
Limit coverage is originally a method to find test cases and comes from the category of black box test procedures. It is an extension of the equivalence class formation. When equivalence classes are formed, all possible values of an input parameter of a function are divided into groups which should cause equivalent behavior in the function. Thus, a single representative of the group should be sufficient to test the entire behavior. The limit value coverage goes one step further and requires that test values must be used immediately at each boundary of two adjacent equivalence classes. This is based on the observation in practice that errors in the software are often found at the transitions from one equivalence class to the next. Furthermore, testing at the boundaries is better at detecting errors from future changes.
For my bachelor thesis I used the code coverage tool JaCoCo as a basis, since its source code is publicly available on GitHub and is also used in the development of FaktorZehn itself. JaCoCo is, with the help of bytecode manipulation, able to collect execution data of the software (tests) and generates a quality report based on the collected data, which includes various test metrics. Since the collection of execution data is massively based on bytecode manipulation, a profound knowledge of JVM bytecode as well as the ASM framework (framework for bytecode manipulation) is necessary. Additionally, JaCoCo uses Java agents and the Instrumentation API to perform these manipulations at runtime. In order to now also include the limit value coverage in this process, adjustments had to be made to the source code of JaCoCo.
As mentioned before, the limit value coverage is a black box process. However, JaCoCo is limited to byte/source code. For this, the limit value coverage had to be redefined first. However, this has the side effect that the tool only roughly approaches the actual limit value coverage instead of exactly mirroring it. Furthermore, the bachelor thesis is mainly limited to integer primitive types and the types char. Not only during redefinition, but also during application to the JVM platform, problems arise which are addressed with a pragmatic approach.
Due to the certain imprecision that arises, the tool is not suitable as an acceptance criterion for a CI platform (such as Jenkins). However, it does help to uncover insufficiently tested code, which led the paper to the conclusion that the calculation of the threshold coverage should rather be a separate tool that is executed on demand.
I would like to thank Faktor Zehn for the opportunity to write my bachelor thesis with you. I am also grateful for the freedom and flexibility in choosing the topic of my bachelor thesis and its elaboration, which was made available to me.”
Author: Aykut Yilmaz