Paper DTM tool voor conferentie in Almeria (Spanje) - september 2012
Model-Based Testing in Early Software Development Phases
Silvio Cacace and Tanja E.J. Vos
Dialogues Technology, Foppingadreef 22, 1102 BS Amsterdam - The Netherlands Universidad Politecnica de Valencia, Camino de vera s/n 46022 Valencia - Spain
Abstract. Despite the clear advantages of using test models, drawing them is not common practice in industry. This is not in the last place because, up to date, no easy to use tool existed that enables the creation of test models and implies less maintenance when requirements change. In this paper we will discuss a tool that does do these things, yet is very simple, light-weight, easy to learn and does not require experience and or knowledge of difficult formal methods. Basically the tool helps you draw a model, by providing a drawing canvas with a range of shapes and the possibility of connecting them. Secondly, the coverage algorithms underlying the tool, will extract all the combinations of the connections between the elements, and display a minimum set of test cases that can be used for functional or acceptance testing purposes.
In the eighties, Boris Beizer published his book on blackbox testing techniques [1]. The ideas in this book are simple: draw a model of the expected behaviour of the software system, and cover it. In the book, Beizer advocates that such test suite design activities should start the sooner the better. The presence of the tester in the initial development phases increases the rate of error identification, since ambiguities and unknown issues will be made clear in a very early stage and can help the business to clarify the requirements with the involvement of the stakeholders. Moreover, when testing starts early, the tester can also define the acceptance criteria in that early stage.
Using models or graphs during the early phases of testing has many identified advantages [1, 4, 6]:
– Reviewing a model is more manageable and less error-prone, than reviewing hundreds of test cases.
– Since a model is created from the requirements, it gets easier to trace the model and test cases back to
– Models represents the tester’s understanding of what the software should do In this way these models
can serve as a tool for improving the communication with other stakeholders about the functionalities
– Many times, just creating a model from the requirements and showing it to the client will already find
important errors in the requirements.
- Using a model to review requirements, takes the ’personal element’ out of reviewing. With traditional reviews, people might feel attacked about the quality of their work. A review based upon a model that the tester himself has made, reduces this element since his own model is being questioned.
- If we have a model, test case generation can be automated. If the model changes, you just generate them again.
- The scope and coverage of a test suite generated from a model becomes clear.
- A model is easier to maintain than test cases when requirements change.
However, despite the clear advantages of using test models, drawing them is not common practice in industry[2]. This is not in the last place because, up to date, no easy to use tool existed that enables the creation of test models. According to a survey done by Binder last year [2], most Model-Based Testing tools that are around and used by industry are found to be too complex, too formal and have a steep learning curve. In this paper we will introduce a tool called DTM (Dialogues Testing Method) [5] a very simple, light-weight and easy to learn tool that enables testers to eas- ily create models that can be used for effective communication and automated test case generation. Basically the tool helps you draw a model, by providing a drawing canvas with a range of shapes and the possibility of connecting them. Secondly, the coverage algorithms underlying the tool, will extract all the combi- nations of the connections between the elements, and display a minimum set of test cases that can be used for functional or acceptance testing purposes. In the following sections we will describe DTM’s methodology, the control flow models, test case generation. Finally, Section 5 concludes with future work. The length of this paper only permits illustration of the models and the generated test cases through very simple examples. However, DTM has been used on various real-world software systems and their requirements.
2 Model-Based Testing Methodology
The MBT methodology behind the DTM tool consists of the following steps:
1. Use the requirements documents that are available to create a model. This is an iterative process that consists of:
(a) Understand the requirements and make a mental model of the system
(b) Translate the mental model in a flow model and leave open ends for the items that were unclear, inconsistent, ambiguous, and need more explanation from business people, designers or other stakeholders.
(c) If there are open ends, organize a brainstorm meeting with the necessary stakeholders to close them.
(d) Update the requirements and update your test model with the obtained information and clarifications.
(e) If there are no open ends left, review the model with the stakeholders.
(f) If the model is found complete during the review meeting, continue with
step 2. If not, go back to step 1d.
Model-Based Testing in Early Software Development Phases 3
Fig. 1. A simple DTM model for calculating a bonus based on amount of years worked.
Generate test cases from the model using a preferred coverage criteria.
Execute the tests, debug and fix if defects are found, and update the require-
ments and the model if necessary.
If the requirements do not need to be updated, the test suite can be saved
3 Creating a model with DTM
The models used in the DTM tool are control flow graphs. The DTM tool only offers 6 symbols that you can use to draw your test model:
– start (a circle with the word start in the centre), is used for indicating the beginning of the control flow.
– end (a circle with the word start in the centre), is used for indicating the ending of the control flow.
– decision (a diamond) for making binary decisions. The right-arrow is the YES option, the down-arrow is the NO option.
– action (an oval), is used to model actions or states.
– result(awavedrectangle),isusedforstoringvaluesinvariablesordatabases.
– reference (a pentagon), is used to reference another test model.
An example of a simple model is in Figure 1. The model does not contain actions, nor references and serves merely as an example for the next Section.
4 DTM’s Test Case Generation capabilities
When the test model is drawn, abstract test cases can be generated just by one- click. The DTM tool will automatically place numbers beside the arrows coming out from each decision symbol and all the possible test cases will be generated based on a decision coverage algorithm that ensures that each combination of two decisions will be tested at least one time. If you generate your test cases on a test model which is not correct, e.g. the model is missing an end symbol, a
path is not complete, etc., you will be notified by a pop up message. The format of the test cases is in plain text and can be exported to Excel (see Figure 2 for the test cases generated form the model in Figure 1).
Fig. 2. DTM test case descriptions in Excell
5 Conclusions and Future Work
We have briefly introduced the philosophy, the methodology, the control flow models and the test case generation capabilities of the DTM tool. When using models during early phases of a development project, the mere fact of creat- ing a model can reveal inconsistencies, ambiguities and incompleteness in the requirements. This saves time and money, as Boehm [3] indicated in 1976.
Future work that is being contemplated for extending the tool is related to sensitization of the abstract test cases with concrete test data, additional coverage criteria for generating test cases, integration and connection with tools for test management and bugtracking, and empirical evaluations of the usability (i.e. effectiveness, efficiency and subjective satisfaction) of the tool.
Boris Beizer. Black-box testing - techniques for functional testing of software and systems. Wiley, 1995.
Robert V. Binder. Model-based testing user survey: Results and analysis, 2011.
B. W. Boehm. Software engineering. IEEE Trans. Comp., 25(12):1226–1241, 1976.
Alan Richardson. Practical experiences with graph based software testing. In Star-
East Software Testing Conference, 2003.
DTM tool. http://www.dtmtool.com/, last visited 27 april 2012.
B. Visser and B. Vrenegoor. First model, then test! Testing Experience, 3, 2012.
