Don't ...just... automate!
This autumn we started a series of events around the Test Automation subject, organized with the help of the Software Testing Community Macedonia. As part of the organizational group, in order to determine if this was the best idea, i had to do a bit of research on how to structure the events and what benefits this program would bring, who's our target audience and what it would contain.
And besides the conclusion to these questions, and as the events unfolded, i've got the time to reflect a bit upon this whole automated testing trend.
We all know the big need for learning it and using it so you can put it into your resume, nowadays every second job requires experience in test automation one way or the other. Which is perfectly understandable for me, and some other people that had contact with it a bit more in their careers: the market is ever changing, there are more and more new technologies and the development process is going through the roof (devops becoming so popular) - continuous delivery on daily basis, maybe even more times a day, trying to put out there more and more features, faster and faster, to keep up with the user’s increasing needs and expectations. And in this kind of world, the testers need to somehow keep up as well, with testing new features faster, doing regression so often and for so many functionalities, browsers, devices. In such a worlds of MORE and FASTER, sometimes an army of testers can’t handle the amount of work required, by the book, in these situations. And without automated tests, they will drawn into the chaos of delivering high quality services in a practically non-existing time frame.
So, naturally, companies, managers turn to this thing they’ve kept hearing that unanimously proclaimed life saver, bearer of all fortunes and glory, that is Automated Testing. This magic that, once implemented, will work by itself, will delivery faster and higher quality software, and will reduce the number of resources. They’ve heard that it’s expensive, but that in time it will have a good return of investment. It has to! - i mean, look at what it can do!! So they just need to find that person that could do this for them right away.
And that’s how the race for people with experience starts. But there are no more experienced testers out there. The same it is in most of the other fields, the demand for skilled resources in Testing and QA discipline is so high and there is a serious deficit of people with experience. Probably as mentioned before, the knowledge and skills that are expected from the perfect candidates haven’t existed until now, so these people need to keep up with the new technologies and requirements, which is not easy and it’s a whole separate subject in itself. But getting back to finding someone suitable for implementing an automation framework for your project it’s a whole lot harder. You could assume that finding someone with some degree of experience might actually give you good odds of succeeding, because most of the time you won’t exactly have much options eitherway. However this might not be quite so. And allow me to explain why. There are some common accepted notions about the Automated Testing like being done for UI mostly or first thing to automate is your regression pack in order to free up your time for exploratory testing or just get down to it and automate the suitable tests from the manual testing. Now, not that all this are totally wrong, but it really depends on the project itself and you can never, you should never start doing automation without a clear and thorough analysis and planning phase. And, as much as i am against documentation and strategies, you should really look at doing a strategy for your automation project, especially when you don’t have enough experience, because it will force you to take into consideration all the aspects and, implicitly, the real costs of implementing the automation.
And this is a major factor that should make you consider and reconsider it: the costs. Automation is a very expensive service. What everybody should understand is that you can’t invest so much in just automating your smoke tests and general regression pack, it just doesn’t pay off. The need for automation is so much bigger than that and, when done right, it takes so much time to get people and projects up and running. Mostly because there are not enough resources, skilled and experienced, to be able to start full power. So it takes too much training for ramping up people. This, in most cases, means delaying the automation from the development process, dropping functionality, making compromises, reducing coverage, which all in all are damaging the outcome of the project.
Because there they are, the expectations. Few people understand the complexity of such project and most of the stakeholders are not part of these people. Stakeholders generally don’t understand the concept and the lifecycle of the automation process (and maybe they shouldn’t!?), and why should we be surprised, when they barely understand the manual testing. They think that manual testing is a safety net that will keep them from having issues in production, even when they don’t provide with enough info and time to actually be done properly. Same with automation: if you tell them that you automated a small regression pack, they already think you can start going live every second day without doing anything else, cause you’re covered. And maybe you can also fire 2 manual testers cause they won’t have anymore work to do from now on. :) But what’s the worst part in all this is that the return of investment is not as fast as the managers expect, because obviously this is not what’s actually happening, and these are unrealistic expectations. But maybe, then again, it’s not completely their fault.
You should feed them the right information to get the correct feeling about this whole thing. And, as i'm not in favor of metrics, you can’t provide the stakeholders with a realistic estimation unless you do the math honestly. How fast are you gonna get the framework up and running? What tests will you automate? How many times are you gonna run them per release vs how much time would it take to a manual tester to test them? And all those fun stuff you find all over the internet as automated testing success metrics. This will really help you understand if it’s worth spending that time on it and not throw it away after one year cause it doesn’t bring you any value.
However, as far as i managed to see, the most common reason the automation projects fail is not necessarily the costs (as in the project runs out of budget) but more that is gives the feeling that those money are wasted. And this is because, no matter how much you struggled and you managed to automated a great deal, it’s not visible to the rest of the team, the management, the client. They don’t feel the benefits, they don’t see the results of your tests, they don’t trust the power of the tests. So you’ve done it for no real benefit.
This comes related to the fact that you can’t do automation in isolation. It needs to be included in some kind of CI/CD system and integrated with some kind of reporting tool, to make it available and clear for everybody of the state of the application/software/website etc. And there we are again talking about costs and effort to do a proper automation framework.
So, at the end of the day, Automation is not a project for everybody. If you think you can do it with one single person that would eventually replace a couple of manual testers, then you do it wrongly. If you think it will replace full regression before doing a new release, then you do it wrongly. I know that, same as Agile, where everybody gets proud to be able to offer it as a service, but so few really understand and implement it rightly, automation is exactly the same: a desirable feature, but so few people deliver quality automated testing. So please think again before you decide you will start implementing it.
If you’d like some more articles around this issue, i can warmly recommend you these. Enjoy!