The most significant benefits of traditional test automation are the decreased test execution time and cost, and increased test coverage for each testing cycle. The downside of test automation is the increased test development cost. In a typical test automation scheme, each test is first documented by a test engineer then scripted by an automation engineer. The cost benefit of automated testing comes through the repeated execution of the test automation suite. Since the cost of executing an automated test is far less than the cost of executing a manual test, eventually the decreased test execution cost will make up for the increased development costs.
For example, suppose that a test engineer costs your organization $60/hr, and an automation engineer costs your organization $100/hr. The test engineer can create 5 new test cases per hour during test plan development, and can execute 10 test cases per hour during test execution. The automation engineer can develop 2 test cases per hour. Once those test cases are completed, a test engineer can execute and analyze the results of 50 tests in one hour (typically the test engineer will run the tests overnight, or while he is doing some other work, then analyze the results once the test is done).
Suppose you create a test plan with 1000 test cases:
For manual testing, your initial test development cost is:
1000 tests * 1 hour/5 test cases * $60/hour = $12,000
The cost of executing all of the tests manually is:
1000 tests * 1 hour/10 test cases * $60/hour = $6,000
For automated testing, your initial cost is:
$12,000 + 1000 tests * 1 hour/2 test cases * $100/hour = $65,000
The cost of executing all of the test cases w/ test automation is:
1000 tests * 1 hour/50 test cases * $60/hour = $600
The cost benefit of test automation is realized by running the tests several times (i.e. with each weekly or nightly build of your application).
In this case, we begin to see a cost savings if we run the tests more than 12 times.
There are several reasons why people choose to automate tests:
o Speed up testing to accelerate releases
o Allow testing to happen more frequently
o Reduce costs of testing by reducing manual labor
o Improve test coverage
o Ensure consistency
o Improve the reliability of testing
o Allow testing to be done by staff with less skill
o Define the testing process and reduce dependence on the few who know it
o Make testing more interesting
o Develop programming skills
So here we go with the top challenges:
1) Testing the complete application:
Is it possible? I think impossible. There are millions of test combinations. It’s not possible to test each and every combination both in manual as well as in automation testing. If you try all these combinations you will never ship the product
2) Misunderstanding of company processes:
Some times you just don’t pay proper attention what the company-defined processes are and these are for what purposes. There are some myths in testers that they should only go with company processes even these processes are not applicable for their current testing scenario. This results in incomplete and inappropriate application testing.
3) Relationship with developers:
Big challenge. Requires very skilled tester to handle this relation positively and even by completing the work in testers way. There are simply hundreds of excuses developers or testers can make when they are not agree with some points. For this tester also requiresgood communication, troubleshooting and analyzing skill.
4) Regression testing:
When project goes on expanding the regression testing work simply becomes uncontrolled. Pressure to handle the current functionality changes, previous working functionality checks and bug tracking.
5) Lack of skilled testers:
I will call this as ‘wrong management decision’ while selecting or training testers for their project task in hand. These unskilled fellows may add more chaos than simplifying the testing work. This results into incomplete, insufficient and ad-hoc testing throughout thetesting life cycle.
6) Testing always under time constraint:
Hey tester, we want to ship this product by this weekend, are you ready for completion? When this order comes from boss, tester simply focuses on task completion and not on the test coverage and quality of work. There is huge list of tasks that you need to complete within specified time. This includes writing, executing, automating and reviewing the test cases.
7) Which tests to execute first?
If you are facing the challenge stated in point no 6, then how will you take decision which test cases should be executed and with what priority? Which tests are important over others? This requires good experience to work under pressure.
8 ) Understanding the requirements:
Some times testers are responsible for communicating with customers for understanding the requirements. What if tester fails to understand the requirements? Will he be able to test the application properly? Definitely No! Testers require good listening and understanding capabilities.
9) Automation testing:
Many sub challenges – Should automate the testing work? Till what level automation should be done? Do you have sufficient and skilled resources for automation? Is time permissible for automating the test cases? Decision of automation or manual testing will need to address the pros and cons of each process.
10) Decision to stop the testing:
When to stop testing? Very difficult decision. Requires core judgment of testing processes and importance of each process. Also requires ‘on the fly’ decision ability.
Over to you:
Many of you are working in manual and/or automation testing field. Though I have addressed many of above challenges in our previous articles, I want your views on handling these software testing challenges. Feel free to express your views in comment section below.
I am Masud Parvez. Working as IT Senior Project Manager for RMIT University. Previously I built and run a distributed Test Center. My success was to turn that in to one of the most successful business units of the company.