There’s normally a question on one of the testing forums about what makes a good test strategy (or plan). And there are always loads of responses from people all over the world and in many different sectors all explaining what they put in their test plan. And this is awesome. The fact you can get this information without getting up and going anywhere is brilliant. Even if 70%* of the responses reckon their way is the Best Practice.
However, 99%* of the responses always centre around the following items:
- The format of the actual document (word, font size, font, sections)
- The entry and exit criteria plus other quality gates
- The test environment, platform and stack (or lab)
- The automation strategy and tools to be used
And I guess this is fair enough. If you are asked to write a strategy then these would be great topics to think about. Most people probably download the IEEEIEIEISIIIEISISTIEEEEE 8.4.4.4.2..21.3.21.1.2 template – or whatever it is. They butcher it for months and then sit back and relax. I know they do, because they admit it. I even used to do that myself. ‘A work of art’ – that’s what I called a test strategy once.
But over the years the one very real lesson I have learnt is that the best Test Plan or Strategy in the entire world, begins and ends with the people in your team.
It’s not about the mega framework of doom or the uber gating criteria of ruin that you’ve documented on 8000 trees. It’s not about the test environment or tools you use. It’s about people. The greatest plan is the one that covers the following items:
- Who is on the team
- What skills these team members bring to the project and how they complement each other
- Understand what skills are lacking from my existing mix of people and understand how am I going to get this skill (learning, bringing someone with the skill, or working around it somehow). <– Thanks to Joel Montvelisky for this addition (http://qablog.practitest.com/)
- How the team communicate
- How the team are awesome, motivated, passionate, dedicated, confident and self assured in what they do.
And that’s all you need. Because if you have a team that meet all of the above four points, you will succeed in any environment. With or without test plans and strategies.
After all, how many of us have created a mega framework of doom test strategy, had it reviewed, had it signed off and then broken a fundamental gating rule on the first release to test? Everyone say ‘yeah’.
So before you consider the mega framework of doom approach, think carefully about the team you have on board. The right team don’t need wooden plans and strategies, documents and pointless plans. What they need is some software to kick, some requirements to soak up and some customers to please.
*Statistics may have been made up. But then 30% of people know that.
(EXCEPTION: Before the comments fill up with abuse about how we *have* to produce these mega frameworks of doom I thought I would acknowledge that under some circumstance you are required to write test plans outlining key information – but even these documents should be dedicated to the people and how they can solve a problem. Not the tools. A tool in the wrong hand is lethal.)
Hey Rob,
I think I agree with your post, but I would also add an additional point between 2 and 3, and it would look something like:
2.5 Understand what skills are lacking from my existing mix of people and understand how am I going to get this skill (learning, bringing someone with the skill, or working around it somehow).
In any case, a great perspective on an old practice!
Cheers!
Rob,
I agree that the paper-intensive (and time intensive) “IEEEIEIEISIIIEISISTIEEEEE 8.4.4.4.2..21.3.21.1.2 template – or whatever it is” approach to putting test plans together can be highly counter-productive. The more time people are spending documenting the test plan, the less time they have available to do the actual testing. In addition, the longer a test plan is, the more likely it is that feedback on it will be incomplete and ineffective.
However, let’s not get too carried away and throw the baby out with the bathwater…. It is often useful to document what conditions should be tested for in multiple test cases (if for no other reason than to ensure that tester 1, tester 2, and tester 3 test different things and that they test what appear to be the most important parts of the application).
If you are a fan of light (non-paper intensive) test plans that include some information about what test conditions should be tested in combination with one another to achieve maximum coverage with as few test plans as possible, I would encourage you to take the Hexawise test design tool out for a spin. It will help testers quickly identify what should be tested in each test script. Using it to put together a pairwise (or 2-way) test plan can be an extremely efficient way to find a lot of defects quickly. It can be used for all kinds of different testing projects, but delivers especially good efficiency and quality benefits when you are testing for potential defects that could be triggered by interactions between one or more functions or configurations.
A free version of Hexawise is available for immediate access at http://www.hexawise.com for anyone interested in exploring it further. In addition, a 10-slide pdf file is available on the home page which explains in more detail how, e.g,. 17 billion possible test cases can be reduced to 35 pairwise test cases (that are – consistent with the theme of your post – “paper-light” and summarized in a way that tends to solicit more meaningful feedback in the review process than the examples you mentioned above.
Good luck with your blog.
- Justin
______________
Justin Hunter
Founder and CEO
Hexawise – http://www.hexawise.com
“More coverage. Fewer tests.”
- Justin
Rob,
Good, thought-provoking post.
A couple thoughts:
1) I agree with you that it is counter-productive to blindly “download the IEEEIEIEISIIIEISISTIEEEEE 8.4.4.4.2..21.3.21.1.2 template – or whatever it is” and slavishly fill out lengthy test plans that resemble telephone books by the time they’re completed. The more time people spend documenting their test plans, the less time they’ll actually have to test.
2) I also agree that the skills and attitudes of the people on the team are more important than the particular format of the test plan used.
3) I wouldn’t be wise, however, to “throw the baby out with the bathwater.” Some documentation about test plans is necessary. Documenting what conditions will be tested (and in what order) in multiple test cases is worthwhile, particularly if it can be done: (a) without taking too much time to document it, (b) in a way that encourages meaningful feedback when reviewed, (c) concisely, and (d) in a way that will allow rapid modifications to the test plan as new information is discovered and/or as requirements change.
4) One possible solution about the “right” level of detail (that generates ~2 pages worth of information about what tests should be executed and what conditions should be included in each of the test cases instead of 200 pages of lengthy content that will likely be ignored) is to use a test design tool like Hexawise to generate and document test cases in concise summaries.
5) Reviewing this approach against the four goals above, (a) it doesn’t take much time to document such test cases since part of the test selection and documentation process is automated, (b) Hexawise plans encourage meaningful feedback about what should be tested and how because they are focused solely on… well, what is tested and how (so it would be easy for a reviewer to identify, for example, “Hey, you’re not testing Opera as a browser, let’s include it”), (c) Hexawise plans are very concise, and (d) Hexawise plans can be changed very quickly (e.g., add a new parameter then push “Create Tests” and all of the conditions will instantly be re-run).
5) For more information about this approach, including screen shots of concise sample test plan outputs, please see Hexawise.com. A free version of the tool, and free trial versions of fully-featured versions of the tool are available at http://www.hexawise.com/users/new
Justin Hunter
Founder and CEO of Hexawise
Hi Justin,
Thanks for the feedback. I am never an advocate of ‘no documentation’ but I do believe, as testers, we spend far too long trying to plan detailed tests and document it all for no other reason than everyone else does. (unless you have explicit customer requirements)
I agree a short document outlining the ‘plan’ of what we aim to do that iteration/sprint is crucial but more important to me is the team involved. Certain team members bring with them positive and negative traits which all affect the success of the test team. A great team can achieve great things without a plan.
I don’t refer to the actual tests that will be run in a test plan. These are so fluid that I see it as pointless to even attempt to include them in a test plan. That is a sure fire way of making the documentation out of date.
A think a simple page of information about the ‘aim’ or ‘goal’ of the testing with a cross reference to the test case information (like a low tech dashboard or test management tool dashboard). This keeps the plan accurate(ish).
Thanks again
Rob
Rob,
Thanks for your comment and for posting this thought-provoking topic.
I have some thoughts about two of the things you said in your reply of July 27th.
(POINT ONE) You said: “I do believe, as testers, we spend far too long trying to plan detailed tests and document it all.”
—> I totally agree – up to that point in the sentence anyway
The answer to this problem, in my view, is to (a) spend less time documenting test cases, (b) make sure that it is possible to change those test plans within a matter of seconds to account for any changes to requirements, and (c) as a significant added bonus (which is actually not even required for my position to be valid) use an optimization engine – based on proven Design of Experiments methods – to select the conditions that go into each of test cases so that they achieve maximum coverage in a minimum number of test cases.
(POINT TWO) – - – You said: “I don’t refer to the actual tests that will be run in a test plan. These are so fluid that I see it as pointless to even attempt to include them in a test plan. That is a sure fire way of making the documentation out of date.”
—> That’s where you lose me. While that statement has some significant valid concerns addressed in it, I see things a bit differently:
1) The argument against documenting test cases presupposes that it would take a lot of time to document the test cases that should be executed. It doesn’t need to.
2) Particularly when you are talking about trying to do testing in a situation that involves multiple potential functional interactions, and/or multiple potential configurations and/or multiple browser types and settings (e.g., in a pretty significant percentage of testing efforts), then if you choose not to document your test cases, you are extremely likely to have efficiency compromised when Tester A inadvertently retraces some of the steps of Tester B already took in some of their tests and – in addition – you are extremely likely to compromise thoroughness when neither Tester A nor Tester B thinks to test a particular combination of two (or more) things that, when tested together in the same test case, would have triggered a fault had they tested it.
2A) A good illustration of the efficiency-compromising point is made by James Bach’s quote that: “Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s foot-prints minimizes the chance of being blown up by a land mine.” While Bach was talking about this in broader terms (e.g., repeating entire regression tests), the same principle is true with sub-parts of each test case. All-else being equal, it would be a bad idea to have Tester A and Tester B both test the first six steps of an application in the same way. By creating optimized test scripts, you can easily accomplish this.
2B) A good example of the quality-compromising issue that is seen whenever tests are not outlined ahead of time is explained in “3″ below. Things will tend to fall through the cracks more than they would as compared to using a test design tool.
3) Let’s consider a defect I encountered a few weeks ago…. when you log onto Expedia.com using the Google Chrome browser, you can book your flights just fine (including seat selection, reviewing alternate dates and airlines, etc.) … at least until you try to change the credit card you are using to pay for the transaction (at which point you lose your entire transaction). Giving Expedia the benefit of the doubt, they presumably tested whether the change credit card feature worked on at least one browser and had testers checking out Chrome as a browser to make sure would work on some “happy path” use cases. No one, though, apparently tested for “Browser Type” = Chrome and “Change Credit Card?” = Yes. Creating a test plan in 30 minutes on Hexawise (and selecting 2-way or “pairwise” strength) would have ensured that that that specific scenario would have been covered.
4) By quickly inputting in a simplified version of the application into a test design tool (like Hexawise) – which can often be done within 30 minutes if you’re familiar with your application and have a good idea, going in, of what you’re testing – and then pushing the “Create Tests” button, you can very quickly create a few dozen tests than can be split up between multiple testers. These wouldn’t be the only tests you would want to run, and I would personally encourage the testers to do some Exploratory Testing mid-way through their test case execution if they saw something that made them think… “hmmm… that’s interesting…”, but I have seen 3 dozen testing projects that have used this method and every single project felt that introducing this method felt it helped them (i) address the efficiency compromising issue described above, (ii) address the thoroughness compromising issue described above, and catch more defects than they previously had that were similar to the Expedia / Chrome / Change Credit Card example listed above.
(5) Once the list of the conditions that should be tested for in each test case is created, it would just be a matter of a couple minutes to add or delete things to be tested and recreate an updated list.
(6) For anyone who hasn’t tried out creating test cases with a test design tool, I would strongly encourage you to experiment with a test design tool to help quickly generate a few sets of test cases. Ideally, also and measure the defects found per tester hour and number of overall defects found to see whether this approach would work for you. The efficiency and quality benefits achieved are often dramatic (e.g., more than twice as many defects found per hour). I’m not just saying that as a test tool vendor; try any test design tool. Hexawise has a free version but so does, say, AllPairs by James Bach.
(7) I’m getting a bit too verbose here. I’m passionate about efficient test design strategies and sometimes can’t help myself from going on a bit too long. I feel computer-aided test design, as a topic, does not get enough attention as an powerful strategy to improve testing efficiency and effectiveness (not to mention a great solution to the problems you highlighted of (i) traditional manual methods of identifying and documenting test plans can be too time-consuming, and (ii) “a sure fire way of making the documentation out of date.”)
- Justin
______________
Hexawise
http://www.hexawise.com/users/new
“More coverage. Fewer tests.”
Hi Justin,
I think we may be talking about the same thing but the good old thing of Semantics is getting in the way.
In my experiencing of writing test plans/strategies over the last 10 years I’ve never once listed the ‘actual tests’ that will be run. This list is too dynamic and changing.
I have however, written these tests and used an appropriate test management system to document, report and store these.
A test plan (in my use of the word) is the plan of how I’m going to test, what resources I may need, environments etc. The list of actual tests is seperate. There may be some cross over with ‘areas’ or ‘requirements’ but no lists of tests in the test plan.
A matrix of pairs or combinations is always a seperate document for me.
My point in the blog is that the test plan itself is losing it’s relevance in my testing world. It’s all about the people, not the paper document that is out of date almost immediately.
I’ve spent too many hours rewriting test plans to cater for new information (just because we had to write it) that now I work in a more efficient way – I simply keep the plan short. This does not mean however, that I don’t write tests or combination matrices or test coverage documents or any of the other artefacts needed for testing.
Like you mention – I keep it simple, easy to update, accessible, uncomplicated, always easy to read/print/download and always available on a network point (and backed up).
I’m streamlining my process because at the end of the day I’ve worked with too many bad testers who can’t follow a plan and too many great testers who don’t need a plan.
Great test ideas and questions I’d like to ask the system under certain circumstances come out whilst testing and so the pre-requisites of resources, plans, kit and approach are not needed anymore. Well – they are, they just don’t need writing down in a static document. They need keeping on a wiki, in my mind, on a scrap of paper – where-ever my audience, purpose and context needs them.
Your product looks very good by the way
but we have a similar thing interal here and is doing the trick nicely.
Cheers
Rob