Tuesday, October 7, 2014

ISO29119 - And its possible future

Let’s rejoice!
Test and Quality Assurance can no longer be called a second hand job or something that just needs to pass, we have a ISO standard now to prove this.
Great news everyone! Or?
Pretty much every conference, blog and especially Twitter has had some sort of comment or information regarding ISO29119 these last months. This has ranged from analyzing the people involved in creating the standard to the content. It’s been a heated debate at some points and also a fairly one sided debate where the opponents have had the loudest voices, and the pro-ISO have been surprisingly quiet.
So what is actually going on?
I will give my views on things as far as I know right now given that information is still coming and the complete standard has not yet been released.
If we start off with the content in the standard, there is a slight problem right away. One must buy it from either ISO or IEEE, which is quite normal procedure in regards to standards. But once you have acquired it, it is yours and yours only, this according to the licensing agreement: “licensee's use only. No further reproduction or networking is permitted.” So how to discuss the content then? Below is my take on the standard.
Part 1
Part 1 is fairly straight forward it focuses on definitions and concepts of testing. If you have read or done the ISTQB Foundation course it will be a familiar read.
There are some interesting passages to note that sets the level for the coming parts.
It is recognized that software testing is a process and that the standard will present and describe a generic process. This process should be planned, monitored and controlled. The result of this process would be to provide information regarding the quality of the software under test and find issues as early as possible
A clarification might be in order here and that is that it is quite clear that the standard is based on the waterfall project model with sequential flow for the project and testing. Also the standard uses risk-based testing as the foundation for the dynamic test process.
Will cover the implications this might have later in the article, and the risk-based testing will be covered in Part 4 of the standard which is to be released late 2014
Part 2
This part of the standard covers the test process in more detail and its intended usage
Looking at the presented test process model, image below [1], it once again seems fairly standard, excuse my pun.
Detailed View of the ISO/IEC/IEEE 29119 Test Process model
We have the organizational process that sets the overall setting for test across the organization. The Management process goes a bit deeper and focuses on how each project should handle testing and reporting. And finally the actual testing is covered by the dynamic process.
When looking at the overview of the process it feels simple and logical but once you continue reading the complexity increases, below is a diagram [1] on how the test plan process is described
ISO/IEC/IEEE 29119 Test Planning Process
Again at first glace this is normal work process for creating a test plan, i.e. gather and analyse information, present the analysis and get a sign off.  So no biggie then?
Well if you are to conform to the standard this is the way to go and as we are talking about a standard its the only way and you must follow all the steps regardless of the size of your project. And if changes are needed one is supposed to go through many of the steps to get the change approved.
Then comes the monitoring and control process and yet again things get complex to the point where it looks like the process will take more time than the actual testing.
As stated before the standard is focused on test cases and therefore this is described in detail, the writers acknowledges that writing test cases is a time consuming activity but also maintains that it may lead to savings over time.
The dynamic test process covers how to identify, create and review test cases, setup the environment and report incidents and much more
The complete process described is a lengthy one and involves many steps and iterations from identifying the requirements to creating and reviewing test cases and on to creating test procedure specifications before testing may start. Not to be left out is the process of reporting incidents and a closing report.
Part 3
This part of the standard covers the documentation that the processes described in Part 2 should have as output. As the processes is quite complex with multiple steps the documentation follows suite and there are a lot of documents to create. The fact that this part is 138 pages long should give you quite a good idea on the amount of document and the detail needed in each.
A detailed test policy and test strategy for the organization might be a good idea as these are normally only created when major changes happens within the organization.
The more you follow the process set in part 2 even more documents are needed or recommended. So for each test phase a new test plan should be created and then a separate test plan for each of the subsequent systems under test.
When we finally reach the actual test there are recommendations on what needs to be documented here as well, ranging from test design specifications to environment readiness reports and on to incident reports.
Much of this can and is handled by the test reporting tool one might use (QC, Jira etc.) but the standard still focuses on test cases and the process of creating, reviewing and executing these
Part three contains an appendix where examples for the documentation can be found and they have used both a waterfall project as well as an agile project, so the possibility is there to see how this can be applied to agile projects
So what will all this mean for test in general and me as a tester?
As I started the article it’s a good thing a standard has been created as this only further shows the importance of quality assurance. Today the quality is more important than time-to-market, as people are prepared to wait a bit longer for something that really works. Quality over quantity etc. and it is here that a test standard might fit.
Unfortunately I believe ISO29119 will not help our profession and especially not in its current state and with its claims. It states that this standard will help manage and perform software testing within ANY organization.
According to the standard there are different levels of conformity, either Full conformance which is as it sounds: one must follow the standard to the fullest. Or Tailored conformance which is where it’s up to the organization to justify where and why it has chosen to step away from the standard.
The tailored variant is there so that the standard can be adapted to a more agile work process. This to me is a bit of an oxymoron: create a standard that one can claim to follow as long as you justify why one has selected to not use or disregard the standard. Sure there must be some leeway even for a standard but not to the extent possible within ISO29119.
One concern regarding ISO29119 is that: Can one actually standardize an intellectual process? In my opinion this is quite difficult and will hamper the ability to actually test. this given the fact that each project and system is different, and needs different approaches depending on the situation. Now each tester and test manager must conform to a set standard and practice regardless of situation and system under test.
Another concern that I see is on a more organizational level which the standard also affects given the way it is written, with the major focus and structure on waterfall and test cases. This means that for an organization to implement and adopt the standard major organizational changes must be made.
The last concern is also the one that most likely will kill this standard as companies will not be willing to take the cost and risk needed to change and adopt to it. The unfortunate backside of this might be that it will leave a bitter aftertaste for those who has read it and then discarded it.
The standard is written and presented as yet another attempt to define a process and way-of-working for test. Had it just been another book or white paper this would have been read and in some cases used as well. But now as it is presented as a Standard with the backing of ISO/IEEE it puts things in a whole new light. Suddenly things are set in stone (maybe not, given the Tailored compliance) and this is how testing should be done. This is also what will be the standards downfall. Few or possibly no organization will be willing to take the time and cost to change its current setup to conform to the standard. The risk that I see is that new organizations or possibly government run departments will adopt this, but this will be a minor risk.
As an example just look at the ISO12207 - Systems and software engineering — Software life cycle processes this has been around since 1995 with the latest revision in 2008. In the years i have been working with SW development, support and test i have never seen or heard about this before. And that what i believe will happen to ISO29119 as well. Unfortunately before it vanishes into the annals of civilization it will give a bitter taste and give quality assurance a bad and backward striving impression.


  1. One reason why "...the pro-ISO have been surprisingly quiet." can be What you Resist, Persists. In other words, the more fuel that is added to the fire, the longer it will burn. I think the ISO WG did the right thing in issuing a response to the petition and then go dormant. I actually expected that.

    Regarding "...the opponents have had the loudest voices", this does not surprise me either. In order to have an opposition noise has to be made so that the ones being opposed are driven to react.

    This has certainly been an interesting petition and corresponding debate. Interesting because its not just a two sided debate; its a three sided debate where the pro's and the agnostics are debating while the side being opposed remains quiet and not "resisting".

    1. I agree it can be a bad thing to fan the flames of discontent, but being silent is also a statement.
      There might be a need to create a better forum for this discussion than using blogs and twitter.

      Having the loudest voice isn't always a good thing, yes you will get the attention but there is also the risk that people back away. so once again a more mediated debate is called for.

      I agree that there are three sides to this, but the way i see it is: Pro-ISO, the opposition and the quiet mass.
      And its this quiet mass that's the interesting part, why are they quiet?
      No interest in this, they don't believe the standard will affect them, or just tired of the debate in general?

      I guess we'll just have to wait and see what the coming months will show, as the last parts are to be released soon and possibly some company/companies will adopt the standard and then there will be a use-case to study.