Slipping on test

As winter has arrived for a short while in Sweden a discussion got me thinking.

A friends mother has apparently decided that it will be slippery everywhere, as there is about 1 mm of snow on the ground and therefore is a bit scared to actually go out.

Do we experience similar behavior when testing ?

Do we test in different ways depending on the state of the software?

If the software is in a bad state, or early in development, where issues are common:

You are expecting issues and therefore test in a different way, as: “it will not work anyway”, and you catch the small issues and might miss the big one. Or even worse give up before even trying one scenario.

I.e one is expecting to slip on the ice and therefore stiffens up and walks completely different and is probably more likely to actually slip on each little patch of ice

Or do you “walk” normally but is aware of the ice and prepared that the foot might skid a bit. ie you test as you normally do and if you “slip” are prepared for it and can handle that issue and continue to the really important ones

Keeping the focus

After doing some interviews for possible new assignment, the approaches have been quite different. They have ranged from the “traditional” question - answer to one that felt more like I held a presentation. And given the interaction with the interviewer my own performance have been quite different.

Is there a similar interaction when you test ?

Regardless if you are running test cases or using an ET approach, the way the system under test responds might dictate they way you continue to interact with the system.

for example if the test scenario takes a long time, ie running large sql queries, one might lose focus during the waiting time and might miss issues once the system actually responds. On the contrary if the system responds quickly you can keep focus better and more likely to spot the possible issues.

For me the more challenging the interview is and if there is a good interaction with the interviewer I seem to perform better. Its a similar feeling when running tests. Early in the project the system is new and its challenging to test. One must get “to know” the system/function and issues are often common. You get continuous feedback and expand you knowledge.

But the longer the testing runs you start to know the system and issues have been handled so the interaction and response might not be that immediate. which in turns might mean the focus is dropping of and issues can be missed

How do you keep focus if the interaction and response from the system drops, or when the test phase is long ?

Handing over the project to maintenance

Finally the project is nearing the end and handovers are being planned, kick-out dinner is booked and the next project is being staffed up. But there are still some open defects, how can this be?

Shouldn't we deliver a complete software, there should not be any defects, or ?

As stated over and over testing can continue until the fingers bleed, and yet there will be new issues found or some issues might not have been solved. So how do you handle this ?

A good handover is the key, and as usual good communication. The receiving organization must be aware of the current status and if they can accept a delivery with open issues. Also there should be a plan for handling these open issues.

Usually some tool is used for tracking issues during development, QC, Jira or similar. But when it's handed over to maintenance, they might not use the same tool, or the issue handling process is completely different as the product is now in production.

Make sure the open defects are clearly described with steps to recreate.

Ensure that when handover is done that the receiving organization understands the issue and the possible impacts. Also make sure that they understand the issue, what seems obvious to you might not be for another person.

Good practice is also to do a follow up later to see how these issues have been handled. As it is easy to forget and if not fixed they will come back in one way or the other.

Emergency room

After spending an afternoon and evening in the hands of healthcare due to bike crash and resulting fracture in the arm, i started thinking on how this is similar to development and test.

The scenario.

I am biking home from work and colliding with a pedestrian resulting i a short flight and a stiff and throbbing arm.

Defect report was raised to “first line support” and providing scenario and symptoms. The initial investigation showed that it most likely was nothing serious and rest was prescribed.

So a minor defect and nothing that needed to be re-written or was a show stopper.

A few days later the arm still was not better so a new call to First-line was made and as the symptoms had changed slightly it was decided that a doctor should look at it.

The minor defect now re-appeared, and was re-opened, but in a slightly different way and a little different outcome.

Finally arrived at the doctors office and was meet by the nurses who made an initial investigation and decided that x-ray was needed. This was done and the images show to the doctor.

Investigating the defect to find the seriousness of it.

Quick meeting with the doctor confirms the fracture and a cast is needed

Product owner checks the defect and decides it needs to be fixed.

Once again meeting the nurses and getting a cast. Information on what to think about and how to handle the cast.

Defect fix information and a workaround provided by the developers. Final fix to be delivered in one to two weeks.

Conclusion

It is important to verify the defect, can it be recreated, and if so will it affect more than one area. A minor defect in one system might affect another part and then become a major.

So remember to test around the defect, as they tend to live in groups

Stop29119 - A Tempest in a Teapot ?

It has been a few months now since CAST 2014 which seems to have been the starting point for the “great ISO29119 debate”. But what has actually happened since then ?

• The Stop29119 petition was started and got over 1000 signatures and this was then presented to the ISO-group.
• Numerous blog posts have been written
• Mostly posts against the standard, but a few pro-ISO have appeared
• Debate on Twitter and other media
• Ranging from personal beef to factual discussions
• Local discussions appears to have happened
• Probably quite a few people have bought the standard (me included)
• No information if any company has actually started to implement this

But what now ?

• The debate seems to have abated and things have gone a bit silent.
• No response to the petition, at least to my knowledge
• We are still waiting for the two final chapters of the standard.
• Awaiting to see if any organization will pick this up or not

So was the whole debate a storm in a water glass?

Well in a way it was. Probably because only a small number of people did the open discussions.

In another way it was not. It brought the attention to the general community on what is going on and even if most stayed silent, many local discussions seem to have happened. And the information has been spreading.

Some discussions might have gotten a bit out of hand and might have lost focus, but thats normal when you really stand for something. Then again we are testers and discussing is what we do, and question how things are.

Looking forward to read the last chapters and also to see what will happen both within the community as well as within organizations in regards to the standard

Hiking Analogy

After doing a hike up a mountain in near white-out conditions it got me thinking that this is a good analogy for approaching a new system-under-test.

You know what it is supposed to look like, what the system should do.

Up ahead is the goal, the end product.

Right now you can only see 100 meters ahead, you see the current function.

You will encounter cliff face - defects or issues

Once at the top you’ll see the big picture can say you have a good view of the system

In order to really know the trail/system you need to do the hike several times in different conditions, it might be hard going but once done it is a rewarding feeling

(Hiking trail: Cairn Gorm and the northern corries)

Titles and definitions

In an earlier post i talked a little about my thoughts on test manager and test team lead.

There is a underlying discussion here that I would like bring up.

Why is it that the career path for testers is usually to move on to become TM or similar?

And how come a TM or TTL usually gets better pay?

As I stated in the last post: to me a TM or TTL is a tester with some additional responsibilities and tasks. This usually mean that less time is left to actual testing, ie the value to the company/organisation might be less than for the regular testers within the team.

To me the most valuable person is the tester and the TTL is there to enable them to do their job.

So why is it that TM seems to be the way forward rather than becoming THE expert tester ?

Development have had a similar issue a few years ago when the path was: developer to dev team lead to project manager. Not sure if this still applies today. One can become a brilliant developer and get the recognition for it.

So why is it not the same in test?

Test as a profession might be a bit younger and has really come into its own in the last couple of years, while dev has a few decades to do the same.

It also might have something to do with the title, put manager there and the paygrade jumps.

But as said above a great tester will provide more value to the organization.

Maybe there is a need to better define Tester in order to better see the difference between A tester and THE tester.

De-focus and Re-focus

We have probably all been there one time or another where the meeting or discussion takes a turn away from the agenda and end up somewhere else and everyone loses interest.

This is what seems to have happened with the current ISO29119 debate. It started with a petition to stop the standard but is currently a yelling battle between a few persons about the personal agendas and personal reason for either creating or stopping the standard. What happened to the discussions on the facts in the standard and how this will affect the testing trade?

Its time to get back on track and focus on the fact at hand:

• Is there any validity for the standard?
• Can the standard be adopted to a non-waterfall project?
• Is it really that document heavy?
• ….and so on.

Bring these types of questions to light and help people understand what the standard will do or not do, and take the discussion for there. hopefully this will be a discussion that the “quiet masses” will have an interest in.

Sure there is no denying that there is a monetary aspect to this as well, but show me something today that do not have this. And there is a personal aspect as well, but the main focus should still be on the standard itself.

Don't let this discussion end up being another example of Godwin’s law  

What's in a Title

I have had a few assignments where my role has been Test Manager or Test Team Lead. 

Common titles, but what has the actual job description been behind these titles ?

A common description is usually: Manage a team to ensure that quality is delivered.

But what does this really mean ?

My take is that as a TM or TTL you are responsible for the team and the testing you do. Yes I mean you, not them, as to me the TM or TTL should also be involved in the testing as much as the team. If you end up just attending meetings and setting policies or work processes your title should probably be Test Strategist or even Project manager. If you are to lead a team one must know what is being tested and how, and this means getting your hands dirty and run tests alongside the team. Doing this will get you the information needed to lead the team, make good decisions and hopefully feel confident when reporting the status to the project.

For me one of the most important roles as TM/TTL is to shield the team from unnecessary discussions, especially politics, and allow them to do the work that is needed and what they excel at. This might mean that it is on you to explain why the team is working in a certain way. Or act as a mediator between the team and the project management when questions arise.

One important factor to remember when changing from different teams and/or assignments is that no two teams are the same. What worked last time might not work here. Different project methodology, different system, different people and skills etc.

This is where communication comes in. Listen to the team, discuss the expectations, talk about the way forward and so on. But DO NOT forget to have a similar discussion with the organization as well. What you and the team decides must be communicated so that managements expectations are also met.  

So what should the job description be, or contain?

Simple: Mediator - consolidate the requests from management with the needs and wishes of the team

Simpler: Gatekeeper - protect the team and let them work in peace.

Simplest:Team Member

Final comment:

Titles should not define you, it's the work you do that defines you.

Review of a review

Recently I was asked to review an article written by a colleague and afterwards it made me think about how I actually read and review text and how I approach test and testing.

The article itself was written as a discussion on the different approaches for a company to solve the webpage / mobile site issues. It was written for the general public and therefore did not contain massive amounts of technical details, rather focusing on the pros and cons of each solution.

My analysis of my own review (self-review?) is as follows, and can be split into three parts:

First i read through the article and just made some minor notes on where i got stuck some way or another. One read through, pretty much skimming the article

Next came the focused read, where I comment each part on a technical and understanding level. Do this make sense? is this actually correct? etc. Each time i got stuck or started questioning things i went back and reread the sentence or the whole chapter, and also checked statements on their validity. I also discussed some parts with the writer to make sure I understood what the aim or goal was.

Finally I once again started over and read the article as the person that the article was aimed at. So this time the focus was more on: Would I understand the text? Will this help me decide which solution to go for?

Once all this was done i cleaned up my notes and handed the article back to be updated. But the important thing here is that I made sure I was available to take questions regarding my comments.

So the conclusion ?

• For me to feel done with my review i read the article three times
• Different focus each time
• Checked and/or discussed questions with writer during review
• Active collaboration
• Written comments was also explained once the review was done
• Is my comments understood correctly?

Some notable observations are that not once did I actively check spelling, this due to the fact that I know another more suitable reviewer (teacher) would focus on the language. It took me three reads to feel done with the review. It was imperative to have the opportunity to discuss my comments directly with the writer to clarify questions and avoid misunderstandings.

How can this be translated to test then ?

In general terms my review are in three phases. First a quick smoke test, is this stable for the actual test? Secondly system test or even system integration test, do each part function as expected? Finally the acceptance test, is this what was actually ordered? The spell checking can be seen as the regression test, standard functionality that should work.

Another way of looking at it is the approach. First I read the requirements or User Story, do I have enough to start test? Secondly I execute the testing, focusing on different parts, digging into possible bugs etc. Lastly I rerun some of the test to get the big picture before finalizing my test report.

And as pointed out time and time again, communication is imperative. And this must be both with the business as well as the developers to avoid misunderstandings, and in the long run to avoid false negatives being reported as defects.

Trying something new and unknown is always good and will provide new insights, even if it just shows that you work in a similar manner regardless of situation.

Have you reviewed your work approach recently?

Give it 10 minutes and it might give you a big “oh-moment”

Analogies - Another way of reporting

During my last projects I have been using analogies when reporting status and during project team discussions. This started when the project I was working on as a test manager was running according to the tried and true waterfall model with many, many, many test cases. So why should I report the same number that everyone could find by a quick look in Jira? I did create a nice dashboard for this.

We had a discussion within the team and came up with the car analogy as a good starting point and something that everyone can relate to. The analogy took hold and shortly was the standard starting point for discussions or answer to questions.

Some examples of how this was used

• The body has been delivered but we are still missing the wheels
• Lacking deployments from supplier
• Brake pedal starts the wipers
• the integration's are faulty
• Again we have a SUV when its a compact we need
• Misunderstood requirements
• Supplier tests the airbag, buyer test drives the car
• Difference between System test and acceptance test
• Many more examples exist, and of course other analogies as well

We used this within the team but also when reporting to management as they also had full access to Jira and could get the actual numbers, so this gave them the quick status and also explained to those who did not have or needed the technical knowledge and full disclosure.

I have seen and used similar analogies in other projects and talked to others who have used this as well.

How do you report status? Discuss the latest progress? Or just talk shop within the team?

ISO29119 - And its possible future

Let’s rejoice!

Test and Quality Assurance can no longer be called a second hand job or something that just needs to pass, we have a ISO standard now to prove this.

Great news everyone! Or?

Pretty much every conference, blog and especially Twitter has had some sort of comment or information regarding ISO29119 these last months. This has ranged from analyzing the people involved in creating the standard to the content. It’s been a heated debate at some points and also a fairly one sided debate where the opponents have had the loudest voices, and the pro-ISO have been surprisingly quiet.

So what is actually going on?

I will give my views on things as far as I know right now given that information is still coming and the complete standard has not yet been released.

If we start off with the content in the standard, there is a slight problem right away. One must buy it from either ISO or IEEE, which is quite normal procedure in regards to standards. But once you have acquired it, it is yours and yours only, this according to the licensing agreement: “licensee's use only. No further reproduction or networking is permitted.” So how to discuss the content then? Below is my take on the standard.

Part 1

Part 1 is fairly straight forward it focuses on definitions and concepts of testing. If you have read or done the ISTQB Foundation course it will be a familiar read.

There are some interesting passages to note that sets the level for the coming parts.

It is recognized that software testing is a process and that the standard will present and describe a generic process. This process should be planned, monitored and controlled. The result of this process would be to provide information regarding the quality of the software under test and find issues as early as possible

A clarification might be in order here and that is that it is quite clear that the standard is based on the waterfall project model with sequential flow for the project and testing. Also the standard uses risk-based testing as the foundation for the dynamic test process.

Will cover the implications this might have later in the article, and the risk-based testing will be covered in Part 4 of the standard which is to be released late 2014

Part 2

This part of the standard covers the test process in more detail and its intended usage

Looking at the presented test process model, image below [1], it once again seems fairly standard, excuse my pun.

We have the organizational process that sets the overall setting for test across the organization. The Management process goes a bit deeper and focuses on how each project should handle testing and reporting. And finally the actual testing is covered by the dynamic process.

When looking at the overview of the process it feels simple and logical but once you continue reading the complexity increases, below is a diagram [1] on how the test plan process is described

Again at first glace this is normal work process for creating a test plan, i.e. gather and analyse information, present the analysis and get a sign off.  So no biggie then?

Well if you are to conform to the standard this is the way to go and as we are talking about a standard its the only way and you must follow all the steps regardless of the size of your project. And if changes are needed one is supposed to go through many of the steps to get the change approved.

Then comes the monitoring and control process and yet again things get complex to the point where it looks like the process will take more time than the actual testing.

As stated before the standard is focused on test cases and therefore this is described in detail, the writers acknowledges that writing test cases is a time consuming activity but also maintains that it may lead to savings over time.

The dynamic test process covers how to identify, create and review test cases, setup the environment and report incidents and much more

The complete process described is a lengthy one and involves many steps and iterations from identifying the requirements to creating and reviewing test cases and on to creating test procedure specifications before testing may start. Not to be left out is the process of reporting incidents and a closing report.

Part 3

This part of the standard covers the documentation that the processes described in Part 2 should have as output. As the processes is quite complex with multiple steps the documentation follows suite and there are a lot of documents to create. The fact that this part is 138 pages long should give you quite a good idea on the amount of document and the detail needed in each.

A detailed test policy and test strategy for the organization might be a good idea as these are normally only created when major changes happens within the organization.

The more you follow the process set in part 2 even more documents are needed or recommended. So for each test phase a new test plan should be created and then a separate test plan for each of the subsequent systems under test.

When we finally reach the actual test there are recommendations on what needs to be documented here as well, ranging from test design specifications to environment readiness reports and on to incident reports.

Much of this can and is handled by the test reporting tool one might use (QC, Jira etc.) but the standard still focuses on test cases and the process of creating, reviewing and executing these

Part three contains an appendix where examples for the documentation can be found and they have used both a waterfall project as well as an agile project, so the possibility is there to see how this can be applied to agile projects

So what will all this mean for test in general and me as a tester?

As I started the article it’s a good thing a standard has been created as this only further shows the importance of quality assurance. Today the quality is more important than time-to-market, as people are prepared to wait a bit longer for something that really works. Quality over quantity etc. and it is here that a test standard might fit.

Unfortunately I believe ISO29119 will not help our profession and especially not in its current state and with its claims. It states that this standard will help manage and perform software testing within ANY organization.

According to the standard there are different levels of conformity, either Full conformance which is as it sounds: one must follow the standard to the fullest. Or Tailored conformance which is where it’s up to the organization to justify where and why it has chosen to step away from the standard.

The tailored variant is there so that the standard can be adapted to a more agile work process. This to me is a bit of an oxymoron: create a standard that one can claim to follow as long as you justify why one has selected to not use or disregard the standard. Sure there must be some leeway even for a standard but not to the extent possible within ISO29119.

One concern regarding ISO29119 is that: Can one actually standardize an intellectual process? In my opinion this is quite difficult and will hamper the ability to actually test. this given the fact that each project and system is different, and needs different approaches depending on the situation. Now each tester and test manager must conform to a set standard and practice regardless of situation and system under test.

Another concern that I see is on a more organizational level which the standard also affects given the way it is written, with the major focus and structure on waterfall and test cases. This means that for an organization to implement and adopt the standard major organizational changes must be made.

The last concern is also the one that most likely will kill this standard as companies will not be willing to take the cost and risk needed to change and adopt to it. The unfortunate backside of this might be that it will leave a bitter aftertaste for those who has read it and then discarded it.

The standard is written and presented as yet another attempt to define a process and way-of-working for test. Had it just been another book or white paper this would have been read and in some cases used as well. But now as it is presented as a Standard with the backing of ISO/IEEE it puts things in a whole new light. Suddenly things are set in stone (maybe not, given the Tailored compliance) and this is how testing should be done. This is also what will be the standards downfall. Few or possibly no organization will be willing to take the time and cost to change its current setup to conform to the standard. The risk that I see is that new organizations or possibly government run departments will adopt this, but this will be a minor risk.

As an example just look at the ISO12207 - Systems and software engineering — Software life cycle processes this has been around since 1995 with the latest revision in 2008. In the years i have been working with SW development, support and test i have never seen or heard about this before. And that what i believe will happen to ISO29119 as well. Unfortunately before it vanishes into the annals of civilization it will give a bitter taste and give quality assurance a bad and backward striving impression.

Ref

[1] http://softwaretestingstandard.org/part2.php