quality assurance

Twenty years goes by quick. It seems like yesterday that I was at Deloitte & Touche supporting their time and expense system based on that cutting edge 32-bit platform known as PowerBuilder. For the record, it had to be one of the most defect-ridden internally built software packages known to the human race. I became such an expert at knowing all the little “secrets” to get around the bugs that I became very popular for Deloitte partners having trouble submitting their expense reports. Soon after that, I moved out of technical support to the Quality Assurance department. I was first involved in change/configuration management, but eventually became trained on LoadRunner because no one else wanted to do it. The rest of my career is history. I’ve been involved in every area of the software development life cycle since then, but I have never forgotten where I came from: support. Quality has always been important to me because it means less support calls and tickets. It made my life easier as a support desk resource when an application worked well. These days, I always think of the support desk when I am working on a project and how a performance issue might impact them. Maybe that is why I am as passionate about what I do today.

Software development has definitely changed. It went from that outdated, horrible, death-on-a-stick waterfall software development approach, to be replaced with the Agile/Lean/DevOps culture that has seemingly solved every IT problem that exist. Now we have all of this software with zero bugs, right? I mean, all I can think of when someone says “Continuous Delivery” is rainbows and lollipops. Said no one. Ever.

You Can’t Say You Are QA If…

In September 2016, I attended and presented at the Software Test Professional Conference (STPCON). I make mention of Eric Proeglers presentation “Early Performance Testing: What Else Shifts Beside Left” in this article, and specifically a statement he made that really struck a chord with me. He said, “If you think you are in Quality Assurance, you are wrong. You can’t say you assure quality if you can’t stop a release. You are an advisor on quality to those who run the project and make those decisions. QA could be Quality Advisor? Maybe… but not Quality Assurance.”

As we say in the southern US, this has been “sticking in my craw” for some time. In my past article, I stated that I would follow up with this and never have…until now. If QA isn’t assuring quality, then what are we doing? Why are we doing what we are doing in QA? Are we just going through some checklist to make someone feel better about the release? Twenty years ago I thought I remember all the developers complaining that QA was the department of “no”, and they were holding back product releases and strangling the creative juices of developers. Development managers who also managed the Quality Assurance department ALWAYS favored the developers and were notorious for fanning the flames of the “us versus them” mentality. QA is just standing in the way of progress. While QA were claiming that they were the only ones who were committed to doing things right and protecting the business. With Agile, it is a developers world, and it is about good enough being good enough. Ship, baby, ship. Fix it in the mix. IMHO – of course.

I know this goes for functional testing, but I am affected on the performance side of things. Many times on a project, I have felt as though my performance testing effort was a waste of time. The client just told me they will roll the software out no matter how bad it performs. Half the team doesn’t understand what I do, or the results I present. In a year they will have a new version of the software, a “prettier” UI, a new backend (“we’re going serverless!”), and all of these test results won’t even matter. I have used the illustration in conversations many times that I sometimes feel like the SDLC is simply building sand castles that are consumed by a sea of users for a short period, only to be replaced with a newer, bigger, different sand castle. Kick one down and build another in a few months. Rinse and repeat. This is when I begin to fantasize about getting out of the IT industry and doing something that has to do with the beach, fishing, BBQ, and things like that.

Back to my questions above, I want to observe the state of Quality Assurance and its function, comparing it to when I started working in QA through present day. Where are we? Are we getting any better at software quality in 2018 versus twenty years ago? If so, how do we know? If not, what are we still doing wrong? What has changed?

Quality Assurance Versus Release Control

Let’s start with semantics. In general principle, let’s agree to these admittedly over simplified definitions:

  • Quality Control – A “check box” activity. This is basically testing at the end of a release cycle. It simply ensures some level of testing occurred to see if the product matches specifications and requirements, including non-functional requirements.
  • Quality Assurance – building quality into a product throughout the lifecycle.
  • Release Control – holding up a release if something isn’t right.

What I think Eric is describing as QA being in a position to control production release would be if they were Release Control as well as Quality Assurance – or had that additional responsibility. What many organizations think they are doing as Quality Assurance is actually Quality Control. I think you need all three of these, but the functions may be split between different groups. For performance testing and engineering performance into the lifecycle, I think there should be a distinction between those who merely verify and validate performance against specifications (read performance testers), and those who engineer performance into a product through its life span. This is what I call a Software Engineer Across the Lifecycle (or SEAL) and it is how I describe myself on my LinkedIn Profile. In fact, I wrote an article about it. Wanna read it?

The QA Dilemma

It’s important to understand how we got here and I realize I don’t know it all. Fortunately, I am connected to some of the brightest minds in software quality assurance. I decided to start discussions with some of them to help me understand the subject of improving quality across the IT industry as a whole. One person I have had several conversations with about this is Theresa Lanowitz. Theresa has been an industry analyst since 1999, first with Gartner and since 2006 as the founder of voke, inc. I highly respect and admire her work and asked her to comment on this. She states:

“The lack of software quality is a global problem that is only getting worse. The problem is multi-faceted. One area of concern is the lack of advocacy for software quality professionals by software vendors themselves.

Since Mercury (the former undisputed leader in automated software quality products was acquired by HP in 2006), there has been a void in the vendor market. Mercury was a strong advocate for software quality professionals and built solid products that helped organizations in a quest for better software to delight the customer. In addition to strong products, Mercury was relentlessly focused on its customers – the software quality professional. Eventually, Mercury was able to take the software quality conversation to the executive level where it really matters because the software and its quality is part of the brand promise of an organization.

Circa 2004/2005, the need for software quality was resonating with enterprise organizations. And, there was a vendor ready to fill the needs of those enterprises. Just as that conversation about quality was gaining traction, Mercury was scooped up for $4.5 billion by HP.

Since that time, there has been no software quality assurance vendor that has stood up and advocated for the software quality professional. That silence from the software quality assurance vendors is deafening.

There is a failure to lead and advocate for the software quality professional. That is one reason we are in the current state. The market is ripe for a new conversation about what software quality means and why the software quality professional is a valuable member of the software lifecycle.”

I can only relate my own anecdotal experiences. I continue to encounter Agile and DevOps as more of a theology than a software development methodology. One could argue Agile and DevOps aren’t methodologies at all. Where in the Agile Manifesto are the words “scrum”, “lean”, “KanBan”, etc? Even in many Agile focused companies, the same issues (and new issues) are encountered within QA and testing functions:

  • Requirements to test against are non-existent (user stories are not requirements)
  • There is a huge deficit of software quality engineers within IT today
  • Manual testers are abundant, but are only conversant in the GUI layer
  • Extreme testing is a fallacy
  • Developers do not like to conduct unit testing (some things never change)
  • Continuous Integration (CI) is not universal
  • Environments for development, testing, performance, and production are misaligned (the phrase “it works on my machine” should not exist in 2018)
  • Buzzwords rule the day
  • Transformations (Agile, DevOps, SAFe, etc) are DIFFICULT and expensive
  • Companies tend to pick only the parts of Agile/DevOps they like and then they wonder why they are not successful.
  • Release frequency with quality and certainty requires modern tooling such as service virtualization and release management.

Is software performance getting any better? Look no further than News of the Damned with James Pulley for all the major failures over the past few years. The 2017 Black Friday and Cyber Monday failures were some of the most epic to date. In my opinion, we should know better by now. The dress maker for Meghan Markle (now Meghan, Duchess of Sussex) is just the latest to find out the hard way. Space won’t allow me to write about all the quality failures in the airline industry, healthcare, etc. I find it interesting that in 2018, TSB (The Savings Bank) in the UK is still reeling after over a month of fighting with software “glitches”, topping the Royal Bank of Scotland’s IT meltdown in June 2012. Are we not learning from the past? Why is this still happening? I believe that it is fundamentally because we are writing more software than ever before, and because of this we are writing more poor quality software than ever before. As I stated earlier – it is a skill set deficit. There are few true quality engineers.

Some of my conversations with leading experts in the testing field these days tend to wander down rabbit trails trying to blame one group or another. It’s the developers! It’s outsourcing! While I love to poke fun at developers more than just about anyone, I realize there is plenty of blame to go around. It still tends to be a stereotype that QA is considered “entry level” for IT and that becoming a developer is somehow a promotion to do “more important and cool” stuff. Those who can’t code well, stick them in the corner and let them suffer as the QA person. This has to stop.

There are obviously exceptions. Some organizations tout a complete, positive transformation when digging in and doing Agile/DevOps correctly and not hand selecting the qualities about it that they like. I stress the word “correctly”, but this can be confusing when you try to dissect each company success story, because they often define Agile and DevOps slightly differently. These terms are not yet as fully defined or prescriptive as we would like.

What Can We Do About It?

A lot. Here are a few of my suggestions to start with:

  • Fire all the marketing people at software tool vendors…. OK. Maybe not. Educate marketing departments at testing product vendors about presenting their product in the best light, while not ignoring the fact of life that sometimes testing is a complex process because sometimes software development is a complex thing. Face reality. That should not cost anyone product sales. Anyone can do testing with your product? No, anyone cannot do testing with it and there are several roles I don’t want doing the testing because they suck at it. You know who you are! No product UI or easy workflow can account for that.
  • Beat recruiters senseless until they… Let me rephrase that. Educate recruiting departments to stop looking for key buzzwords and titles and look for signs of experiential knowledge. This experience may not be directly on a specific technology (stop asking for 5 years of experience for technology that has only existed 2 years for example) and look for skill sets that translate. Just because I haven’t performance tested some new SAP module and know all the business rules, doesn’t mean I wouldn’t be great at it if I understand the underlying technology. Just because a performance engineer has no JMeter on their resume doesn’t mean they couldn’t use it just as well as the tool sets they have listed.
  • Tell the dev managers at some companies to shove this pencil… – er… I mean, educate development managers and upper management that software testing isn’t just a place for entry level resources and development certainly isn’t any kind of rite of passage. It is as complex as coding, and resources should really be better at coding than the software developers. Stop hiring QA people to functionally test web applications that cannot even read HTML or create their own web site from scratch. Project managers are seldom qualified to be testers. Developers generally make horrible testers. Functional requirements are not the ONLY requirements to worry about when it comes to quality.

I’m not so naive to think these are the only things that need to be done. I am sure there are a lot more that needs to be addressed. This is where YOU come in. What do you think? Is software quality getting better or worse? Are the number of defects per release going down? Is it better than 20 years ago? How would we know? Where are the statistics that would prove or disprove this? Has Agile made a difference in the software quality (for good or for bad)?

If you would like to dive deeper into some controversial research from voke, I would recommend the Agile Dilemma reports from their site. It might be rather eye opening. I hope some of the really smart people I am connected to will read this and respond in a way that will help me understand if I am just missing something and everything really is great or if it is as bad or worse than I think it is. I’d love to hear your feedback.


About the Author

Scott Moore

Scott Moore specializes in application performance engineering and testing. This includes education, hands-on implementation, and application performance monitoring.


  1. Mike Hawkins

    I think the author has hit on something rather profound. Just as the Industrial Revolution took several generations to reach the high levels of production and quality that we enjoy today, the Information Revolution is going through the same process. “Best practices” in a major sense are being redefined before our very eyes. I believe that generally the commitment to improving the software development lifecycle within the industry as a whole is as strong as ever. Unfortunately, that commitment often does not translate into the resources needed for the task. With shrinking development cycles and ever increasing demand for a viable product sooner rather than later, development inevitably gets the lion share of those resources while the leftovers are tossed to QA.

    The author is correct in that this is as much about education outside of QA as it is about increased skills within QA. I am convinced that the most efficient and ultimately best ROI for medium and large-scale development organizations is to build modular, automated, and scalable frameworks for both functional testing and performance engineering. Some strides towards this goal have been made within the functional testing realm but a shockingly small amount of progress has been made within performance engineering. I could expound further on this problem but that would require at least one if not several articles.

    Ultimately it begins with the following two objectives:

    – Education within the industry in order to align perspective with reality.
    – Emphasis on the development of higher-skilled QA professionals who are capable of designing and building the needed frameworks and have the analytical skills to correctly interpret test results

    Companies who are able to reach this level sooner will no doubt enjoy a huge competitive edge over their industry counterparts.

  2. automike

    Awesome article! Provocative and relevant – thanks for taking the time to write this.

    In terms of education – I’d venture to say that it might not be a bad idea to educate the QA department, or even the QA community at large, as well. Some QA engineers have been so inundated with the fallacy that they are second class citizen in the software development, that they have convinced themselves to believe the lie! Indeed software testing is hard. Doing it well and be recognized as critical success criteria is even more difficult. We are way pass due for Quality engineering be treated as an afterthought, as the “fall guy”, as a speed bump and as noise. As the Quality community we really need to hone in, sharpen our craft and assert our value proposition in the software development process.