The first time I ever heard the term “performance testing”, I worked at Deloitte just outside Nashville, Tennessee. I was selected to be on the newly formed “Implementation Team”. This was an internal group at the home office that would assist branch locations in their adoption of new applications rolled out at the higher corporate level. This was the 90’s, and deployments of desktop client/server apps means a lot of individual machines had to be touched. This was even before Citrix deployed applications were a thing.
Application adoption was never a small undertaking. The sheer size of Deloitte meant rolling out applications to a few “pilot” offices first, and then to the remaining offices over time. Until our team was proposed, some offices could lag behind for years before they decided to migrate to a new system. Each office ran somewhat independently in that regard, and had their own IT staff per location. They could decide when they moved, which would cause a big headache trying to support multiple systems that did the same thing. The Implementation Team was there to help speed the adoption process up and support those local IT people by helping them with the process and training them so they could train their own office. It was a new concept to the firm.
Entering The Multi-dimensional Universe
One of the new applications being rolled out was Hyperion Essbase https://en.wikipedia.org/wiki/Essbase . It was a plug-in to Microsoft Excel. There was a stand alone database server for Essbase – which was a MULTI-DIMENSIONAL database rather than a relational database – which only meant to me I understood nothing about it. From any satellite office, a user with Excel could click a button that would pull data from the central database server (physically at the main corporate office) and get financial report information. Seems simple enough. Unfortunately, pulling all that data across the LAN/WAN to satellite locations proved to be a performance issue, and many complaints came in to the support desk about it.
To overcome this, the manager of the application decided that we should conduct a performance test. Great idea! Except none of us knew how to do that. No one even knew the concept, much less that we should automate it. Obviously, we did what anyone would do. We decided to do things the wrong way. Remember, at this moment in time I am not a performance engineer. I am just another typical IT person on a new team that helps install new software at remote locations and trains the local office on how to use it. I have no concept of performance testing. If only I had Doc Brown’s Delorean, I could have been a hero!
Houston. Really?
The application administrator created a Word document that listed the steps that users would do within the application on a regular basis. These were the business processes that people were complaining about. Ten (10) people were selected to travel to various remote offices with these sheets of paper, along with a stopwatch (yes, a physical stopwatch), and a laptop with the application on it. The cities were also based on locations with the most complaints about performance. One member of my team was selected to go to Miami. One was selected for New York. One was selected for Los Angeles. I got selected for Houston. No offense, Houston, but C’MON!!!
My travel was handled by a third party. I showed up at the hotel, which was a 2-star dive literally a stone’s throw from an Interstate overpass. The homeless were setting up camp for the night underneath – with their own mattresses! The hotel was being used as a recruiting location for the army, and there were a bunch of military vehicles in the parking lot. The lobby had a very eclectic mix of people, let’s leave it at that. I went to my room, and there was a large crack in the wall that looked like someone took a metal chain and whipped it across the room, creating tears in various places on the drywall. Think of a room from the movie “SAW”. I went back to the lobby to ask for another room. After being propositioned by a couple of interesting ladies (I -think- they were ladies but…) outside the lobby, I summoned a taxi and went to the Galleria Mall. Fast. I wasn’t worried about a refund on my credit card at that point. I learned a lot about third party travel arrangements that day, and have never made that mistake again. But I digress….
Conference Call Chaos
The Essbase consultant would monitor the database server while we were out in the remote offices executing the application. We had multiple conference calls where we would all get on together and manually walk through the steps on the documents, and write our timings on the paper. The conference call would go something like this:
Admin: “OK everyone, are we ready to begin?”
5 seconds of room noise. We heard a siren going off in New York, what sounded like a latin music festival in Miami, and an earthquake happening in Los Angeles – all at the same time.
Admin: “OK, everyone type in your name in the field as the document states”
Conference line: Keyboard typing sounds in a flutter
Admin: “OK, everyone hit the submit button on 3. 1….2…..3!
5 seconds of silence
Conference line: “Hey everyone, this is Bob. Sorry I am late to the call, have we started yet?”
This happened at almost every step of the business process as we painfully walked through it and logged the timings. We did this for two days. Then we all flew back home and came into the main office with our sheet of paper.
I had conversations with other team members asking them about their trip experiences. Obviously, my trip to Houston was interesting, but boring once I switched hotels. The person in Miami apparently had a great time. Same with several other cities. We weren’t even sure the timings we all wrote down were accurate, or perhaps some of them were made up. Maybe it was a great boondoggle for some, I don’t know. What I do know is that this was not a performance test, and definitely not repeatable. It was more akin to voodoo than testing.
Is That Repeatable?
We handed our timings into the administrator, who created a standard spreadsheet, and made graphs from the data we provided. This took a couple of days. The “timings report” was provided to the partner in charge of this application at the firm. At which time, he said, “This is interesting. Now can we do this for 50 people and see what the results are?” I thought the administrator was going to faint!
Shortly after this, the firm decided to buy an automated performance testing tool. LoadRunner 4.5 CD’s were found on multiple desks! They hired two consultants from Mercury to come in to help get things started by automating the business processes we had walked through manually. I distinctly recall walking down the hallway where these mysterious engineers were tucked away doing their magic. It would be years later before I learned I had walked right past James Pulley, who was one of those engineers. We never met the whole time he was there, and we passed each other in and out of the office daily. I’ve got more “Pulley” encounters to talk about, but that is for another day.
The Takeaway
Think about that next step. Could you imagine hiring 50 college students and placing them in a room with some temporary laptops with the same image and coordinating this elaborate test, and the additional cost of travel to remote locations to see the effect of the WAN? These days, we take for granted that automated performance testing and load testing is the default. Most IT professionals involved with software development understand the benefits of automated load testing, and the value it brings. We wouldn’t think of doing a manual exercise like this. But it wasn’t always that way. Sometimes, it’s a good thing to think about how far we’ve come. While I still don’t think performance is given the same respect at functional testing, we have come a long way.
What’s your first/worst load testing story?
About the Author
