Testing is Rocket Science Not Brain Surgery

If I had known that so much would be required to even perform my duties as a performance tester I might have gone the other way.  Working on a triple-digit million dollar government project held by a company whose parent was a multi-billion dollar revenue earner was probably the best place I could have started this career path.  But in some respects it may have been the worst as my expectations were heightened from that point on.  We had every piece of software for testing you could want from two of the biggest application vendors in the space, and I would say “my cup runneth over” with hardware capacity for testing.  Fortunately for me, the costs of not doing performance testing had become well known and the perception issue was of the utmost importance to my employer.  This only served to set me up for the fall later, as I would discover that for every one client who cares about perception and correctness there are three who do not.

I was wrapped up in a sort of delusion that this was the state of the industry, performance testing and automation testing were must haves with big budgets and time to build frameworks and labs and the like.  Those public failures among the e-commerce giants and other web heavyweights had done all the ROI calculations for me.  No one wanted to deploy a poorly performing application, no one wanted an app that couldn’t scale, it would be disastrous, think apocalypse.

Instead what I’ve come to find is that my analysis failed to look at these three questions:

1. How much things actually cost and how they are paid for?
2. Is your audience for the application captive or emancipated?
3. Is your organization a quality organization?

The long and short of it  with respect to the first question:

If time is money, like the checkout line being served by a retail point-of-sale application and financial transaction systems then you better be ready to get the job done because resources will definitely be available.  If they aren’t available now they will be made available as soon as possible.  It’s clear to everyone involved that there is a direct relationship between application’s performance and scaleability and the amount of revenue that can be generated or received.

If your company does not generate revenue directly from the sale or use of the application under test then cost is probably going to be a big issue.  Now if the application has a single degree of separation from the revenue stream (i.e. call center application whose goal is to allow for more calls to make more sales) then the cost tends to be manageable.  If however, the application fulfills a support function like a content management site for an organization in the group, forget about it, pull your stopwatches out andenlist some help for a manual testing effort aimed at performance testing(I wish this were said jokingly, but I’ve seen it happen recently). 

Why this is the case goes towards the second question:

When the audience for the application is captive, usually meaning employees of the company, their desire for a fast system is understood maybe even noted.  But if productivity is not negatively impacted by an order of magnitude and the system’s merits meet other criteria like better tracking, more efficient processes(not necessary faster, they are two different things), or fulfilling some corporate mandate then performance will take a backseat.

In the case of the emancipated or free audience, performance should be at the forefront of everyone’s mind.  These users have a choice, and if it takes twenty seconds to update my shopping cart I’m going to make a summary judgment about doing business with that entity.  Unless you have the only application or site of its kind your application will need to perform well to retain existing and gain prospective clients.

The true driving force behind this actvity comes down to the answer you provide for the third question:

In regards to the type of organization you are involved with, even if all those aforementioned indicators lead to performance not being a concern, a quality-oriented company will seek to do the correct and appropriate testing.  If your company develops code in an unstructured ad-hoc fashion under tight deadlines and budget constraints it goes without saying that the test effort will suffer with automation and performance suffering disproportionately.

None of this may help your situation, but the aim is to help your understanding of the forces at play when it comes to investing in technology for testing.  When your generator box is a desktop trying to create a thousand virtual users you will know why you’re not getting another gig of memory anytime soon much less a new server.  The silver lining in this message is that if you come to grips with your reality early it will help you find more inventive and cost-effective ways of performing your duties, which will in turn broaden your exposure and experience.  I’m happy to say that having the best and the worst of resources has made me a better tester as I now have a breadth of tools, technologies, and test methods to pull from to help clients of all sizes.  But gaining that experience wasn’t pleasant and had its failings, so remember you’re not alone and step away from the ledge.

Cheers!