Testing is Rocket Science Not Brain Surgery

LR-Citrix_Presentation

You need to understand that any project requiring a performance test of an application deployment via Citrix is not going to be your casual record and playback affair. So begin by digging up your old C reference manuals, actually they should be close by at all times when working with this tool, but for the neophyte a good reference is a must. Next get a lay of the land for what you can monitor over the network and what key Citrix performance counters you need. Then start to steel yourself for the onslaught of maintenance, you’re going to be using snapshots of defined screen areas to validate whether something has loaded on the screen of finished. What this means is any GUI changes or minor differences in the way the GUI renders (something that happens even when there are no screen changes, go figure) will require at least another run through of all your scripts and making the necessary bitmap hash code updates. LoadRunner provides these for you; it’s just a labor-intensive affair. Factor in that each load injector may render a little differently and you have the potential for a lot of debug pass through.

My colleague and friend Chetan Rahul of Fusion Alliance based in Indianapolis was nice enough to take me up on putting together a PowerPoint outlining some of the issues and lessons learned when we began our LoadRunner Citrix engagement. Enjoy!

NOTE: HP has provided some enhancements through a runtime setting that allows for the tool to accept some differential in what is rendered versus the snapshot originally taken, unfortunately despite patch 9.52 I’m still not seeing any benefit from changing the image tolerance.

(2 Factor Orthogonal Array Example) The ties that bind us together, our networks, connecting the disparate dots on our application landscape allowing us to process transactions from around the globe.  Imagine a circle of people holding hands moving in unison around the camp fire, a perfect symphony of movement as eveyone dances around faster and faster until the circle becomes a blur of movement.  It’s not until someone’s shoestings come untied, followed by a stumble and a loud thud that we start to realize that we might have been standing too close to the open flames.  This is the type of analogy that comes from a mind warped on videogames that add flamethrowers to your arsenal, games which I only play to discover software bugs and not as mindless entertainment for hours on end by the way.  Back on subject, our application network is only as robust as its weakest link, an adage that has held true for thousands of years.  Unless confronted with hard performance requirements this is another one of those areas that are so easy to take for granted in the context of of having tested intranet applications that reside behind the corporate firewall.  In reality this is even more precarious than testing an external facing application whose end user is on the other side of the world wide web.  I say that because if you want a simple worse case scenario all you need to do is govern the bandwidth that you conduct your test with when simulating an external user.  You can take the broadband penetration numbers and scale accordingly using the lesser of the average connection speeds.  Right now Nielsen/NetRatings pegs broadbacnd penetration at about 33% of American households, this is an abysmal number compared to other countries leaving us at an overwhelming majority of households at less than broadband or at 56kbps worst case.  So if you wanted to cover your bets, all of the external users in your test would run at 56kbps.  Obviously all of this could be trumped by a hard technical requirement that states the minimum connectivity speed. But when it comes to internal apps behind the firewall we tend to not even gather network stats because we are on the corporate LAN. This false sense of security can be shattered with one mis-configured router, an incorrect duplex setting on a NIC, bad cables, or poor network connection handling in the application. Complicating this is the infrastructure behind the corporate firewall then on the other side of the firewall you have the possibility of caching servers and services(Akamai), and possibly local caching on the client-side of the application puzzle.

In a vacuum a component test of a piece of business logic may be acceptable, but the result of that business logic execution could end up in a datagrid in the UI that is waiting to be rendered because the data hasn’t made it across the wires in a timely fashion.  This means you absolutely have to take the render time into account from an end-user perpective and create a 360-degree view of your application performance which includes the network performance. 

This now adds an additional factor to your testing array where you can represent various types of network speeds using less-than broadband, broadband and enterprise levels of connectivity.  This then opens up to the addition of WAN topology and modes of transport such as wireless.

Don’t get burned.

In helping various collegues and companies manage their performance testing efforts I’ve come across a reuseable set of common activities that have helped me explain and divide the work necessary for a successful engagement.

The following activies:
Engaging in Risk Identification
Developing a Performance-Testing Strategy
The application of the Performance-Testing Framework which produces a Performance-Testing Plan

Assessment
Modeling
Execution
Analysis

Assessment

Performing the assessment is critical to the justification of the performance-testing effort. The result of this activity forms the basis for the charter of the entire effort. There are a number of issues that need to be addressed per the pre-requisites.

Critical questions to ask concerning application performance:

• What is the real cost of system downtime?
• What is the impact on the perception of the organization’s ability to develop, test, implement, or maintain applications?
• What is the impact on the internal or external customer?
• What is the cost of correcting performance issues post-implementation?

Just as important, an assessment of the capability and capacity to execute performance-testing activities is just as important. This includes a deep dive into the measurements that can be collected, the hardware capacity of the hardware comprising the performance testing infrastructure and the network bandwidth to support the generation of transaction load and polling the respective performance counters.

The follow-up activities are:
Determining Performance Criteria
Verify the Measurements
Validate that the Metrics are Measurable
Determine Measures of Success

Modeling

The performance test lead(s) in collaboration with the business analyst(s), system analyst(s), and test lead(s) will determine the combination of test methodologies and test types that are applicable. Since this performance-testing framework does not rely on a completed application the appropriate methodology has to be determined in order to leverage this flexibility.

A performance test plan will be written to document the results of the assessment and the methodology chosen. In addition to the performance test plan, a performance test specification detailing the user and usage profiles, the scripts being developed, the traceability matrix, performance test environment diagram, SUT diagram and scheduling will accompany the plan.

User Profile
Usage Profile
Test Data
Developing Performance Test Scenarios
Performance Test Script and Scenario Traceability
Developing Performance Test Scripts

Execution

Performance Test Infrastructure Evaluation
Apply the appropriate Test Methodologies
Apply the appropriate Test Type(s)

Analysis
Sample Size, Inclusions, and Exclusions
Verify Quality of Test Execution
In Test Analysis
Post Test Analysis Deliverables
Top Down Analysis
Root Cause Analysis

Note the separation of the Test Methodology and Test Types from the Modeling activities. A single model can be used between methods(Profiling, Benchmarking) and Test Types(Load, Stress, Volume, Baseline).