To this day I often laugh when I recall hearing the phrase “All you gave me was a bunch of numbers.” I mean honestly, what is the resultant output of the Performance Testing activity? There is no subjectivity here, this is a hardcore exercise in gathering measurements, applying metrics, and interpreting meaning. It doesn’t get more Dexter’s Lab than this, yet time and time again I’ve been dinged by the need to extrapolate the numbers into high-level executive summaries. While my more arrogant side says produce a powerpoint with one slide, a thumbs up or a thumbs down picture for “Management”, I’m compelled to try to de-mystify the whole analysis reporting side of things. I’ve never been the artisitic type, relegating web design to marketing and art majors, but I’ve found some tools that make it a little easier to add some color and flash.
Here is an example produced by my team for a client:
Tags: General · Performance Testing · Testing
(2 Factor Orthogonal Array Example) The ties that bind us together, our networks, connecting the disparate dots on our application landscape allowing us to process transactions from around the globe. Imagine a circle of people holding hands moving in unison around the camp fire, a perfect symphony of movement as eveyone dances around faster and faster until the circle becomes a blur of movement. It’s not until someone’s shoestings come untied, followed by a stumble and a loud thud that we start to realize that we might have been standing too close to the open flames. This is the type of analogy that comes from a mind warped on videogames that add flamethrowers to your arsenal, games which I only play to discover software bugs and not as mindless entertainment for hours on end by the way. Back on subject, our application network is only as robust as its weakest link, an adage that has held true for thousands of years. Unless confronted with hard performance requirements this is another one of those areas that are so easy to take for granted in the context of of having tested intranet applications that reside behind the corporate firewall. In reality this is even more precarious than testing an external facing application whose end user is on the other side of the world wide web. I say that because if you want a simple worse case scenario all you need to do is govern the bandwidth that you conduct your test with when simulating an external user. You can take the broadband penetration numbers and scale accordingly using the lesser of the average connection speeds. Right now Nielsen/NetRatings pegs broadbacnd penetration at about 33% of American households, this is an abysmal number compared to other countries leaving us at an overwhelming majority of households at less than broadband or at 56kbps worst case. So if you wanted to cover your bets, all of the external users in your test would run at 56kbps. Obviously all of this could be trumped by a hard technical requirement that states the minimum connectivity speed. But when it comes to internal apps behind the firewall we tend to not even gather network stats because we are on the corporate LAN. This false sense of security can be shattered with one mis-configured router, an incorrect duplex setting on a NIC, bad cables, or poor network connection handling in the application. Complicating this is the infrastructure behind the corporate firewall then on the other side of the firewall you have the possibility of caching servers and services(Akamai), and possibly local caching on the client-side of the application puzzle.
In a vacuum a component test of a piece of business logic may be acceptable, but the result of that business logic execution could end up in a datagrid in the UI that is waiting to be rendered because the data hasn’t made it across the wires in a timely fashion. This means you absolutely have to take the render time into account from an end-user perpective and create a 360-degree view of your application performance which includes the network performance.
This now adds an additional factor to your testing array where you can represent various types of network speeds using less-than broadband, broadband and enterprise levels of connectivity. This then opens up to the addition of WAN topology and modes of transport such as wireless.
Don’t get burned.
Tags: General · Performance Testing
In helping various collegues and companies manage their performance testing efforts I’ve come across a reuseable set of common activities that have helped me explain and divide the work necessary for a successful engagement.
The following activies:
Engaging in Risk Identification
Developing a Performance-Testing Strategy
The application of the Performance-Testing Framework which produces a Performance-Testing Plan
o Assessment
o Modeling
o Execution
o Analysis
Assessment
Performing the assessment is critical to the justification of the performance-testing effort. The result of this activity forms the basis for the charter of the entire effort. There are a number of issues that need to be addressed per the pre-requisites.
Critical questions to ask concerning application performance:
• What is the real cost of system downtime?
• What is the impact on the perception of the organization’s ability to develop, test, implement, or maintain applications?
• What is the impact on the internal or external customer?
• What is the cost of correcting performance issues post-implementation?
Just as important, an assessment of the capability and capacity to execute performance-testing activities is just as important. This includes a deep dive into the measurements that can be collected, the hardware capacity of the hardware comprising the performance testing infrastructure and the network bandwidth to support the generation of transaction load and polling the respective performance counters.
The follow-up activities are:
Determining Performance Criteria
Verify the Measurements
Validate that the Metrics are Measurable
Determine Measures of Success
Modeling
The performance test lead(s) in collaboration with the business analyst(s), system analyst(s), and test lead(s) will determine the combination of test methodologies and test types that are applicable. Since this performance-testing framework does not rely on a completed application the appropriate methodology has to be determined in order to leverage this flexibility.
A performance test plan will be written to document the results of the assessment and the methodology chosen. In addition to the performance test plan, a performance test specification detailing the user and usage profiles, the scripts being developed, the traceability matrix, performance test environment diagram, SUT diagram and scheduling will accompany the plan.
User Profile
Usage Profile
Test Data
Developing Performance Test Scenarios
Performance Test Script and Scenario Traceability
Developing Performance Test Scripts
Execution
Performance Test Infrastructure Evaluation
Apply the appropriate Test Methodologies
Apply the appropriate Test Type(s)
Analysis
Sample Size, Inclusions, and Exclusions
Verify Quality of Test Execution
In Test Analysis
Post Test Analysis Deliverables
Top Down Analysis
Root Cause Analysis
Note the separation of the Test Methodology and Test Types from the Modeling activities. A single model can be used between methods(Profiling, Benchmarking) and Test Types(Load, Stress, Volume, Baseline).
Tags: Performance Testing · Testing
