Performance is Not a Check Box

For the Intralinks Performance Engineering team, our ultimate goal is application performance that yields an excellent user experience. Such a broadly defined goal requires a broad approach. In order to effectively deliver an excellent user experience, performance must be managed throughout the application life-cycle.


6 August 2009

For the Intralinks Performance Engineering team, our ultimate goal is application performance that yields an excellent user experience.

Such a broadly defined goal requires a broad approach. In order to effectively deliver an excellent user experience, performance must be managed throughout the application life-cycle. Taking a larger view of application performance opens up additional possibilities for improving user satisfaction that might be missed with a narrower focus. Application Performance is not just a testing activity, it is built-in to the product and therefore must be addressed throughout the product life-cycle in order to be effective and achieve the best results. Let's look at some of the ways application performance is managed during the software development process.

Requirements

From the very beginning, the definition of requirements can have a tremendous impact on application performance. Traditionally falling into the general bucket of "non-functional requirements," performance requirements are typically used as pass/fail criteria for later testing. For example:

• How many concurrent users must be supported?
• What is the expect throughput (transactions per second, bytes per second, etc)?
• What are the required response times?

Having these requirements defined up front is important to making sure the application architecture can support the defined specifications, and they give focus to performance testing efforts. However, taking a broader view of managing user experience, we must also consider the functional requirements and their potential impact. Even though they may not seem like performance requirements, there are many areas that must be evaluated. Here are just a few examples:

• What data sizes must be supported? Many performance issues are data-driven.
• What will be the supported browsers? Recent browsers perform much better than old.
• Can some complexity be simplified? Complexity increases risk of poor performance.
• Can asynchronous requests improve user experience? AJAX is an excellent example of improving user experience using asynchronous requests.
• How much data needs to be presented to the user? Can long lists of data be paginated?

Managing application requirements is the first step in managing performance.

Design and Architecture

Once the requirements are defined and agreed to, the performance engineers must understand the design and architecture of the solution, and identify potential risk areas. For example, a single-threaded process cannot be expected to satisfy hundreds of requests per second. Performance Engineers, Architects and Designers must carefully consider the design choices to ensure application responsiveness, scalability and stability:

• Use of frameworks, libraries, dependencies
• Understand potential architectural bottlenecks - database access, messaging, flow of data
• User Interface design

As is the case with all areas of software development, documentation and communication are key for performance engineers to gauge the impact of design and architecture on application performance. Armed with this knowledge, the performance engineer can more effectively identify root cause of performance issues, saving critical time in problem resolution.

Development

Performance defects are often hard to fix, making early detection a key factor in reducing product development and testing time. Some of the things we do to catch performance defects early include:

• Code analysis - PMD, FindBugs
• Integration Tests - Capture performance data during build integration testing
• Code profiling - Identify poor algorithms, slow SQL queries, etc.

Having performance engineers working closely and communicating with developers during this phase is critical not only for finding issues early, but also for creating effective performance tests. Comprehensive performance test plans are created during this phase, and having the implementation details is vital for identifying areas in the code that require deep testing.

Testing

By the time we get to the testing phase, after managing our application performance through requirements, design and implementation, we will have already identified and resolved many issues. There is much still to do, though, as scalability and concurrency issues can only be found in during large-scale testing in a suitable test environment, with production-like data. Furthermore, as code is changed, defects fixed, new requirements introduced, there is always a risk of introducing performance defects. Effective performance tests are critical to:

• Validate requirements are met
• Identify bottlenecks, concurrency issues
• Test scalability, stability

Equally as important as the actual test execution is the effective communication of the test results. Data + Context = Information, and performance testing activities generate massive amounts of data. Managing application performance requires having the proper context (requirements, design, architecture, etc) to make the performance data useful information.

Production

Application performance management doesn't stop after testing has been completed and the application is deployed and live in production. Once the application is live, the user experience must be measured and monitored in order to identify any performance problems that were not previously caught. Additionally, monitoring in production is absolutely critical for ongoing application performance management in order to validate the accuracy of performance tests, scalability predictions and usage models. Live production data provides key feedback to help refine and improve all of the previous performance management activities - requirements, design, architecture and testing. It is impossible to perfectly replicate or predict exactly how users will access the application and there will always be some issues that arise in production, so it is important to have the ability to detect performance issues as they happen and collect actionable data so the issue can be resolved quickly and at minimal cost.

More than Just Testing

Managing Application Performance is more than just performance testing, it is managing the user experience. It requires a comprehensive approach and involvement throughout the product life cycle. Performance is not a line item in a list of to-dos. Performance is not a check box.