Recently I witnessed the execution of performance test runs that did not include the elements necessary for successful and reliable result generation.
The test run should have included the following activities:
- Test setup
- Seed the database with fresh data
- Rebuild indexes
- Recreate statistics
- Test execution and
- Test tear down
- Delete data generated during the test
- Clear caches
- Recycle application pools
- Remove all database deadlocks
The initial performance run included steps 1-2; subsequent runs only included step 2. When the results from all runs where compared, it was observed that the performance of the system under test(SUT) gradually deteriorated.
The main point is that without proper tear down and setup after each performance run, the tests were measuring a different SUT each time.
For performance comparisons to be meaningful, one has got to compare similar SUTs.
If you can’t perform steps 1-3 because of complexities in your environment, then abandon your performance tests until these prerequisites are resolved.
No comments:
Post a Comment