A look at the Fundamentals in the Cloud

If you are interested in the Cloud and testing the following is a talk I did at GTAC2011 that might be interesting to you.

Part the Clouds and See Fact from Fiction

http://www.youtube.com/watch?v=nXIA3VYN1To&list=PLBB2CAFDDBD7B7265&index=9&feature=plpp_video

Old performance addage... Polling is bad

Its long been known that polling is bad.  It uses a ton of resources.  The challenge is it trips up even great developers.  Check out... http://m.guardian.co.uk/technology/2011/oct/29/iphone-4s-battery-location-services-bug?cat=technology&type=article

One way to catch this is to have a good set if resource monitoring tests.  Its very likely Apple had these however its hard to catch with so many ways to configure software.  This is where collecting these same resources from released devices can help (crowd sourcing test).  Check out for example Microsofts SQM (aka Customer Improvement Program) data.

Should you decide to collect telemetry just remember the second addage... Bad collection is like polling.

Crowd sourcing Apple iPhone 4S power performance

Interesting easy to solve the issue...

http://m.guardian.co.uk/technology/2011/oct/28/iphone-4s-battery-apple-engineers?cat=technology&type=article

Performance Test Pattern

One of the biggest challenges in monitoring and tracking performance is getting stable and repeatable numbers.  Check out the following plot of two performance tests.  On which do you think it will be easier to spot regressions?

There are two test here.  Test 1 which looks pretty erratic and test 2 which looks pretty stable and repeatable around 3 ms.  Given this I'm pretty sure you are going to choose test 2 on which to monitor and track performance.  The test is very stable meaning the variance is very low and repeatable because it does not drift off over time.  For an example of drift check out the following:

Here you can see the results are pretty stable in that the overall variance from number to number if pretty low however the results are not very repeatable and seem to drift up over time.  For this particular test high ping time is bad.  This is also and example of "death-by-a-thousand cuts" where from test to test results look good but over long periods of time you see performance is dropping off.

So the question then comes up how do you make stable and repeatable performance tests?  The answer is to follow a test pattern like the xUnit pattern with a couple of extra steps. The pattern is the following:

1. setup 
2. warmup
3. execute
4. something most tests forget
5. publish
6. cleanup
7. teardown

Notice the following additional tests - warmup, step 4, publish, cleanup.  Now let me explain these steps.   

Warmup - This step is here to allow the performance test to "warm-up" the system under test.  For example if you want to measure database queries generally you have to decide if you want hot (most likely the common case) numbers where the database has been in use for a while or cold numbers which is the state right after boot/init/etc.  By having warmup you can test both hot and cold tests by the additional or removal of this step.  An example might be selecting 10 rows from a database before doing the general select tests.

Step 4 - Ahh... the mystery.  What is step 4?  Take a quick look back at the first graph.  Any ideas?  Well the answer is VALIDATE.  Most performance tests forget to validate the results they are getting.  In the previous step of warmup we said to select 10 rows.  Did the test actually return 10 rows?  If not there is likely some error.  Be sure to check your results and dont publish them if there was an error.  Generally on performance graphs invalid results look like super high, 0, or super low numbers.

Publish - This is the act of pushing the result into your tracking infrastructure.  Performance results tend to have a lifttime of usefulness however there is always good cause to look back over time.

Cleanup - Cleanup is like teardown without exiting all layers of initialization.  Generally the role of cleanup is to get things put back in order so the test can be run again with minimal side-effects.  For cold performance results you will need to teardown.

While execute is not a new step in the performance pattern I wanted to mention it because often times in performance tests you want "stable" numbers.  This is generally achieve by running the execute step a number of times and averaging or repeating steps 3 - 6 a number of times.  While averaging is often the right answer it can sometimes hide performance issues.  Perhaps I blog on that another day.

Now that you have a solid performance test pattern go forth and create amazing results....

  Tony

Remember WPR?Ccheck out the drive for better power use at Google

A couple of posts ago I talked about Watt Per Request (WPR) how power is becoming ever more important in http://perfguy.blogspot.com/2011/05/single-most-import-performance-metric.html.  What is cool is Google just released its power consumption to the world and it gives some good insights.  In Google fashion everything was accounted for right down to the Google Street Cars.  Check out the NY times article http://www.nytimes.com/2011/09/09/technology/google-details-electricity-output-of-its-data-centers.html.  To learn more about Google power use and the industry standard metric PUE (Power Usage Effectiveness) check out http://www.google.com/about/datacenters/index.html.

Enjoy,

   Tony

Ahh... the world has evolved. No more 1TB sorts.

The 1TB sort competition has ended because winners take less than a second now. Good news is there are other competitions... http://sortbenchmark.org/

The second most important metric - Location, Location, Location

You might think from the title "Location, location, location" this will be a post about real estate however in the new era of Cloud and Mobile Computing location is going to big a huge factor both for the design of your service as well as testing it and in particular performance.  Cloud Computing is a growing trend which is enabling all kinds of new applications.  If you want to find out more on Cloud Computing you can check our the slides from a talk I did at the Better Software Conference this year in Las Vegas here.

So why does location matter?  Location matters because of physics and no one has figured out how to out run the speed of light.  Ok... that is a little abstract so let me give an example.  Image you live in South Africa and want to deploy your cool new service in California because lots of Cloud providers have data centers there - superfast to do and cheap.  A single packet of data will take a half of a second to go from South Africa to California and back.  Thats how long it takes light to travel (roughly). Now image if your new site serves pages with images to fetch, database rows to read, etc.  Each new object you serve means the user (using a browser) in South Africa has to request data from California.  Each round trip is 0.5 seconds so a page with 10 images could take 5 to 10 seconds just is trying to initiate a fetch.  Wow... its not looking good for your service if you want people in South Africa to use it.

Some generally accepted time constrains for operations to happen are the following -

• User interfaces should respond in around 100 millisec or less.  Human perception is around 30 millisec.
• A user can detect a software hang in round one second and will take action at around 5 seconds to fix it.
• The good news for the web is users are willing to wait up to 15 seconds for a page however they will likely never come back if it takes more than 30 seconds.

So now that you have seen deploying your new service in California for you South Africa users is not such a good idea what should you do?  The answer is to find zones closer to you like Europe or possibly Asia.  At the time of writing this I dont know of anyone providing Cloud resources directly in Africa however the landscape is changing quickly as demand rises.

Another example of how location matters is interactive games.  Image a multi-player games really popular on the East Coast of the US with all the game servers on the West Coast of the US.  In general a packet takes around 50 to 70 millsec to travel there an back.  This means a game can only get around 10 to 15 corrections a second.  These long latencies can show up in you shooting anther player first but for some reason you die.  Gamers hate this.

Given the growing dependence of cool applications on Cloud resources its time really start thinking about where your users are and where you services are located.  The shorter the physical distance the better.

Location also matter for legal and privacy issues however thats a whole topic onto itself.