tag:blogger.com,1999:blog-27204876400929133062024-03-13T20:11:52.978-07:00All topics fundamentalThis blog will be about all topics fundamental to computer science, computer users, IT admins, and much more.Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-2720487640092913306.post-55773858204471441802012-10-26T10:41:00.001-07:002012-10-26T10:41:24.197-07:00<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-size: large;">Google Compute Engine breaks Hadoop Terasort World Record</span></h2>
While I've been light on posts as of recent, I though I'd share a cool result that Google Cloud and MapR accomplished together.<br />
<br />
This has been the result of a lot of performance analysis, tuning, and much more but nothing is specialized. You get this experience without any effort - just boot virtual machines and go.<br />
<br />
Here is a press release on the accomplishment:<br />
<a href="http://insights.wired.com/video/mapr-google-compute-engine-set-new-world-record-for-hadoop-teraso">http://insights.wired.com/video/mapr-google-compute-engine-set-new-world-record-for-hadoop-teraso</a><br />
<br />
The Cloud is powerful.<br />
<br />
Enjoy,<br />
Tony<br />
<br />
PS. My Google Cloud Security, Performance and Test team is hiring :)</div>
Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-78128607719424588522012-10-03T00:56:00.001-07:002012-10-03T00:56:33.995-07:00Testing 2.0: Enter the human<div dir="ltr" style="text-align: left;" trbidi="on">
Over the last couple of weeks I've started writing and speaking on what I'm calling Testing 2.0. To get an overview on this new chapter in test check out this <a href="http://googletesting.blogspot.com/2012/08/testing-20.html">post</a>.<br />
<br />
As a follow on to the Google Testing Blog post I had the honor of speaking at <a href="http://events.yandex.ru/events/yac/2012/">YaC</a> in Russia. The talk was focused on how to improve engineering productivity one of the focuses of Testing 2.0. You can see the talk <a href="http://events.yandex.ru/talks/335/">here</a> (video soon).<br />
<br />
On a side note... visiting Russia was awesome.<br />
<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-69556980231059896852012-03-24T19:00:00.001-07:002012-03-24T19:01:13.566-07:00Ads can steal your power - Mobile trade-offs<div dir="ltr" style="text-align: left;" trbidi="on"><div style="text-align: left;">In reading the following article the other day <a href="http://www.pcmag.com/article2/0,2817,2401797,00.asp">"<span style="background-color: white; font-family: arial; line-height: 32px;">Study: Free Android Apps Can Steal Your Phone's Power"</span></a> I was reminded of all the trade-offs one has to make when designing mobile applications. Before we dig into some of the trade-offs one does have to wonder about the purpose of one company doing a sanctioned study of another companies products. We'll leave digging into that topic for the another day.</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">Now back to the topic of mobile trade-offs. Some might argue this, but the single most important thing to design for is minimal power consumption. Power is so important on mobile platforms because users really don't want to hang out at charging pods in airports, plug in on the train, plugging in at a friends house, ... One of the big delighters of the original Kindle was it could run for weeks on a single charge. The newer Kindle Fire lasts less than a day like most of current gen platforms. In looking at the new iPad3 it has a battery almost twice the size without any addition device life ~10 hours. So where is all that power going? The two biggest draws on power are the screen and radio. You have some control of the power consumption on the screen. The dimmer you go the longer you go. The iPad3 has a super dense screen and new graphics CPUs. I wonder how many people would keep the old screen and CPUs for 20 hours of use on an iPad2? </div><div style="text-align: left;"><br />
</div><div style="text-align: left;">I have a Windows Phone 7 (yes I am willing to admit I have one - I'm a techie) and just use it as an in house wi-fi device. What totally surprised me is the phone lasts for weeks without the radio on. The first time I had that happen I was pretty surprised by just how much impact the radio did have.</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">While I've talked able the screen and radio we can't ignore use of the processor (CPU). Most mobile devices have ARM chips which do all kinds of cool things like clock the CPU lower when not in use, have low power cores for when the phone is idle, etc. The take-away is doing really CPU intensive work like my fractal app at http://www.tonyware.com/fractals will drain your power. It has my phone because zoom in is so cool :)</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">So what are some of the trade-offs for mobile you should be thinking about?</div><div style="text-align: left;"></div><ul style="text-align: left;"><li>Do you really need to send data to your backend server every 10sec or would once an hour be good? Not only will this save power it will also cost your customers less of their network bandwidth. (Rule #1 - Use the radio less)</li>
<li>Does having a white background really make the App better? Can you choose dimmer colors? If the App is idle should you dim the screen? (Rule #2 - lighting up more pixels uses more power)</li>
<li>What about pushing more computation to the server rather than having the phone do it? (Rule #3 - Dont use the CPU for big calculations)</li>
</ul><div>The last point around doing work on the server rather than the phone is one of the reasons <a href="http://www.google.com/appengine">Google AppEngine</a> is growing so quickly for mobile. The more you can do on the server the more power will be saved. Its also very likely the computation will go much faster on the server. The trade-off on the developer end is how to manage cost while getting the best customer experience. By the way computation in the Cloud are far "greener" than on any other computer device. But that is a topic for another day.</div><div><br />
</div><div>Hope this got you thinking....</div><div><br />
</div><div> Enjoy,</div><div> Anthony F. Voellm (aka Tony)</div><br />
<div style="text-align: left;"><br />
</div></div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-28079897327765566872012-03-02T11:29:00.000-08:002012-03-02T11:29:49.483-08:00Old rules of thumb always need to be reconsidered<div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Arial, Helvetica, sans-serif;">After being in the computer industry for a while you begin to appreciate just how much machine capabilities change and the need to change designs along with them. For example just 10 years about developers would spend hours trying to find ways to save a few bytes of memory. Now most of the code the world runs is via interpreters (JiT, PY, PhP, Jscript, ...) and a few bytes is less interesting. I'm not saying to waste them but I personally would not put it as my first priority. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Let me give you an example of how changes in machine capabilities caused the rethinking of an OS. In Windows XP Microsoft Engineers designed the memory manager to aggressively push data from main memory (RAM) to disk. This was done because RAM was costly and very small (~128MB) at the time and so if more memory could be freed up new applications could start faster. If you waited until an application started to free memory users would wait 30 seconds to minutes before the application was usable because of paging RAM to disk. Between the time Windows XP and Vista shipped RAM prices dropped dramatically (from $40 for 128MB to just $2 dollars). </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg87ApaV1qcwGcivLyyBtTnsAM91HRSepqvQq8Zjo6cYmLfL7XB6UEiiRcrW2ocyL16ACoaaAMAJc_wFAc_3uEIrQpvC96ES953tUSMX2YNHSmH_rfTTYCTmxtxhGhW2TvmQ9xxsjQuiI0/s1600/Screenshot-4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" height="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg87ApaV1qcwGcivLyyBtTnsAM91HRSepqvQq8Zjo6cYmLfL7XB6UEiiRcrW2ocyL16ACoaaAMAJc_wFAc_3uEIrQpvC96ES953tUSMX2YNHSmH_rfTTYCTmxtxhGhW2TvmQ9xxsjQuiI0/s640/Screenshot-4.png" width="640" /></span></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEkRqcIR9mJdKcXH7J3kTTnNWUKyDSQSWXPn1FF_q2iZW-178-MbICZxuhof3IldkEiuBDqVWmrYNsF_RGph-CMxXlKbHM2YyzGvgaiA-TwsQGxBs9cYEDhWvXBFKmOWC6hVRcTfZw4ok/s1600/Screenshot-5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEkRqcIR9mJdKcXH7J3kTTnNWUKyDSQSWXPn1FF_q2iZW-178-MbICZxuhof3IldkEiuBDqVWmrYNsF_RGph-CMxXlKbHM2YyzGvgaiA-TwsQGxBs9cYEDhWvXBFKmOWC6hVRcTfZw4ok/s640/Screenshot-5.png" width="640" /></span></a></div><br />
<span style="font-family: Arial, Helvetica, sans-serif;">With the dramatic change in memory prices and the fact disks did not really get any faster Vista fundamentally broke from the past rule of thumb of free up as much RAM as possible and push it to disk to just the reverse. RAM was cheap and relatively plentiful so a feature called <a href="http://blogs.technet.com/b/askperf/archive/2007/03/29/windows-vista-superfetch-readyboost.aspx">SuperFetch</a> was created to aggressively page in data from disk to RAM. Based on the decision to not force RAM to disk overall UI performance seemed to be more snappy in Vista. No more shaking the mouse after lunch with XP and waiting a minute+ before logging in.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Well it looks like with the improved performance of CPU's and networks old rules of thumb around UI responsiveness are starting to be reconsidered. Some early UI research in 1968 by Miller and 1991 by Card lead to rules of thumb for UI regularly cited in <a href="http://www.useit.com/papers/responsetime.html">"Response Times: The 3 important limits"</a> and extended for the World Wide Wait, I mean Web.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Here is a recap of those rules and a few more that have been adopted from experience and very likely paper I've read long ago and forgotten:</span><br />
<span id="internal-source-marker_0.40357438125647604" style="text-align: -webkit-auto;"></span><br />
<ul style="font-weight: bold;"><li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">Users consider 100ms response times fast</span></span></li>
<li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">At around 1 seconds users will notice a delay but are tolerant</span></span></li>
<li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">At 5 seconds users are starting to get impatient and may take action</span></span></li>
<li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">At 10 seconds they lose focus</span></span></li>
<li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">At 15 seconds they are likely to hit “refresh”</span></span></li>
<li style="font-size: 15px; font-weight: normal; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial, Helvetica, sans-serif;">At 30 seconds they generally navigate away and don’t come back if there is an acceptable alternative.</span></span></li>
</ul><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="white-space: pre-wrap;">Well it looks like even hard earned rules of thumb for UI and Web are now falling as seen in a recent NYTIMES article </span><span style="background-color: white; color: #333333; font-weight: bold; text-align: left;"><a href="http://mobile.nytimes.com/article;jsessionid=1637B981EC2094F40D6DCB7A0DA46F97.w6?a=919045&f=24">"For Impatient Web Users, an Eye Blink Is Just Too Long to Wait".</a> </span><span style="text-align: left;">Based on this article it looks like 250millisec is the new goal for web responsiveness rather than 1 second as we had all used.</span></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="text-align: left;"><br />
</span></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="text-align: left;">The overall morale of the story is don't hold on too dear to those rules of thumb and perhaps you should rethink them often. </span></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="text-align: left;"><br />
</span></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="text-align: left;"> -- </span></span><span style="font-family: Arial, Helvetica, sans-serif; text-align: left;">Anthony F. Voellm</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;"><span style="text-align: left;"><br />
</span></span></div></div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-66406800287652508052012-02-27T13:53:00.002-08:002012-02-27T13:53:59.702-08:00Fix security bugs early - Interesting paper<div dir="ltr" style="text-align: left;" trbidi="on"><span style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;">Interesting paper - Find security bugs before they release because of the high cost to fix later. Internet Apps change some of the cost dynamics however the that does not mean fixing early is less important because its hard to fix your reputation.</span><br style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;" /><br style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;" /><span style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;">http://www.stickyminds.com/Files/Automated%20Testing%20With%20Commerical%20Fuzzing%20Tools.pdf</span><br />
<span style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj31bD2ZDrvgVOHdkWlTNo3z-uesJBs8joii2tQP-Scp43ZMvkMi91_Vb1uP4RpT5ntoKpOvyWT_5FVsjr2Kj5HgF1VpU5loC_Z4GLyG42Vd11VPmyR3v_05NK5W20OmP1ID1ycNywXPc/s1600/Screenshot-6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="219" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj31bD2ZDrvgVOHdkWlTNo3z-uesJBs8joii2tQP-Scp43ZMvkMi91_Vb1uP4RpT5ntoKpOvyWT_5FVsjr2Kj5HgF1VpU5loC_Z4GLyG42Vd11VPmyR3v_05NK5W20OmP1ID1ycNywXPc/s320/Screenshot-6.png" width="320" /></a></div><span style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; text-align: -webkit-auto;"><br />
</span></div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-4266419960962210662011-11-07T14:52:00.000-08:002011-11-07T14:52:50.675-08:00A look at the Fundamentals in the CloudIf you are interested in the Cloud and testing the following is a talk I did at GTAC2011 that might be interesting to you.<div><br />
</div><div class="separator" style="clear: both; text-align: center;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUG6gPOitUMz4l478D5BWqzwgrYdJRmYmazftNX3ngqlpoVZNZ0MH5S57jgamhqFFlyzzwIZ1tqroXBUPimdZxtgom7GWaI826vr9Dp3RY4EnsELi2jCwPnPebTrpTVEHH5CHlAW53P5Y/s320/Screenshot-3.png" width="320" /><span id="goog_376718618"></span><span id="goog_376718619"></span><a href="http://www.blogger.com/"></a></div><div class="separator" style="clear: both; text-align: center;"><span style="text-align: -webkit-auto;">Part the Clouds and See Fact from Fiction</span></div><div><br />
</div><div style="text-align: center;"><a href="http://www.youtube.com/watch?v=nXIA3VYN1To&list=PLBB2CAFDDBD7B7265&index=9&feature=plpp_video">http://www.youtube.com/watch?v=nXIA3VYN1To&list=PLBB2CAFDDBD7B7265&index=9&feature=plpp_video</a></div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com1tag:blogger.com,1999:blog-2720487640092913306.post-46334173567872970192011-10-30T07:42:00.001-07:002011-10-30T07:42:11.222-07:00Old performance addage... Polling is bad<div><p>Its long been known that polling is bad. It uses a ton of resources. The challenge is it trips up even great developers. Check out... http://m.guardian.co.uk/technology/2011/oct/29/iphone-4s-battery-location-services-bug?cat=technology&type=article</p>
<p>One way to catch this is to have a good set if resource monitoring tests. Its very likely Apple had these however its hard to catch with so many ways to configure software. This is where collecting these same resources from released devices can help (crowd sourcing test). Check out for example Microsofts SQM (aka Customer Improvement Program) data.</p>
<p>Should you decide to collect telemetry just remember the second addage... Bad collection is like polling.</p>
</div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-55757857135402445092011-10-28T15:04:00.001-07:002011-10-28T16:21:47.271-07:00Crowd sourcing Apple iPhone 4S power performance<div><p>Interesting easy to solve the issue...</p>
<p>http://m.guardian.co.uk/technology/2011/oct/28/iphone-4s-battery-apple-engineers?cat=technology&type=article</p>
</div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-10366409907879355692011-10-17T08:55:00.000-07:002011-10-17T08:55:58.632-07:00Performance Test PatternOne of the biggest challenges in monitoring and tracking performance is getting stable and repeatable numbers. Check out the following plot of two performance tests. On which do you think it will be easier to spot regressions?<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAZ5LZxWeL7lHKrSrCA_BThgQdu6f78U0Z1ZT8uh_XM35Frakm5qB38ruzM4AA6jSsU1A9MA7A9Ba95Mp4Cc4DA2fr19Z_A8_s5VqfVv7IPGPScyjGOA4Zwxp2eZWJWVgHIWdfLJtCBvE/s1600/Screen+shot+2011-10-17+at+8.24.44+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAZ5LZxWeL7lHKrSrCA_BThgQdu6f78U0Z1ZT8uh_XM35Frakm5qB38ruzM4AA6jSsU1A9MA7A9Ba95Mp4Cc4DA2fr19Z_A8_s5VqfVv7IPGPScyjGOA4Zwxp2eZWJWVgHIWdfLJtCBvE/s400/Screen+shot+2011-10-17+at+8.24.44+AM.png" width="400" /></a></div><br />
There are two test here. Test 1 which looks pretty erratic and test 2 which looks pretty stable and repeatable around 3 ms. Given this I'm pretty sure you are going to choose test 2 on which to monitor and track performance. The test is very stable meaning the variance is very low and repeatable because it does not drift off over time. For an example of drift check out the following:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCaEVp3afE-ZhqMVRBT6SqOTf1LsXACeifZ23FTUEWGerXV6ZxVPpqNt5HalIKkdNTVnhUBknNf__yhdPBTE-MlY60iCNSI440jCiXZu1SwPg6FI5Y4Y5Vp4XtC25c1UYBW0SPT6i_6E4/s1600/Screen+shot+2011-10-17+at+8.30.01+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCaEVp3afE-ZhqMVRBT6SqOTf1LsXACeifZ23FTUEWGerXV6ZxVPpqNt5HalIKkdNTVnhUBknNf__yhdPBTE-MlY60iCNSI440jCiXZu1SwPg6FI5Y4Y5Vp4XtC25c1UYBW0SPT6i_6E4/s400/Screen+shot+2011-10-17+at+8.30.01+AM.png" width="400" /></a></div><br />
Here you can see the results are pretty stable in that the overall variance from number to number if pretty low however the results are not very repeatable and seem to drift up over time. For this particular test high ping time is bad. This is also and example of "death-by-a-thousand cuts" where from test to test results look good but over long periods of time you see performance is dropping off.<br />
<br />
So the question then comes up how do you make stable and repeatable performance tests? The answer is to follow a test pattern like the <a href="http://en.wikipedia.org/wiki/XUnit">xUnit </a>pattern with a couple of extra steps. The pattern is the following:<br />
<br />
<ol><li>setup </li>
<li>warmup</li>
<li>execute</li>
<li>something most tests forget</li>
<li>publish</li>
<li>cleanup</li>
<li>teardown</li>
</ol><div>Notice the following additional tests - warmup, step 4, publish, cleanup. Now let me explain these steps. </div><div><br />
</div><div><b>Warmup</b> - This step is here to allow the performance test to "warm-up" the system under test. For example if you want to measure database queries generally you have to decide if you want hot (most likely the common case) numbers where the database has been in use for a while or cold numbers which is the state right after boot/init/etc. By having warmup you can test both hot and cold tests by the additional or removal of this step. An example might be selecting 10 rows from a database before doing the general select tests.</div><div><br />
</div><div><b>Step 4</b> - Ahh... the mystery. What is step 4? Take a quick look back at the first graph. Any ideas? Well the answer is VALIDATE. Most performance tests forget to validate the results they are getting. In the previous step of warmup we said to select 10 rows. Did the test actually return 10 rows? If not there is likely some error. Be sure to check your results and dont publish them if there was an error. Generally on performance graphs invalid results look like super high, 0, or super low numbers.</div><div><br />
</div><div><b>Publish</b> - This is the act of pushing the result into your tracking infrastructure. Performance results tend to have a lifttime of usefulness however there is always good cause to look back over time.</div><div><br />
</div><div><b>Cleanup</b> - Cleanup is like teardown without exiting all layers of initialization. Generally the role of cleanup is to get things put back in order so the test can be run again with minimal side-effects. For cold performance results you will need to teardown.</div><div><br />
</div><div>While execute is not a new step in the performance pattern I wanted to mention it because often times in performance tests you want "stable" numbers. This is generally achieve by running the execute step a number of times and averaging or repeating steps 3 - 6 a number of times. While averaging is often the right answer it can sometimes hide performance issues. Perhaps I blog on that another day.</div><div><br />
</div><div>Now that you have a solid performance test pattern go forth and create amazing results....</div><div><br />
</div><div> Tony</div><div><br />
</div><div><br />
</div><div><br />
</div><div><br />
</div><div><br />
</div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-10671022556029142882011-09-09T10:03:00.000-07:002011-09-09T10:11:00.106-07:00Remember WPR?Ccheck out the drive for better power use at GoogleA couple of posts ago I talked about Watt Per Request (WPR) how power is becoming ever more important in <a href="http://perfguy.blogspot.com/2011/05/single-most-import-performance-metric.html">http://perfguy.blogspot.com/2011/05/single-most-import-performance-metric.html</a>. What is cool is Google just released its power consumption to the world and it gives some good insights. In Google fashion everything was accounted for right down to the Google Street Cars. Check out the NY times article <a href="http://www.nytimes.com/2011/09/09/technology/google-details-electricity-output-of-its-data-centers.html">http://www.nytimes.com/2011/09/09/technology/google-details-electricity-output-of-its-data-centers.html</a>. To learn more about Google power use and the industry standard metric PUE (Power Usage Effectiveness) check out <a href="http://www.google.com/about/datacenters/index.html">http://www.google.com/about/datacenters/index.html</a>.<br />
<div><br />
</div><div>Enjoy,</div><div> Tony</div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-71131567523375863462011-07-11T16:40:00.001-07:002011-07-11T16:40:56.674-07:00Ahh... the world has evolved. No more 1TB sorts.<span class="Apple-style-span" style="background-color: white; color: black; font-family: arial,sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 18px; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">The 1TB sort competition has ended because winners take less than a second now. Good news is there are other competitions...<span class="Apple-converted-space"> </span><a class="ot-anchor" href="http://sortbenchmark.org/" style="color: #3366cc; cursor: pointer; text-decoration: underline;">http://sortbenchmark.org/</a></span>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com1tag:blogger.com,1999:blog-2720487640092913306.post-2406194780267530932011-06-30T10:29:00.000-07:002011-06-30T10:29:49.378-07:00The second most important metric - Location, Location, Location<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">You might think from the title "Location, location, location" this will be a post about real estate however in the new era of Cloud and Mobile Computing location is going to big a huge factor both for the design of your service as well as testing it and in particular performance. Cloud Computing is a growing trend which is enabling all kinds of new applications. If you want to find out more on Cloud Computing you can check our the slides from a talk I did at the<a href="http://www.sqe.com/conferencearchive/bscwest2011/"> Better Software Conference</a> this year in Las Vegas <a href="https://docs.google.com/present/edit?id=0AZiICDqrB_E9ZGZocHJ0ZnpfM2ZxdjkzamR2&hl=en_US&authkey=CPzYqbUO">here</a>.</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">So why does location matter? Location matters because of physics and no one has figured out how to out run the speed of light. Ok... that is a little abstract so let me give an example. Image you live in South Africa and want to deploy your cool new service in California because lots of Cloud providers have data centers there - superfast to do and cheap. A single packet of data will take a half of a second to go from South Africa to California and back. Thats how long it takes light to travel (roughly). Now image if your new site serves pages with images to fetch, database rows to read, etc. Each new object you serve means the user (using a browser) in South Africa has to request data from California. Each round trip is 0.5 seconds so a page with 10 images could take 5 to 10 seconds just is trying to initiate a fetch. Wow... its not looking good for your service if you want people in South Africa to use it.</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Some generally accepted time constrains for operations to happen are the following -</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><ul><li>User interfaces should respond in around 100 millisec or less. Human perception is around 30 millisec.</li>
<li>A user can detect a software hang in round one second and will take action at around 5 seconds to fix it.</li>
<li>The good news for the web is users are willing to wait up to 15 seconds for a page however they will likely never come back if it takes more than 30 seconds.</li>
</ul><div>So now that you have seen deploying your new service in California for you South Africa users is not such a good idea what should you do? The answer is to find zones closer to you like Europe or possibly Asia. At the time of writing this I dont know of anyone providing Cloud resources directly in Africa however the landscape is changing quickly as demand rises.</div><div><br />
</div><div>Another example of how location matters is interactive games. Image a multi-player games really popular on the East Coast of the US with all the game servers on the West Coast of the US. In general a packet takes around 50 to 70 millsec to travel there an back. This means a game can only get around 10 to 15 corrections a second. These long latencies can show up in you shooting anther player first but for some reason you die. Gamers hate this.</div><div><br />
</div><div>Given the growing dependence of cool applications on Cloud resources its time really start thinking about where your users are and where you services are located. The shorter the physical distance the better.</div><div><br />
</div><div>Location also matter for legal and privacy issues however thats a whole topic onto itself. </div>Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-22839254514981025022011-05-08T08:43:00.000-07:002011-05-08T08:43:06.553-07:00The single most import performance metric - WPRBefore we dive into WPR I'd like to take a moment to write about metrics because without metrics there is nothing to measure or tune. Metrics are the quantities you are going to measure on software and hardware. More formally a metric is a unit of measure. There are tons of interesting metrics like %CPU for CPU utilization, Packets Per Second, QPS (Queries per second), FPS (frames per second), RTT (round trip time), and so on.<br />
<br />
In general metrics are thought of in two classes - Utilization and Throughput/Latency. Utilization is a measure of how much something is used so from the previous example %CPU is a utilization metric. Throughput metrics are a measure of the rate at what things are getting done like QPS. Latency is how long it takes for an individual piece of work to complete like RTT. Another example of Throughput/Latency is while Google might do millions of queries a second (Throughput) you the end user are concerned with how fast your query runs (latency). Server software tends to tune for throughput while interactive software like mobile phone apps tune for latency.<br />
<div><br />
</div>Another term you might have heard is efficiency. Efficiency is a measure of wasted work. The more efficient something is the less work is wasting (ie driving around the block twice before parking is likely wasted work). I dont list it in the metric classes above because both utilization and throughput/latency metrics can be used to derive efficiency.<br />
<br />
In my experience throughput/latency measures are more reliable than utilization metrics. There are lots of reasons for this like Virtual Machines and advanced CPU's tend to skew utilization but not throughput. You can see a past post of mine that talks about skew on virtual machines <a href="http://blogs.msdn.com/b/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx">here</a>. If there is interest I can write more on this topic.<br />
<br />
Now back to WPR... WPR is Watts per request. Watts is a measure of the power used. You might have seen references to Power Performance or Power Utilization etc over the last couple of years but why does it matter so much? Power utilization is so important these days because of portable devices and data centers. <br />
<br />
Ten years ago most computing was under the desk and prior to that it was in a central room. Power in the central room was interesting however important issues like the speed of computation drove engineering. Under the desk the costs of inefficient computations (High WPR) was so spread out most people did not notice or care. However we all have really fast processors now (and yes they can be faster) and a computer in your pocket[book]. In your pocket[book] watts = surfing/talking/dorking time and in the data center watts = heat. Heat means you have to pay a lot for space and cooling. The biggest costs for a data center is not the computers but rather the space and power used for cooling.<br />
<br />
Now in simpler concepts as to why WPR is so important - lower WPR can make your phone/tablet/laptop last longer and save you money in the datacenter.<br />
<br />
So now you might be wondering how to measure power. On Windows machines you can use "c:\windows\system32\powercfg -energy" and on Linux machines you can read sensors (lm-sensors). The internal computer sensors can be useful however most engineers looking to drive WPR are using external measurement tools like <a href="http://www.extech.com/instruments/product.asp?catid=14&prodid=206">Extech</a> or <a href="http://www.intech21.com/products/pm_2104_3_v6.html">Intech</a> which are more accurate and have computer readouts which can be used for automation.<br />
<br />
Little did you know that Qsort [O(n*log(n))] vs. BubbleSoft [O(n^2)] was making happier users, saving money, and making the world a little greener. Happy power hunting.<br />
<br />
Tony VoellmAnonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-69567431833714649952011-05-01T16:51:00.000-07:002011-05-01T16:51:10.475-07:00Three steps to making great performant softwareOver the last 20+ years I have been teaching and learning about performance and its time to return more of what I've learned to the public domain. My knowledge is based in OS (was a Windows and IRIX kernel developer as well as the Hyper-V perf lead), DB (lead the SQL Server perf team), web apps, compilers, image processing, optimization, and much more. I've worked at the best companies like SGI, MSFT, and now Google which has also given me a wider perspective.<br />
<br />
So now that you have a little of my background I'm going to teach you the three steps to making performant software. You Ready?<br />
<br />
Step 1: Have a plan<br />
Step 2: Instrument<br />
Step 3: Measure and Track<br />
<br />
Yep... thats it. Now to put this into perspective the diet industry also has three steps<br />
<br />
Step 1: Eat less<br />
Step 2: Exercise more<br />
Step 3: Keep doing 1 and 2<br />
<br />
However simple those three steps are there is a mutil-billion dollar business out there to teach them to us. The steps are not easy and there are a lot of nuances like "What should I eat less of", etc. The three steps to great performance is a lot like the diet steps. There are a lot of nuances and in coming posts I'll detail them more. For now I'll give you a quick rundown.<br />
<br />
Step 1 is to have a plan. This means you have an idea of why you are trying to improve the software and how you want to improve it. You have some goal in mind. If you have no goal then why are you performance tuning?<br />
<br />
Step 2 is to instrument. This means you will be putting markers into the code you are measuring in such a way you can figure out how close you are to your goal. There are lots of ways to instrument with the simplest being a printf, performance counters, Windows ETW, etc.<br />
<br />
Step 3 is to measure and track. This mean with each change you make you'll measure the impact it has on your performance goals and track it overtime. If a regression shows up you'll be ready to fix it.<br />
<br />
I can't wait to dig into the three steps more with you...<br />
<br />
Tony VoellmAnonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com1tag:blogger.com,1999:blog-2720487640092913306.post-84480505743208126082011-04-25T11:52:00.001-07:002011-04-25T11:52:51.956-07:00Ask your Hyper-V or any other software fundamentals questions here...If you have questions about Hyper-V or other topics around software fundamentals in general you can post a comment here and in due time I'll answer it.Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com6tag:blogger.com,1999:blog-2720487640092913306.post-1677341443056969842011-04-25T10:32:00.000-07:002011-04-25T10:32:19.924-07:00What's up with "/", "\", ";", "," ?Since this blog is dedicated to all things fundamental I thought I put out a quick post on usability. Over the years of computing there have been many common meme's to come about such as using "CRTL-C" (or Apple Key-C on Mac) for copy, holding the power key for 8+ seconds to do a hard power off, and much more. These meme's work across all platforms and really make computing accessible. This is great.<br />
<br />
Why is it then that email clients like Outlook don't accept both ";" or "," to separate names? To Gmail's credit while it prefers "," it accepts ";" without issue. It seems like such a simple usability improvement for Outlook and Windows Phone 7. If anyone knows the history behind the choice of ";" vs. "," and why or why not to accept both I'd love to hear it.<br />
<br />
As for "/" and "\" for directory naming we could go on for ages on this topic. Fortunately most users never have to worry about this now because directory access is abstracted via GUI's.<br />
<br />
This post was really intended to just help you think about the little things... they really matter!Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com0tag:blogger.com,1999:blog-2720487640092913306.post-23510919761920139962011-03-02T06:59:00.000-08:002011-03-02T06:59:20.438-08:00Which is faster Windows Hyper-V or VMWare ESX?Catchy title and I am sure you want to know the simple answer - Hyper-V or ESX. The challenge is the answer is not so simple. This blog will help you understand why the answer is not so simple and should help you in asking some questions when trying to decide.<br />
<br />
Before I go too far I should let you know I am definitely biased toward Hyper-V after being the Performance Lead for three releases. You can check out my Hyper-V only blog on http://blogs.msdn.com/tvoellm. However now that I am no longer with Microsoft I'll do my best to give some good balanced insights.<br />
<br />
While ESX has been around longer than Hyper-V I dont think you should use this to determine how fast for functional one is over the other. For example is MySpace better than Facebook? What you will likely see from a more mature product will be higher reliability just because the engineers for the mature product has had more time to shake out bugs. I can't really speak to ESX reliability however I can save Hyper-V runs on everything from 1 processor up to 64 processors and we literally tested it day and night for thousands of hours of up time fault free. I think it is more than ready for your mission critical applications.<br />
<br />
Now back to performance. First you need to understand there are a couple of types of virtualization. You can get all the details on <a href="http://en.wikipedia.org/wiki/Virtualization">http://en.wikipedia.org/wiki/Virtualization</a> however for purposes of this article you just need to understand Hyper-V and ESX virtualize the CPU, network, storage, and graphics using hardware support, emulation and binary patching. Hardware support means hardware vendors like Intel/AMD have added capabilities to their hardware that allows certain operations like page resolution or packet routing to a Virtual Machine (VM) to be done in hardware (hardware helps because it does not require expensive switches into the hypervisor). Emulation means the CPU/etc instructions to be executed are being emulated in software rather than running on real hardware (you might wonder how this can be faster - it can be when there is a lot of events in the hardware that keep engaging the hypervisors). Last there is binary patching where the original software being run in a VM is changed in some why - only ESX does this (binary patching is a useful technique because it allows for more direct control of what a piece of software - the virtualized operating system in this case - should do rather ahead of time rather than trying to determine it when an event happens). <br />
<br />
Now that you have the basics of vitualization you can begin to understand some of the questions to ask and you might also realize that there is a lot more than the Hyper-V and ESX bits that determine how performant your virtualized workloads will be.<br />
<br />
Really there is more than just Hyper-V and ESX to worry about? Yes. For example Hyper-V and ESX both support special instructions in the CPU's to support things like second-level address translation, etc however that does not mean the CPU you have supports that function. For example CPU's that are three years old likely do not support second level address translation which is a key feature to making VM's run fast expecially VM's running memory intensive operations. So the first question you should ask when looking to virtualize is "What machine should I buy and in particular what CPU does it have?". The simple answer is to look for a Intel Nehalem based processor such as the Xeon 5500+ or Core i7+ and for AMD look for recent Opteron or Phenom II processors. You can see more on Intel virtualization <a href="http://www.intel.com/technology/virtualization/technology.htm?wapkw=(virtualization)">here </a>and AMD virutalization <a href="http://sites.amd.com/us/business/it-solutions/virtualization/Pages/amd-v.aspx">here</a>. I can't stress enough how much the CPU virtualization features have an impact. Both Hyper-V and ESX make good use of the CPU features and none really has an advantage over the other. Choose your CPU with the workload you want to run in mind.<br />
<br />
This leads us to question #2 - What do the workloads look like that you are planning to run? The reason the workload matters so much is how Hyper-V and ESX virtualize networking and storage. In for example if you have a fully cached web server you want to run in a virtual machine its very likely that Hyper-V will run better because its networking virtualization is better although ESX is catching up. If however if you are running a database it may be ESX will be better because it has more support from big hardware vendors like EMC and NetApp to improve storage performance. As for 3D graphics I dont have a clear winner for you. If you are a Microsoft shop mostly you should go with Hyper-V because of the deep integration with Terminal Services. Your workloads is largely determined on how it uses the network, storage and graphics primarily. More use more a more intensive workload. For example databases are storage intensive, web servers are generally network intensive, and simulations like Weather modeling are CPU and Graphics intensive.<br />
<br />
So on to question #3 - What is your storage environment? The environment not only includes the host machine where the virtual machines will be running but also the storage infrastructure. For example will you be running a SAN or iSCSI storage network? If you want iSCSI to a VM then Hyper-V will likely be better because its networking performance is better overall however if you run a SAN than ESX might be the better choice. There are also other questions to ask around storage like LUN provisioning, snapshotting, and migration (moving storage between host machines). The deeper the level on integration of the solution the more performant it is likely to be. Hyper-v has great basic I/O performance however I've seen more integration of VMWare with storage solutions.<br />
<br />
Given we touched on storage we need to cover the importance of networking. It would be worth asking what virtualization networking features does your NIC support? Believe it or not Intel and Broadcom have both adopted certain features like <a href="http://en.wikipedia.org/wiki/VMQ">VMQ</a> (aka Netqueue), <a href="http://en.wikipedia.org/wiki/TCP/IP_offload_engine">TCP offload</a> (checksum and large send), <a href="http://en.wikipedia.org/wiki/Jumbo_frames">Jumbo Frames</a>, and <a href="http://en.wikipedia.org/wiki/Rdma">RDMA</a>. Hyper-V has traditionally been ahead here but there has been some leap frogging.<br />
<br />
Another important performance dimension is power so the question is what power management features does your virtualization solution support? This is important because lots of power use means lots of heat and lots of heat means lots of cost for cooling. Both Hyper-V and ESX have power management features however at the time of this article VMWare is a bit ahead on this front. Overall virtualization with either solution will save power because of true hardware to virtual machine consolidation. What I am taking about is once you have virtualized which solution will use less power per VM operation. They are both very competitive here.<br />
<br />
Last but not least what virtualization features do Hyper-V and VMWare support. For Hyper-V you can see <a href="http://www.microsoft.com/windowsserver2008/en/us/hyperv-features.aspx">here </a>and for ESX see <a href="http://www.vmware.com/files/pdf/key_features_vsphere.pdf">here</a>.<br />
<br />
So to recap the questions:<br />
<br />
#1 - What machine should I buy and in particular what CPU does it have? Intel <a href="http://www.intel.com/technology/virtualization/technology.htm?wapkw=(virtualization)">here</a> and AMD <a href="http://sites.amd.com/us/business/it-solutions/virtualization/Pages/amd-v.aspx">here</a><br />
#2 - What do the workloads look like that you are planning to run?<br />
#3 - What is your storage environment? Check out the <a href="http://www.netapp.com/us/technology/storage.html">NetApp</a> and <a href="http://www.emc.com/utilities/search.esp">EMC </a>sites.<br />
#4 - What virtualization networking features does your NIC support?<br />
#5 - What power management features does your virtualization solution support?<br />
<br />
There are many many more questions you could ask however the real purpose of this article was to help you understand that asking "Which is faster Windows Hyper-V or VMWare ESX?" is not such an easy question to answer and to arm you with some question you should ask.<br />
<br />
In the end my recommendation is to try before you buy. You can ask all the questions you want however in the end you need to make a decision. My suggestion is to borrow an environment if you can and try your workloads on it. Whichever is better for you go for it (PS.... dont forget the cost).<br />
<br />
Tony VoellmAnonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com3tag:blogger.com,1999:blog-2720487640092913306.post-59288757927081145572011-02-13T22:07:00.000-08:002011-02-13T22:20:27.853-08:00Welcome!Welcome to my new blogs where I'll be exploring a range of topics from performance to security of real systems along with the occasional topics like software testing and computer science in general. My previous blog http://blogs.msdn.com/tvoellm was dedicated to Microsoft technologies however this blog will have wider coverage. If there are questions you would like me to answer just send a message and I'll get to them as time allows.Anonymoushttp://www.blogger.com/profile/08059418838373215504noreply@blogger.com2