AJAX performance stats, ROI, and business value

How do you build a system that can deliver and update content to 100,000 people simultaneously? Via Ajaxian.com I saw this article from MacRumors on what traffic they got when Steve Jobs delivered his keynote on MacWorld a few days ago.

On January 10th, 2006, MacRumors.com successfully delivered live updates of the Macworld San Francisco Keynote speech to over 100,000 people simultaneously using the latest web technologies.

“By far the best mac keynote coverage I have ever seen. No constant page refreshing. No ‘page will update every three minutes’ even though they never do. I just let the page update every minute like it was supposed to. It was actually weird to get coverage without some sort of connection failure.” – MacRumors member, war

The live coverage web solution is basically a transscript updated frequently during the keynote. One article is about the traffic they experienced:

Our site was linked to widely across the web, including prominent links from Digg and Slashdot — two popular technology sites. These links caused no noticable “spike” in our number of visitors, illustrating the magnitude of traffic that MacRumorsLive was already receiving.

We peaked at approximately 103,000 simultaneous web visitors and 6,000 IRC viewers during the Keynote speech and transmited over 32 GB of data in a three hour period. If not for the efficiency of the MacRumorsLive AJAX update system, the same webcast would have required approximately twice as many servers and would have had to transfer almost 6 times as much data (196 GB).

With peak loads that reaches insane levels of almost 4,000 hits per second:

Hits/second (2) 3,812
IRC users 6,079
Simultaneous visitors 109,000
Bandwidth (3) approx. 77Mbit/sec
MacWorld keynote hits graph (from macrumors.com)
Hits per second from ~8am to ~12:30pm. Average over each minute.

They also posted an article about how they built the system which is insanely interesting

Traditionally, websites have coped with visitors during the keynotes by swapping to text-only pages, using HTML Meta Refresh to reload the page at set intervals, usually every three minutes, for example, MacRumors’ coverage from MacWorld in January 2005. As a result, the sites lose their identity and branding, and cannot make best use of the additional traffic for site promotion.

Even with all this simplification the sheer number of visitors can still take the server down. To provide the best coverage during live events an improved system is required.

As this system is retaining the normal layout of the site, all the necessary objects (CSS, images & Javascript) also need to be loaded. This increases the load placed on the web servers considerably in comparison to the old system, which only had a single text file to display. Even with the ability to use multiple servers, the web server software needs to be improved.

By using PHP to generate the necessary HTML & XML files every ten seconds, rather than each time the file is requested, it is possible to use small & simple web servers to deliver the public site, without the overhead of database usage. Two web server software packages are employed – lighttpd and thttpd – with Apache still being used for the administration interface.

The efficiency of these packages over Apache is considerable and allows quite modest hardware to be used to serve a large number of visitors. Benchmarks of lighttpd compared to various other packages can be found on their website. In our tests, thttpd was able to sustain in excess of 3,300 hits per second, whereas Apache managed only 600 hits per second for 20 seconds, before overloading the server due to memory usage. [1]

The use of these more specialised web servers allows a vast number of simultaneous visitors to be supported – as the system stands at the moment, in excess of 200,000.

This reminds me of Ajaxinfo (the guys behind AJAX usability metrics), AJAX ROI faceoff, where a traditional webapp is compared to an AJAX webapp. For the visually oriented, there is a video comparing the two applications at the end of the article.

Recently on Developer.com, I looked at ways for developers to communicate the value of Ajax in dollars and cents. Arguing for or against a technology usually creates some stress for developers, who tend to speak a different language from business decision makers.

Developers talk about

  • Usability
  • Bandwidth
  • Ramp-up
  • Cost of Ownership
Managers talk about

  • Efficiency
  • Transaction costs
  • Training costs
  • Return on Investment (ROI)

Right at the point. Performance website or not, the metrics he mentions above are almost like something Jakob Nielsen could have done. Conclusions:

Although every new technology should be greeted with a healthy amount of skepticism, there are clearly demonstrable, quantifiable advantages to using an Ajax architecture in a Web application. These cost savings originate primarily from time savings, but also from reductions in bandwidth requirements.

A representative test case showed that a business can save between 500 and 2,800 man hours per year on a 10-step hypothetical process, saving roughly 4 seconds per step (a between 30% and 70% reduction in labor costs).

Although the benefits of improved application architecture extend beyond mere time savings, when included in the decision making process, an ROI approach such as this can help make a solid business case for Ajax.

Total time taken (AJAX vs traditional webapp)Total bytes transferred (AJAX vs traditional webapp)

The benefits listed here are very similar to benefits when we’re arguing for web-standards: Reduced bandwith, training costs, cost of ownership, and increased return on investment. But for the time being I still believe that AJAX has increased learning costs for developers. (Although most developers are really keen on using AJAX and webstandards it still requires expert knowledge regarding libraries, backend integration and browser compatibility — to name a few).
I’d be happy to hear your comments on this. Please share more info here. Also check out Ajaxinfo’s Call for AJAX Studies.

UPDATE: Just saw Jonathan Boutelle’s Bandwidth Savings with AJAX blog post, that covers the same story and has a good point on AJAX fit frequently used applications:

The old-school RIA technology vendors always made the claim that using the RIA paradigm saves you bandwidth. Numbers on this were always hard to get, and hard to trust.

Theoretically it always made sense that an RIA would consume less bandwidth (in exchange for a larger initial download). The longer a user continues to use the application, the greater this benefit (so an RIA word processor might make more sense than an RIA news site).

Now, watching a keynote presentation takes at least an hour, so it’s definitely long enough to provide benefit. What we see in this case is really different though: AJAX is basically being used as a broadcast technique, grabbing new data every minute so that the user doesn’t have to refresh the entire page. For covering live events that have a large audience, this makes a ton of sense. Expect to see more content sites “pushing” content out to their users this way.

Technorati Tags: , , , , , ,

21 Responses to “AJAX performance stats, ROI, and business value”

  1. Andre Charland Says:

    Great post Jesper! I’m glad to see the conversation around performance and usability metrics is evolving and going deeper! I think things will get exciting this year as more main stream developers and managers start grasping these concepts.

    I’d like to offer both you and Thomas free copies of our AJAX components and betas as our suite evolves in 2006. Please email me acharland [at] ebusinessapps [dot] com and I’ll get them over to you.

    Keep up the great blog and be sure to let us know about your case studies!

    Andre.

  2. Jesper Rønn-Jensen Says:

    Andre,
    thanks for the flattering words. I’ll send you a mail with contact details. Thomas and I would love to check out your suite (but of course we cannot guarantee to use it).

    I really hope your foresight comes true regarding more AJAX and usability metrics. Presently, I think that the material your company posted is very good and can probably serve as a template for anybody that want to document performance of AJAX applications.

    So back to you: YOU keep up the good work!

  3. WebWord » Blog Archive » AJAX performance stats, ROI, and business value Says:

    […] Is Ajax a smart move? (Source: JustAddWater.dk, Authos: Jesper Rønn-Jensen) — The benefits listed here are very similar to benefits when we’re arguing for web-standards: Reduced bandwith, training costs, cost of ownership, and increased return on investment. But for the time being I still believe that AJAX has increased learning costs for developers.” […]

  4. Jesper Rønn-Jensen Says:

    Just to clarify my opinion: I still AJAX is a smart move. There are significant productivity benefits (as documented) when using an AJAX application.

    My point is that the development costs may be increased because of learning costs and that this extra cost should not be ignored.

  5. Jesper Rønn-Jensen Says:

    For the techies, there are some good hands-on tips from Alexander Kirk:
    10 Realistic Steps to a Faster Website
    http://alexander.kirk.at/2006/02/02/10-steps-to-a-faster-web-site/

  6. justaddwater.dk | AJAX businesscase: Reduce development costs and increase usability Says:

    […] As a followup to my earlier post “AJAX performance stats, ROI, and business value“, I decided that I’d share with you some considerations on a recent project I was involved in. I can’t give you all the juicy details, and I might never be able to show you the final solution or tell, who the customer is. […]

  7. Ajaxian » Ajax Case Study - Simple Validations Reduce Costs/Increase Usability Says:

    […] In a follow-up to his previous post, Jesper Rønn-Jensen looks yet again at business uses of Ajax technologies – specifically a case study looking at one of his recent usability issues. […]

  8. kai Says:

    Thanks for giving such a detailed view on that matter.
    It is always nice to have that kind of information in one place and your blog post provides a nive overview of the possibilites that lie ahead.

    I especially liked the fact that this high-performance system also stretched to the design of the webserver platform. I guess that AJAX saves enough bandwidth to increase the speed and application reaction for smaller sites and AJAX is a big step in the right direction. But it seems to me that also the combination of lighttpd and thttpd did a lot for the user experience in this scenario. And it’s nice that you take it all into account.

    So Jonathan is definitely right in pointing out that “For covering live events that have a large audience, this makes a ton of sense”, but that this goes even beyond AJAX…

    Just my 2cents

  9. ntschutta.com Says:

    The Upside of Ajax

    Near as I can tell, none of the major Ajax sites have referenced Tim Bray’s piece entitled The Real AJAX Upside. Tim hits on some of my company’s reasons for adding Ajax to our applications – it makes them perform better (or, more precise…

  10. justaddwater.dk | Happy Birthday Justaddwater :) Says:

    […] AJAX performance stats, ROI, and business value […]

  11. amp Says:

    We came across serious performance problem in some of our projects. After reading this information we made design to rewrite some of functions with help of AJAX. Intermediate results show that “bottle-neck” almost disappear. Thanks a lot!

  12. Sanjeev Says:

    The Microsoft AJAX Library 3.5 is a standalone collection of the standards-based JavaScript classes that are automatically included when using ASP.NET AJAX in ASP.NET 3.5 SP1. It is supported by the most popular browsers and can be used to build client-centric Web applications without using ASP.NET.

  13. Just Pens Says:

    Wow. That is some traffic that was received there. Most servers would bottom out with that amount of load in such a short space of time. I think the technology that is available is improving drastically.

    I still see mixed opinions whether the internet is getting over stretched or not. Some say it’s all speculation whilst other experts say there’s a real possibility that the internet could bottom out due to excessive load.

  14. Tech Blog Says:

    I had no idea that Ajax could produce such a reduction in bandwidth – gonna have to learn more about it for sure.

  15. Toni Says:

    Hey Jesper!
    Thanks for this words. I never relized that it could produce such a reduction, too. Very interesting thou.

  16. namaste tshirt Says:

    I think everyone should go to college and get a degree and then spend six months as a bartender and six months as a cabdriver. Then they would really be educated.

  17. Trend Scout Says:

    Thanks for sharing this!
    I bookmarked this page because your post is knowledgeable.

  18. press release script Says:

    Can you provide more information on this? and where i can find rssof your blog?

  19. locksmith Says:

    Reduction on bandwidth — AJAX. Can’t believe it.

  20. Donnie Says:

    Nice Info Jesper!
    Yes, I believe it’s totallly insane with almost 4,000 hits/second and still in the good performance ..
    seems this AJAX suits for many dev and mgrs.

  21. adiw Says:

    I especially liked the fact that this high-performance system also stretched to the design of the webserver platform. I guess that AJAX saves enough bandwidth to increase the speed and application reaction for smaller sites and AJAX is a big step in the right direction. But it seems to me that also the combination of lighttpd and thttpd did a lot for the user experience in this scenario. And it’s nice that you take it all into account.