Monthly Archives: February 2008

Thoughts on Amazon S3 Outage and Cloud Computing

I am pretty sure that by now everyone has heard about today’s outage of Amazon Web Services, their S3 service in particular.

amazonweb I noticed there was a problem this morning when the avatars on Twitter were not showing up.  I know Twitter uses Amazon S3 to host the avatars but thought the issue was with Twitter because they constantly have problems keeping the service running.  I gave little thought that the problem might have been Amazon S3 service instead.

The coverage on the Internet spread like wildfire and the details and speculation can be found many places, including TechMeme which has lots of links and the list is growing.

I am not posting to report the Amazon outage (a million before me have already done so) but more to reflect on it and give some of my thoughts about using services like S3 and EC2 versus other ways of accomplishing the same tasks.  I don’t have the answers, just some thoughts and opinion.  I would like to hear from readers who may have more experience than I with this. 

I have been thinking lately about the best way to host my applications, either in part or full.  Right now I use a hosting provider, RailsPlayground, to host a couple Ruby on Rails applications.  I have a single VPS setup with them and have never had a problem, but I could.  The applications hosted there are small and if the VPS was down for a period of time I doubt it would matter much.  The pricing is right but there is no redundancy.

I have a couple additional applications I have been working on and the thought of hosting them is where my concerns are.  Hosting them could be done one of several ways:

  • Slices/VPS – Slicehost or RailsPlayground
  • Clustered Slices – Engine Yard
  • Dedicated Server – co-located and managed
  • Compute Cloud – Amazon EC2

All options have their strengths and weaknesses with regards to costs, scalability and reliability.   My biggest concern at this point is balancing reliability with costs.  We can easily through money at the problem if money was no object.  Even with virtually unlimited funds such as those Amazon has, they are obviously not excluded from downtime issues.

I don’t know the cause of Amazon’s downtime but do they have S3 or any of the other services running in a single data center?  Even if the data center is huge it still has a single point of failure.  It could also be their disaster recovery procedures were not tested well enough.  I think we will hear more and more coming from Amazon about the exact cause of the problem, any disaster recovery issues and a statement to the effect of why this will not happen again.

Relying on the Cloud

Can we rely on what we put in the cloud to be available?  How will we be effected by having our applications unavailable while there is some downtime?

I think I can characterize my use of the cloud by application level, which is probably like other users as well:

Level 1 Application – used daily but can live without, won’t really miss if gone.  An example of this type of application is Twitter.  Lately, this service has been down more than it has been up, but life goes on.

Level 2 Application – used daily but depending on the day, will be a productivity loss if down.  An example of this is Google Docs or GMail.

Level 3 Application – used daily and cannot be down without financial loses.  An example of this may be my own web applications I make a living on.

So how can I position my applications to give me the most uptime without throwing money at the problem unnecessarily?    I would think a single location with a single database is asking for problems.  The solution that stands out in my mind for a solution to my particular problem would be to used managed services hosting the applications on VPS slices on a cluster across multiple data centers, in different states or even continents.  I know Engine Yard does this but I don’t know if Amazon does this with EC2.  If they don’t, they should.  If EC2 was fully down today then Amazon needs to rethink their strategy a bit.

Service Level Agreements

Problems happen to the best of companies and we know uptime is not 100% guaranteed. 

Looking at part of Amazon’s Service Level Agreement (SLA), they admit they can’t ensure 100% uptime and will pay customers back for anything less than 99.9% uptime.

Service Commitment

AWS will use commercially reasonable efforts to make Amazon S3 available with a Monthly Uptime Percentage (defined below) of at least 99.9% during any monthly billing cycle (the “Service Commitment”). In the event Amazon S3 does not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.

SLAs are a fair statement as to the importance of uptime to the company but no real guarantee.  The company gives us their best and will refund anything less, which is more than fair.

Final Thoughts

Amazon’s S3 downtime today just really got me thinking how risky it could be for users to rely on the cloud for computing.  The concept is great but even large companies like Amazon with massive infrastructure are still prone to problems. 

It makes me think how I run my business and how I can minimize downtime.  I certainly can guarantee 100% that the services I host will never be down.  I can only do my best and have a recovery plan.

The same is true for those services like Twitter who takes its share of slams for not being up all the time.  Heck, it’s a free service and they have a huge number of users, they are growing and we will experience pain.  Amazon S3 has experienced its real issues today and they will get better.  We can only learn from mistakes and weaknesses, this will be a learning experience.  It has opened my eyes.

Technorati Tags: ,,,

links for 2008-02-15

Good Merb and DataMapper Tutorial from Atlantic Dominion Solutions

There’s no doubt about it, Ruby on Rails is hot, but not the only game in the Ruby web framework town.  Many Ruby developers are Ruby developers because of Ruby on Rails but these developers are missing out on a web framework just a year old, Merb.

Merb is a web framework created by Ezra Zygmuntowicz, founder of The Engine Yard which provides its own solution to building web applications based on their work with Rails and leveraging some of the downsides to build something better.   Rails uses ActiveRecord to access data but Merb lets the developer choose from others, including DataMapper.  DataMapper is an ORM increasing in popularity due to its simplicity.

I have found a blog over at Atlantic Dominion Solutions that provides some good information on using some of the latest coming from the Ruby world.  One such nugget was an article by Chris Kaukis titled Merborial: Getting Started with Merb and DataMapper.

The article covers:

  • Installing DataMapper
  • Installing Merb
  • Installing DataMapper drivers for Merb
  • Installing Merb Helpers
  • Creating a Merb application
  • Connecting up to DataMapper
  • Database setup and Configuration with Merb
  • Running the Merb application

This is one of those tutorials I had thought to create and blog about but ADS has done this already, so I just wanted to point it out and let you in on a great resource in the ADS blog.

Overview of Latest Leopard Update from Wired

Apple released Leopard update 10.5.2 a couple days ago which is intended to fix many of the common complaints with the operating system.  I ran the update as soon as I had heard it was available and it installed without problem.  I did hear a few people having problems but they seemed like isolated incidents and I am not aware of the details.

Wired Blog Network had an article titled Latest Leopard Update Resolves Common Complaints which highlights some of the major updates and biggest complaints.  I was going to write up something about the updates but since Scott Gilbertson already beat me too it, I figured I would just quote some of the details he gives.

When I ran the update on my MacBook Pro it weighed in a whopping 370MB, bigger than I expected.  The shear size alone should be a good indication at the number of fixes.  The release notes on this update from Apple are worth looking at for details on particular fixes, but the important ones from Wired:

  • AirPort – improves connection reliability and stability
  • Mail – numerous bug fixes and better handling of SMTP servers
  • RAW image support for several new cameras (Canon, Nikon and more)
  • Safari – patches a security flaw that affected only Leopard’s version of Safari 3
  • Third Party Apps – Improves general stability when running third-party applications.

Below are some additional links to some other sources of information on the update.

See Also:

I think this update is big in working toward fixing some of the items people complained about.  I have not had any real issues with Leopard since I installed it but my use is not 100% of my daily computing time.

Technorati Tags: , ,

links for 2008-02-14

What I learned from Charles Nutter about JRuby

JRubyLogo-140_63pxThe acts_as_conference has passed and was a great time to relax and learn some things related to Ruby.  One of talks I was looking forward to was the talk about JRuby from Charles Nutter.  I was aware of the JRuby project but never really paid close attention to it but with Mr. JRuby himself going to talk about it, I wanted to listen.

JRuby includes features such as:

    • A 1.8.6 compatible Ruby interpreter written in 100% pure Java
    • Most built-in Ruby classes provided
    • Support for interacting with and defining java classes from within ruby
    • Bean Scripting Framework (BSF) support
    • Distributed under a tri-license (CPL/GPL/LGPL)

    When I thought of JRuby I thought of Java and I am not a Java programmer, so JRuby must not be for me.  Charles said he heard this reason many times and he said:

    Not liking JRuby because it is written in Java is like not liking Ruby because it is written in C.

    It may seem obvious, but I never thought of it that way before.

    JRuby Performance Information

    One of the more interesting pieces of information Charles provided was that of the different versions of JRuby and how they related to

    • JRuby 1.0 was 2x slower than Ruby 1.8.6
    • JRuby 1.1 Beta1 2x faster than Ruby 1.8.6
    • JRuby trunk 2-5x faster and often faster than 1.9

    Performance improvements in trunk due to a JIT compiler implementation.  This looks like a huge boost for JRuby and really makes one think about running Ruby in this runtime over native Ruby 1.8.6.

    JRuby Performance Demonstration


    There has been a ton of performance-related bits of information posted on the web lately pertaining to Ruby vs. JRuby vs. other implementations.  These are sometimes hard to get your head around and relate to.  I anticipated Charles would simply gives us some performance numbers from previous tests but he instead run some live tests against different Ruby implementations he had on his laptop.

    Three different types of tests were run, live for us to witness:

    Method Dispatch (10 million invocations)

    • Ruby 1.8.6 ~2.1 secs
    • Ruby 1.9 ~0.95 secs
    • JRuby trunk 1.29 secs for first call and then 0.6 secs (JRuby trunk on Soy Latte)


    • Ruby 1.9 0.4 secs
    • Ruby 1.8.6 ~1.54 secs
    • JRuby ~.77 secs first, 0.4 secs subsequently

    Hilbert Matrix of Dimension 2, 4, … 64 times its inverse

    • Ruby 1.9 ~21secs
    • JRuby ~21secs

    These numbers are really encouraging and well beyond what I expected.   This information alone gives good reason for me to look deeper at using JRuby as a Ruby implementation I can use for some Rails applications I have running today. 

    Sun had a good presence at the conference and they seem committed to making Ruby successful on the Java JVM.  They have already converted some of their web sites to run Ruby on Rails on JRuby and they are looking for people to use JRuby and talk about it.  The NetBeans IDE was created by a group at Sun as well.  This is a great IDE for Ruby development and overall project development.

    The talk by Charles really made me more aware of JRuby and convinced me to try it out in conjunction with NetBeans as an IDE on my Windows or Mac machines.

    Technorati Tags: ,

    acts_as_conference is now history

    It’s Saturday night here in Orlando, FL and I just finished listening to Obie Fernandez give the final keynote of a busy two days of talks about Ruby and Ruby on Rails development.

    The conference was very good, it was hard to believe it was the first time the Rails for All folks put in a conference like this one. The event was sold out with about 150 developers from all over the U.S. I think a conference of this size is really nice where you can sit down with speakers, mingle and get to know many of the attendees.

    Meeting New People

    I met a lot of new people from all over the country including some people I only knew from blogs or podcasts.

    Great Speakers

    There were many great speakers at this event, none that I did not enjoy. It was really good to hear the latest about Merb from Ezra Zygmuntowicz, Rubinius from Evan Phoenix and JRuby from Charles Nutter. The presentation from Charles was probably my favorite technical talk. It could be because I learned a lot about JRuby, not knowing too much about it before today. I plan on trying out JRuby along with Netbeans. Sun was a great sponsor as well, having people manning the booth both days.

    I also enjoyed the talk from Charles Brian Quinn about how to approach training people. This was a talk from real experience and makes training more of a possibility for me to offer down the road.

    There was also a great talk on working with teams from Luke Fracl, who I was not familiar with prior to his talk. Luke presents very well and his style gives you confidence he knows his stuff.

    Two of my favorite talks were the two keynotes we had this weekend, the first from Dan Benjamin who talked about building and selling Cork’d. Dan talked about practical views of creating a business, running it, and all around suggestions for running your company. Obie Fernandez was the second keynote speaker, who is one of my favorite authors from his book to his blog. His presentation on practical aspects of being the best developer you can was truly inspiring advice that got me thinking about my own approach to my development work.


    Well, it was a great time! I am glad I spent the time to come down to warm Orlando from my home in CT, where it has been snowing for days.

    We had a great party sponsored by Rails Machine at the local ale pub and they gang really had their share of ale. The best part was the conversations we had and the friends made. I look forward to seeing some of these same speakers at RailsConf 2008 in Portland, OR May 29 – June 1, 2008.

    The conference was well organized and I thank all those involved in putting it on.  There was talk about having this event again next year.   I certainly hope so as I plan on attending.

    Technorati Tags: