Blendr Blog


Keepin' your feed blending real.

Ongoing Amazon EC2 Observations

It’s been a couple of months now that FeedBlendr has been hosted with Amazon EC2, and some people have shown interest in hearing more about that experience, so I thought I’d follow up with some observations about EC2 in general and my experiences/configuration in particular.

Here goes:

  1. Dynamic DNS is a SLOW way of faking load balancing. It’s reasonably functional from a normal management perspective, but an emergency situation would not be pretty. Setting your TTLs down to 300 should mean that changes happen pretty quickly, but 48 hours after making changes and removing an instance from my DNS records, I still see requests coming direct to that instance. Amazon, please offer us internal load balancing between instances somehow!
  2. Disposable Instances are something you need to get used to. I’ve already had one instance get itself into trouble because of a “degraded ephemeral data store” (according to an email I received from Amazon). If you’re in the habit of making instances completely disposable then you can just launch a new one and terminate the one with problems.
  3. Instance Cycling is something I’m starting to believe in – periodically just starting new, clean instances, and moving your operations over to them, then shutting down the previous ones. This is also the case with a new code release for me. Rather than upgrading the code on my instances that are already running, I slide onto new instances via DNS changes.
  4. Alternate AMIs are something I need to deal with, so that I can have a new AMI being bundled and tested, without throwing away my previous one. If there’s a problem with the new one, how do you roll back otherwise? This is something I need to figure out in my own deployment process.
  5. Inconsistent Performance seems to be quite common on instances. I wasn’t expecting this one, but if you think about it, it probably makes sense. I can launch identical instances, and load them equally in DNS, but they will perform quite differently. I believe this is probably due to resource sharing/exhaustion on specific instances due to other users’ instances on the same physical machine within EC2. This is something Amazon probably needs to look into.
  6. Slow External Connections are not something you want to be a required part of your core system. Currently my database is hosted outside the cloud (on DreamHost) and this seems to be a bit of a bottle-neck in the system. Also, I pay for the bandwidth generated by those communications which is a cost I could avoid by hosting the database within the cloud.
  7. Pricing on EC2 is something interesting to consider. Right now, I run 2 instances, 24/7, which means I’m up for $144 per month plus bandwidth, so let’s just call it $200 pm for a self-managed service. Is that actually worth it compared to getting a dedicated machine somewhere else? More thoughts on this below.

EC2 vs. “Traditional Dedicated Host”

Potentially “FOR” EC2

  • You can easily launch as many instances as you need, rather than just having one server to work with
  • Setting up unique AMIs is relatively easy, so you can scale any part of your application
  • Pay-as-you-drink means you’re not paying for infrastructure you’re not using
  • No up-front/set-up costs
  • Scalability through commodity computing
  • More choice over operating system, because it’s entirely up to you

Potentially “AGAINST” EC2

  • Because you are technically sharing hardware, you’re subject to resource-starvation/sharing with other users!
  • No control over your network configuration (e.g. no availability of hardware load balancers, choice to put instances on the same switch etc, you get what you’re given and that’s it)
  • No choices for custom hardware, so you can’t get better hard drives, more RAM etc
  • You need to learn how to build an AMI and work with the Amazon tools for bundling and launching instances etc (not that it’s hard)
  • You have to use Amazon S3 to store your AMIs in
  • There is no bundled bandwidth with the service, so you only pay for what you use, but you pay for all of it
  • No control panels, monitoring, reporting or statistics or anything like that provided (unless you install them yourself) other than very basic bandwidth numbers and “instance-hours” reporting
  • No SLA at all, so you have no guarantees on uptime

Anyone else have any other ideas/additions to those lists?

6 Responses to “Ongoing Amazon EC2 Observations”

  1. khang Says:

    hey, man. thanks for sharing your experiences. great stuff.

    given the lack of an SLA, what’s your take on the “reliability” of EC2? what kinda things/precautions do you put in place to live with any reliability problems?

    i just started my site about a week ago on one instance and things were going well until today. i woke up to the first hiccup with my instance: for some reason, it decided to reboot itself, which freaked me out because i hadn’t yet bound my changes to an AMI.

  2. fthiery Says:

    You should check rpath Linux (http://www.rpath.com/corp/) which offers rollback functionnality and automatic AMI generation.

    Cheers,

  3. beau Says:

    khang: As you mention, not having an SLA of any kind is a concern for reliability. I’m working on some “external” monitoring for instances, because basically I think the assumptions here need to be 1. You’re on your own for keeping track of connectivity, 2. Things will fail. Another option for all of this is that you could look into something like WeoCEO which provides all sorts of monitoring/automated scaling features if that’s pricing-appropriate for you.

    fthiery: Thanks for the pointer – someone else actually mentioned rPath to me yesterday, so I’ll have to check them out.

  4. Bert Armijo Says:

    Thanks for sharing your experiences. Service like EC2 and ours at 3tera are opening up new avenues for developers and having users like yourself that will share openly as you’ve done helps everyone to build better services.

  5. Stelios Says:

    SLA is there now. So one prob is going out of this list.

  6. Gil Megidish Says:

    Four more reasons AGAINST hosting from Amazon EC2:

    1. No fixed IP (yet). This is too scary for me. Everybody serving long enough had their servers rebooted. Now think what happens when you need to modify dns records and wait 24-48 until all dns servers update.

    2. No phone number or direct email. Just grab a look at the forums. It’s really sad. People list their ami id, and instance-id and ask amazon guys to please-please-please check it now. I wouldn’t mind to pay extra $$$ for a phone support.

    3. Virtual environment is not suitable for everybody. I spent $250 for EC2 uptime (several servers) while moving one of my website into a completely load-balanced and distributed environment. Of course, this doesn’t include the countless hours I spent rewriting code. At the end of everything, when I launch it, it blew in my face. I’m using a custom server which uses unix pipes to send data between two processes. Hmf! EC2’s Xen is capped at like 10,000 context-switches a sec! 🙁 So stuff that used to take 0.5 sec on dedicated, now take 15 sec (which is considered timeout on my javascript side). I really get a nice distributed environment, but at 1/10 of the processing capacity 🙂

    4. Compare prices. You can get far better deals on vps from other companies. Amazon has a minimum price of $72 (0.10 * 24 * 30) per month, without single use of bandwidth. Companies offer large traffic packages bundled in price. 500gb worth of IN traffic is worth a lot of extra money. For that price you can already purchase a dedicated server at godaddy (which has really good 24/7 phone support.)

    But that’s just my opinion. :o)
    – gil