FeedBlendr.com

I am currently looking for someone who is willing to acquire FeedBlendr and take over the operations of the service. It would be up to them whether they kept it publicly-accessible or took the technology and rolled it into part of something else.

Here’s some information if you’re interested in taking over (or just if you like numbers 🙂 )

• Runs from Amazon EC2, so if you’re willing to add additional instances to the pool, it can scale out.
• Currently almost 16,000 blends in the database.
• Those blends are compiled from over 65,000 source feeds.
• In February (short month), FeedBlendr pushed 150 Gig of data “in” and 200 Gig “out”.
• Approximately 50 new blends are created each day.
• According to Compete.com, FeedBlendr gets more than a couple thousand hits a day and growing.
• The site actually sees a couple hundred page impressions per day — the vast majority of requests are for feeds/blends from the system (that’s the point)
• The (MySQL) database is hosted externally (on DreamHost!) but you could move it somewhere else (and I’d recommend that you do)
• DNS is managed from DynDNS, but you could transfer it to anywhere you prefer.

Whoever acquires FeedBlendr will get the following:

1. The domain “feedblendr.com”,
2. All code (PHP) that powers the site including the app itself, WordPress templates, site UI,
3. The database behind the service (including this WordPress-powered blog)
4. Amazon AMI used to host the site (to be moved to your own S3 bucket and changed over to using your credentials),
5. My assistance transferring everything over to your servers/accounts/control,
6. Gratitude from the people using the service (and from me).

If you’re interested, please email beau@feedville.com directly.

It’s been a couple of months now that FeedBlendr has been hosted with Amazon EC2, and some people have shown interest in hearing more about that experience, so I thought I’d follow up with some observations about EC2 in general and my experiences/configuration in particular.

Here goes:

1. Dynamic DNS is a SLOW way of faking load balancing. It’s reasonably functional from a normal management perspective, but an emergency situation would not be pretty. Setting your TTLs down to 300 should mean that changes happen pretty quickly, but 48 hours after making changes and removing an instance from my DNS records, I still see requests coming direct to that instance. Amazon, please offer us internal load balancing between instances somehow!
2. Disposable Instances are something you need to get used to. I’ve already had one instance get itself into trouble because of a “degraded ephemeral data store” (according to an email I received from Amazon). If you’re in the habit of making instances completely disposable then you can just launch a new one and terminate the one with problems.
3. Instance Cycling is something I’m starting to believe in – periodically just starting new, clean instances, and moving your operations over to them, then shutting down the previous ones. This is also the case with a new code release for me. Rather than upgrading the code on my instances that are already running, I slide onto new instances via DNS changes.
4. Alternate AMIs are something I need to deal with, so that I can have a new AMI being bundled and tested, without throwing away my previous one. If there’s a problem with the new one, how do you roll back otherwise? This is something I need to figure out in my own deployment process.
5. Inconsistent Performance seems to be quite common on instances. I wasn’t expecting this one, but if you think about it, it probably makes sense. I can launch identical instances, and load them equally in DNS, but they will perform quite differently. I believe this is probably due to resource sharing/exhaustion on specific instances due to other users’ instances on the same physical machine within EC2. This is something Amazon probably needs to look into.
6. Slow External Connections are not something you want to be a required part of your core system. Currently my database is hosted outside the cloud (on DreamHost) and this seems to be a bit of a bottle-neck in the system. Also, I pay for the bandwidth generated by those communications which is a cost I could avoid by hosting the database within the cloud.
7. Pricing on EC2 is something interesting to consider. Right now, I run 2 instances, 24/7, which means I’m up for $144 per month plus bandwidth, so let’s just call it $200 pm for a self-managed service. Is that actually worth it compared to getting a dedicated machine somewhere else? More thoughts on this below.

EC2 vs. “Traditional Dedicated Host”

Potentially “FOR” EC2

• You can easily launch as many instances as you need, rather than just having one server to work with
• Setting up unique AMIs is relatively easy, so you can scale any part of your application
• Pay-as-you-drink means you’re not paying for infrastructure you’re not using
• No up-front/set-up costs
• Scalability through commodity computing
• More choice over operating system, because it’s entirely up to you

Potentially “AGAINST” EC2

• Because you are technically sharing hardware, you’re subject to resource-starvation/sharing with other users!
• No control over your network configuration (e.g. no availability of hardware load balancers, choice to put instances on the same switch etc, you get what you’re given and that’s it)
• No choices for custom hardware, so you can’t get better hard drives, more RAM etc
• You need to learn how to build an AMI and work with the Amazon tools for bundling and launching instances etc (not that it’s hard)
• You have to use Amazon S3 to store your AMIs in
• There is no bundled bandwidth with the service, so you only pay for what you use, but you pay for all of it
• No control panels, monitoring, reporting or statistics or anything like that provided (unless you install them yourself) other than very basic bandwidth numbers and “instance-hours” reporting
• No SLA at all, so you have no guarantees on uptime

Anyone else have any other ideas/additions to those lists?

I’ll explain in detail once I’m moved, but if you’re having problems with FeedBlendr, it’s probably because I’m in the process of moving servers (for real this time).

UPDATE 2007-05-13: As promised, here’s some more information about this Amazon business.

After running FeedBlendr on DreamHost for a year and bit, and having a few problems along the way with causing too much load, they finally pulled the pin on me. They’ve been very good about things, and worked with me to help identify and solve the problems I was causing (I was on a shared server, so my usage was affecting other customers). Basically, in the end there wasn’t a “fix” per se, because I just had too many people requesting blends too many times a day (over a MILLION times a month!), and had to do something about it.

In comes Amazon. For those of you who don’t know, Amazon has started getting into the Web Services world, and 2 of their offerings are of particular interest to me (us!):

1. Elastic Compute Cloud (EC2): A system whereby I can request a new “copy” of a complete server on-demand, and use it as part of a cluster of machines to power FeedBlendr.
2. Simple Storage Solution (S3) Unlimited, fully-redundant (in the good way) online storage, allowing me to keep copies of things out there in “Amazon-space” where my new EC2 servers can get at it.

I won’t bore you with all the details (although feel free to get in touch if you’re interested), but FeedBlendr is now running on 2 Amazon EC2 “instances” in a balanced manner (requests go to both machines), so hopefully performance is a lot better, and things will be more reliable. You may also have noticed that I fixed some caching bugs, so blends load faster, and should be more stable. I’ve also bumped up the minimum age for blends slightly, so you may notice now that blends can get slightly older before they will get refreshed. This is mainly because people were just requesting their blends too often, causing my servers to have to rebuild a complete blend every 5 minutes, just because a single feed changed. I’ll be looking at better ways to address this, but in the mean time, please let me know if this causes any problems.

Here’s to looking forward, and seeing FeedBlendr continue to improve and serve your feed-blending needs better!

With increasing popularity come a number of problems that are only to be expected. Right now, FeedBlendr is experiencing one of those problems – scalability/load issues.

After a number of warnings/notifications from DreamHost, I’ve been asked to either figure out a way to lower my processor usage immediately, or risk having my account disabled until I can do so. I’m using too much of the processor on my shared server and it’s unfair to the other users – fair enough.

Now I’m faced with a decision: do I shut FeedBlendr down (I don’t want to)? is there something I can do to lower usage (not that I’ve found yet)? can I justify upgrading my hosting (without making any money from FeedBlendr under the current model)?
At the moment I’m considering upgrading to a dedicated server from Hosting.com, which should give me the power/flexibility I need, but it’s a lot of extra management/set up etc that I’d have to deal with just to get it all happening, and life is just plain busy right now. So as a stop-gap, and either way, I’ve decided to open up for donations, allowing anyone who would like to do so to show their appreciation for the service FeedBlendr offers, by dropping a few bucks in my virtual tip-jar.

This may well affect my decision on whether or not to keep FeedBlendr live – if people aren’t even willing to drop a few dollars once-off in appreciation of the service, then I don’t know if I can continue providing it, and pay extra for the privelege of keeping FeedBlendr public and free.

It’s up to you people – so please show how you feel about FeedBlendr and donate now!

FeedBlendr has been experiencing a bit of popularity growth, and so I’m handling more feeds/blends now. This has started raising some eyebrows over at DreamHost (where this site is hosted), and fair enough – looks like I’m using a lot more processor time than is fair on a shared server.

I’m making some modifications to ensure that caching is working locally every time, and trying to avoid remote calls when possible because they seem to be causing the problem, so please bear with me in the meantime if there are any problems. If you’d like to do your part, then please reassess how often you’re requesting your blend – you really don’t need to be hitting them every minute (which it appears some users are doing!). In the future I may have to throttle certain blends if they are requested too often from the same IP and just block some requests.

On a brighter note, if you love FeedBlendr and want to support its future development (bandwidth costs, development time etc), then please jump over here and Make A Donation!

Please let me know via comments or email if something is broken from the changes that I’ve made, or the ones I make in the near future.

UPDATE 2006-04-12: Previous changes didn’thave enough of an effect, so I’ve made some more caching changes and will wait to see what effect they have on my processor load problems.