Dealing with dumb programs (or Why our site has been sluggish recently)

Some people have noted the recent slowness of the Openomy web site. To be clear, the API has been fine, as have been downloading files, but the web site has been extremely sluggish. The reason is we've essentially been DDOS'd by a dumb program. I think we've begun to get it under control, but I wanted to shed some light on what happened and how we've controlled it. We're no experts in this field, so I'd love to hear about what we did right/wrong or could improve upon in these situations.

A while ago, we noticed a not-so-clever program which basically said:

while (true) {
  get_public_file_from_openomy();
  ...
}

They didn't do this to be evil (we know because we were able to find the application and see what it was doing). We were unable to get them to fix it for a variety of reasons.

Over time, more and more people downloaded this application, making it effectively a DDOS attack on Openomy. The funny thing is, because it's just a user's public file and therefore subject to bandwidth limitations, it's virtually useless 99% of the time. We've tried a variety of strategies to deal with this file, so I figured we'd talk about them.

Step 1: Use .htaccess to block some IPs
At the beginning, just a few people were using the program. The IPs remained relatively static, so we would just add the IP to an .htaccess file which we then blocked. This was an extremely simple solution that we hoped would stop the proliferation of the program and make things back off.

Unfortunately, it didn't. People continued to download it, and the list of IPs grew. Adding IPs to the .htaccess file really only helped us for a week or so.

Step 2: 404ing the file
Since the file is virtually useless (due to the bandwidth overages) anyways, we decided we would just make the URL 404. We could do this at the mod_rewrite level and then no further processing would need to be done.

This actually lasted for a couple months, until the increased load once again caused the site to perform sluggishly.

Step 3: Stopping KeepAlives
One thing we noticed about 404ing the file was that although the requests weren't taking long to process, many of our available sockets were being taken up for long periods of time by these requests due to HTTP Keep-Alives. So, we turned off all KeepAlives for the site. This could have hurt some performance for our other pages, but because our site is so lightweight, the net gain was actually very large. We were once again able to process many more connections.

This, too, helped us for another couple months. That is, up until these past few days.

Step 4: (Now testing) iptables
The amount of clients requesting this bogus, unavailable file is again becoming too many for our web servers to handle. Moreover, the clients are all accessing the file many times. However, it does seem as though many of the IPs stick around for a few days, which is nice.

So, we've decided to run a periodic cron job which parses our log files for the past week, find the IP addresses that have requested our non-existant file (with some threshold), and create an iptable filter to drop all requests from said IP addresses. This obviously won't cover everyone requesting this file, but hopefully it will keep the amount of requests low enough that our servers can continue to serve requests to others with good performance.

This works better than the .htaccess file because our web server doesn't even begin to touch these requests.

I ended up writing a ruby script to perform this for us, but I started out with just some simple shell-foo. An example of how to find all the IPs accessing a certain log file would be like:

grep pattern filename.log | cut -d' ' -f1 | sort | uniq -c

The site seems to be running pretty well right now, but we've only had this out for a few hours. We'll need to watch this over time to see how it continues to perform and scale. One issue we currently see is a large amount of connections in the TIME_WAIT state. I'm unsure how this will affect us, but we'll be keeping an eye on it.


Hopefully this gives some insight into what we're facing, helps some people who face this problem in the future, and perhaps we'll learn something about how to make this even better going forward!

Openomy In The Press

Since we've started, we've never done any real PR work attempting to get press mentions. Luckily, we've been fortunate enough to be featured in some of the best news sources, including blogs, online news sites, magazines, and being able to participate in conferences.

I thought I'd take a second to appreciate all the nice words we've received, and point out some nice press we've recently received.

Web Worker Daily
On March 5, Mike Gunderloy of Web Worker Daily said in a post entitled Openomy: Going Beyond Free Storage:

...Openomy has a different strategy: they’ll give you a gigabyte of storage and a way for you to build your own features.

Specifically, Openomy has implemented a REST-based API with (at the moment) Ruby language bindings. Anyone can write an application that gets authenticated access to files stored on Openomy, then perform operations like uploading, downloading, and tagging. If you’re a developer looking to add online storage to your own applications, this gives you a potential way to do that without running your own disk farm.

PC World
And, on March 23, PC World included us in their list of 101 Fantastic Freebies, as one of their top File Sharing and Storage sites. The list should be in the May, 2008 issue of the print version of PC World. We're very humbled and thankful for this distinction and thank everyone involved.

Thanks for all the kind words, folks! We hope we can exceed the high expectations set by these articles.

Openomy: Now with unlimited storage and even more control over your data (via your own S3 account)!

Today is another big day here at Openomy: You can now use your own Amazon S3 account within Openomy! This is important for many reasons: you now have unlimited storage (just pay Amazon for what you use), full control over where your files are stored, and the ability to manage your S3 account through the Openomy interface.

How It Works

An example: Every day, I back up some files to S3 using the S3Fox Firefox extension. I use my Openomy account for storing files, too.

All I have to do is set up my S3 account with Openomy and now I'll be able to upload and download even more files through Openomy -- the 1gb storage limit is removed when I use my own S3 account. Every file I upload will be stored in my own S3 account. My older Openomy files won't change and will be available just as they were before.

Syncing
Moreover, anything that I upload to my S3 account outside of Openomy can be imported into my Openomy account with a single click. For example, if I add a file through S3Fox, I can Sync my account, making it available via Openomy.

Likewise, if I delete a file from S3Fox, I can Sync and it will no longer be shown within Openomy.

Setup

To setup, just click the Your Account link at the top right of every page, and fill in your AWS credentials, including the bucket you'd like to use. That's it! All new files will be uploaded to that bucket.

If you'd like to Sync, just click the Sync button. Any differences between what Openomy knows about and S3 contains will be resolved. You may Sync your account at any time, but beware that Syncing may take a few minutes.

Caution!

This is a Beta feature, and we are making it publicly available for testing. We are using this feature internally already, and are confident in the quality. That said, you should be aware of a few things:

• Once you begin using S3, you can not stop. We plan on adding this ability shortly, but for now, there's no turning back.


• If you delete a file from Openomy that is stored in your S3 account, it is also deleted from S3. Make sure this is what you actually want!

Feedback

As always, we love to hear feedback about our features, so please do send me email. Comments, suggestions, bug reports, and flames are all welcome!

Expected Downtime: 2/5/08 at 6:30AM PST

Tomorrow morning, at 6:30AM PST (9:30AM EST), we are taking a planned outage for approximately 2 hours.

Recently, our colocation provider was acquired by Your Net Connection who has to reassign IP addresses. On Wednesday, our old IP addresses will cease to work. Therefore, tomorrow is the drop-dead day to get switched over.

Realistically, we expect this to take about an hour. However, I'm notifying you that it will take 2 hours in case of an unexpected hiccup.

During this, we'll also be updating our DNS records. DNS typically propagates quickly these days, but it could take a little while for all users to see these changes.

Regardless of how long it takes to switch our IPs and DNS, your files will be safely and reliably stored thanks to our friendly partners at Amazon S3.

I'll be updating our Twitter account throughout the outage, so if you're unable to access the site after 8:30AM PST, please check Twitter to determine whether we should be available yet. If you still have trouble, please email me personally at iseff@iseff.com.

Thanks for your patience and understanding!

Ruby library v0.6: Updated for APIv2.0

Today I checked-in and released the latest version of our Ruby library. This version of the library is compatible with our new APIv2.0 (recently launched).

It's not quite a drop-in replacement for the previous version, with method name changes and some other things, but it should be fairly easy to get going.

You can get the new library at our Openomy SourceForge project page. You can also check out our general Openomy Open Source page.

Here's a little help to get started with the library:

Openomy::OpenomyAPI.setKeys("abcdesasdf", "1234567890")

a = Openomy::Auth::new()
u = Openomy::Objects::User::new
u.username = "foo"
u.password = "bar"
conf_token = a.create_confirmed_token(u)

# create a new tag
t = Openomy::Tags::new(conf_token)
new_tag = Openomy::Objects::Tag.new
new_tag.name = "New Tag"

resp = t.create(new_tag)

# get all tags
all = t.find(:all)

# get tag id 1
tag = t.find(1)

With that, the rest should be fairly straightforward to figure out. Enjoy!

Case Study: Using Haskell and HAppS for Openomy API v2.0

At Openomy we are continuously trying out new technologies and, in particular, programming languages. We've been pretty successful thus far (with some perl background scripts, a couple Ruby on Rails services, a C#/mod_mono frontend, a Java service, etc) and continue to innovate.

The latest release of our API is written in Haskell using the new HAppS framework. This turned out to be a really interesting project and one that fundamentally changed how I think about programming. I wanted to highlight some of the reasons we chose Haskell, what worked, and what didn't.

Why Haskell/HAppS

We initially chose Haskell/HAppS for a few reasons. First, we were getting a little tired of the same old OOP languages, and wanted an entirely different way of thinking. We're geeks and we like to try new things, especially those which make us think. Haskell offers that with functional programming. But, more than just being a "cool" language, it did have advantages that made it the right tool for the job.

Our backend is a collection of REST-based internal services. For example, we have a "files service" which deals with the uploading/downloading/etc of actual binary file data. We have a "metadata service" which deals with the metadata about a file (name, size, tags, etc). We have a "user service" which manages user accounts, sessions, etc. Each of these is fully independent and interact with each other through normal REST service calls. The front-ends (the web site or the API) tie them together as necessary.

Because of this, the API is essentially just a transformer: from the XML our internal services produce to XML that our developers can use.

Thoughts

Haskell's type system (and HaXml) work very nicely for our transformer, making it a great language choice for this project. It was simple to define our transformers and connect with HAppS to serve back to the client.

But there were many problems along the way. When we first started, we were working with HAppS v0.8.x. Quite frankly, there was a lot not to like about it at the time. There were many quirks and bugs and in general, things we just simply couldn't figure out. HAppS comes packed with a lot of features (such as an S3 library, a Facebook library, a stellar state management system, etc), most of which were totally unnecessary for us. This caused a bit of clunkiness as we figured out what we needed and how we could avoid using the rest. A lot of this comes from the fact that the HAppS documentation was (and is) absolutely NOT up to par. In my opinion, this problem is two-fold: (1) Haddock (the Haskell documentation engine) is just an odd interface and one I don't particularly like, and (2) HAppS' Haddock and other docs are almost non-existent.

However, by the time we were nearing our launch, HAppS 0.9.1 was launched and we decided to upgrade with it. There were some substantial changes, including basic things such as how to define routes for URLs. However, in the end, we were much happier spending some time getting up to date and using the new, nicer HAppS API. It ended up being much cleaner and much simpler to understand an maintain.

One thing about Haskell in general that really tripped us up initially was finding a non-code littering, easy to understand method of handling errors. After many iterations, we finally decided -- since we spent much of our time within the IO monad already -- upon using Haskell's Control.Exception. This ended up working out very well for us, and keeps our code relatively clean and concise.

Once we had everything implemented and tested, we moved towards figuring out how to deploy the project in production. We decided to do a very similar process as we did with our Java service, using Capistrano to build and deploy our service to multiple boxes. We then simply run round-robin DNS to load-balance these machines.

Since being in production, we've so far found very few issues and everything runs quickly and smoothly. Haskell and HAppS have turned out to be a very nice addition to our stable of tools and services.

Announcing the Openomy API v2.0

I'm very pleased to announce that today we're formally launching our new API (v2.0)! This is a huge step forward for Openomy, the developers using Openomy to develop applications, and the users of Openomy and participating applications.

The new API is much more "REST-compliant" (as opposed to our older, GETSful API) and should be much easier for developers to write applications against. It will also be much easier for us at Openomy to add features. We think our developers will greatly enjoy this new simplicity and ability to quickly iterate on your apps!

S3 Storage

Also with this launch, we're formally announcing that our default
storage engine is now Amazon's Simple Storage Service (S3). In addition to cutting costs, we believe using S3 will bring more peace of mind to our users and developers that their files are safe and secure in Amazon's data centers and will be highly available.

Achieving Our Vision

Launching both the new API as well as our new storage system completes a long process of rearchitecting our backend to allow us to grow more easily and iterate more quickly. The short-term benefits of this are quicker responses and higher uptime. The long-term benefits are allowing us to focus more closely on our vision of becoming the Online File System.

Over the past couple of years, as we predicted, the world is moving to an almost entirely web-based software model. We post our photos to Flickr, write our documents on Zoho, etc. Unfortunately, it's much too difficult for us to use our photos posted on Flickr within our documents on Zoho.

This is solved in the traditional desktop-based software model by the file system. Any application can access any file via a single API in the file system. There are (virtually) no data silos on the desktop. Any file is available for use by any application.

We're working to achieve this on the web. Our goal is to break down the data silos and allow applications to have access to all of a user's data. Moreover, we want the user to have full control over this. You deserve to be able to choose what applications have access to which data and have full data portability.

We believe this is the right model, and with these latest launches, you'll start to see this happening shortly!

P.S. Don't worry! The old APIs will stay on for a while, and we'll cover the deprecation process at a later time in another post.