Getting Back To Bug Fixes

Any medium-scale web application will have bugs. Openomy is no different. I know of many small, non-showstopper bugs that I would love to fix, but just haven't. There's a delicate balance between knowing when to fix a bug, and when to work on a new feature that matters more. In the past, I've worked on new features too much. so, I've been spending a few days now fixing some bugs -- mostly things you as a user can't see.

But, I did know that the Flash uploader we launched a few months ago had some annoyances and new users had trouble understanding it. So, yesterday I spent a couple hours making it slightly more user friendly. The changes were by no means earth-shattering (likely they didn't even register on the Richter scale), but I thought I'd take a second to just let our users know that we are continually improving Openomy.

Now when you bring up the flash uploader after logging in, you'll see a note explaining how to best use it. This guide makes it slightly easier for new users to understand. You'll also see that I fixed the annoying bug with the progress label (it used to cut off, saying something like "Uploaded 1mb out of " -- out of what?!). And lastly, clicking on a filename after the file is uploaded now does what you'd expect: sends you to a page where you can download/publicize/share that file.

Hopefully this makes your life just slightly better. We do have some big projects that are in development, but I promise to spend more time fixing bugs as well.

The Windows Explorer of the Web

Even though we're moving all our life to the web, little has been done to make finding our data easy. It's easy to find my videos at YouTube, easy to find my photos at Flickr and Facebook, easy to find my documents at Zoho. However, it's difficult to find all my files in one place, difficult to search across multiple services at once, and difficult to organize my files by theme, rather than just by file type. Solving this problem is the vision for an Online File System.

I wanted to expand on this a little bit and explain Openomy as the Windows Explorer of the web. As many of you know, Windows Explorer (or Finder for us OS X folks) is the application Windows provides to us to view all our files in a single place. We can see all the photos, documents, spreadsheets, applications, and anything else that sits on our hard drive in one place. We can search for different things. We can organize those files. And, we can transfer files directly to our hard drive via Windows Explorer.

Openomy is much the same. You'll be able to see all of your files stored at various web sites. You'll be able to search for files across these different sites. You can organize the files using tags, allowing you to find a single file by different themes. You can upload files directly to Openomy via our interface.

Unlike Windows Explorer, however, we go further by allowing you to share your files with friends, family, and the world. You can share any file with a friend as long as they, too, have an Openomy account. Or, you can allow the entire world to see your file publicly, which also allows you to embed your file into other sites and services.

We're excited to organize your web based world and become the Windows Explorer of the web.

The Vision For An Online File System

Just over two weeks ago, we launched a long-overdue redesign of Openomy. We think the impact of the design has been great and we're proud of what we accomplished -- knowing full well there's a tremendous amount of work still ahead.

We had one driving goal for this new design:

To execute on our expanded vision of becoming the place where your organize your web-based world.

From Centralized To Decentralized

When we first started, we thought the Online File System would be easy: provide the equivalent of a hard drive (online) for a user, and allow applications to store and retrieve files from that "hard drive" using open APIs (the "file system"). This was extremely innovative three years ago, when we started, but it still wasn't exactly right just yet.

As we've grown, we've realized this isn't the model of the web: just because something worked on the desktop, doesn't mean it works on the web. The models are very different; the former is a controlled, centralized space, while the latter is an uncontrolled, decentralized web of loosely connected parts.

To become the Online File System, we needed to look past the centralized "hard drive in the cloud." We had to think hard about how to capture the web's decentralized nature. It occurred to us that the important part of the model wasn't the storage itself, but the simple access to the data, both for the user as well as the developer.

Easy Organization Of Your Web Based World

Right now, we store our photos on Flickr and Facebook, our documents on Google Docs and Zoho, our videos on YouTube and Vimeo. This is the decentralized model of the web, and it works. What doesn't work, however, is easily aggregating and organizing all your different files together, in one place. I can't easily see the pictures of my trip to Vancouver in the same place I can see my videos. I have to remember what is where, and when they were put there. To make an analogy from the desktop model, there's no web-based "Windows Explorer."

Openomy's goal is to organize your web based world. We'll let you store your files where you see fit (Flickr, Facebook, Google Docs, etc) and we'll learn about those files automatically (using the remote services APIs). Once we know about those files, we'll make them searchable, taggable, and shareable. All your data, available in one place, easy to find, and easy to share (with a few friends or the world). And for files that have no other place, we still provide easy-to-use storage!

It's a future we're excited about, and we're eager to keep working towards this world.

Expected Downtime: Tonight (8/22/08) at 7PM PDT

We'll be doing some work on the site this evening that will require us to take Openomy down briefly. The site will be going under the knife starting around 7PM PDT.

In all likelihood, this work will take a few hours to complete, so we expect the site to be back up no later than 12AM. If there are any issues, we'll be updating our Twitter account, so check there often!

Thanks for your patience, we think you'll enjoy what we're bringing out! :)

Dealing with dumb programs (or Why our site has been sluggish recently)

Some people have noted the recent slowness of the Openomy web site. To be clear, the API has been fine, as have been downloading files, but the web site has been extremely sluggish. The reason is we've essentially been DDOS'd by a dumb program. I think we've begun to get it under control, but I wanted to shed some light on what happened and how we've controlled it. We're no experts in this field, so I'd love to hear about what we did right/wrong or could improve upon in these situations.

A while ago, we noticed a not-so-clever program which basically said:

while (true) {
  get_public_file_from_openomy();
  ...
}

They didn't do this to be evil (we know because we were able to find the application and see what it was doing). We were unable to get them to fix it for a variety of reasons.

Over time, more and more people downloaded this application, making it effectively a DDOS attack on Openomy. The funny thing is, because it's just a user's public file and therefore subject to bandwidth limitations, it's virtually useless 99% of the time. We've tried a variety of strategies to deal with this file, so I figured we'd talk about them.

Step 1: Use .htaccess to block some IPs
At the beginning, just a few people were using the program. The IPs remained relatively static, so we would just add the IP to an .htaccess file which we then blocked. This was an extremely simple solution that we hoped would stop the proliferation of the program and make things back off.

Unfortunately, it didn't. People continued to download it, and the list of IPs grew. Adding IPs to the .htaccess file really only helped us for a week or so.

Step 2: 404ing the file
Since the file is virtually useless (due to the bandwidth overages) anyways, we decided we would just make the URL 404. We could do this at the mod_rewrite level and then no further processing would need to be done.

This actually lasted for a couple months, until the increased load once again caused the site to perform sluggishly.

Step 3: Stopping KeepAlives
One thing we noticed about 404ing the file was that although the requests weren't taking long to process, many of our available sockets were being taken up for long periods of time by these requests due to HTTP Keep-Alives. So, we turned off all KeepAlives for the site. This could have hurt some performance for our other pages, but because our site is so lightweight, the net gain was actually very large. We were once again able to process many more connections.

This, too, helped us for another couple months. That is, up until these past few days.

Step 4: (Now testing) iptables
The amount of clients requesting this bogus, unavailable file is again becoming too many for our web servers to handle. Moreover, the clients are all accessing the file many times. However, it does seem as though many of the IPs stick around for a few days, which is nice.

So, we've decided to run a periodic cron job which parses our log files for the past week, find the IP addresses that have requested our non-existant file (with some threshold), and create an iptable filter to drop all requests from said IP addresses. This obviously won't cover everyone requesting this file, but hopefully it will keep the amount of requests low enough that our servers can continue to serve requests to others with good performance.

This works better than the .htaccess file because our web server doesn't even begin to touch these requests.

I ended up writing a ruby script to perform this for us, but I started out with just some simple shell-foo. An example of how to find all the IPs accessing a certain log file would be like:

grep pattern filename.log | cut -d' ' -f1 | sort | uniq -c

The site seems to be running pretty well right now, but we've only had this out for a few hours. We'll need to watch this over time to see how it continues to perform and scale. One issue we currently see is a large amount of connections in the TIME_WAIT state. I'm unsure how this will affect us, but we'll be keeping an eye on it.


Hopefully this gives some insight into what we're facing, helps some people who face this problem in the future, and perhaps we'll learn something about how to make this even better going forward!

Openomy In The Press

Since we've started, we've never done any real PR work attempting to get press mentions. Luckily, we've been fortunate enough to be featured in some of the best news sources, including blogs, online news sites, magazines, and being able to participate in conferences.

I thought I'd take a second to appreciate all the nice words we've received, and point out some nice press we've recently received.

Web Worker Daily
On March 5, Mike Gunderloy of Web Worker Daily said in a post entitled Openomy: Going Beyond Free Storage:

...Openomy has a different strategy: they’ll give you a gigabyte of storage and a way for you to build your own features.

Specifically, Openomy has implemented a REST-based API with (at the moment) Ruby language bindings. Anyone can write an application that gets authenticated access to files stored on Openomy, then perform operations like uploading, downloading, and tagging. If you’re a developer looking to add online storage to your own applications, this gives you a potential way to do that without running your own disk farm.

PC World
And, on March 23, PC World included us in their list of 101 Fantastic Freebies, as one of their top File Sharing and Storage sites. The list should be in the May, 2008 issue of the print version of PC World. We're very humbled and thankful for this distinction and thank everyone involved.

Thanks for all the kind words, folks! We hope we can exceed the high expectations set by these articles.

Openomy: Now with unlimited storage and even more control over your data (via your own S3 account)!

Today is another big day here at Openomy: You can now use your own Amazon S3 account within Openomy! This is important for many reasons: you now have unlimited storage (just pay Amazon for what you use), full control over where your files are stored, and the ability to manage your S3 account through the Openomy interface.

How It Works

An example: Every day, I back up some files to S3 using the S3Fox Firefox extension. I use my Openomy account for storing files, too.

All I have to do is set up my S3 account with Openomy and now I'll be able to upload and download even more files through Openomy -- the 1gb storage limit is removed when I use my own S3 account. Every file I upload will be stored in my own S3 account. My older Openomy files won't change and will be available just as they were before.

Syncing
Moreover, anything that I upload to my S3 account outside of Openomy can be imported into my Openomy account with a single click. For example, if I add a file through S3Fox, I can Sync my account, making it available via Openomy.

Likewise, if I delete a file from S3Fox, I can Sync and it will no longer be shown within Openomy.

Setup

To setup, just click the Your Account link at the top right of every page, and fill in your AWS credentials, including the bucket you'd like to use. That's it! All new files will be uploaded to that bucket.

If you'd like to Sync, just click the Sync button. Any differences between what Openomy knows about and S3 contains will be resolved. You may Sync your account at any time, but beware that Syncing may take a few minutes.

Caution!

This is a Beta feature, and we are making it publicly available for testing. We are using this feature internally already, and are confident in the quality. That said, you should be aware of a few things:

• Once you begin using S3, you can not stop. We plan on adding this ability shortly, but for now, there's no turning back.


• If you delete a file from Openomy that is stored in your S3 account, it is also deleted from S3. Make sure this is what you actually want!

Feedback

As always, we love to hear feedback about our features, so please do send me email. Comments, suggestions, bug reports, and flames are all welcome!