1. Start with price. There is a pretty wide variance in price among vendors in the industry.  This mostly correlates to how big of a fish they want to go after.  If you can push a lot of traffic over a CDN (I want to say maybe hundreds of terabytes in a month), then you’re likely to find that many CDNs are going to give you great pricing.  If you’re a small fish with a small budget, however, you’ll immediately be able to exclude some of the larger vendors from your shopping list.

2. Setup a trial account. When you get down to three or four vendors, arrange a two week window for you to test their service.  You’ll need to have a little time to do basic configuration for the CDN, so plan for about half a day when you can work with vendors to iron out any unexpected issues with getting setup.

I give additional points to vendors who have a technical lead available to help setup your account and review your configuration.  Ask as many questions as possible and make sure you get answers, even if it means the sales team has to get their engineers involved.

3. Use “origin pull”. I touched on this briefly in my talk, but I believe that this is a better solution than CDN storage.  It should become obvious while setting up your account that slipping a CDN in front of your origin servers is supposed to be easy.  If you’re spending a lot of time getting your account setup at this point, imagine the frustration you’re going to face if you CDN has an outage and you need to switch in another provider.

4. Compare speeds from various locations. I used ab and http_load to do some basic speed tests on individual files being served from your CDN.  Run the same tests against your origin servers.  You should run these tests in a few different geographic locations. If you don’t have access to test locations of your own, talk to friends who may be able to run tests for you and send you results.

If you have access to tools like Keynote or Gomez, add those into your testing.  Let them monitor load speeds from the CDNs your testing.

5. Test cold cache speeds. Response for objects that are in cache are supposed to be very fast, and you’ll probably find no problems there.  What you really should be concerned about is how slow the latency is for objects not already in cache — i.e. a full round-trip from end-user to origin by way of the CDN.  There’s no reason latency should be exceptionally high here, but if you find any surprises escalate them to the vendor.  It could be a problem with your configuration, or it could be that they are someone you shouldn’t be working with.

6. Understand the support model. An outage is a really bad time to be figuring out how to reach support for the first time.  Find this out during the evaluation period.  Do you open trouble tickets with a phone call, an email, or a control panel?  Test this out.  Open a ticket to ask a few questions.  Become familiar with how tickets are resolved — do you get an immediate follow up, do you have visibility into the status of your ticket, or does your request go into a black hole?

Here’s a fun test for support: Intentionally set an invalid “cache-control” header for a test URL and tell support that you’re having trouble with caching that object.  Hopefully you get a quick, intelligent, and polite response.  If you’re missing any of those three characteristics, maybe you should reconsider whether you want to be working with this vendor.  Remember that the next time you’ll be opening a trouble ticket, you’re site may be on fire.

7. Look for transparency in status. I’ve wrote a bit about transparency in operations earlier this year.  I haven’t seen this with most of the vendors I’ve evaluated myself, but I think it’s worth extra points to vendors who get this right.

8. Look for the pretty pictures. Most vendors have pretty crappy reporting, but check it out an know what you’re going to get.  Run a light load against your test account for four or five days and then review the reports (reports without data aren’t very useful).

9. Get close with a technical lead. Find the smartest technical guy on your vendor’s team and talk with him/her regularly.  Tell them what you’re up to, how you’re using the CDN, what is working and what is not.  This person can teach you a lot about the more subtle features of caching after you get past the initial setup.  Ask for new features, no matter how absurd they may seem.  Vendors are looking to differentiate themselves in a space where most look like commodity services, so use this to your advantage.

It’s really important, as with most vendors, to avoid getting locked-in too tightly.  If you’re using a new video steaming service that is only available through one provider, then you’re going to have to select a niche vendor. On the other hand, if you are just serving typical site assets (images, CSS, JavaScript, etc.) then you should keep things flexible enough where you can move from CDN to CDN when necessary.

Hopefully these tips are useful to you.  Please post additional suggestions in the comments below.  I’m specifically interested in hearing about additional monitoring or testing tools that you have found useful in evaluating a CDN yourself.

Apparently I posted the deck that included some additional slides past the “thank you”, which were left in just in case there were questions that could have been addressed with some visual support.  So, enjoy those out-of-context extras.

For users new to One tsp., the app runs in the web browser and registering for an account is free.  This is better than recipe software that you would install on your phone because you can also access and manage your recipes from your desktop computer or anywhere else you have access to a web browser.

Thanks for the screen shots, Anthony!

Last weekend I was looking at my stereo and thinking, “When was the last time I actually put a CD in this deck?”  I also thought, “Why does this stereo have a tape deck, anyway?” (it’s a 2003 model, by the way).

The article states a number of potential issues with it which I’m hoping will get cleared up.  Of course, I don’t have a bluetooth receiver for my car either, which could be a drawback.

Today, I just found Google’s recommendation for leveraging proxy caches, which states, “to enable proxy caching for these resources, remove query strings from references to static resources, and instead encode the parameters into the file names themselves.”

What is a good alternative?  One method I like is to use a version number that is part of the URL path:

  /12345/css/site.css

The “12345″ is prepended in your page templates using a variable.  The value can be taken from anywhere you want… it could be part of a timestamp, a revision number in your source control, or maybe even manually changed in a constants file anytime your are updating files in your “css” directory.  If you’re using Apache, this portion of the path can be easily stripped off the request using mod_rewrite.

  RewriteEngine On
  RewriteRule ^/\d{5,}(/.+)$ $1

With this method, you’re not literally using directories with version number in them.  Rather, you just use your regular old /htdocs/css directory. The same can be applied to JavaScript, images, or other directories with static assets.

What you gain is the ability to break your cached versions of files when necessary, but to also set long-term cache times on these assets (while they’re not changing) and expect cache servers between you and your users to properly store these objects.

More to come…

I’ll be speaking at Velocity in two weeks about how to optimize caching with your CDN.  I have a lot of content that I’d like to present, but only 20 minutes to do it.  Over the coming weeks I’ll be writing a number of posts covering topics that don’t make it into the presentation.

My general approach goes something like this:

1. Point your document root at a static location, such as /var/www/html/prod.
2. Promote new code (from stage, dev, what have you) to a versioned directory on your prod server, such as /var/www/html/release_12345.
3. Use a symlink, “prod” in this case, to point at a specific release (e.g. prod -> release_12345).

The goal here is that you have a handful of versions on your prod servers that make it very quick to revert to a previous version.  You can revert your code, right?  (I’ll admit that only after a number of years I’ve finally gotten reverts working quickly and consistently.)  Our build process is not too involved, but it takes about two minutes to get code out to all of our servers and then flip over the symlinks to make the new code live.  The flip only takes one or two seconds, so all of the web servers we are deploying to remain consistent.

We have used a similar technique for nearly the last year in our build process at CafeMom.  I stress the word used, because we just got rid of the symlinks.

It turns out that we ran into an issue with the way that Apache handles symlinks while under load.  At the point when we would flip our symlinks to a new release version, files (especially PHP) from the previous release were still being executed or served by Apache.  It seems that Apache caches the real path to a file after determining where the symlink is pointing, and doesn’t necessarily re-check the symlink.  I don’t know how long this occurs, or whether it is specific to individual Apache children, but I do know that it has been a regular pain in the ass (PHP fatals, blank white pages, yuck).

To be clear about details, we also run APC on our servers to speed things up in our PHP processing.  When we promote a new version of code we also flush the PHP cache, thinking that this would help clear out cached versions of files.  The only thing that would absolutely clear up this issue was to cycle Apache, which we would prefer not to do on every deployment.

What now?  Rather than repointing a symlink, we move entire directories.  Something like this:

1. Sync code out to a new versioned directory on our prod servers, say release_12346.
2. Switch release directories doing something like this: “mv prod release_12345; mv release_12346 prod”.  (We use a state file to keep track of where “prod” originally lived, in case you’re wondering where release_12345 came from.)

Renaming these directories takes about the same time as swapping the old symlink we were using.  The Apache virtual host configurations (document roots) don’t need to be updated, either.  Looking at our APC dashboard and the affect on our site, it seems like this approach is working much more smoothly.

If you’ve got questions, alternative suggestions, or if you know what the root issue is with Apache caching our symlinks, sound off in the comments below!

A status page could be as simple as some of the third-party magic eight balls out there like Down For Everyone Or Just Me.  But an effective status page will give more information than “yes” or “no”, and will communicate:

• Is the service or site down?
• How long has it been down?
• What is being done to fix it?
• When is the service expected to be back up?

Oh, and it probably warrants stating that you shouldn’t run these status pages on the same infrastructure that runs your site/service.  If your servers, network, etc. go out of service, you want to be sure that your customers can still reach your status page.

Let’s get on to some examples:

status.aws.amazon.com

Status page for Amazon’s Web Services which shows a separate status display for each of their infrastructure services.  There are RSS feeds available for each service allowing you to subscribe to monitor service outages.  One thing I like about this page is the “report an issue” link on the page, especially since it doesn’t require to to have your login credentials to quickly note an issue.

www.google.com/appsstatus

Similar in format to Amazon’s status page, this one also includes a running history of when the last issues occurred.

status.mosso.com and status.dreamhost.com

Status pages for Mosso and Dreamhost are fairly similar, they are both simple blogs.  Support techs post issues when they arise and follow up with time-stamped messages when services are diagnosed and fixed.

This is by no means an exhaustive list but I think it highlights a few good examples.  Why is this important?  As much as I hate to use the phrase “controlling the message,” it is really important that your customers know that you’re on top of things… especially when they are paying you.

Besides, in this day and age if you don’t tell your customers what’s going on, somebody else will, and you never know what they are going to say about you.

If you know of other good status pages, please feel free to add them to the comments below.