There was interest after my talk at Velocity on which vendor is the best. Since I haven’t been exposed to many of the vendors on the market, it’s hard for me to answer that question. I can, however, give some insight onto what you should look for when choosing a vendor for yourself.
1. Start with price. There is a pretty wide variance in price among vendors in the industry. This mostly correlates to how big of a fish they want to go after. If you can push a lot of traffic over a CDN (I want to say maybe hundreds of terabytes in a month), then you’re likely to find that many CDNs are going to give you great pricing. If you’re a small fish with a small budget, however, you’ll immediately be able to exclude some of the larger vendors from your shopping list.
2. Setup a trial account. When you get down to three or four vendors, arrange a two week window for you to test their service. You’ll need to have a little time to do basic configuration for the CDN, so plan for about half a day when you can work with vendors to iron out any unexpected issues with getting setup.
I give additional points to vendors who have a technical lead available to help setup your account and review your configuration. Ask as many questions as possible and make sure you get answers, even if it means the sales team has to get their engineers involved.
3. Use “origin pull”. I touched on this briefly in my talk, but I believe that this is a better solution than CDN storage. It should become obvious while setting up your account that slipping a CDN in front of your origin servers is supposed to be easy. If you’re spending a lot of time getting your account setup at this point, imagine the frustration you’re going to face if you CDN has an outage and you need to switch in another provider.
4. Compare speeds from various locations. I used ab and http_load to do some basic speed tests on individual files being served from your CDN. Run the same tests against your origin servers. You should run these tests in a few different geographic locations. If you don’t have access to test locations of your own, talk to friends who may be able to run tests for you and send you results.
If you have access to tools like Keynote or Gomez, add those into your testing. Let them monitor load speeds from the CDNs your testing.
5. Test cold cache speeds. Response for objects that are in cache are supposed to be very fast, and you’ll probably find no problems there. What you really should be concerned about is how slow the latency is for objects not already in cache — i.e. a full round-trip from end-user to origin by way of the CDN. There’s no reason latency should be exceptionally high here, but if you find any surprises escalate them to the vendor. It could be a problem with your configuration, or it could be that they are someone you shouldn’t be working with.
6. Understand the support model. An outage is a really bad time to be figuring out how to reach support for the first time. Find this out during the evaluation period. Do you open trouble tickets with a phone call, an email, or a control panel? Test this out. Open a ticket to ask a few questions. Become familiar with how tickets are resolved — do you get an immediate follow up, do you have visibility into the status of your ticket, or does your request go into a black hole?
Here’s a fun test for support: Intentionally set an invalid “cache-control” header for a test URL and tell support that you’re having trouble with caching that object. Hopefully you get a quick, intelligent, and polite response. If you’re missing any of those three characteristics, maybe you should reconsider whether you want to be working with this vendor. Remember that the next time you’ll be opening a trouble ticket, you’re site may be on fire.
7. Look for transparency in status. I’ve wrote a bit about transparency in operations earlier this year. I haven’t seen this with most of the vendors I’ve evaluated myself, but I think it’s worth extra points to vendors who get this right.
8. Look for the pretty pictures. Most vendors have pretty crappy reporting, but check it out an know what you’re going to get. Run a light load against your test account for four or five days and then review the reports (reports without data aren’t very useful).
9. Get close with a technical lead. Find the smartest technical guy on your vendor’s team and talk with him/her regularly. Tell them what you’re up to, how you’re using the CDN, what is working and what is not. This person can teach you a lot about the more subtle features of caching after you get past the initial setup. Ask for new features, no matter how absurd they may seem. Vendors are looking to differentiate themselves in a space where most look like commodity services, so use this to your advantage.
Hopefully these tips are useful to you. Please post additional suggestions in the comments below. I’m specifically interested in hearing about additional monitoring or testing tools that you have found useful in evaluating a CDN yourself.