Guess which version?

Posted by Mike Brittain on January 26, 2010
Microsoft / Comments Off on Guess which version?

Just what do you suppose I should make of this user-agent string?  It’s real.  I found it in my logs today.

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

Seriously.  We see “MSIE 7.0” in there, which you would initially think of as Internet Explorer 7.  But of course, since we also see “Trident/4.0”, we can deduce that this is Internet Explorer 8 running in Compatibility View.  So what the hell is “MSIE 6.0” still doing in here?

P.S. Thanks for including the version of .NET CLR… 6 different ways. That certainly is helpful.

Tags: , ,

Choosing a CDN

Posted by Mike Brittain on January 15, 2010
WWW / 1 Comment

I thought this was a good article covering 8 Things to Consider When Choosing a CDN.  It’s worth a read.

The only bit I would disagree with is that this reads a bit too video-centric for me.  It felt like the main argument for using a CDN is that you don’t have enough bandwidth at your own data center to handle all of the requests being made to your servers.

I use CDNs for delivering static objects like images, CSS, and JavaScript.  An additional consideration I have is how fast cached objects will reach different locations across the country or around the world.  I’m dependent (as most sites are) on one central data center where my pages are being generated.  Every page view needs to make that trip over the network from the browser to my data center.  However, if the 20-100 successive objects can be requested from a regional CDN node, the performance in the end-user’s browser will be much better than if every request made a full trip across the country.

Tags: , ,

Two Munin Graphs for Monitoring Our Code

Posted by Mike Brittain on December 17, 2009
PHP, WWW / Comments Off on Two Munin Graphs for Monitoring Our Code

We’re using a few custom Munin graphs at CafeMom to monitor our application code running on production servers.  I posted two samples of these to the WebOps Visualization group at Flickr.  The first graph measures the “uptime” of our code, a measure of how many minutes it’s been since our last deployment to prod (with a max of 1 day).  The second graph provides a picture of what’s going on in our PHP error logs, highlighting spikes in notices, warnings, and fatals, as well as DB-related errors that we throw of our own.

When used together, these graphs give us quick feedback on what sort of errors are occurring on our site and whether they are likely to be related to a recent code promotion, or are the effect of some other condition (bad hardware, third-party APIs failing, SEM gone awry, etc.).

I figured someone might find these useful, so I’m posting the code for both Munin plugins.

Code Uptime

When we deploy code to our servers, we add a timestamp file that is used to manage versioning of deployments to make things like rolling back super easy.  It’s also handy for measuring how long it’s been since the last deployment.  All this plugin does is reads how long ago that file was modified.

We run multiple applications on the same clusters of servers. I wrote our code deployment process in a manner that allows for independent deployment of each application. For example, one team working on our Facebook apps can safely deploy code without interfering with the deployment schedule another team is using for new features that will be released on the CafeMom web site.

Each of these applications is deployed to a separate directory under a root, let’s say “/var/www.”  This explains why the plugin is reading a list of files (directories) under APPS_ROOT.  Each app has it’s own reported uptime on the Munin graph.

#!/bin/sh
#
# Munin plugin to monitor the relative uptime of each app
# running on the server.
#

APPS_ROOT="/var/www/"

# Cap the uptime at one day so as to maintain readable scale
MAX_MIN=1440

# Configure list of apps
if [ "$1" = "config" ]; then
 echo 'graph_title Application uptimes'
 echo "graph_args --base 1000 --lower-limit 0 --upper-limit $MAX_MIN"
 echo 'graph_scale no'
 echo 'graph_category Applications'
 echo 'graph_info Monitors when each app was last deployed to the server.'
 echo 'graph_vlabel Minutes since last code push (max 1 day)'

 for file in `ls $APPS_ROOT`; do
 echo "$file.label $file"
 done
 exit 0
fi

# Fetch release times
now_sec=`date +%s`

for file in `ls $APPS_ROOT`; do
 release_sec=`date +%s  -r $APPS_ROOT/$file/prod/release_timestamp`
 diff_sec=$(( $now_sec - $release_sec ))
 diff_min=$(( $diff_sec/60 ))
 ((diff_min>MAX_MIN?diff_min=MAX_MIN:diff_min))
 echo "$file.value $diff_min"
done

Error Logs

The second plugin uses grep to search for occurrences of specific error-related strings in our log files. In our case, the graph period was set to “minute” because that gives the best scale for us (thankfully, it’s not in errors per second!).

If you’re wondering about using grep five times against large error files every time Munin runs (every five minutes), I want to point out that we rotate our logs frequently which ensures that these calls are manageable. If you run this against very large error logs you may find gaps in your Munin graphs if the poller times out waiting for the plugin to return data points.

Even if you don’t care about PHP logs, you may find this to be a simple example of using Munin to examine any sort of log files that your application is creating.

#!/bin/bash

#
# Collect stats for the contents of PHP error logs. Measures notice,
# warning, fatal, and parse level errors, as well as custom errors
# thrown from our DB connection class.
#

logs="/var/log/httpd/*.error.log"

# CONFIG

if [ "$#" -eq "1" ] && [ "$1" = "config" ]; then
	echo "graph_title Error Logs"
	echo "graph_category applications"
	echo "graph_info Data is pooled from all PHP error logs."
	echo "graph_vlabel entries per minute"
	echo "graph_period minute"
	echo "notice.label PHP Notice"
	echo "notice.type DERIVE"
	echo "notice.min 0"
	echo "notice.draw AREA"
	echo "warning.label PHP Warning"
	echo "warning.type DERIVE"
	echo "warning.min 0"
	echo "warning.draw STACK"
	echo "fatal.label PHP Fatal"
	echo "fatal.type DERIVE"
	echo "fatal.min 0"
	echo "fatal.draw STACK"
	echo "parse.label PHP Parse"
	echo "parse.type DERIVE"
	echo "parse.min 0"
	echo "parse.draw STACK"
	echo "db.label DB Error"
	echo "db.type DERIVE"
	echo "db.min 0"
	echo "db.draw STACK"
	exit
fi

# DATA

# The perl code at the end of each line takes a list of integers (counts) from grep (one per line)
# and outputs the sum.

echo "notice.value `grep --count \"PHP Notice\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "warning.value `grep --count \"PHP Warning\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "fatal.value `grep --count \"PHP Fatal error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "parse.value `grep --count \"PHP Parse error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "db.value `grep --count \"DB Error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"

I’m open to comments and suggestions on how we use these, or how they were written.  Spout off below.

Tags: , , , , , , ,

Web-based Email: One Example of Cloud Apps Replacing Desktop Apps

Posted by Mike Brittain on August 10, 2009
Cloud Computing / Comments Off on Web-based Email: One Example of Cloud Apps Replacing Desktop Apps

I read this poll on LifeHacker the other day about web-based email vs. desktop email apps.  It reinforced what I believe is the current momentum in web applications these days — that over time, people are going to get more and more comfortable using web apps the way they used to use desktop apps.  It’s been five years since the release of Gmail, which I view as a forerunner in this area, so clearly this isn’t going to be a quick change.

Browser innovations will help users with the perception that web applications are interchangeable with desktop software.  Google Chrome is already working in this direction by reducing the amount of browser UI and allowing the user to focus on sites and apps they are using.  I’m won’t argue that Chrome will be a major browser; it may never be.  I do believe that Google’s intention is to continue swaying the way we look at the Internet. As other browsers follow suit (in some cases), Google’s web applications and sites will all benefit.

As we move forward, many computers, especially in public spaces like libraries and educational computer labs, will use fewer licensed software suites and more subscriptions to web-based applications.  As with Gmail, it doesn’t matter whether I check email on my computer, your computer, a work computer, or a mobile phone — I still have access to the application because modern web browsers provide the baseline of support for these apps.  With stabilization and implementation of features in HTML 5, additional web-based apps will be built and they will continue to look and act like our familiar desktop apps.  Over time, institutions will replace their local file servers, email servers, and parts of their IT staff with outsourced apps that are purchased by subscription and delivered in a browser.

Microsoft’s Lost Vision

Posted by Mike Brittain on July 26, 2009
Microsoft / Comments Off on Microsoft’s Lost Vision

John Dvorak has a great piece on MarketWatch right now about how Microsoft has lost sight of its core business.  He points out a number of great examples of how the company has chased after others in publishing, online networks, search, and music, while none of these areas have anything to do with Microsoft’s history and what it used to do best.  The Xbox isn’t mentioned, and while it might look like a separate business (gaming) I think it probably is pretty close to the idea of building an OS and tools for other developers to build for.  Not a far stretch from building Windows when you think about it.

Dvorak hits on all of the points that bother me about Microsoft.  Personally, I’ve felt that over the years I’ve poured a bundle of money into Windows and Office licenses (for personal and business use) and I can fathom that much of the profit that Microsoft has made on these purchases has been sunk into losing propositions.  And that makes me angry.

Tags:

Learning is Hard

Posted by Mike Brittain on July 19, 2009
Misc / Comments Off on Learning is Hard

I used to do a decent amount of reading about e-learning during my grad school studies. I was a big fan of Clifford Stoll’s book, “Silicon Snake Oil.”. The premise was essentially that people keep trying to sell is on this idea that learning should be fun. Education can be built into some sort of game so that it becomes enjoyable and the things we dislike about learning will happen automatically while we are entertaining ourselves.

I just bumped into a great quote from Dave Benjamin that reminded me of all of this, and I wanted to share that because I think it’s important to remember when raising kids in this age:

“Learning is hard work. Making this seem untrue or avoidable is popular and lucrative. Often, the result of catering toward this interest is something other than learning.”

Tags:

One tsp. Has Now Left the Building

Posted by Mike Brittain on July 19, 2009
Misc / Comments Off on One tsp. Has Now Left the Building

A little matter of house keeping…

The One tsp. blog entries have moved over to their own site at onetsp.com/blog.  I think I’ve covered all of my bases with the original links to posts here.  If you were an RSS subscriber, you may want to just resubscribe over there.

Leaner iPhone Interfaces with CSS Gradients

Posted by Mike Brittain on July 05, 2009
Mobile / Comments Off on Leaner iPhone Interfaces with CSS Gradients

I started playing around with Safari’s CSS gradients yesterday to see whether they would be usable on One tsp.’s mobile interface.  Looks like there has been support in WebKit for about a year now, but I don’t know specifics about how that translates to versions of Safari and other browsers built on top of WebKit.

The demos seemed to work for me in Safari 4 and in the latest version of mobile Safari built into the iPhone 3.0 OS.  I tested the 2.0 OS and it did not support gradients. I don’t know what support the Palm Pre browser has available.

This looked good enough for me, through.  Much of the interface for One tsp. is already taking advantage of a few CSS extensions with varying support.  The interface looks its best on modern browsers (IE excluded) but is still totally usable everywhere else.

So what’s the difference?

I’ve only replaced one gradient background so far, but I’m stunned.  By defining the gradient in CSS, I’ve added just 92 bytes to my style sheet.  This allowed me to remove the background-image rule I had in place to load an image file, which was 50 bytes.  The image file that is no longer needed was pretty small (635 bytes) but also meant another external request that needed to be made.  When we’re talking about a mobile device, extra requests can have a high latency — worse than what we typically think of for the web.

These are pretty small numbers, I’ll admit.  But assuming I have six gradients defined per page, the net savings would be trading around 4 KB and six additional requests for about 260 bytes and no additional requests.  That’s pretty cool.

Faster Mobile Interfaces

Successful mobile web applications need to be super fast. Users trading a native app for a web app will expect it to be responsive. Speed can be improved through faster server responses, low mobile network latency (which we have little control over), fewer and smaller requests to the server, and cacheability on as much content as possible.

Rounded corners and background gradients are two frequently used interface styles that can now be achieved directly in the browser using CSS, eliminating the need for many additional image requests.

Tags: , , , , , ,

Tips on Selecting a CDN Vendor

Posted by Mike Brittain on June 24, 2009
WWW / Comments Off on Tips on Selecting a CDN Vendor

There was interest after my talk at Velocity on which vendor is the best. Since I haven’t been exposed to many of the vendors on the market, it’s hard for me to answer that question.  I can, however, give some insight onto what you should look for when choosing a vendor for yourself.

1. Start with price. There is a pretty wide variance in price among vendors in the industry.  This mostly correlates to how big of a fish they want to go after.  If you can push a lot of traffic over a CDN (I want to say maybe hundreds of terabytes in a month), then you’re likely to find that many CDNs are going to give you great pricing.  If you’re a small fish with a small budget, however, you’ll immediately be able to exclude some of the larger vendors from your shopping list.

2. Setup a trial account. When you get down to three or four vendors, arrange a two week window for you to test their service.  You’ll need to have a little time to do basic configuration for the CDN, so plan for about half a day when you can work with vendors to iron out any unexpected issues with getting setup.

I give additional points to vendors who have a technical lead available to help setup your account and review your configuration.  Ask as many questions as possible and make sure you get answers, even if it means the sales team has to get their engineers involved.

3. Use “origin pull”. I touched on this briefly in my talk, but I believe that this is a better solution than CDN storage.  It should become obvious while setting up your account that slipping a CDN in front of your origin servers is supposed to be easy.  If you’re spending a lot of time getting your account setup at this point, imagine the frustration you’re going to face if you CDN has an outage and you need to switch in another provider.

4. Compare speeds from various locations. I used ab and http_load to do some basic speed tests on individual files being served from your CDN.  Run the same tests against your origin servers.  You should run these tests in a few different geographic locations. If you don’t have access to test locations of your own, talk to friends who may be able to run tests for you and send you results.

If you have access to tools like Keynote or Gomez, add those into your testing.  Let them monitor load speeds from the CDNs your testing.

5. Test cold cache speeds. Response for objects that are in cache are supposed to be very fast, and you’ll probably find no problems there.  What you really should be concerned about is how slow the latency is for objects not already in cache — i.e. a full round-trip from end-user to origin by way of the CDN.  There’s no reason latency should be exceptionally high here, but if you find any surprises escalate them to the vendor.  It could be a problem with your configuration, or it could be that they are someone you shouldn’t be working with.

6. Understand the support model. An outage is a really bad time to be figuring out how to reach support for the first time.  Find this out during the evaluation period.  Do you open trouble tickets with a phone call, an email, or a control panel?  Test this out.  Open a ticket to ask a few questions.  Become familiar with how tickets are resolved — do you get an immediate follow up, do you have visibility into the status of your ticket, or does your request go into a black hole?

Here’s a fun test for support: Intentionally set an invalid “cache-control” header for a test URL and tell support that you’re having trouble with caching that object.  Hopefully you get a quick, intelligent, and polite response.  If you’re missing any of those three characteristics, maybe you should reconsider whether you want to be working with this vendor.  Remember that the next time you’ll be opening a trouble ticket, you’re site may be on fire.

7. Look for transparency in status. I’ve wrote a bit about transparency in operations earlier this year.  I haven’t seen this with most of the vendors I’ve evaluated myself, but I think it’s worth extra points to vendors who get this right.

8. Look for the pretty pictures. Most vendors have pretty crappy reporting, but check it out an know what you’re going to get.  Run a light load against your test account for four or five days and then review the reports (reports without data aren’t very useful).

9. Get close with a technical lead. Find the smartest technical guy on your vendor’s team and talk with him/her regularly.  Tell them what you’re up to, how you’re using the CDN, what is working and what is not.  This person can teach you a lot about the more subtle features of caching after you get past the initial setup.  Ask for new features, no matter how absurd they may seem.  Vendors are looking to differentiate themselves in a space where most look like commodity services, so use this to your advantage.

It’s really important, as with most vendors, to avoid getting locked-in too tightly.  If you’re using a new video steaming service that is only available through one provider, then you’re going to have to select a niche vendor. On the other hand, if you are just serving typical site assets (images, CSS, JavaScript, etc.) then you should keep things flexible enough where you can move from CDN to CDN when necessary.

Hopefully these tips are useful to you.  Please post additional suggestions in the comments below.  I’m specifically interested in hearing about additional monitoring or testing tools that you have found useful in evaluating a CDN yourself.

Slides from Velocity 2009

Posted by Mike Brittain on June 24, 2009
Personal / Comments Off on Slides from Velocity 2009

I’ve got my slides posted from Getting to Second Base with Your CDN, which I presented at Velocity 2009.  It was a much shortened talk than I hoped to give, but I think that some of this was still useful.  Good feedback so far.

Apparently I posted the deck that included some additional slides past the “thank you”, which were left in just in case there were questions that could have been addressed with some visual support.  So, enjoy those out-of-context extras.