Etsy is Hiring, Meet us at Velocity Conf

Posted by Mike Brittain on June 22, 2010
etsy, Job Openings / Comments Off

We still have a number of open positions at Etsy that are located in both New York City and San Francisco.  John Allspaw and Mike Brittain will be at O’Reilly’s Velocity Conference this week.  If you’re attending Velocity, don’t miss out on the chance to talk to one of us about working at Etsy.

These positions include:

  • Senior Operations Engineer
  • Web Operations Engineer
  • Senior Engineers and Lead Engineers in various areas, including Search, Payments, Social Systems, Community, Internationalization, Internal Tools, Fraud and Risk Management, and Content Platforms
  • Test Automation Engineer
  • Release Manager
  • Technical Project Manager

Check out our recruiting site for more details about each of these positions, or email me directly.  I hate having to say it, but please, no recruiters.

And don’t miss out on John’s talk at Velocity: Ops Meta-Metrics, The Currency You Use to Pay for Change.

Tags: , , , , ,

Zero-Day Deploys

Posted by Mike Brittain on June 10, 2010
etsy / 3 Comments

When you bring new engineers on-board, it’s important to make them productive as soon as possible.  Spending days (or weeks) in training session, locked down permissions, and/or watchful eyes simply constrains your shiny, new developer from getting shit done.  (I’ll be honest and say that I’ve been one of those types in the past.)

At Etsy, every engineer has access to deploy code to our production site.  We use a tool called “Deployinator” to do this quickly and easily.  It’s one button.  A culture of unit and functional testing (know when your code done broke), transparency (know when releases are happening), operational metrics (know when your servers are crying), and personal accountability (know when it’s your fault) keeps the entire process under control.

Earlier this week, we did something brand new for us…

watching a new engineer who started at Etsy today push code RIGHT NOW. awesome.

~chaddickerson, Etsy’s CTO

That’s right. On his very first day, Jason got setup in his development environment, made a change to our code base (albeit a small one), tested, committed, and deployed to the production web servers.

I can just hear it: “Why would you ever let someone deploy code on their first day?” But if you’re asking this question, I have to assume that you’ve also found your self wondering at times, “How long is it going to be before Joe Newbie is going to be up to speed?

Sure, it’s going to take time for every new hire to learn all of the ropes. But it’s better for them to be productive, confident and experienced in releasing code (even if it’s small changes!) while they’re learning all of the other details than it is for them to sitting on their hands waiting for coach to put them into the game.

And in case you’re wondering, Jason has already released code three times in his first week… possibly four by time I finish writing this.

Want to get in on the action? We’re hiring in both Engineering and Operations.

Tags: , , , ,

Moving Billions of Objects Within S3

Posted by Mike Brittain on May 25, 2010
Misc / Comments Off

This is an interesting write-up on how to move enormous numbers of objects within S3.  I have a similar post coming up about large-scale batch processing that I’m looking forward to sharing in the next couple of weeks.  But give this a read.  It’s interesting!

Tags:

Twitter’s Photo Storage (from the outside looking in)

Posted by Mike Brittain on May 24, 2010
Cloud Computing, WWW / 1 Comment

I’ve been working on some photo storage and serving problems at Etsy, which is exciting work given the number of photos we store for the items being sold on the site.  This sort of project makes you wonder how other sites are handling their photo storage and serving architectures.

Today I spent a few minutes looking at avatar photos from Twitter from the outside.  This is all from inspection of URLs and HTTP headers, and completely unofficial and unvalidated assumptions.

Two things I found interesting today were (1) the rate of new avatar photos being added to Twitter, and (2) the architecture for storing and serving images.

Avatars

Avatar photos at Twitter have URLs that look something like the following:

http://a3.twimg.com/profile_images/689887435/my_photo.jpg

I’m assuming the numeric ID increments linearly with each photo that is uploaded… two images uploaded a few minutes apart showed a relatively small increase between these IDs.  I compared one of these IDs with the ID of an older avatar, along with the “Last-Modified” header that was included with its HTTP response headers:

Last-Modified Tue, 26 Feb 2008 03:15:46 GMT

Comparing these numbers shows that Twitter is currently ingesting somewhere over two million avatars per day.

Stock, or library, avatars have different URLs, meaning they are not served or stored the same way as custom avatars.  This is good because you get the caching benefits of reusing the same avatar URL for multiple users.

http://s.twimg.com/a/1274144130/images/default_profile_6_normal.png

Storage and Hosting

Running a “host” look up on the hostname of an avatar URL shows a CNAME to Akamai’s cache network:

$ host a3.twimg.com
a3.twimg.com is an alias for a3.twimg.com.edgesuite.net.
a3.twimg.com.edgesuite.net is an alias for a948.l.akamai.net.
a948.l.akamai.net has address 96.6.41.171
a948.l.akamai.net has address 96.6.41.170

If you’re familiar with Akamai’s network, you can dig into response headers that come from their cache servers.  I did a little of that, but the thing I found most interesting is that Akamai plucks avatar images from Amazon’s CloudFront service.

x-amz-id-2: NVloBPkil5u…
x-amz-request-id: 1EAA3DE5516E…
Server: AmazonS3
X-Amz-Cf-Id: 43e9fa481c3dcd79…

It’s not news that Twitter uses S3 for storing their images, but I hadn’t thought about using CloudFront (which is effectively a CDN) as an origin to another CDN.  The benefit here, aside from not pounding the crap out of S3, is that Akamai’s regional cache servers can pull avatars from CloudFront POPs that are relatively close, as opposed to reaching all the way back to a single S3 origin (such as the “US Standard Region”, which I believe has two locations in the US).  CloudFront doesn’t have nearly as many global POPs as Akamai. But using it does speed up image delivery by ensuring that Akamai’s cache servers in Asia are grabbing files from a CloudFront POP in Hong Kong or Singapore, rather than jumping across the Pacific to North America.

I suspect that Twitter racks up a reasonably large bill with Amazon by storing and serving so many files from S3 and CloudFront.  However, it takes away the burden of owning all of the hardware, bandwidth, and man power required to serve millions upon millions of images… especially when that is not a core feature of their site.

Tags: , , , , , ,

Mobile Sites Proliferate by Category

Posted by Mike Brittain on April 21, 2010
Mobile / Comments Off

An article on RWW about mobile apps and browser-based sites highlights the different concentrations in categories between native apps and mobile sites.  Native apps are heavily weighted toward games and entertainment.  Mobile web sites are heavily weighted toward shopping and social categories:

… 19% of the mobile sites measured were Shopping & Services sites; compared to 3.6% in the same category in the App Store. Content in the ‘Social’ category also has a higher chance of being a browser-based mobile site, rather than an app (12.9% to 1.7%).

Additionally, Taptu estimates that mobile site growth far outpaces the growth of native apps on any other platform, including the iPhone and App Store.

My position for a long while has been that mobile sites have much better reach due to the ability to access them from any mobile device with a decent browser, without having to download an app.  This makes cross-platform development much easier for existing web teams.  As more mobile platforms take up the WebKit rendering engine, including this week’s report that BlackBerry 6.0 will include a touch browser backed by WebKit, the baseline for development across the myriad of mobile devices is actually much better than what we had with the first web browsers in the late ’90s.

Still, the perception by consumers that apps are hip, as well as aggressive app-centric marketing by carriers, sets a higher barrier for consumers to understand the wealth of mobile sites available to them on their existing handsets.  Visibility is still an issue here.

Tags: , , , ,

NBCOlympics.com Delivers Quality TV to Web Crossover

Posted by Mike Brittain on February 16, 2010
WWW / Comments Off

I write this because I expected it not to work. Tonight while watching coverage from the Olympics, an ad popped up promotion additional background video available online. I opened the site on my iPhone, browsed through the videos, and watched the video that was highlighted on the broadcast coverage.

Great experience. No Flash Required.

Tags: , , , ,

Batch Process Image Optimizations

Posted by Mike Brittain on February 14, 2010
WWW / 2 Comments

A couple weeks ago I wrote a post about a script I had put together for batch processing JPEGs with jpegtran.  This week I extended that script so that it handles processing GIF and PNG images as well.  It’s now a project on GitHub called “Wesley“.

Wesley is a single Perl script that you run from the command line, supplying the path to a single file name or a directory where you keep your site’s images.  If you work on a Linux or Mac development server, you can quickly run this script against all new images that you add to your site code.  Additionally, you could tie this into your build process or pre-commit hook for your preferred source control.  I haven’t spent time on this yet, but expect to add a write up on it soon.

The script strips meta data and comments from image files and tries to optimize images using lossless techniques.  You should be able to run Wesley on your images without any reduction in quality.

Wesley makes use of locally installed copies of ImageMagick, jpegtran, pngcrush, and gifsicle.  Some of these are probably already installed on your own machine (or shared hosting service).  If you are missing one or more of these packages, you can still run Wesley and it will use as many packages as you have available.

Usage

   wesley.pl  /path/to/images/

Sample Output

  ----------------------------
    Summary
  ----------------------------

  Converting the following GIFs to PNG would save additional file size.
  Bytes saved: 19173 (orig 149404, saved 12.83%)

    ./hd-sm.gif
    ./top_navigation.gif
    ./logo.gif

  Inspected 226 JPEG files.
  Modified 190 files.
  Huffman table optimizations: 138
  Progressive JPEG optimizations: 52
  Bytes saved: 408508 (orig 2099658, saved 19.45%)

  Inspected 105 PNG files.
  Modified 99 files.
  Bytes saved: 84618 (orig 315056, saved 26.85%)

  Inspected 129 GIF files.
  Modified 70 files.
  Bytes saved: 57535 (orig 1393120, saved 4.12%)

  Total bytes saved: 550661 (orig 3807834, saved 14.46%)

Tags: , , ,

What I Want in a Tablet

Posted by Mike Brittain on February 10, 2010
Gadgets / 3 Comments

For the last week I have been laid up in bed and on the couch while recovering from ACL reconstructive surgery. I would estimate that I’ve been using my iPhone an average of 7 hours a day for online tasks:

– Reading articles on newspaper sites
– Reading RSS subscriptions (in Reeder)
– Posting links to Twitter and Delicious
– Reading and writing on Twitter
– Reading and writing emails
– Drafting blog posts in WordPress and Tumblr (this post is, in fact, being written on the WordPress app)
– Writing in online forums and posting bug reports for apps
– Instant Messaging (AIM)
– Searching for web content on Google and Wikipedia
– Taking photos and posting to Flickr
– Posting events on my calendar
– Checking the weather
– Ordering dinner (Seamlessweb)
– Ordering groceries (FreshDirect)
– Setting timers (post-surgery medication and exercise schedules)
– Reading and writing messages to friends on Facebook
– Trying out new native apps and web apps

I feel kind of like an expert at ingesting and creating content on a “tablet” computer. An iPhone truly is a mini-tablet.

Knowing that the iPad release is just around the corner, I’m somewhat upset that I don’t have one because a lot of this would be a lot easier on a larger device. Additionally, the suggestive video of a Chrome OS tablet is also intruiging. While a number of the tasks I listed above are being done within iPhone apps, I believe that nearly all of them could be done from web applications.

So here are some thoughts about the experience of doing all of this on an iPhone and what I’d like to see (or not see) in an iPad.

1. Web browsing. I would characterize web browsing on the iPhone as very good. You can san whole pages and zoom in on portions you care about. I can get to about 85% of what I’m interested in right from Safari.

There are some downsides.

Some sites don’t scale well when zooming. The text is just too small when the column is spread full width, either in portrait or landscape. My workaround is to send these pages to Instapaper and allow that app to reformat the text for easier reading.

Sites that use Ajax and fancy UI designed for desktop browsing don’t always translate well in Mobile Safari. The touch interface and the select/copy tools on the iPhone often get triggered inappropriately when using some of these interfaces.

Even with zooming, text is often too small. Horizontal scrolling back and forth to read an article is downright annoying.

Most of what I do on my phone is done in the browser, so I look forward to a larger display that will make it easier to read whole pages at a time without zooming in and out.

2. Missing Flash Content. “I wouldn’t say I’ve been missing it, Bob.” Sure, it’s kind of annoying that a lot of articles I’m reading embed a video player written in Flash. It’s actually a little frustrating to me that a core use of Flash for so many sites is simply to deliver videos. The same uploaders, transcoders, storage services, and video players have been written by so many sites. It really is time for video playback directly in the browser. So I think the web is at a growing pain. It may suck for a short while as so many Flash-based videos will be inaccessible to iPhones and iPads, but the prototype sites from YouTube and Vimeo show promise of in-browser video playback for HTML5.

I’ll admit there are still some areas, especially gaming, where Flash is a solid platform. As for so called Rich Internet Applications, I think most developers would agree that Ajax toolsets make these more capable directly in the browser today.

3. Battery Life. Battery life on the iPhone has always been pretty bad. If you have an iPhone, then you’ve trained yourself to recharge your phone every morning or evening. It’s simply a habit. When you’re using your phone for 7 hours a day, then you’re charging it a few times a day. A battery life at least three times as long as this phone would be appropriate.

4. Watching Video. There are probably a lot of conference presentations I would consider watching from my iPhone which I’m not doing now. I think this relates most directly to the battery life issue. My guess is that many (but not all) are already available for viewing on the YouTube app for the iPhone.

5. Multi-tasking. Gotta have it. One thing I’ll say that I like about iPhone apps is that they are fullscreen and keep your focus. It would be nice in a larger display to be able to run two apps side by side, but I don’t know that I want user-sizable windows.

What I do miss is having an IM client running in the background so I can be reached by people I chat with while working.

6. Webcam or Camera. I could care less about both of these. I can’t see aiming a device the size of a textbook at someone to take a picture. And as for the webcam, I agree it would be convenient, but picture the following case. You are relaxing in your favorite easy chair reading the news on your tablet which is in your hands but resting on your legs. You get invited to a video chat and accept. The way I see it, from the position you are sitting the webcam on your tablet will have a great shot straight up your nostrils. This is not the image that sells video conferencing on tablet devices.

7. Virtual Keyboard. I tend to think the iPhone keyboard is pretty good and easy to adjust to. I’ve written this entire post from my phone, and it likely has a few spelling mistakes to prove it.

An improvement for a larger device would be to have a more functional keyboard with command/alt/option keys. This would make SSH clients more approachable on a tablet device.

8. Functioning “File” Form Controls. One thing I hate on the iPhone is signing up for some new online app ad not being able to do something simple like uploading an avatar. Would be great to see file fields in Mobile Safari that could pull from files stored by various apps (camera roll photos, documents, etc.)

Decision time.

I’m very excited about tablet computers and expect that unless there is a good looking product announced prior to the ipad release, I’m 90% certain I’ll buy one.

There are still some open questions about iPad functionality that may be addressed prior to it’s release. How exactly do you get files in and out of it -all via iTunes sync? Will there be background apps in the next iPhone OS and will that roll out onto the iPad immediately?

But even if the product is not perfect out of the gate, Apple has already demonstrated a successful model of revising and deploying updates. Like the iPhone, the iPad is a product that will continually improve while you own it.

Tags: , , , ,

Reaction to PPK’s “The iPhone obsession”

Posted by Mike Brittain on February 08, 2010
Mobile / Comments Off

PPK wrote a thought provoking post drawing a parallel between iPhone-specific mobile development and IE6-specific development we saw begin a decade ago and is still, somewhat surprisingly, biting us in the ass.  It’s an important warning to heed, but I think he misses some critical points about mobile web usage trends and dwells too much on a snapshot of current statistics about mobile handset market share.

Why are developers obsessed with the iPhone?

Let’s not mince words.  Mobile web browsing has traditionally been a horrible experience.  I’ve had a couple of what seemed like great smartphones (a BlackBerry 8700 and a Palm Treo 550).  They made me feel like I could use the web, and were better than nothing, but they were still a terrible experience when compared with today’s mobile web.  To deal with the fact that these handsets couldn’t consume all of the image formats, plugins, or even standard HTML produced for the web, the browsers either munged their rendering or made use of proxies that would reformat a web site’s HTML content to be more WAP-ready.  Transfer speeds were also doggedly slow.  It made much of the web accessible, but was a rough ride.

It’s not even worth talking about the people who were making mobile-friendly sites three years ago.  It’s not to say they didn’t exist, but the number of sites customized for mobile browsers at that point is negligible.

The iPhone coming to market changed the game.  Why?  Because Mobile Safari allowed users to see real web sites and not dumbed down version of web pages (well, except for Flash).  Additionally, AT&T contracts for iPhones require data plans.  Most smartphones on the market can be purchased with a contract that doesn’t include web data usage.   I just priced out BlackBerry Tour and Droid Eris (HTC) phones on Verizon and their base package includes no data usage.  You pay an additional $1.99/MB for data transfer.  BlackBerry handsets are huge in the U.S., but they’re simply not for web usage, they’re purchase primarily for email and texting.  (I believe this will change, but RIM needs to massively overhaul their browser architecture to catch up to the current state of mobile browsers being released on other phones.)

The graph below demonstrates the effect of the iPhone on mobile web usage (global).  Click for full size.

Source: Quantcast 2009 Mobile Web Trends Report

I am not advocating that we only design for the iPhone.  Newer mobile operating systems are incorporating WebKit into their default offerings.  Even though the versions of WebKit released on iPhone, Android, Symbian, Palm, etc. differ, they are all starting miles ahead of the mobile browsers from two or three years ago.  If PPK is concerned about developers favoring a technology (as they did IE6), should we be concerned about a dominance of WebKit and an ignorance of everything else?

Trends for mobile web usage

Another problem I have with the article is the statistics that were chosen.  There is too much focus on mobile handset/OS market share and not on mobile web usage.  All of the statistics that PPK chose to show are a snapshot of the current market.  If a web developer took this sort of approach to their web development strategy, they may have concluded in April 2009 that “Internet Explorer 8 only has 3.6% market share, let’s not worry too much about it.” (Source: Net Applications)

Mobile handset/OS market share has no correlation to mobile web usage.  Looking at the stats presented in PPK’s post, Symbian commands 45% of the smartphone market share (I’m fairly sure this is global market share, he made a point of not allowing developers to focus on the U.S. alone).  Contrast that with Quantcast’s 2009 Mobile Web Usage Report which shows that Symbian makes up just over 3% of global mobile web usage (all handsets included).

This graph, again from Quantcast’s 2009 report, demonstrates the mobile web market share (global) of the top handset operating systems. Click to enlarge.

Source: Quantcast 2009 Mobile Web Trends Report

Bottom line: The mobile web is tiny

There’s one more point I want to make, also supported by Quantcast’s report.  Statistics on mobile phone usage and mobile web penetration by market can get pretty heady.  It’s exciting to see all of this growth in mobile.  But the fact is that despite all of the people in the world walking around with their mobile phones, they’re not yet consuming very much of the web.  According to Quantcast, by the end of 2009 mobile web usage accounts for just 0.95% of total web usage worldwide.

Much respect

To be clear, my intention is not to bash PPK. His research, code samples, and compatibility tables are invaluable to the web development community.  I know I’ve used them countless times.  Many thanks!

I also agree with his overall rant, which is that building iPhone-specific sites is a dumb approach.  I hope to write more on that soon.

Additional resources and data

I’ve put a lot of emphasis on numbers I found in a Quantcast report.  Their findings are based on a pixel-based measurement of web usage and report only from sites where their instrumentation is deployed. Concerns I have are whether their measurements are effective on all mobile browsers (including those using transcoding proxies), and whether their measurements would be skewed by higher deployment of their pixels on sites in one global region over others.  It seems, however, that most of the best known operating systems are represented.

I didn’t call out any of the numbers from the AdMob report (listed below) because I don’t believe an ad serving network would have solid coverage of the mobile web, especially not on a global scale.

Below are some additional resources worth reviewing which I read while researching these trends.

Mobile Metrics Report, Dec 2009 – AdMob

State of the Mobile Web, Sept 2009 – Opera

The Mobile Internet Report, Dec 2009 – Morgan Stanley

Tags: , , , , , , , , , ,

Mobile Sites Outpace Native Apps… But What About Web Apps?

Posted by Mike Brittain on February 05, 2010
Mobile / Comments Off

I was excited to read this article about mobile web sites growing faster than native apps, a trend I believe will continue in the future.

Taptu estimates that there are 326,000 Mobile Touch Web sites worldwide, which they say compares to 148,000 iPhone apps in the App Store and 24,000 apps in the Android market. Taptu expects the browser-based mobile web market to grow much faster than the app market.

I’m excited about browser-based apps and the potential of HTML5 as a platform for cross-platform app development.  It’s too bad there’s not a distinction in these numbers between web applications and content sites that simply have layouts customized for display on mobile devices.

Mobile-Friendly, Web Apps, and Native Apps

When the iPhone was launched, Safari became the first mobile browser to really represent web sites as you would see them on a desktop browser.  Before then, surfing from your phone’s browser was a clunky experience with unsupported HTML and CSS features, and web proxies that would reformat or strip out content deemed unfriendly for the mobile browser.  On rare occasions, you might stumble into a site that would push your phone’s browser to a version of the site formatted specifically for display on a small screen.  Today, the term “web app” gets confused with “mobile-friendly” web sites.  I say they are wholly different products.

Native apps are touted for their advantages over web-based apps, which include: 1. better exposure to hardware- or platform-specific APIs, 2. potential for instant revenue streams (payment via app stores), and 3. ability to operate in an offline capacity.

It’s difficult to argue against native apps and games that tie into platform-specific APIs unavailable to web apps.  However, there are plenty of services available for making payments to web sites, but developers need to keep in mind their competition in native apps.  You can’t charge $5/month for access to your web app when a similar app sells for a one-time $0.99 fee through an app store.  Finally, the point of offline access is fairly ironic given that so many native apps are simply wrappers for network APIs and cannot run without a network connection.

So when I talk about web apps, I don’t just mean a mobile-friendly web site.  Rather, I’m talking about the potential to build rich applications that can run in a browser.  Using HTML5 capabilities like offline data storage and object caches, web apps can mimic many of the same features built in native apps but have the ability to reach across platforms and devices.

My own definition of web app an application that:

  1. Is written with core web technologies and runs in a browser,
  2. Provides more utility than simply reading pages of content,
  3. Is responsive to user interaction and gives the same impression that an installed app gives, and
  4. Is device and platform independent, indeed it should be irrelevant whether it is displayed on a mobile phone, a desktop computer, or any other device.

The Web App Findability Problem

Platform-specific app stores have risen up as clearing houses for apps. Users are being conditioned that the term “app” means something you download and install on your phone.  App stores are backed by big marketing dollars and ad campaigns because they provide a competitive advantage for their platform and create new revenue streams.  Web apps, on the other hand, provide neither.

There are online directories of web apps to solve this problem, but these don’t have nearly the visibility or marketing of today’s app stores.  It would be difficult for consumers to even consider web apps when they have a nice, easy shortcut to their platform’s app store built right into their phone. Even Apple, who claimed at the release of the first iPhone that web applications were enough, do not provide a means for finding web apps from their phone.  Indeed, their own web app directory doesn’t render in a mobile-friendly format.

Visibility is a key barrier for web apps.

Where Will Web Apps Get Their Exposure?

I’ve thought over a few possibilities for building the web app market, which are outlined here.

First, app stores could expand their listings to include web apps. There are already plenty of free apps in the stores.  Web apps would not create any new low price points.  I see business concerns with this direction, though.  App stores have their own services to charge for paid apps at the point of purchase.  A web app is never downloaded, so it is difficult to charge a toll for access.  Furthermore, web apps might require subscriptions or usage fees, but would be open to choose from a variety of payment gateways.  Owners of app stores would likely see this as losing out.

Search engines could provide niche listings for web applications in the same manner that news, shopping and image search reasults are handled by Google.  Search engines are supposed to provide unbiased rankings and could probably fit an app directory well.

The last idea would be to identify web applications using a new “.app” top-level domain, similar to the .mobi TLD used for mobile compatible sites.  The term “app” has already become part of the vernacular and its use as a TLD could easily identify a URL as a pointer to an application (desktop or mobile) as opposed to just mobile-friendly web pages.

Tags: , , , , , ,