Twitter’s Photo Storage (from the outside looking in)

Posted by Mike Brittain on May 24, 2010
Cloud Computing, WWW / 1 Comment

I’ve been working on some photo storage and serving problems at Etsy, which is exciting work given the number of photos we store for the items being sold on the site.  This sort of project makes you wonder how other sites are handling their photo storage and serving architectures.

Today I spent a few minutes looking at avatar photos from Twitter from the outside.  This is all from inspection of URLs and HTTP headers, and completely unofficial and unvalidated assumptions.

Two things I found interesting today were (1) the rate of new avatar photos being added to Twitter, and (2) the architecture for storing and serving images.

Avatars

Avatar photos at Twitter have URLs that look something like the following:

http://a3.twimg.com/profile_images/689887435/my_photo.jpg

I’m assuming the numeric ID increments linearly with each photo that is uploaded… two images uploaded a few minutes apart showed a relatively small increase between these IDs.  I compared one of these IDs with the ID of an older avatar, along with the “Last-Modified” header that was included with its HTTP response headers:

Last-Modified Tue, 26 Feb 2008 03:15:46 GMT

Comparing these numbers shows that Twitter is currently ingesting somewhere over two million avatars per day.

Stock, or library, avatars have different URLs, meaning they are not served or stored the same way as custom avatars.  This is good because you get the caching benefits of reusing the same avatar URL for multiple users.

http://s.twimg.com/a/1274144130/images/default_profile_6_normal.png

Storage and Hosting

Running a “host” look up on the hostname of an avatar URL shows a CNAME to Akamai’s cache network:

$ host a3.twimg.com
a3.twimg.com is an alias for a3.twimg.com.edgesuite.net.
a3.twimg.com.edgesuite.net is an alias for a948.l.akamai.net.
a948.l.akamai.net has address 96.6.41.171
a948.l.akamai.net has address 96.6.41.170

If you’re familiar with Akamai’s network, you can dig into response headers that come from their cache servers.  I did a little of that, but the thing I found most interesting is that Akamai plucks avatar images from Amazon’s CloudFront service.

x-amz-id-2: NVloBPkil5u…
x-amz-request-id: 1EAA3DE5516E…
Server: AmazonS3
X-Amz-Cf-Id: 43e9fa481c3dcd79…

It’s not news that Twitter uses S3 for storing their images, but I hadn’t thought about using CloudFront (which is effectively a CDN) as an origin to another CDN.  The benefit here, aside from not pounding the crap out of S3, is that Akamai’s regional cache servers can pull avatars from CloudFront POPs that are relatively close, as opposed to reaching all the way back to a single S3 origin (such as the “US Standard Region”, which I believe has two locations in the US).  CloudFront doesn’t have nearly as many global POPs as Akamai. But using it does speed up image delivery by ensuring that Akamai’s cache servers in Asia are grabbing files from a CloudFront POP in Hong Kong or Singapore, rather than jumping across the Pacific to North America.

I suspect that Twitter racks up a reasonably large bill with Amazon by storing and serving so many files from S3 and CloudFront.  However, it takes away the burden of owning all of the hardware, bandwidth, and man power required to serve millions upon millions of images… especially when that is not a core feature of their site.

Tags: , , , , , ,

Mobile Sites Proliferate by Category

Posted by Mike Brittain on April 21, 2010
Mobile / Comments Off on Mobile Sites Proliferate by Category

An article on RWW about mobile apps and browser-based sites highlights the different concentrations in categories between native apps and mobile sites.  Native apps are heavily weighted toward games and entertainment.  Mobile web sites are heavily weighted toward shopping and social categories:

… 19% of the mobile sites measured were Shopping & Services sites; compared to 3.6% in the same category in the App Store. Content in the ‘Social’ category also has a higher chance of being a browser-based mobile site, rather than an app (12.9% to 1.7%).

Additionally, Taptu estimates that mobile site growth far outpaces the growth of native apps on any other platform, including the iPhone and App Store.

My position for a long while has been that mobile sites have much better reach due to the ability to access them from any mobile device with a decent browser, without having to download an app.  This makes cross-platform development much easier for existing web teams.  As more mobile platforms take up the WebKit rendering engine, including this week’s report that BlackBerry 6.0 will include a touch browser backed by WebKit, the baseline for development across the myriad of mobile devices is actually much better than what we had with the first web browsers in the late ’90s.

Still, the perception by consumers that apps are hip, as well as aggressive app-centric marketing by carriers, sets a higher barrier for consumers to understand the wealth of mobile sites available to them on their existing handsets.  Visibility is still an issue here.

Tags: , , , ,

NBCOlympics.com Delivers Quality TV to Web Crossover

Posted by Mike Brittain on February 16, 2010
WWW / Comments Off on NBCOlympics.com Delivers Quality TV to Web Crossover

I write this because I expected it not to work. Tonight while watching coverage from the Olympics, an ad popped up promotion additional background video available online. I opened the site on my iPhone, browsed through the videos, and watched the video that was highlighted on the broadcast coverage.

Great experience. No Flash Required.

Tags: , , , ,

Batch Process Image Optimizations

Posted by Mike Brittain on February 14, 2010
WWW / 2 Comments

A couple weeks ago I wrote a post about a script I had put together for batch processing JPEGs with jpegtran.  This week I extended that script so that it handles processing GIF and PNG images as well.  It’s now a project on GitHub called “Wesley“.

Wesley is a single Perl script that you run from the command line, supplying the path to a single file name or a directory where you keep your site’s images.  If you work on a Linux or Mac development server, you can quickly run this script against all new images that you add to your site code.  Additionally, you could tie this into your build process or pre-commit hook for your preferred source control.  I haven’t spent time on this yet, but expect to add a write up on it soon.

The script strips meta data and comments from image files and tries to optimize images using lossless techniques.  You should be able to run Wesley on your images without any reduction in quality.

Wesley makes use of locally installed copies of ImageMagick, jpegtran, pngcrush, and gifsicle.  Some of these are probably already installed on your own machine (or shared hosting service).  If you are missing one or more of these packages, you can still run Wesley and it will use as many packages as you have available.

Usage

   wesley.pl  /path/to/images/

Sample Output

  ----------------------------
    Summary
  ----------------------------

  Converting the following GIFs to PNG would save additional file size.
  Bytes saved: 19173 (orig 149404, saved 12.83%)

    ./hd-sm.gif
    ./top_navigation.gif
    ./logo.gif

  Inspected 226 JPEG files.
  Modified 190 files.
  Huffman table optimizations: 138
  Progressive JPEG optimizations: 52
  Bytes saved: 408508 (orig 2099658, saved 19.45%)

  Inspected 105 PNG files.
  Modified 99 files.
  Bytes saved: 84618 (orig 315056, saved 26.85%)

  Inspected 129 GIF files.
  Modified 70 files.
  Bytes saved: 57535 (orig 1393120, saved 4.12%)

  Total bytes saved: 550661 (orig 3807834, saved 14.46%)

Tags: , , ,

What I Want in a Tablet

Posted by Mike Brittain on February 10, 2010
Gadgets / 3 Comments

For the last week I have been laid up in bed and on the couch while recovering from ACL reconstructive surgery. I would estimate that I’ve been using my iPhone an average of 7 hours a day for online tasks:

– Reading articles on newspaper sites
– Reading RSS subscriptions (in Reeder)
– Posting links to Twitter and Delicious
– Reading and writing on Twitter
– Reading and writing emails
– Drafting blog posts in WordPress and Tumblr (this post is, in fact, being written on the WordPress app)
– Writing in online forums and posting bug reports for apps
– Instant Messaging (AIM)
– Searching for web content on Google and Wikipedia
– Taking photos and posting to Flickr
– Posting events on my calendar
– Checking the weather
– Ordering dinner (Seamlessweb)
– Ordering groceries (FreshDirect)
– Setting timers (post-surgery medication and exercise schedules)
– Reading and writing messages to friends on Facebook
– Trying out new native apps and web apps

I feel kind of like an expert at ingesting and creating content on a “tablet” computer. An iPhone truly is a mini-tablet.

Knowing that the iPad release is just around the corner, I’m somewhat upset that I don’t have one because a lot of this would be a lot easier on a larger device. Additionally, the suggestive video of a Chrome OS tablet is also intruiging. While a number of the tasks I listed above are being done within iPhone apps, I believe that nearly all of them could be done from web applications.

So here are some thoughts about the experience of doing all of this on an iPhone and what I’d like to see (or not see) in an iPad.

1. Web browsing. I would characterize web browsing on the iPhone as very good. You can san whole pages and zoom in on portions you care about. I can get to about 85% of what I’m interested in right from Safari.

There are some downsides.

Some sites don’t scale well when zooming. The text is just too small when the column is spread full width, either in portrait or landscape. My workaround is to send these pages to Instapaper and allow that app to reformat the text for easier reading.

Sites that use Ajax and fancy UI designed for desktop browsing don’t always translate well in Mobile Safari. The touch interface and the select/copy tools on the iPhone often get triggered inappropriately when using some of these interfaces.

Even with zooming, text is often too small. Horizontal scrolling back and forth to read an article is downright annoying.

Most of what I do on my phone is done in the browser, so I look forward to a larger display that will make it easier to read whole pages at a time without zooming in and out.

2. Missing Flash Content. “I wouldn’t say I’ve been missing it, Bob.” Sure, it’s kind of annoying that a lot of articles I’m reading embed a video player written in Flash. It’s actually a little frustrating to me that a core use of Flash for so many sites is simply to deliver videos. The same uploaders, transcoders, storage services, and video players have been written by so many sites. It really is time for video playback directly in the browser. So I think the web is at a growing pain. It may suck for a short while as so many Flash-based videos will be inaccessible to iPhones and iPads, but the prototype sites from YouTube and Vimeo show promise of in-browser video playback for HTML5.

I’ll admit there are still some areas, especially gaming, where Flash is a solid platform. As for so called Rich Internet Applications, I think most developers would agree that Ajax toolsets make these more capable directly in the browser today.

3. Battery Life. Battery life on the iPhone has always been pretty bad. If you have an iPhone, then you’ve trained yourself to recharge your phone every morning or evening. It’s simply a habit. When you’re using your phone for 7 hours a day, then you’re charging it a few times a day. A battery life at least three times as long as this phone would be appropriate.

4. Watching Video. There are probably a lot of conference presentations I would consider watching from my iPhone which I’m not doing now. I think this relates most directly to the battery life issue. My guess is that many (but not all) are already available for viewing on the YouTube app for the iPhone.

5. Multi-tasking. Gotta have it. One thing I’ll say that I like about iPhone apps is that they are fullscreen and keep your focus. It would be nice in a larger display to be able to run two apps side by side, but I don’t know that I want user-sizable windows.

What I do miss is having an IM client running in the background so I can be reached by people I chat with while working.

6. Webcam or Camera. I could care less about both of these. I can’t see aiming a device the size of a textbook at someone to take a picture. And as for the webcam, I agree it would be convenient, but picture the following case. You are relaxing in your favorite easy chair reading the news on your tablet which is in your hands but resting on your legs. You get invited to a video chat and accept. The way I see it, from the position you are sitting the webcam on your tablet will have a great shot straight up your nostrils. This is not the image that sells video conferencing on tablet devices.

7. Virtual Keyboard. I tend to think the iPhone keyboard is pretty good and easy to adjust to. I’ve written this entire post from my phone, and it likely has a few spelling mistakes to prove it.

An improvement for a larger device would be to have a more functional keyboard with command/alt/option keys. This would make SSH clients more approachable on a tablet device.

8. Functioning “File” Form Controls. One thing I hate on the iPhone is signing up for some new online app ad not being able to do something simple like uploading an avatar. Would be great to see file fields in Mobile Safari that could pull from files stored by various apps (camera roll photos, documents, etc.)

Decision time.

I’m very excited about tablet computers and expect that unless there is a good looking product announced prior to the ipad release, I’m 90% certain I’ll buy one.

There are still some open questions about iPad functionality that may be addressed prior to it’s release. How exactly do you get files in and out of it -all via iTunes sync? Will there be background apps in the next iPhone OS and will that roll out onto the iPad immediately?

But even if the product is not perfect out of the gate, Apple has already demonstrated a successful model of revising and deploying updates. Like the iPhone, the iPad is a product that will continually improve while you own it.

Tags: , , , ,

Reaction to PPK’s “The iPhone obsession”

Posted by Mike Brittain on February 08, 2010
Mobile / Comments Off on Reaction to PPK’s “The iPhone obsession”

PPK wrote a thought provoking post drawing a parallel between iPhone-specific mobile development and IE6-specific development we saw begin a decade ago and is still, somewhat surprisingly, biting us in the ass.  It’s an important warning to heed, but I think he misses some critical points about mobile web usage trends and dwells too much on a snapshot of current statistics about mobile handset market share.

Why are developers obsessed with the iPhone?

Let’s not mince words.  Mobile web browsing has traditionally been a horrible experience.  I’ve had a couple of what seemed like great smartphones (a BlackBerry 8700 and a Palm Treo 550).  They made me feel like I could use the web, and were better than nothing, but they were still a terrible experience when compared with today’s mobile web.  To deal with the fact that these handsets couldn’t consume all of the image formats, plugins, or even standard HTML produced for the web, the browsers either munged their rendering or made use of proxies that would reformat a web site’s HTML content to be more WAP-ready.  Transfer speeds were also doggedly slow.  It made much of the web accessible, but was a rough ride.

It’s not even worth talking about the people who were making mobile-friendly sites three years ago.  It’s not to say they didn’t exist, but the number of sites customized for mobile browsers at that point is negligible.

The iPhone coming to market changed the game.  Why?  Because Mobile Safari allowed users to see real web sites and not dumbed down version of web pages (well, except for Flash).  Additionally, AT&T contracts for iPhones require data plans.  Most smartphones on the market can be purchased with a contract that doesn’t include web data usage.   I just priced out BlackBerry Tour and Droid Eris (HTC) phones on Verizon and their base package includes no data usage.  You pay an additional $1.99/MB for data transfer.  BlackBerry handsets are huge in the U.S., but they’re simply not for web usage, they’re purchase primarily for email and texting.  (I believe this will change, but RIM needs to massively overhaul their browser architecture to catch up to the current state of mobile browsers being released on other phones.)

The graph below demonstrates the effect of the iPhone on mobile web usage (global).  Click for full size.

Source: Quantcast 2009 Mobile Web Trends Report

I am not advocating that we only design for the iPhone.  Newer mobile operating systems are incorporating WebKit into their default offerings.  Even though the versions of WebKit released on iPhone, Android, Symbian, Palm, etc. differ, they are all starting miles ahead of the mobile browsers from two or three years ago.  If PPK is concerned about developers favoring a technology (as they did IE6), should we be concerned about a dominance of WebKit and an ignorance of everything else?

Trends for mobile web usage

Another problem I have with the article is the statistics that were chosen.  There is too much focus on mobile handset/OS market share and not on mobile web usage.  All of the statistics that PPK chose to show are a snapshot of the current market.  If a web developer took this sort of approach to their web development strategy, they may have concluded in April 2009 that “Internet Explorer 8 only has 3.6% market share, let’s not worry too much about it.” (Source: Net Applications)

Mobile handset/OS market share has no correlation to mobile web usage.  Looking at the stats presented in PPK’s post, Symbian commands 45% of the smartphone market share (I’m fairly sure this is global market share, he made a point of not allowing developers to focus on the U.S. alone).  Contrast that with Quantcast’s 2009 Mobile Web Usage Report which shows that Symbian makes up just over 3% of global mobile web usage (all handsets included).

This graph, again from Quantcast’s 2009 report, demonstrates the mobile web market share (global) of the top handset operating systems. Click to enlarge.

Source: Quantcast 2009 Mobile Web Trends Report

Bottom line: The mobile web is tiny

There’s one more point I want to make, also supported by Quantcast’s report.  Statistics on mobile phone usage and mobile web penetration by market can get pretty heady.  It’s exciting to see all of this growth in mobile.  But the fact is that despite all of the people in the world walking around with their mobile phones, they’re not yet consuming very much of the web.  According to Quantcast, by the end of 2009 mobile web usage accounts for just 0.95% of total web usage worldwide.

Much respect

To be clear, my intention is not to bash PPK. His research, code samples, and compatibility tables are invaluable to the web development community.  I know I’ve used them countless times.  Many thanks!

I also agree with his overall rant, which is that building iPhone-specific sites is a dumb approach.  I hope to write more on that soon.

Additional resources and data

I’ve put a lot of emphasis on numbers I found in a Quantcast report.  Their findings are based on a pixel-based measurement of web usage and report only from sites where their instrumentation is deployed. Concerns I have are whether their measurements are effective on all mobile browsers (including those using transcoding proxies), and whether their measurements would be skewed by higher deployment of their pixels on sites in one global region over others.  It seems, however, that most of the best known operating systems are represented.

I didn’t call out any of the numbers from the AdMob report (listed below) because I don’t believe an ad serving network would have solid coverage of the mobile web, especially not on a global scale.

Below are some additional resources worth reviewing which I read while researching these trends.

Mobile Metrics Report, Dec 2009 – AdMob

State of the Mobile Web, Sept 2009 – Opera

The Mobile Internet Report, Dec 2009 – Morgan Stanley

Tags: , , , , , , , , , ,

Mobile Sites Outpace Native Apps… But What About Web Apps?

Posted by Mike Brittain on February 05, 2010
Mobile / Comments Off on Mobile Sites Outpace Native Apps… But What About Web Apps?

I was excited to read this article about mobile web sites growing faster than native apps, a trend I believe will continue in the future.

Taptu estimates that there are 326,000 Mobile Touch Web sites worldwide, which they say compares to 148,000 iPhone apps in the App Store and 24,000 apps in the Android market. Taptu expects the browser-based mobile web market to grow much faster than the app market.

I’m excited about browser-based apps and the potential of HTML5 as a platform for cross-platform app development.  It’s too bad there’s not a distinction in these numbers between web applications and content sites that simply have layouts customized for display on mobile devices.

Mobile-Friendly, Web Apps, and Native Apps

When the iPhone was launched, Safari became the first mobile browser to really represent web sites as you would see them on a desktop browser.  Before then, surfing from your phone’s browser was a clunky experience with unsupported HTML and CSS features, and web proxies that would reformat or strip out content deemed unfriendly for the mobile browser.  On rare occasions, you might stumble into a site that would push your phone’s browser to a version of the site formatted specifically for display on a small screen.  Today, the term “web app” gets confused with “mobile-friendly” web sites.  I say they are wholly different products.

Native apps are touted for their advantages over web-based apps, which include: 1. better exposure to hardware- or platform-specific APIs, 2. potential for instant revenue streams (payment via app stores), and 3. ability to operate in an offline capacity.

It’s difficult to argue against native apps and games that tie into platform-specific APIs unavailable to web apps.  However, there are plenty of services available for making payments to web sites, but developers need to keep in mind their competition in native apps.  You can’t charge $5/month for access to your web app when a similar app sells for a one-time $0.99 fee through an app store.  Finally, the point of offline access is fairly ironic given that so many native apps are simply wrappers for network APIs and cannot run without a network connection.

So when I talk about web apps, I don’t just mean a mobile-friendly web site.  Rather, I’m talking about the potential to build rich applications that can run in a browser.  Using HTML5 capabilities like offline data storage and object caches, web apps can mimic many of the same features built in native apps but have the ability to reach across platforms and devices.

My own definition of web app an application that:

  1. Is written with core web technologies and runs in a browser,
  2. Provides more utility than simply reading pages of content,
  3. Is responsive to user interaction and gives the same impression that an installed app gives, and
  4. Is device and platform independent, indeed it should be irrelevant whether it is displayed on a mobile phone, a desktop computer, or any other device.

The Web App Findability Problem

Platform-specific app stores have risen up as clearing houses for apps. Users are being conditioned that the term “app” means something you download and install on your phone.  App stores are backed by big marketing dollars and ad campaigns because they provide a competitive advantage for their platform and create new revenue streams.  Web apps, on the other hand, provide neither.

There are online directories of web apps to solve this problem, but these don’t have nearly the visibility or marketing of today’s app stores.  It would be difficult for consumers to even consider web apps when they have a nice, easy shortcut to their platform’s app store built right into their phone. Even Apple, who claimed at the release of the first iPhone that web applications were enough, do not provide a means for finding web apps from their phone.  Indeed, their own web app directory doesn’t render in a mobile-friendly format.

Visibility is a key barrier for web apps.

Where Will Web Apps Get Their Exposure?

I’ve thought over a few possibilities for building the web app market, which are outlined here.

First, app stores could expand their listings to include web apps. There are already plenty of free apps in the stores.  Web apps would not create any new low price points.  I see business concerns with this direction, though.  App stores have their own services to charge for paid apps at the point of purchase.  A web app is never downloaded, so it is difficult to charge a toll for access.  Furthermore, web apps might require subscriptions or usage fees, but would be open to choose from a variety of payment gateways.  Owners of app stores would likely see this as losing out.

Search engines could provide niche listings for web applications in the same manner that news, shopping and image search reasults are handled by Google.  Search engines are supposed to provide unbiased rankings and could probably fit an app directory well.

The last idea would be to identify web applications using a new “.app” top-level domain, similar to the .mobi TLD used for mobile compatible sites.  The term “app” has already become part of the vernacular and its use as a TLD could easily identify a URL as a pointer to an application (desktop or mobile) as opposed to just mobile-friendly web pages.

Tags: , , , , , ,

Browser Testing for Mobile Web Applications

Posted by Mike Brittain on January 31, 2010
Misc / Comments Off on Browser Testing for Mobile Web Applications

Developers working on mobile web apps need to be able to test their apps or sites in all of the major mobile platforms.  Unfortunately, there are not a lot of good resources online for how to go about this.  You could pay for a service from a mobile testing company, like DeviceAnywhere, who provides access to a wide selection of real devices (using a virtual client).  Or you could go the free route by installing a variety of SDKs and mobile phone simulators.

Base Setup

I’m developing on a Mac.  Many of these emulators are Windows-specific.  Additionally, this could end up being a lot of work to setup, so putting all of these tools onto a Windows virtual machine will let me move the VM to another machine in the future and save me from reinstalling from scratch.  On my Mac I’m running Parallels Desktop with a copy of Windows XP.

The only exception is the iPhone SDK which includes a simulator for the iPhone.  The iPhone SDK is only available for installation on the Mac OS and I’ll be leaving it out of the instructions below.  I also haven’t installed or tested the 3.2 (beta) yet which includes an iPad simulator.

Java

Big surprise that Java is used for some of these SDKs and simulators, so we might as well get started by installing the JDK and runtime if you don’t already have it installed.  You should be able to download a copy of the JDK from here:

http://java.sun.com/javase/

I grabbed “JDK 6 Update 18”.

Android

Download the Android SDK from Google’s Android Developer site and run the enclosed SDK Setup program.

Unzip the .zip package you download and put in a location you want to keep the files (perhaps within Program Files) and then run the SDK Setup program.

In the “Installed Packages” section of the setup program, click “Update All…” to download the platforms and APIs that will be run by the SDK.

To create an emulator for a specific version of the Android OS, select the “Virtual Devices” option, then click the “New…” button.  In the dialog box that opens, enter a name for your emulator and select a target OS (e.g. “Android 2.0.1”).  Then click the “Create AVD” button.  Select your new emulator from the list and click the “Start” button.  When the emulator starts you’ll find an icon for “browser” on the main screen.

Symbian

In the past, I have setup Symbian emulators and SDKs to do local testing.  When I returned to their site to download new software I was pleased to find that they now provide a service for accessing virtual devices over the Internet using a Java application.  Visit http://apu.ndhub.net/ to register and access a wide selection of devices.

Easy!

BlackBerry

BlackBerry simulators for various models and OS versions can be downloaded from their developer site.  Each simulator is downloaded as its own installer package.  So download all of the  emulators you want to test (Storm, Bold, Pearl, etc.) and run their installers.

http://na.blackberry.com/eng/developers/resources/simulators.jsp

This seems simple enough, but if you start up one of these simulators and open the web browser you’ll quickly find that you’re missing a critical piece: network access.  To access the web from these emulators you need to also download the MDS Services Simulator package.  Find a link for this download from the resources page:

http://na.blackberry.com/eng/developers/resources/

Also note that BlackBerry makes available some documentation specific to web application development for the BlackBerry platform.  You can find these resources at the address below (registration required):

http://na.blackberry.com/eng/developers/browserdev/

Palm webOS SDK

The Palm SDK includes a web browser within its phone simulator which is useful for testing the browser that runs on the Palm Pre and Palm Pixi.

Go to Palm’s developer site and look for a button or link to download the SDK.

The emulator runs within Sun’s VirtualBox software.  There is a link from the Palm download site for downloading VirtualBox.  Follow the instructions and install VirtualBox first.

Next, download and install the webOS SDK.

Once you’ve completed the installation, you can start up Palm emulator from your Start Menu: Programs > Palm > SDK > Palm Emulator.

When you run VirtualBox, it may prompt you to download an updated version.  The Palm Emulator (as of January 30, 2010) will not run on the latest version of Virtual Box.  Stick to the version that you download from the Palm Developer web site (3.0.10).

Wrap up

This should get your started with a testing environment for a few of the top mobile browsers on the market today.

If you’ve got other tips to share about testing mobile browsers, share them in the comments below.

Tags: , , , , , , , , , , , ,

A Web Without Flash?

Posted by Mike Brittain on January 31, 2010
WWW / 1 Comment

Scoble has an interesting post asking Can Flash Be Saved? His central point is that developers will make use of whatever technologies that are widely available to their users which, of course, makes sense.  He draws an analogy between the growth of the iPhone OS, which does not include Flash, and Firefox, which does not include Microsoft-centric technologies like ActiveX.  Within the first years that Firefox began to steal share from Internet Explorer in the consumer market, developers changed the manner that they coded their sites by embracing Web Standards.

The interesting thing about that change was that Firefox’s growth was fairly slow outside of the developer community.  Many developers loved Firefox, but it took a few years for the general public to get hip to the browser.  I would have to say that the iPhone has far higher brand recognition and uptake.  You could attribute this to Apple’s strong marketing machine.

Mobile web browsing is still a small market compared to desktop browsers.  It’s a tiny, but growing market that can’t be ignored.  I believe that in just a few years mobile Internet services (including networked apps)  will be more important than the desktop Internet for communication, entertainment, and consuming information.  Devices like the iPhone and iPad are driving this forward.  Android is growing and finally seeing a wider number of devices offering Google’s mobile OS.  BlackBerry is a widely used platform, but it doesn’t contribute much to mobile web usage (yet).

If these (iPhone and Android) operating systems become the leading platforms in the mobile space, developers will build sites and apps based on the technologies available across them, namely HTML5 support that is part of the WebKit engine.  If Flash continues to be blockaded from the iPhone OS, developers will certainly look to other technologies to replace it.

Tags: , , , , ,

Batch Processing your JPEGs with jpegtran

Posted by Mike Brittain on January 27, 2010
WWW / 1 Comment

UPDATE: Please read my post about a new version of this image processing script.

Stoyan Stefanov wrote up a nice post last year about installing jpegtran on a Mac or Unix/Linux system so that you can run optimizations on your JPEG files.  His conclusion on jpegtran is that you can save about 10% on your JPEG file sizes for “about a minute of work, or less.”

Sounds great!  I looked it over and, indeed, jpegtran cuts some of the junk out of the JPEG files I tested.  The only holdup, however, is that at CafeMom we have a few thousand JPEG files in our site code, and that number grows every week.  The only reasonable solution was to automate this process.

The following Perl script should work right out of the box for you, assuming you already have jpegtran installed on your server or shared hosting account.

optimize_jpegs.pl

#!/usr/bin/perl

#
# Lossless optimization for all JPEG files in a directory
#
# This script uses techniques described in this article about the use
# of jpegtran: http://www.phpied.com/installing-jpegtran-mac-unix-linux/
#

use strict;
use File::Find;
use File::Copy;

# Read image dir from input
if (!$ARGV[0]) {
    print "Usage: $0 path_to_images\n";
    exit 1;
}
my @search_paths;
my $images_path = $ARGV[0];
if (!-e $images_path) {
    print "Invalid path specified.\n";
    exit 1;
} else {
    push @search_paths, $ARGV[0];
}

# Compress JPEGs
my $count_jpegs = 0;
my $count_modified = 0;
my $count_optimize = 0;
my $count_progressive = 0;
my $bytes_saved = 0;
my $bytes_orig = 0;

find(\&jpegCompress, @search_paths);

# Write summary
print "\n\n";
print "----------------------------\n";
print "  Sumary\n";
print "----------------------------\n";
print "\n";
print "  Inspected $count_jpegs JPEG files.\n";
print "  Modified $count_modified files.\n";
print "  Huffman table optimizations: $count_optimize\n";
print "  Progressive JPEG optimizations: $count_progressive\n";
print "  Total bytes saved: $bytes_saved (orig $bytes_orig, saved "
       . (int($bytes_saved/$bytes_orig*10000) / 100) . "%)\n";
print "\n";

sub jpegCompress()
{
    if (m/\.jpg$/i) {
        $count_jpegs++;

        my $orig_size = -s $_;
        my $saved = 0;

        my $fullname = $File::Find::dir . '/' . $_;

        print "Inspecting $fullname\n";

        # Run Progressive JPEG and Huffman table optimizations, then inspect
		# which was best.
        `/usr/bin/jpegtran -copy none -optimize $_ > $_.opt`;
        my $opt_size = -s "$_.opt";

        `/usr/bin/jpegtran -copy none -progressive $_ > $_.prog`;
        my $prog_size = -s "$_.prog";

        if ($opt_size && $opt_size < $orig_size && $opt_size <= $prog_size) {
            move("$_.opt", "$_");
            $saved = $orig_size - $opt_size;
            $bytes_saved += $saved;
            $bytes_orig += $orig_size;
            $count_modified++;
            $count_optimize++;

            print " -- Huffman table optimization: "
				. "saved $saved bytes (orig $orig_size)\n";

        } elsif ($prog_size && $prog_size < $orig_size) {
            move("$_.prog", "$_");
            $saved = $orig_size - $prog_size;
            $bytes_saved += $saved;
            $bytes_orig += $orig_size;
            $count_modified++;
            $count_progressive++;

            print " -- Progressive JPEG optimization: "
				. "saved $saved bytes (orig $orig_size)\n";
        }

        # Cleanup temp files
        if (-e "$_.prog") {
             unlink("$_.prog");
        }
        if (-e "$_.opt") {
            unlink("$_.opt");
        }
    }
}

How to use this script

For starters, copy this script into a text file (such as optimize_jpegs.pl) and set it to be executable (chmod 755 optimize_jpegs.pl).

After the script is setup, pull the trigger…

$ ./optimize_jpegs.pl  /path/to/your/images/dir

That’s it.  The output should look something like this:

Inspecting ./phpXkWlcW.jpg
 -- Progressive JPEG optimization: saved 1089 bytes (orig 13464)
Inspecting ./phpCnBRri.jpg
 -- Progressive JPEG optimization: saved 1155 bytes (orig 34790)
Inspecting ./phpx6G3lD.jpg
 -- Progressive JPEG optimization: saved 742 bytes (orig 11493)

...

----------------------------
  Sumary
----------------------------

  Inspected 21 JPEG files.
  Modified 21 files.
  Huffman table optimizations: 0
  Progressive JPEG optimizations: 21
  Total bytes saved: 63918

Wrap up

Many thanks to Stoyan for his post on jpegtran, and all of the other performance ideas he has been sharing on his blog.  This script was easy to write, knowing the right techniques to be running on our images.

optimize_jpegs.pl took about a minute or so to run on our thousands of images and shaved a few megabytes from all of those files combined.  This will be a great savings for us.

Tags: , , , ,