Mike Brittain

I was excited to read this article about mobile web sites growing faster than native apps, a trend I believe will continue in the future.

Taptu estimates that there are 326,000 Mobile Touch Web sites worldwide, which they say compares to 148,000 iPhone apps in the App Store and 24,000 apps in the Android market. Taptu expects the browser-based mobile web market to grow much faster than the app market.

I’m excited about browser-based apps and the potential of HTML5 as a platform for cross-platform app development.  It’s too bad there’s not a distinction in these numbers between web applications and content sites that simply have layouts customized for display on mobile devices.

Mobile-Friendly, Web Apps, and Native Apps

When the iPhone was launched, Safari became the first mobile browser to really represent web sites as you would see them on a desktop browser.  Before then, surfing from your phone’s browser was a clunky experience with unsupported HTML and CSS features, and web proxies that would reformat or strip out content deemed unfriendly for the mobile browser.  On rare occasions, you might stumble into a site that would push your phone’s browser to a version of the site formatted specifically for display on a small screen.  Today, the term “web app” gets confused with “mobile-friendly” web sites.  I say they are wholly different products.

Native apps are touted for their advantages over web-based apps, which include: 1. better exposure to hardware- or platform-specific APIs, 2. potential for instant revenue streams (payment via app stores), and 3. ability to operate in an offline capacity.

It’s difficult to argue against native apps and games that tie into platform-specific APIs unavailable to web apps.  However, there are plenty of services available for making payments to web sites, but developers need to keep in mind their competition in native apps.  You can’t charge $5/month for access to your web app when a similar app sells for a one-time $0.99 fee through an app store.  Finally, the point of offline access is fairly ironic given that so many native apps are simply wrappers for network APIs and cannot run without a network connection.

So when I talk about web apps, I don’t just mean a mobile-friendly web site.  Rather, I’m talking about the potential to build rich applications that can run in a browser.  Using HTML5 capabilities like offline data storage and object caches, web apps can mimic many of the same features built in native apps but have the ability to reach across platforms and devices.

My own definition of web app an application that:

1. Is written with core web technologies and runs in a browser,
2. Provides more utility than simply reading pages of content,
3. Is responsive to user interaction and gives the same impression that an installed app gives, and
4. Is device and platform independent, indeed it should be irrelevant whether it is displayed on a mobile phone, a desktop computer, or any other device.

The Web App Findability Problem

Platform-specific app stores have risen up as clearing houses for apps. Users are being conditioned that the term “app” means something you download and install on your phone.  App stores are backed by big marketing dollars and ad campaigns because they provide a competitive advantage for their platform and create new revenue streams.  Web apps, on the other hand, provide neither.

There are online directories of web apps to solve this problem, but these don’t have nearly the visibility or marketing of today’s app stores.  It would be difficult for consumers to even consider web apps when they have a nice, easy shortcut to their platform’s app store built right into their phone. Even Apple, who claimed at the release of the first iPhone that web applications were enough, do not provide a means for finding web apps from their phone.  Indeed, their own web app directory doesn’t render in a mobile-friendly format.

Visibility is a key barrier for web apps.

Where Will Web Apps Get Their Exposure?

I’ve thought over a few possibilities for building the web app market, which are outlined here.

First, app stores could expand their listings to include web apps. There are already plenty of free apps in the stores.  Web apps would not create any new low price points.  I see business concerns with this direction, though.  App stores have their own services to charge for paid apps at the point of purchase.  A web app is never downloaded, so it is difficult to charge a toll for access.  Furthermore, web apps might require subscriptions or usage fees, but would be open to choose from a variety of payment gateways.  Owners of app stores would likely see this as losing out.

Search engines could provide niche listings for web applications in the same manner that news, shopping and image search reasults are handled by Google.  Search engines are supposed to provide unbiased rankings and could probably fit an app directory well.

The last idea would be to identify web applications using a new “.app” top-level domain, similar to the .mobi TLD used for mobile compatible sites.  The term “app” has already become part of the vernacular and its use as a TLD could easily identify a URL as a pointer to an application (desktop or mobile) as opposed to just mobile-friendly web pages.

Developers working on mobile web apps need to be able to test their apps or sites in all of the major mobile platforms.  Unfortunately, there are not a lot of good resources online for how to go about this.  You could pay for a service from a mobile testing company, like DeviceAnywhere, who provides access to a wide selection of real devices (using a virtual client).  Or you could go the free route by installing a variety of SDKs and mobile phone simulators.

Base Setup

I’m developing on a Mac.  Many of these emulators are Windows-specific.  Additionally, this could end up being a lot of work to setup, so putting all of these tools onto a Windows virtual machine will let me move the VM to another machine in the future and save me from reinstalling from scratch.  On my Mac I’m running Parallels Desktop with a copy of Windows XP.

The only exception is the iPhone SDK which includes a simulator for the iPhone.  The iPhone SDK is only available for installation on the Mac OS and I’ll be leaving it out of the instructions below.  I also haven’t installed or tested the 3.2 (beta) yet which includes an iPad simulator.

Java

Big surprise that Java is used for some of these SDKs and simulators, so we might as well get started by installing the JDK and runtime if you don’t already have it installed.  You should be able to download a copy of the JDK from here:

http://java.sun.com/javase/

I grabbed “JDK 6 Update 18″.

Android

Download the Android SDK from Google’s Android Developer site and run the enclosed SDK Setup program.

Unzip the .zip package you download and put in a location you want to keep the files (perhaps within Program Files) and then run the SDK Setup program.

In the “Installed Packages” section of the setup program, click “Update All…” to download the platforms and APIs that will be run by the SDK.

To create an emulator for a specific version of the Android OS, select the “Virtual Devices” option, then click the “New…” button.  In the dialog box that opens, enter a name for your emulator and select a target OS (e.g. “Android 2.0.1″).  Then click the “Create AVD” button.  Select your new emulator from the list and click the “Start” button.  When the emulator starts you’ll find an icon for “browser” on the main screen.

Symbian

In the past, I have setup Symbian emulators and SDKs to do local testing.  When I returned to their site to download new software I was pleased to find that they now provide a service for accessing virtual devices over the Internet using a Java application.  Visit http://apu.ndhub.net/ to register and access a wide selection of devices.

Easy!

BlackBerry

BlackBerry simulators for various models and OS versions can be downloaded from their developer site.  Each simulator is downloaded as its own installer package.  So download all of the  emulators you want to test (Storm, Bold, Pearl, etc.) and run their installers.

http://na.blackberry.com/eng/developers/resources/simulators.jsp

This seems simple enough, but if you start up one of these simulators and open the web browser you’ll quickly find that you’re missing a critical piece: network access.  To access the web from these emulators you need to also download the MDS Services Simulator package.  Find a link for this download from the resources page:

http://na.blackberry.com/eng/developers/resources/

Also note that BlackBerry makes available some documentation specific to web application development for the BlackBerry platform.  You can find these resources at the address below (registration required):

http://na.blackberry.com/eng/developers/browserdev/

Palm webOS SDK

The Palm SDK includes a web browser within its phone simulator which is useful for testing the browser that runs on the Palm Pre and Palm Pixi.

Go to Palm’s developer site and look for a button or link to download the SDK.

The emulator runs within Sun’s VirtualBox software.  There is a link from the Palm download site for downloading VirtualBox.  Follow the instructions and install VirtualBox first.

Next, download and install the webOS SDK.

Once you’ve completed the installation, you can start up Palm emulator from your Start Menu: Programs > Palm > SDK > Palm Emulator.

When you run VirtualBox, it may prompt you to download an updated version.  The Palm Emulator (as of January 30, 2010) will not run on the latest version of Virtual Box.  Stick to the version that you download from the Palm Developer web site (3.0.10).

Wrap up

This should get your started with a testing environment for a few of the top mobile browsers on the market today.

If you’ve got other tips to share about testing mobile browsers, share them in the comments below.

Scoble has an interesting post asking Can Flash Be Saved? His central point is that developers will make use of whatever technologies that are widely available to their users which, of course, makes sense.  He draws an analogy between the growth of the iPhone OS, which does not include Flash, and Firefox, which does not include Microsoft-centric technologies like ActiveX.  Within the first years that Firefox began to steal share from Internet Explorer in the consumer market, developers changed the manner that they coded their sites by embracing Web Standards.

The interesting thing about that change was that Firefox’s growth was fairly slow outside of the developer community.  Many developers loved Firefox, but it took a few years for the general public to get hip to the browser.  I would have to say that the iPhone has far higher brand recognition and uptake.  You could attribute this to Apple’s strong marketing machine.

Mobile web browsing is still a small market compared to desktop browsers.  It’s a tiny, but growing market that can’t be ignored.  I believe that in just a few years mobile Internet services (including networked apps)  will be more important than the desktop Internet for communication, entertainment, and consuming information.  Devices like the iPhone and iPad are driving this forward.  Android is growing and finally seeing a wider number of devices offering Google’s mobile OS.  BlackBerry is a widely used platform, but it doesn’t contribute much to mobile web usage (yet).

If these (iPhone and Android) operating systems become the leading platforms in the mobile space, developers will build sites and apps based on the technologies available across them, namely HTML5 support that is part of the WebKit engine.  If Flash continues to be blockaded from the iPhone OS, developers will certainly look to other technologies to replace it.

Stoyan Stefanov wrote up a nice post last year about installing jpegtran on a Mac or Unix/Linux system so that you can run optimizations on your JPEG files.  His conclusion on jpegtran is that you can save about 10% on your JPEG file sizes for “about a minute of work, or less.”

Sounds great!  I looked it over and, indeed, jpegtran cuts some of the junk out of the JPEG files I tested.  The only holdup, however, is that at CafeMom we have a few thousand JPEG files in our site code, and that number grows every week.  The only reasonable solution was to automate this process.

The following Perl script should work right out of the box for you, assuming you already have jpegtran installed on your server or shared hosting account.

optimize_jpegs.pl

#!/usr/bin/perl

#
# Lossless optimization for all JPEG files in a directory
#
# This script uses techniques described in this article about the use
# of jpegtran: http://www.phpied.com/installing-jpegtran-mac-unix-linux/
#

use strict;
use File::Find;
use File::Copy;

# Read image dir from input
if (!$ARGV[0]) {
    print "Usage: $0 path_to_images\n";
    exit 1;
}
my @search_paths;
my $images_path = $ARGV[0];
if (!-e $images_path) {
    print "Invalid path specified.\n";
    exit 1;
} else {
    push @search_paths, $ARGV[0];
}

# Compress JPEGs
my $count_jpegs = 0;
my $count_modified = 0;
my $count_optimize = 0;
my $count_progressive = 0;
my $bytes_saved = 0;
my $bytes_orig = 0;

find(\&jpegCompress, @search_paths);

# Write summary
print "\n\n";
print "----------------------------\n";
print "  Sumary\n";
print "----------------------------\n";
print "\n";
print "  Inspected $count_jpegs JPEG files.\n";
print "  Modified $count_modified files.\n";
print "  Huffman table optimizations: $count_optimize\n";
print "  Progressive JPEG optimizations: $count_progressive\n";
print "  Total bytes saved: $bytes_saved (orig $bytes_orig, saved "
       . (int($bytes_saved/$bytes_orig*10000) / 100) . "%)\n";
print "\n";

sub jpegCompress()
{
    if (m/\.jpg$/i) {
        $count_jpegs++;

        my $orig_size = -s $_;
        my $saved = 0;

        my $fullname = $File::Find::dir . '/' . $_;

        print "Inspecting $fullname\n";

        # Run Progressive JPEG and Huffman table optimizations, then inspect
		# which was best.
        `/usr/bin/jpegtran -copy none -optimize $_ > $_.opt`;
        my $opt_size = -s "$_.opt";

        `/usr/bin/jpegtran -copy none -progressive $_ > $_.prog`;
        my $prog_size = -s "$_.prog";

        if ($opt_size && $opt_size < $orig_size && $opt_size <= $prog_size) {
            move("$_.opt", "$_");
            $saved = $orig_size - $opt_size;
            $bytes_saved += $saved;
            $bytes_orig += $orig_size;
            $count_modified++;
            $count_optimize++;

            print " -- Huffman table optimization: "
				. "saved $saved bytes (orig $orig_size)\n";

        } elsif ($prog_size && $prog_size < $orig_size) {
            move("$_.prog", "$_");
            $saved = $orig_size - $prog_size;
            $bytes_saved += $saved;
            $bytes_orig += $orig_size;
            $count_modified++;
            $count_progressive++;

            print " -- Progressive JPEG optimization: "
				. "saved $saved bytes (orig $orig_size)\n";
        }

        # Cleanup temp files
        if (-e "$_.prog") {
             unlink("$_.prog");
        }
        if (-e "$_.opt") {
            unlink("$_.opt");
        }
    }
}

How to use this script

For starters, copy this script into a text file (such as optimize_jpegs.pl) and set it to be executable (chmod 755 optimize_jpegs.pl).

After the script is setup, pull the trigger…

$ ./optimize_jpegs.pl  /path/to/your/images/dir

That’s it.  The output should look something like this:

Inspecting ./phpXkWlcW.jpg
 -- Progressive JPEG optimization: saved 1089 bytes (orig 13464)
Inspecting ./phpCnBRri.jpg
 -- Progressive JPEG optimization: saved 1155 bytes (orig 34790)
Inspecting ./phpx6G3lD.jpg
 -- Progressive JPEG optimization: saved 742 bytes (orig 11493)

...

----------------------------
  Sumary
----------------------------

  Inspected 21 JPEG files.
  Modified 21 files.
  Huffman table optimizations: 0
  Progressive JPEG optimizations: 21
  Total bytes saved: 63918

Wrap up

Many thanks to Stoyan for his post on jpegtran, and all of the other performance ideas he has been sharing on his blog.  This script was easy to write, knowing the right techniques to be running on our images.

optimize_jpegs.pl took about a minute or so to run on our thousands of images and shaved a few megabytes from all of those files combined.  This will be a great savings for us.

Just what do you suppose I should make of this user-agent string?  It’s real.  I found it in my logs today.

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

Seriously.  We see “MSIE 7.0″ in there, which you would initially think of as Internet Explorer 7.  But of course, since we also see “Trident/4.0″, we can deduce that this is Internet Explorer 8 running in Compatibility View.  So what the hell is “MSIE 6.0″ still doing in here?

P.S. Thanks for including the version of .NET CLR… 6 different ways. That certainly is helpful.

I thought this was a good article covering 8 Things to Consider When Choosing a CDN.  It’s worth a read.

The only bit I would disagree with is that this reads a bit too video-centric for me.  It felt like the main argument for using a CDN is that you don’t have enough bandwidth at your own data center to handle all of the requests being made to your servers.

I use CDNs for delivering static objects like images, CSS, and JavaScript.  An additional consideration I have is how fast cached objects will reach different locations across the country or around the world.  I’m dependent (as most sites are) on one central data center where my pages are being generated.  Every page view needs to make that trip over the network from the browser to my data center.  However, if the 20-100 successive objects can be requested from a regional CDN node, the performance in the end-user’s browser will be much better than if every request made a full trip across the country.

We’re using a few custom Munin graphs at CafeMom to monitor our application code running on production servers.  I posted two samples of these to the WebOps Visualization group at Flickr.  The first graph measures the “uptime” of our code, a measure of how many minutes it’s been since our last deployment to prod (with a max of 1 day).  The second graph provides a picture of what’s going on in our PHP error logs, highlighting spikes in notices, warnings, and fatals, as well as DB-related errors that we throw of our own.

When used together, these graphs give us quick feedback on what sort of errors are occurring on our site and whether they are likely to be related to a recent code promotion, or are the effect of some other condition (bad hardware, third-party APIs failing, SEM gone awry, etc.).

I figured someone might find these useful, so I’m posting the code for both Munin plugins.

Code Uptime

When we deploy code to our servers, we add a timestamp file that is used to manage versioning of deployments to make things like rolling back super easy.  It’s also handy for measuring how long it’s been since the last deployment.  All this plugin does is reads how long ago that file was modified.

We run multiple applications on the same clusters of servers. I wrote our code deployment process in a manner that allows for independent deployment of each application. For example, one team working on our Facebook apps can safely deploy code without interfering with the deployment schedule another team is using for new features that will be released on the CafeMom web site.

Each of these applications is deployed to a separate directory under a root, let’s say “/var/www.”  This explains why the plugin is reading a list of files (directories) under APPS_ROOT.  Each app has it’s own reported uptime on the Munin graph.

#!/bin/sh
#
# Munin plugin to monitor the relative uptime of each app
# running on the server.
#

APPS_ROOT="/var/www/"

# Cap the uptime at one day so as to maintain readable scale
MAX_MIN=1440

# Configure list of apps
if [ "$1" = "config" ]; then
 echo 'graph_title Application uptimes'
 echo "graph_args --base 1000 --lower-limit 0 --upper-limit $MAX_MIN"
 echo 'graph_scale no'
 echo 'graph_category Applications'
 echo 'graph_info Monitors when each app was last deployed to the server.'
 echo 'graph_vlabel Minutes since last code push (max 1 day)'

 for file in `ls $APPS_ROOT`; do
 echo "$file.label $file"
 done
 exit 0
fi

# Fetch release times
now_sec=`date +%s`

for file in `ls $APPS_ROOT`; do
 release_sec=`date +%s  -r $APPS_ROOT/$file/prod/release_timestamp`
 diff_sec=$(( $now_sec - $release_sec ))
 diff_min=$(( $diff_sec/60 ))
 ((diff_min>MAX_MIN?diff_min=MAX_MIN:diff_min))
 echo "$file.value $diff_min"
done

Error Logs

The second plugin uses grep to search for occurrences of specific error-related strings in our log files. In our case, the graph period was set to “minute” because that gives the best scale for us (thankfully, it’s not in errors per second!).

If you’re wondering about using grep five times against large error files every time Munin runs (every five minutes), I want to point out that we rotate our logs frequently which ensures that these calls are manageable. If you run this against very large error logs you may find gaps in your Munin graphs if the poller times out waiting for the plugin to return data points.

Even if you don’t care about PHP logs, you may find this to be a simple example of using Munin to examine any sort of log files that your application is creating.

#!/bin/bash

#
# Collect stats for the contents of PHP error logs. Measures notice,
# warning, fatal, and parse level errors, as well as custom errors
# thrown from our DB connection class.
#

logs="/var/log/httpd/*.error.log"

# CONFIG

if [ "$#" -eq "1" ] && [ "$1" = "config" ]; then
	echo "graph_title Error Logs"
	echo "graph_category applications"
	echo "graph_info Data is pooled from all PHP error logs."
	echo "graph_vlabel entries per minute"
	echo "graph_period minute"
	echo "notice.label PHP Notice"
	echo "notice.type DERIVE"
	echo "notice.min 0"
	echo "notice.draw AREA"
	echo "warning.label PHP Warning"
	echo "warning.type DERIVE"
	echo "warning.min 0"
	echo "warning.draw STACK"
	echo "fatal.label PHP Fatal"
	echo "fatal.type DERIVE"
	echo "fatal.min 0"
	echo "fatal.draw STACK"
	echo "parse.label PHP Parse"
	echo "parse.type DERIVE"
	echo "parse.min 0"
	echo "parse.draw STACK"
	echo "db.label DB Error"
	echo "db.type DERIVE"
	echo "db.min 0"
	echo "db.draw STACK"
	exit
fi

# DATA

# The perl code at the end of each line takes a list of integers (counts) from grep (one per line)
# and outputs the sum.

echo "notice.value `grep --count \"PHP Notice\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "warning.value `grep --count \"PHP Warning\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "fatal.value `grep --count \"PHP Fatal error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "parse.value `grep --count \"PHP Parse error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"
echo "db.value `grep --count \"DB Error\" $logs | cut -d':' -f2 | perl -lne ' $x += $_; END { print $x; } ' `"

I’m open to comments and suggestions on how we use these, or how they were written.  Spout off below.

I read this poll on LifeHacker the other day about web-based email vs. desktop email apps.  It reinforced what I believe is the current momentum in web applications these days — that over time, people are going to get more and more comfortable using web apps the way they used to use desktop apps.  It’s been five years since the release of Gmail, which I view as a forerunner in this area, so clearly this isn’t going to be a quick change.

Browser innovations will help users with the perception that web applications are interchangeable with desktop software.  Google Chrome is already working in this direction by reducing the amount of browser UI and allowing the user to focus on sites and apps they are using.  I’m won’t argue that Chrome will be a major browser; it may never be.  I do believe that Google’s intention is to continue swaying the way we look at the Internet. As other browsers follow suit (in some cases), Google’s web applications and sites will all benefit.

As we move forward, many computers, especially in public spaces like libraries and educational computer labs, will use fewer licensed software suites and more subscriptions to web-based applications.  As with Gmail, it doesn’t matter whether I check email on my computer, your computer, a work computer, or a mobile phone — I still have access to the application because modern web browsers provide the baseline of support for these apps.  With stabilization and implementation of features in HTML 5, additional web-based apps will be built and they will continue to look and act like our familiar desktop apps.  Over time, institutions will replace their local file servers, email servers, and parts of their IT staff with outsourced apps that are purchased by subscription and delivered in a browser.

John Dvorak has a great piece on MarketWatch right now about how Microsoft has lost sight of its core business.  He points out a number of great examples of how the company has chased after others in publishing, online networks, search, and music, while none of these areas have anything to do with Microsoft’s history and what it used to do best.  The Xbox isn’t mentioned, and while it might look like a separate business (gaming) I think it probably is pretty close to the idea of building an OS and tools for other developers to build for.  Not a far stretch from building Windows when you think about it.

Dvorak hits on all of the points that bother me about Microsoft.  Personally, I’ve felt that over the years I’ve poured a bundle of money into Windows and Office licenses (for personal and business use) and I can fathom that much of the profit that Microsoft has made on these purchases has been sunk into losing propositions.  And that makes me angry.

I used to do a decent amount of reading about e-learning during my grad school studies. I was a big fan of Clifford Stoll’s book, “Silicon Snake Oil.”. The premise was essentially that people keep trying to sell is on this idea that learning should be fun. Education can be built into some sort of game so that it becomes enjoyable and the things we dislike about learning will happen automatically while we are entertaining ourselves.

I just bumped into a great quote from Dave Benjamin that reminded me of all of this, and I wanted to share that because I think it’s important to remember when raising kids in this age:

“Learning is hard work. Making this seem untrue or avoidable is popular and lucrative. Often, the result of catering toward this interest is something other than learning.”