Effective Ajax With TurboGears

Kevin Dangoor, Blazing Things LLC

February 26, 2006

(Note that this should not be confused with [Rob Sanheim's][http://www.robsanheim.com/] Effective Ajax presentation.)

Introduction

It's been a year since Jesse James Garrett of Adaptive Path coined the term "Ajax" and nearly two years since Google showed everyone that browsers could be a lot richer than most people were taking advantage of (with Gmail). Ajax, for those not following this sort of thing, stands for Asynchronous JavaScript + XML. Ajax is all about providing a more interactive web application by having the browser request additional information without a page refresh. XML is not always used, as we'll see here, but you can't escape JavaScript and it'll definitely be asynchronous if you don't want to lock up the user's browser.

Speaking of things that you don't want to do to your users, that's what "Effective Ajax with TurboGears" is all about. Ajax is still new territory for many, and it's a wild frontier of experimentation. It seems like every day that there's a ne.w.si.te showing off some technique or other that makes your browser jump into the 21st century. Experiments are fine, but much of our time is spent making applications that are used by normal people who are just trying to get their jobs done.

This article explores some thoughts, with examples, of how you can effectively use Ajax today to provide a better user experience. In the brief space available, this article is going to completely ignore the large and important issue of accessibility. There are real concerns with Ajax and accessibility, and that is a separate topic.

Current User Expectations

People have been experimenting with Ajax-like techniques for years. I remember coming across Brent Ashley's JavaScript Remote Scripting library in 2001. But, Gmail was the application that brought the idea to the forefront and Google Maps showed that the idea was not a one-trick pony. Google developers were certainly not the only ones using this technique, but they made the biggest splash. The Adaptive Path article came at just the right time to give the technique a name, just as developers all over were starting to embrace the technique.

Depending on your target audience, it's hard to ignore Ajax for many applications these days. Even something as low-tech as reading a news story on Yahoo! has become Ajaxified, highlighting key terms and running searches in-page for you.

Yahoo! News screenshot

So, the bar has been raised. Users want Ajax, even if they're not going to ask for it by name. More specifically, though, users want good use of Ajax.

State of the Browser

If we were still living in an Internet Explorer 5/Netscape 4 world, I think that web developers would be more inclined to jump into a flaming, alligator-filled canyon than write something like Gmail.

Picture of a flaming, alligator-filled canyon

Luckily, IE6, Firefox and Safari together add up to nearly 95% of the browsers in use today. These three browsers are far more standards-compliant than their predecessors, which means that there's now a non-zero chance that code you write for one might work vaguely similarly on the others. Actually, it's a bit better than that. Many things look and work the same between those three browsers, and diligent folks working on projects like Dojo and MochiKit have encapsulated many of the differences for you.

The state of the browser is much, much better than it used to be. If you happen to be in a large corporate environment where IT has mandated that everyone must run Netscape 4.7, the canyon is right over there to your left.

Enhancer or Confuser

There are certain uses of Ajax that are pure wins. They enhance the user experience and are intuitively obvious to people, based on their prior experiences.

But, the use of Ajax is still new and things that may seem easy to you may be confusing to others. If something is a little confusing but can be a big boon to productivity, you might want to use that approach, particularly if your users are people that can be trained. A site like Amazon.com that needs to be easy for everyone will avoid confusion as much as possible. How do they know what confuses people? Amazon.com tests out their ideas to see what effect they have on sales.

Then there are things that might seem really cool, but they just confuse people without providing them with a productivity boost. Do those things if you're tired of running a profitable business or being employed.

Ajax as an Enhancer

Some Ajax is a pure win. Do more of that and people will love you for it. (Please think of me at your IPO.)

Improving Text Entry

One of Google's many perennial betas is Google Suggest. As you type into the text box, a drop-down list of search terms appears, letting you choose from the menu. Suggest even lists the number of matches for each search term, which helps give you an idea if one set of terms may be better than another for the search you're doing.

This kind of feature is definitely a user-interface enhancer. It provides the user with a convenient pick list, even among large numbers of entries. It reduces the amount of typing the user needs to do. It also reduces the amount of thinking they need to do, by avoiding possible misspellings.

And, importantly, this kind of autocompletion is something that users are already used to. This is a vitally important thing to keep in mind when you're working on a spiffy new Ajax control: if you start doing things that your users aren't used to, you've got a big education battle ahead of you. And you'll probably lose. Desktop applications already have a number of useful controls, metaphors and ideas that you can bring across to your webapps. Google Suggest's autocompletion is something that Quicken, for one, has had for years.

Of course, even when dealing with something that people are familiar with, you still need to pay attention to how you implement the feature. Thomas Baekdal points out that a live search can distract people from what they're trying to do.

Autocompletion Example

The software running the visuals for the Effective Ajax with TurboGears presentation was itself written in TurboGears. The autocompletion example performs simple searches of all of the names in the Python namespace. As of this writing, that is about 13,300 names. That's far too many for a pick list, but it works great with autocompletion.

In addition to using Ajax to perform autocompletion among those 13,000 names, this demo will also do an Ajax request to look up the stringified value of the object you pick.

Let's take a look at how the autocompletion is set up:

1class AutoCompleteExample(controllers.Controller): 
2    @expose(template="effectiveajaxturbogears.templates.symbols"
3    def index(self): 
4        searchbox = widgets.AutoCompleteField( 
5            name="symbol"
6            search_controller="symsearch",  
7            search_param="search_for"
8            result_name="symbols"
9        return dict(searchbox=searchbox, mochi=mochikit, 
10                    all_symbols=all_symbols, 
11                    prev="text-entry", next="beenthere-donethat"
view plain | print | ?

The important thing to note in this example is that we're using an AutoCompleteField widget. This type of widget is commonplace enough and important enough that TurboGears includes it out of the box. To use it, we need to do just a little bit of configuration. search_controller provides the URL (a relative URL in this case) that will perform the search for the widget. search_param specifies the name of the parameter that needs to be passed to the search_controller. result_name tells the widget what name to look up in the resulting hash table in order to find the list of matches.

The dictionary returned by the method will get passed to the template. By including the AutoCompleteField widget and the mochikit widget, the required JavaScript will automatically be included in the final pages.

Note that the "next" and "prev" values returned in the dictionary are used for navigation through the presentation visuals. Those have no function for the demo itself.

Next, we'll take a look at the template body that generates the page for this demo:

1<body bgcolor="white"
2    <h1 class="exampleheader">Python Symbol Search</h1> 
3    <p>There are $${len(all_symbols)} symbols.</p> 
4    $${searchbox.display()} 
5    <div> 
6      <input type="button" onclick="get_symbol()"  
7        value="Show Value"/> 
8    </div> 
9    <pre id="value"></pre> 
10</body> 
view plain | print | ?

In this template snippet, we see $${searchbox.display()}. That is all it takes to display the text field and the little spinner image next to it. After the autocomplete widget, we have the "Show Value" button, which calls a JavaScript function called get_symbol, which we'll look at shortly. We include an empty <pre> tag that we fill in later with the value of the Python symbol we're looking up.

1@expose(format="json"
2def symsearch(self, search_for): 
3    if len(search_for) < 3: 
4        return dict(symbols=[]) 
5    search_for = search_for.lower() 
6    matches = ["%s.%s" % (item.mod, item.name)  
7                for item in all_symbols  
8                if search_for in item.name.lower()] 
9    return dict(symbols=matches) 
view plain | print | ?

Above, we see the Python controller method that performs the search on behalf of the AutoCompleteField widget. We had told the AutoCompleteField that its search_controller is called symsearch, and you'll note that that is the name of this method. At the top, you see that the format provided for this method is "json". The dictionary we return at the end of the method is automatically turned into a JSON string. As we had specified when we created the widget, the parameter with the search string is called search_for and the result we provide in the JSON output is named "symbols". (Clearly, the "run straight through the entire list" search algorithm being used is not the most efficient. That's not the point of the demo app, though.)

The AutoCompleteField widget includes all of the JavaScript it needs to work its magic. Let's take a look at the JavaScript used to retrieve the value of the symbol:

1function get_symbol() { 
2  d = loadJSONDoc("getsym?" +  
3      queryString( 
4          {"symbol" :  
5              $("autoCompleteform_symbol").value})); 
6  d.addCallback(show_value); 
7
8 
9function show_value(result) { 
10  replaceChildNodes("value", result["value"]); 
11
view plain | print | ?

get_symbol uses MochiKit's loadJSONDoc function to call a method called getsym on the server. It looks up the value of the AutoCompleteField widget we created and passes that string over in a parameter called symbol. When the result comes back (asynchronously), MochiKit will call the show_value function.

show_value is very simple. It just uses MochiKit's replaceChildNodes function to fill in our empty <pre> tag with the stringified value of the Python object. MochiKit's loadJSONDoc function handles the conversion of the JSON string that arrives in the browser into JavaScript objects.

Finally, we see that the getsym method on the server side is very simple:

1@expose(format="json"
2def getsym(selfsymbol): 
3    ldot = symbol.rfind("."
4    mod = symbol[:ldot] 
5    var = symbol[ldot+1:] 
6    value = getattr(sys.modules[mod], var, None) 
7    return dict(value=str(value)) 
view plain | print | ?

Real Time Display Updates

The web was originally an information delivery vehicle and not an application platform. Though we think of Ajax in terms of bridging the gap between webapps and desktop apps, Ajax can also help in information delivery. The web today displays quite a bit of rapidly changing information: stock quotes, election results, news headlines, and Steve Jobs keynote addresses are just a few examples. By making requests to the server periodically, you can make portions of your page stay up to date while your reader is viewing a single item.

This technique also works for webapps. Imagine a document review application. With the proper infrastructure, you can notify the user that the document has changed while they're viewing it.

This technique is not without peril, however. For a real-world implementation, you need to carefully orchestrate these updates to not swamp your server and to not go over IE's 2 connection limit. Dojo's Alex Russell has written about the challenges of implementing low-latency update code that works within the boundaries of what a browser can do. For a departmental application behind a firewall, odds are that these issues don't matter. If you're planning to compete with Google, you'll probably want to give it some serious thought.

It's also important to make sure that you don't have so many parts of your page updating in different ways and at different times that the user is unable to focus on their primary task at hand.

The Document Update Example

For this example, imagine a database in which it's possible that two users could edit the same record at the same time. The decision is made to inform the user when this happens and give them a chance to reload the data fresh from the server.

In order to allow this demo to run on a stock TurboGears install (no database required), the database used for this simple demo is a Python dictionary. The data looks like this:

1class PersonPaid(object): 
2    def __init__(selfid, name, paid): 
3        self.name = name 
4        self.paid = paid 
5        self.id = id 
6         
7    def __setattr__(self, attr, value): 
8        super(PersonPaid, self).__setattr__(attr, value) 
9        super(PersonPaid, self).__setattr__("last_update",  
10                                            time.time()) 
11 
12poormansdb = { 
13    1: PersonPaid(1, "Gandalf""paid"), 
14    2: PersonPaid(2, "Albus Dumbledore""dog"), 
15    3: PersonPaid(3, "Merlin""deadbeat"), 
16    4: PersonPaid(4, "Raistlin Majere""shortchanged"
17
view plain | print | ?

There's nothing too exciting happening there. Let's take a look at the form that we use for the data entry:

1paidoptions = dict
2    paid="Paid in Full",  
3    shortchanged="Shortchanged Us"
4    dog="Dog Ate the Bill",  
5    deadbeat="Deadbeat"
6 
7class PaidStatusWidgets(widgets.WidgetsDeclaration): 
8    name = widgets.TextField() 
9    paid = widgets.RadioButtonList(options=paidoptions.items()) 
10    id = widgets.HiddenField(validator=Int()) 
11    last_update = widgets.HiddenField(validator=Number()) 
12 
13paymentform = widgets.TableForm(fields=PaidStatusWidgets()) 
view plain | print | ?

This form consists entirely of standard HTML form control widgets. There is no JavaScript involved in this. Here is the controller code that displays the edit form:

1@expose(template="effectiveajaxturbogears.templates.paidedit"
2@validate(validators=dict(id=Int())) 
3def paidedit(selfid): 
4    return dict(person=poormansdb[id], form=paymentform, 
5        mochi=mochikit, 
6        next="rtumodel", prev="jobs"
view plain | print | ?

The interesting thing that we see here is a validator. The id value that is passed in by the browser is converted into an int on the way in.

1$${form.display(person, action="saveperson")} 
2<div id="changemsg" class="statusmsg" style="display:none"
3    Data has been changed.  
4    <href="javascript:window.location.href=window.location"
5      Click to refresh</a>
6</div> 
view plain | print | ?

Above, we see the body of the editing template. We start by displaying the edit form, using the person value passed in by the edit method. We also tell the form where it will be posting to (saveperson).

The change message that appears below the form is not displayed until the JavaScript on the page detects the page. The JavaScript holds the key to the update example:

1function checkchanged() { 
2  d = loadJSONDoc("checkchanged?" + queryString({ 
3    "id" : $${person.id}, 
4    "last_update" : $${person.last_update} 
5  })); 
6  d.addCallback(displaychange); 
7
8 
9function displaychange(result) { 
10  if (result["changed"]) { 
11    showElement("changemsg"); 
12  } else { 
13    window.setTimeout("checkchanged()", 1000); 
14  } 
15
16 
17window.setTimeout("checkchanged()", 1000); 
view plain | print | ?

At the bottom of the JavaScript, we see a timeout set. We tell the browser to call the checkchanged function after one second. checkchanged uses loadJSONDoc to call the server to find out if the data has changed, passing along the id and last_update time for that person. The server will determine if the data has changed and, when it does, the JavaScript displaychange function is called.

displaychange just looks at the result to see whether the data has changed. If it has, we show our change message and no longer check for changes. If it hasn't we tell the browser to check again in a second.

Clearly, doing a check for new data every second can get expensive. There are different ways to request updated data from the server (some of which keep the connection to the server open persistently to make latency as low as possible). This type of feature is something you'll definitely need to tune for your own application and deployment requirements.

The checkchanged method on the server is very similar to others that we've seen:

1@expose(format="json"
2@validate(validators=dict(id=Int(), last_update=Number())) 
3def checkchanged(selfid, last_update): 
4    person = poormansdb[id
5    has_changed = int(person.last_update) != int(last_update) 
6    return dict(changed=has_changed) 
view plain | print | ?

Ordered Lists

Reorganizing a list of items is a lot easier with drag-and-drop than with the old "click until you're in the right place" method.

My Yahoo! customization screen

Nowadays, you can drag and drop:

But, wait! This isn't Ajax! Actually, in the case of Google's personalized page it is Ajax. Google, correctly in my opinion, decided that there was no need for some additional confirmation once you had dragged the item to the place you wanted it. How annoying would Microsoft Word be if you had to click "Confirm" every time you moved an image on the page?

To Yahoo!'s credit, My Yahoo! does support drag and drop placement of items. Arguably, Yahoo!'s implementation is better because it gracefully supports older browsers and browsers without JavaScript enabled in addition to doing drag and drop for recent browsers.

Cookie Preference Example

Our example for Ajaxified drag and drop is a list of Girl Scout cookie preferences. It looks like a simple unordered list, but you can drag and drop the items from one position to another. When you do so, a "Changes Saved" message will appear after the data has been saved on the server.

This example uses Scriptaculous. MochiKit 1.3 will include a port of Scriptaculous' features, but that wasn't quite ready at the time of this writing.

The template shows just how much functionality is included in the JavaScript libraries we have today. The list of cookie names is created as a standard unordered list. After we create the list, we call Sortable.create, passing in the ID of our list and the instruction to call update_server whenever the user completes a change to the list. That one call is all it takes to add drag and drop to our list!

1<h1 class="exampleheader">Girl Scout Cookie Preference</h1> 
2<p>Please rank your preference of Girl Scout cookie  
3varieties:</p> 
4<ul id="cookielist"
5  <li py:for="cookie in cookies" py:content="cookie" 
6   id="c_$${cookie}">Cookie Name Here</li> 
7</ul> 
8<script type="text/javascript"
9  Sortable.create('cookielist',  
10    { "onUpdate": update_server }); 
11</script> 
12<div id="saved" class="statusmsg">Changes saved!</div> 
view plain | print | ?

Let's look at the JavaScript for update_server that is responsible for the Ajax part of this demo:

1function update_server() { 
2  var children = $("cookielist").childNodes; 
3  var cookies = new Array(children.length); 
4  for (var i = 0; i < children.length; i++) { 
5      cookies[i] = children[i].firstChild.nodeValue; 
6  } 
7  jsoncookies = serializeJSON(cookies); 
8  d = loadJSONDoc("saveorder?"  
9      + queryString({cookielist : jsoncookies})); 
10  d.addCallback(update_done); 
11
12 
13function update_done() { 
14  showElement("saved"); 
15  window.setTimeout(function() { 
16      new Effect.SwitchOff("saved", {queue: "end"})}, 
17      1000); 
18 
19
view plain | print | ?

The first part of update_server goes through the DOM and picks up the value of each element in our list. It's essentially turning the list displayed onscreen into a JavaScript array. That array is then handed off to MochiKit's serializeJSON function, which turns it into a JSON string. We then use the familiar loadJSONDoc call to save the new order on the server.

Once the save is complete, we display the "Changes Saved" message to the user.

There's one interesting bit to the server side of this save operation. TurboGears includes a JSONValidator that will convert an incoming JSON string into proper Python objects. That allows us to just set the server's cookie list directly to the incoming value. Here's the code for that:

1@expose(format="json"
2@validate(validators={"cookielist" : JSONValidator()}) 
3def saveorder(self, cookielist): 
4    global cookies 
5    cookies = cookielist 
6    return dict() 
view plain | print | ?

Ajax as a Confuser

The potential for Ajax abuse is staggering. Remember when everyone started making their own homepages with sickening backgrounds, color cycling rainbow borders and whatever dancing hamster-like web tchotzkes they could get their hands on? If we're not careful, we'll enter into the Ajax equivalent of that. Imagine dancing hamsters that change their dance depending on the state of the stock market. That's the kind of horror I'm imagining.

In your webapps, you're not likely to get into something quite that bad. But, it's pretty easy to do things that will make your users unhappy.

Hey! What happened to my back button?

Most people don't like having to learn a new piece of software. After several years of web browsing, people are comfortable with the controls in a web browser. I was working on a desktop application where we had the full might of a GUI toolkit at our disposal. What our users wanted, though, was a back button, just like the one in their browsers. People like their back button.

The whole point of Ajax is to be able to update the page without changing the user's perspective in their browser. For real-time data display or "application-like" things, this may be fine (ask your users!). But, sometimes you really want the back button to work the way people expect it to.

Dojo Toolkit, for one, has explicitly baked in hooks and helper code to make the back button work as expected. The kind of pain that they went through to make that work is something that you likely don't want to try to do under your project's deadlines. If you need Ajax and the back button, use Dojo or a toolkit with similar capability.

For some cases, there's another solution: don't use Ajax. Just because you can roundtrip to the server without updating the page doesn't mean that you have to or you should. If Ajax isn't actually adding any value to your app, don't use it. Avoiding it will let your user's browser work like they expect a web browser to work.

Why can't I bookmark this?

Closely related to the back button problem is the bookmarks problem. The same ideas apply. For Google Maps, their solution was to put a special link on the page that you could copy and paste. Dojo also helps with bookmarks. If you read the Dojo page, you'll find that Alex Russell is wisely letting you know that there's deep voodoo going on.

Repeating what I said in the last section: unless Ajax is adding serious value, don't let the voodoo creep into your software and don't let your users be confused because their browsers aren't behaving like they normally do.

That's not how that control works!

Google Suggest-style autocompletion is an obvious win. People are used to that kind of interaction. Quicken is used by millions of people. If Quicken does it, you're probably safe doing it, too. There are lots of other programs that use this kind of autocompletion, so you're really safe with that one.

Just because you can spruce up a text field with autocompletion doesn't mean that all widgets are ripe for Ajaxification. The principle of least surprise applies here: people have expectations of what a widget is supposed to do. Over time, those expectations can change. Once upon a time, a select box was just for selecting an item from a list. Now, select boxes are generally accepted as a way to navigate, without a button click, to another page on a site.

Shifts like that don't happen overnight and, as simple as it may seem, users get confused by it until they are used to it. Maybe, by now, users are so used to the lack of interface standards on the web that they're willing to accept anything.

If you're trying to reach a broad or non-technical audience, your best bet is to leave things as people are used to them.

Evil Voting Example

Here's an example of a misused control: a confusing voting system in which the radio button control automatically submits the vote without any confirmation. This is not normal behavior for a radio button. Even worse, this particular example does not let you change your vote after you've set it! This is a combination of unexpected behavior and exceedingly unfriendly behavior.

The code for this example is very simple. We put a call to a function called cast_vote as the onclick event for the radio buttons.

1<td valign="top" id="EQS">Edward Q. Scissorhands</td> 
2<td><input type="radio" name="vote" value="EQS" 
3 onclick="cast_vote(this)"/> 
view plain | print | ?

The JavaScript that's called sends the vote along to the server. When the response comes back, voteresponse will highlight the user's vote if they haven't already voted. If they have voted, it will switch the radio button back to the one they voted for and display that crazy error message.

1function cast_vote(vote) { 
2    d = loadJSONDoc("savevote?vote=" + vote.value); 
3    d.addCallback(voteresponse); 
4
5  
6function voteresponse(result) { 
7    if (result["already"]) { 
8        buttons = document.controlexample.vote; 
9        votevalue = result["voted"]; 
10        for (var i = 0; i < buttons.length; i++) { 
11            if (buttons[i].value == votevalue) { 
12                buttons[i].checked = true
13                break
14            } 
15        } 
16        showElement("already"); 
17        window.setTimeout(function() { 
18            new Effect.Fade("already", {queue: "end"})},  
19            3500); 
20    } else { 
21        new Effect.Highlight(result["voted"]); 
22    } 
23
view plain | print | ?

Conclusions

Know your audience

This is what it all boils down to. You need to consider who your users are: what is their level of technical expertise? How much do they like to learn and do new things on the computer? Can you train them?

Those kinds of questions will narrow down the sorts of things that you can get away with. But, just because you can get away with something doesn't mean that you should do it.

You should use Ajax in a way that enhances the user experience: allowing users to work faster, do more with less clicks, need to learn less or think less about the actions required.

Build on what your users have experienced previously and avoid breaking what they already know, and you'll be able to use Ajax fearlessly to make the world a better place. Or, at least, to give your app a better face.

Further Reading

Yahoo! Design Pattern Library by Yahoos!

Ajax Mistakes by Alex Bosworth.

XMLHttpRequest Usability Guidelines by Thomas Baekdal.

Usable XMLHttpRequest in Practice by Thomas Baekdal.

Design Patterns for Ajax Usability by Michael Mahemoff