Thursday, February 16, 2006

Yet another Google Suggest Clone - Take 2

Arthur C. Clarke's Third Law was related to me recently in the context of Google Suggest, it goes something like:

Any sufficiently advanced technology is indistinguishable from magic

I first saw Google Suggest around the 17th December 2004. At the time I had no idea how it worked, it was as magic to me. In the intervening period the AJAX phenomenon has grown, especially following the arrival of Google Maps. In November last year, I put together a modest Google Suggest clone of my own using Periodic Table data (see: Yet another Google Suggest Clone). This was largely based upon the ObjectGraph Dictionary, accompanying explanation and articles from phpRiot and XmlProNews. My initial implementation was a poor imitation of the rich interface that Google Suggest offers.

I have since improved upon that effort to a point where it is almost usable (bear with me self deprecation is a proud British tradition)! The following can be said about almost any human endeavour but I still feel the need to say it anyway. This effort represents the combination of ideas and techniques gathered, stolen and borrowed from several other authors. Although, I do hope that it is sufficiently novel in its own right as to merit some attention.

A valuable reference for this was Chris Justus' Google Suggest Dissected. Google Suggest has been explored by many others. My primary aim was to learn about how Google Suggest works to try and replicate its functionality for myself. Some methods I have found concentrate too much on the use of particular library or backend technology (e.g. JSP, ColdFusion, ASP, PHP) as an integral part of the solution. Google Suggest is all about JavaScript, how the backend is produced should be largely immaterial. I have opted to use JSON as the backend data format. JSON can apparently be produced using a wide range of technologies including some that are relatively obscure, such as Chicken Scheme and Squeak.

The evolution of my Google Suggest clone

Starting with my Periodic Table Suggest application the first step I took was to fix the CSS in order to stop the screen jumping about as a result of the appearing and vanishing "dropdown" suggestion box.

The original version used crude text separators as a backend data format rather than any recognised format. I have modified it to reply with JSON format data (this is easily modified to output larger arrays of data for more complicated functionality).

In the original application the onkeyup event triggered a new query, meaning a query was sent after every key press. I modified the application to use the submission throttling approach, this polls the textbox obtaining a value periodically rather than after every keypress.

A colleague of mine had rewritten my initial simple example to allow the use of the up and down keys to navigate between the suggestion options. I took his code on board and with the help of another example I found (here, found via Find a U.S. City - Suggest Two) I reintegrated this code into my original suggest application.

Inspired by Creating an Autosuggest Textbox with JavaScript, Part 1, I added autocomplete functionality. I refactored the behaviour to make it as simpler and removed duplicate functionality where I could. Finally, I wrapped the whole thing inside object oriented JavaScript (in a vain attempt that I could run two suggest windows on a single page, needs more work!).

It is still not perfect yet but AFAICT it pretty much emulates Google Suggest behaviour. The example in Creating an Autosuggest Textbox with JavaScript, Part 2 achieves the same thing and probably does a better job of it than I do! At least I have the satisfaction of having gone through each step and as a result I have a clear idea of what is going on. I have created an example application using the Cities table from the MONDIAL database (a larger dataset than the elements of the periodic table that I previously used). This can be seen here:

An example application: City Suggest

For a finishing touch I created a Google Suggest style logo using the logo maker tool at http://googlefor.com/.

How my suggest implementation works:

The JavaScript behind this application can be accessed here. Once the JavaScript Suggest class is instantiated, the program continually checks for textbox changes, key presses and mouse events.

  • onkeydown events trigger suggestion highlighting and setValues()
  • onmousemove events trigger highlight() suggestion
  • onmousedown events trigger setValues()
  • onkeyup events trigger an autosuggest option refresh

It is all encapsulated inside a class called Suggest. An instance of Suggest is created (with suitable parameters) and then the startup() method is invoked.

startup()
assigns an onkeydown event handler (keypressHandler() - for the up/down/enter key functionality) and an onkeyup event handler (keyupHandler() - for the autocomplete functionality). It invokes an infinite polling loop (requestLoop()).
requestLoop()
each iteration of this loop it first obtains the "unselected" portion of the textbox (using query()) and if the query has changed since the last request it makes a new request for data from the backend (using sendQuery()).
query()
retrieves unselected portion of the textbox
sendQuery()
calls initialize() in order to establish a usable XMLHttpRequest, on retrieving a result it passes it to process()
initialize()
establish XMLHttpRequest
process()
obtains JSON object and makes a call to htmlFormat() to render the result or reports an error
htmlFormat()
renders the suggestion box which includes adding listeners for onmousemove (to enable mouse driven suggestion highlighting - calling highlight()) and onmousedown (to enable mouse driven suggestion selection - calling setValues()). At the end of the rendering a function call is made that assembles suitable autocomplete options (requestSuggestions()).
keyupHandler()
handles keyup events, principally used for autocomplete functionality
keypressHandler()
handles keydown events, principally for key up, key down and enter key
highlight()
invoked by onmousemove event, highlights current selected suggestion
setValues()
invoked by onmousedown and onkeydown, sets textbox value to currently selected suggestion
showAutocompleteDiv()
shows autocomplete div
hideAutocompleteDiv()
hides autocomplete div
requestSuggestions()
builds an array of currently matching autocomplete options, calls autosuggest
autosuggest()
calls typeAhead if a non-empty autocomplete array exists
typeAhead()
inserts a suggestion into the textbox, highlighting the suggested part of the text.
selectRange()
selects a range of text in the textbox.
trim()
trims whitespace from a given string

Monday, February 13, 2006

How did I miss Type-Ahead behaviour?

I have been doing some further work on my simple Google Suggest clone recently (I've not finished doing this yet, more on this another time). A colleague of mine brought the autocomplete or (type ahead) behaviour of Google Suggest to my attention. I have to admit that I'd never really noticed this before, it is so easy to take clever AJAX behaviour for granted (one of the reasons for AJAX's success is it just feels natural). Whilst trying to figure out how it works I tripped over a JavaScript feature that I never knew existed before. If you are viewing this as a standalone entry you will be able to see an example of it working below (fixed! I think my blog front page is too JavaScript laden for it to work there). The highlighting is achieved via JavaScript. JavaScript is also used to tell you what text is highlighted and what text is not highlighted. Here is the JavaScript.

The mechanism used (createTextRange/setSelectionRange) differs between IE and Firefox. Whilst investigating this I found the following references useful:

Google Suggest Dissected..., Creating an Autosuggest Textbox with JavaScript, Part 1 & Get Selection (from quirksmode.org)

Sunday, February 12, 2006

An XML to JSON webservice

On a blog entry I was reading somebody commented that all we need now is a general purpose XML to JSON webservice. I thought a syndicated feed XML to JSON service would be really easy to produce, so I thought I'd knock one up for fun! Essentially this is a guise of the proxy that is necessary in AJAX apps to get around the XMLHttpRequest security restrictions. This uses the On Demand JavaScript approach, advocated by the Yahoo JSON API and used by Google Maps, meaning it doesn't suffer the same server security constraint and could therefore be used by remote servers (the transport format could be XML or JSON but I've chosen to use JSON to satisfy my own curiosity).

I recently questioned the suggestion that JSON is easier to use than XML (especially since E4X has arrived on the scene). This outburst followed an article on XML.com that seemed to suggest that JSON was an essential part of a recipe that could be used to get around some of the short comings of AJAX. I hope I showed that XML could have equally well have been used in the place of JSON using the same approach.

I also pointed out that pretty much Yahoo were the only chaps to offer JSON as an output format. I have been playing with JSON recently and my position has softened towards it, I still don't think that is a ready made replacement for XML in all circumstances but it does have its place.

Looking at Yahoo's JSON API I thought that the easiest way to do this is with a plain old java servlet (I suppose it is called a POJS nowadays ;)). First we need someway of accessing multiple syndication formats. I'm going to use Rome and Rome's Fetcher subproject to retrieve the feeds. Rome can access multiple feed formats (including Atom) and converts them to a common internal object representation. Having obtained an object representation all we need to do is output this information in the JSON format. There is Java JSON API available to help with the JSON conversion. After a short amount of coding I have my basic XML to JSON servlet (okay there is still room for improvement but it does the job).


package xmltojson;

import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.fetcher.FeedFetcher;
import com.sun.syndication.fetcher.FetcherException;
import com.sun.syndication.fetcher.impl.FeedFetcherCache;
import com.sun.syndication.fetcher.impl.HashMapFeedInfoCache;
import com.sun.syndication.fetcher.impl.HttpURLFeedFetcher;
import com.sun.syndication.io.FeedException;
import java.io.IOException;
import java.io.PrintWriter;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Iterator;
import java.util.List;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

public class XMLtoJSONServlet extends HttpServlet {

FeedFetcherCache feedInfoCache;
FeedFetcher feedFetcher;

public void init()
throws ServletException {
feedInfoCache = HashMapFeedInfoCache.getInstance();
feedFetcher = new HttpURLFeedFetcher(feedInfoCache);
}

protected void doGet(HttpServletRequest req,
HttpServletResponse res)
throws ServletException,
IOException {
String callback = req.getParameter("callback");
String url = req.getParameter("url");
if (callback==null)
callback = "";
if (url==null)
url="http://content.mark-mclaren.info/rss.xml";
PrintWriter out = res.getWriter();
out.print(callback + "("+fetchFeed(url)+")");
}

private String fetchFeed(String url)
throws IOException {
try {
SyndFeed feed = feedFetcher.retrieveFeed(new URL(url));
return syndFeed2JSON(feed);
} catch (MalformedURLException e){
} catch (FeedException e){
} catch (FetcherException e){
}
return "Not working";
}

private String syndFeed2JSON(SyndFeed feed) {
JSONObject jsonFeed = new JSONObject();
try {
String title = feed.getTitle();
String link = feed.getLink();
jsonFeed.put("title", title);
jsonFeed.put("link", link);
JSONArray jsonFeedEntries = new JSONArray();
List entries = feed.getEntries();
Iterator i = entries.iterator();
while (i.hasNext()) {
SyndEntry entry = (SyndEntry) i.next();
String itemTitle = entry.getTitle();
String itemLink = entry.getLink();
JSONArray jsonFeedEntry = new JSONArray();
jsonFeedEntry.put(itemTitle);
jsonFeedEntry.put(itemLink);
jsonFeedEntries.put(jsonFeedEntry);
}
jsonFeed.put("entries",jsonFeedEntries);
return jsonFeed.toString();
} catch (JSONException e) {
}
return "";
}

}

You can download the above servlet source here. See this example of what the above servlet outputs.

Add a smattering of JavaScript and some CSS borrowed from Listutorial and we have a working example! If you haven't guessed already the RSS list at the top left of this blog entry is produced using the XML to JSON approach (if you don't have JavaScript enabled you won't see it).

For this blog entry I have taken a static copy of what the servlet would output as I don't want to provide the XML to JSON conversion gateway for the world.

There is no reason to stop at XML to JSON. Add a little Spring Framework magic and it could quite easily become a database => JSON or webservice => JSON gateway. Let the mashups begin!

Thursday, February 09, 2006

A Mention in Google Maps Hacks!

I'm famous...well my blog is! It was recently referenced in the recently published O'Reilly book called Google Maps Hacks by Rich Gibson, Schuyler Erle (January 2006).

In section 4.10.6:

Hack 37. View Your GPS Tracklogs in Google Maps

My blog is referenced as:

How to use Google Maps' XSLT voodoo to process the GPX file

How cool is that?!!

Spring's LDAPTemplate

The Spring Framework is my new favourite thing! I toyed with LDAP and Spring recently. There is a Sourceforge project called LDAPTemplate maintained by Mattias Arthursson and Ulrik Sandberg of Jayway which looks to be very impressive. It is built on the same principles as Spring JDBCTemplate and other DAO templates. Unfortunately since I am working with LDAP version 2 (rather than version 3) I couldn't get it to work but the guys are working on fixing this for a future release. I can see this project becoming part of the core Spring framework in the not too distant future.

I did however manage to get an earlier version of the code from LDAPTemplate written by Olivier Jolly working with the following example.

Spring Config file (applicationContext.xml):


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">

<beans>
<bean id="rootContextSource"
class="org.springframework.ldap.support.UrlContextSource">
<property name="ldapUrl" value="ldap://hostname:389/o=Blah, c=GB"/>
</bean>
<bean id="ldapTemplate"
class="org.springframework.ldap.core.LdapTemplate">
<property name="contextSource">
<ref bean="rootContextSource" />
</property>
</bean>
</beans>

LDAPTest.java


import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import javax.naming.NamingEnumeration;
import javax.naming.directory.Attributes;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;
import org.springframework.beans.factory.xml.XmlBeanFactory;
import org.springframework.core.io.ClassPathResource;
import org.springframework.ldap.core.LdapTemplate;
import org.springframework.ldap.core.SearchResultCallbackHandler;

public class LDAPTest {

public static void main(String[] args) throws Exception {

// Manually create the Spring Bean Factory
// Note: The factory is driven by the Bean definitions held in ApplicationContext
// Note: There are also other implementations of Resource and Factory, but
// the following combination work well for standalone apps.

ClassPathResource res = new ClassPathResource("applicationContext.xml");
XmlBeanFactory factory = new XmlBeanFactory(res);
{
SearchControls ctls = new SearchControls();
ctls.setSearchScope(SearchControls.SUBTREE_SCOPE);
// Specify the ids of the attributes to return
String[] attrIDs = {"sn", "mail", "uid", "cn", "ou", "postalAddress", "postalCode"};
ctls.setReturningAttributes(attrIDs);

LdapTemplate ldapTemplate = (LdapTemplate)factory.getBean("ldapTemplate");
List l = (List) ldapTemplate.search(
"",
"(sn=McLaren)",
ctls, new SearchResultCallbackHandler() {
LinkedList list = new LinkedList();

public void processSearchResult(SearchResult searchResult) throws javax.naming.NamingException {
Attributes attrs = searchResult.getAttributes();
list.add(attrs);
}

public Object getResult() {
return list;
}
});
Iterator i = l.iterator();
while(i.hasNext()) {
Attributes o = (Attributes) i.next();
NamingEnumeration ids = o.getIDs();
while(ids.hasMore()){
String att = (String) ids.next();
System.out.println(att + ":");
System.out.println(o.get(att).get());
}
System.out.println();
}
}
System.out.println("Done");
}

}

What is it about Spring that makes me use anonymous classes all the time?