Thursday, February 16, 2006

Yet another Google Suggest Clone - Take 2

Arthur C. Clarke's Third Law was related to me recently in the context of Google Suggest, it goes something like:

Any sufficiently advanced technology is indistinguishable from magic

I first saw Google Suggest around the 17th December 2004. At the time I had no idea how it worked, it was as magic to me. In the intervening period the AJAX phenomenon has grown, especially following the arrival of Google Maps. In November last year, I put together a modest Google Suggest clone of my own using Periodic Table data (see: Yet another Google Suggest Clone). This was largely based upon the ObjectGraph Dictionary, accompanying explanation and articles from phpRiot and XmlProNews. My initial implementation was a poor imitation of the rich interface that Google Suggest offers.

I have since improved upon that effort to a point where it is almost usable (bear with me self deprecation is a proud British tradition)! The following can be said about almost any human endeavour but I still feel the need to say it anyway. This effort represents the combination of ideas and techniques gathered, stolen and borrowed from several other authors. Although, I do hope that it is sufficiently novel in its own right as to merit some attention.

A valuable reference for this was Chris Justus' Google Suggest Dissected. Google Suggest has been explored by many others. My primary aim was to learn about how Google Suggest works to try and replicate its functionality for myself. Some methods I have found concentrate too much on the use of particular library or backend technology (e.g. JSP, ColdFusion, ASP, PHP) as an integral part of the solution. Google Suggest is all about JavaScript, how the backend is produced should be largely immaterial. I have opted to use JSON as the backend data format. JSON can apparently be produced using a wide range of technologies including some that are relatively obscure, such as Chicken Scheme and Squeak.

The evolution of my Google Suggest clone

Starting with my Periodic Table Suggest application the first step I took was to fix the CSS in order to stop the screen jumping about as a result of the appearing and vanishing "dropdown" suggestion box.

The original version used crude text separators as a backend data format rather than any recognised format. I have modified it to reply with JSON format data (this is easily modified to output larger arrays of data for more complicated functionality).

In the original application the onkeyup event triggered a new query, meaning a query was sent after every key press. I modified the application to use the submission throttling approach, this polls the textbox obtaining a value periodically rather than after every keypress.

A colleague of mine had rewritten my initial simple example to allow the use of the up and down keys to navigate between the suggestion options. I took his code on board and with the help of another example I found (here, found via Find a U.S. City - Suggest Two) I reintegrated this code into my original suggest application.

Inspired by Creating an Autosuggest Textbox with JavaScript, Part 1, I added autocomplete functionality. I refactored the behaviour to make it as simpler and removed duplicate functionality where I could. Finally, I wrapped the whole thing inside object oriented JavaScript (in a vain attempt that I could run two suggest windows on a single page, needs more work!).

It is still not perfect yet but AFAICT it pretty much emulates Google Suggest behaviour. The example in Creating an Autosuggest Textbox with JavaScript, Part 2 achieves the same thing and probably does a better job of it than I do! At least I have the satisfaction of having gone through each step and as a result I have a clear idea of what is going on. I have created an example application using the Cities table from the MONDIAL database (a larger dataset than the elements of the periodic table that I previously used). This can be seen here:

An example application: City Suggest

For a finishing touch I created a Google Suggest style logo using the logo maker tool at

How my suggest implementation works:

The JavaScript behind this application can be accessed here. Once the JavaScript Suggest class is instantiated, the program continually checks for textbox changes, key presses and mouse events.

  • onkeydown events trigger suggestion highlighting and setValues()
  • onmousemove events trigger highlight() suggestion
  • onmousedown events trigger setValues()
  • onkeyup events trigger an autosuggest option refresh

It is all encapsulated inside a class called Suggest. An instance of Suggest is created (with suitable parameters) and then the startup() method is invoked.

assigns an onkeydown event handler (keypressHandler() - for the up/down/enter key functionality) and an onkeyup event handler (keyupHandler() - for the autocomplete functionality). It invokes an infinite polling loop (requestLoop()).
each iteration of this loop it first obtains the "unselected" portion of the textbox (using query()) and if the query has changed since the last request it makes a new request for data from the backend (using sendQuery()).
retrieves unselected portion of the textbox
calls initialize() in order to establish a usable XMLHttpRequest, on retrieving a result it passes it to process()
establish XMLHttpRequest
obtains JSON object and makes a call to htmlFormat() to render the result or reports an error
renders the suggestion box which includes adding listeners for onmousemove (to enable mouse driven suggestion highlighting - calling highlight()) and onmousedown (to enable mouse driven suggestion selection - calling setValues()). At the end of the rendering a function call is made that assembles suitable autocomplete options (requestSuggestions()).
handles keyup events, principally used for autocomplete functionality
handles keydown events, principally for key up, key down and enter key
invoked by onmousemove event, highlights current selected suggestion
invoked by onmousedown and onkeydown, sets textbox value to currently selected suggestion
shows autocomplete div
hides autocomplete div
builds an array of currently matching autocomplete options, calls autosuggest
calls typeAhead if a non-empty autocomplete array exists
inserts a suggestion into the textbox, highlighting the suggested part of the text.
selects a range of text in the textbox.
trims whitespace from a given string


Mark McLaren said...

Hi Mark,

I try to implement a google suggestion function at my website using the idea from "Cloning Google Suggest with AjaxAC". It works fine as a single input suggestion. However in my application I want to have two input suggestions. so even I try to duplicate the java objects for both input suggestions, at the same time only one input suggestion works. I am new to Ajax and I knew I miss something.

Could you give me a hint about where I should check? or is your approach easy for two input suggestion functions? for example, your city suggestion example, could it be easy to duplicate to two inputs and each of them works when you type?

really appreciate your help.

Note: Comment imported. Original by Franky at 2006-03-17 06:43

Mark McLaren said...

Hi Franky,

With my approach I cut no corners (I hand coded or stole! all the JavaScript). I was interested in finding out exactly how Google Suggest actually works so it was important for me to perform all the steps by hand.

For a beginner you are probably better off using a toolkit like AjaxAC. The problem with AjaxAC is that it does not appear to use an Object Oriented (OO) JavaScript approach. In order to have two suggest boxes on a page you are better off with a toolkit that follows the OO JavaScript approach. I'm not really sure what toolkit you should use as currently you are spoilt for choice (but I'm sure there will something suitable out there somewhere). Since you appear to be using PHP a toolkit like phpAjaxTags looks quite promising. Alternatively you can look through a list of such frameworks at PHP Ajax Frameworks. The worst case scenario is that you have to hand code your solution, even this can be quite easy (as long as you don't want all the Google Suggest behaviour) see this article: Creating an Autosuggest Textbox with JavaScript, Part 2.



Note: Comment imported. Original by markmc website: at 2006-03-17 09:11

Mark McLaren said...

I am so glad tonight I saw your reply. these days I spent quite a lot time google and ask gurus, and so far you are the best. Great resource for a beginner like me to learn the ajax using the gogole suggestion as an application. I feel much better now. :)

thanks again, Franky
Note: Comment imported. Original by Franky at 2006-03-18 07:09

Mark McLaren said...

Found your example a breeze to play with, and managed to get it working with a little application I am writing to autocomplete a Suburb, then also update a State and Postcode field. The one thing I am trying to debug at the moment is to do with query(). While your example works flawlessly when it is the only form item on the page, I find that when it is doing its timed checks to see if the query has changed from the last time, that if you have another form field, and tab into it (meaning the field is highlighted) the query() method will pick it up and use that to compare with the last result instead of the textbox that contains the autocomplete.

EG. Two form fields, one with autocomplete, one without. Use the autocomplete field then tab to the next field. If it is blank, all is good. If it has a value, then it gets highlighted, and will switch back to the autocomplete field.

Just wondering if you had any ideas on how to stop this from happening, and confine the query() method to only the field in which the autocomplete occurs.
Note: Comment imported. Original by Robbo at 2006-03-28 06:44

Mark McLaren said...

Hi Robbo,

I'm very happy that you found this useful. I've tried to write this in an object oriented JavaScript fashion (my first OO JS app!), this is so that it could theoretically support multiple suggest boxes per page (but it still needs work).

The query() method *should* only access the text field with the specified ID (in the example shown the ID is keyword). In theory as long as your two text fields have different IDs it should work, that said it could be broken (I'll need to look into this further).

Note: Comment imported. Original by markmc website: at 2006-03-28 09:22

Mark McLaren said...

I just have this feeling that the following code isn't doing what it is meant to be doing.

var fa=document.selection.createRange().duplicate();


Note: Comment imported. Original by Robbo at 2006-03-28 12:11

Mark McLaren said...

Just to define the error a bit more, it seems there isn't a problem with Firefox.
Note: Comment imported. Original by Robbo at 2006-03-28 13:21

Mark McLaren said...

Hi Robbo, that would make sense. I nicked that bit from Google Suggest dissected. As you say, it looks like it is looking for selected text in the entire document rather than in a specific field.
Note: Comment imported. Original by markmc website: at 2006-03-28 18:01

Mark McLaren said...

Hi Robbo,

It is an IE specific problem. Try this IE specific example page, I’ll need to modify my Google Suggest clone to check the field under selection for IE but hopefully the code in the example will help you get to where you want to go.

Thanks for the feedback; as you can tell I am a Firefox user so it would have been a while before I noticed that, Great stuff,

Note: Comment imported. Original by markmc website: at 2006-03-28 19:40

Mark McLaren said...

That fixed it. Threw a little test in the top of requestLoop() that checks if the id is the same as the autosuggest id. If there they don't match, it just resets the timer.

Usually a fiewfox person myself, but this application needs to run on IE.
Note: Comment imported. Original by Robbo at 2006-03-28 23:55

Mark McLaren said...

Hello, I agree with the others that you've done a great thing here for beginners!

It seems easy enough to plug in a servlet to handle the query url and then off I go with your js code, however, what I can't seem to put my finger on is how to format the response text? I've only built a few AJAX components and have used delimitirs or XML Strings I construct in a Servlet, can I do something like that here?

For example, my servlet may have something like:


sb.append("" + cityVal + "");

sb.append("" + descriptionVal + "");


I'm probably missing something simple here, I appreciate any additional guidance you may offer.
Note: Comment imported. Original by Maroon at 2006-03-30 02:14

Mark McLaren said...

Thank you for your kind words Maroon, I’m really happy that you find this useful. It makes blogging feel worthwhile when you get feedback like that!

In this example I chose to use JSON, there is a free Java JSON API you could use to construct JSON data (it is a bit messy to do so otherwise). Alternatively you could use XML, as you suggest. From a servlet you might like to use David Megginson's XMLWriter to help create your XML. The main thing you'd need to change to get my Google Suggest clone working is to modify htmlFormat() so that it processes XML format data rather than JSON.

Previously I talked about how JSON wasn't better than XML (as some parties were suggesting). My position has softened towards JSON since writing that blog entry but I still don't think JSON is better than XML. You can achieve the same things with XML and JSON, it is just a matter of personal preference.
Note: Comment imported. Original by markmc website: at 2006-03-30 07:32

Mark McLaren said...

Could you post the code you used to test for the 'focused' id?
Note: Comment imported. Original by troy at 2007-05-09 20:39

Mark McLaren said...

Hi Troy,

I am not sure what you mean by "focussed id". Do you mean the Type Ahead behaviour?

Or do you mean something to do with the drop down box? There are various tricks going on with the drop down box to fool you into believing that it is actually a drop down box but it is really just a bit of DHTML.

Note: Comment imported. Original by markmc website: at 2007-05-10 08:44

Mark McLaren said...

Hi Mark, could you post a reference to the codes that produced the JSON reply "string"?

I am a little confused how it might look like. Thanks
Note: Comment imported. Original by Marv at 2008-01-01 04:40

Mark McLaren said...

No problem Marv. The following is the servlet that fetches the data from the database and returns the JSON format output (it is not exemplar code!!!). Usually I would use Spring for this kind of thing (so as not to hardcode database and SQL in the Java code).

JSON format output:


Note: Comment imported. Original by markmc website: at 2008-01-01 12:37

Mark McLaren said...

Thank you Mark ^^

I used everyday's lunch surplus time to figure this out. So glad that you helped.

Note: Comment imported. Original by Marv at 2008-01-05 09:45

Mark McLaren said...

Hi Mark,

I am facing the same problem with Robbo in ie. Is there a workaround to counter this problem?


Note: Comment imported. Original by Marv at 2008-02-23 10:12

Mark McLaren said...

Hi Marv,

You want Google Suggest style autocomplete, right? My advice is use a JavaScript library!

Note: Comment imported. Original by markmc website: at 2008-02-24 19:42