Wednesday, April 30, 2008

YouTube widget: clone of the iGoogle gadget using Google AJAX APIs

I happened upon the Official YouTube Gadget for iGoogle the other day and thought it would be a fun project to replicate it using the myriad of Google API's that are now available. From the developer perspective, this mini application is entirely client side JavaScript - as the Google API's take care of any server side processing.

See my YouTube widget in standalone here

My workings...

I was able to extract some useful utility JavaScript code for formatting the video duration, truncating the titles, etc. from the gadget code itself:

That said, I needed to re-engineer the greater part of the widget using the available Google APIs.

Google AJAX Search API approach

I initially thought the widget would be driven primarily by Google AJAX Search API due to their support for YouTube search access (Direct Access to YouTube Channels). This support means that using a GvideoSearch for ytfeed:recently_featured would return a JSON object containing all the detail you need about that feed. However, the support for YouTube feeds is limited to variants of most_viewed, recently_featured and top_rated (and YouTube channels). The major reason that I chose to abandon this avenue was that the search API provides very limited results ( = 8 results!).

Clicking here you will see the JSON object returned by the Google Search API GvideoSearch for ytfeed:recently_featured.

Google AJAX Feed API approach

I turned my attention to the Google AJAX Feed API which does not have the same severe result set limitations as the Google AJAX Search API.

YouTube feed issues

There seem to be two variants of YouTube feeds knocking about. One in RSS 2.0 and one in Atom:

Clicking here you will see the JSON object returned by the Google Feed API result for (RSS 2.0) using the combined JSON/XML result format [feed.setResultFormat(google.feeds.Feed.MIXED_FORMAT);].

Clicking here you will see the JSON object returned by the Google Feed API result for (Atom) using the combined JSON/XML result format [feed.setResultFormat(google.feeds.Feed.MIXED_FORMAT);].

Using this JSON pretty printer and changing the name of the callback in the GvideoSearch and Gfeed calls I was able to display a live version of the recently_featured feeds for side by side comparison. For some reason this only works properly in Firefox.

The problem with using the YouTube RSS 2.0 feed is that the supplied feed is not valid RSS 2.0. The duration attribute is not a supported part of RSS 2.0. Therefore, when you fetch the feed via the Google AJAX Feed API you get no duration information.

To get the complete behaviour required for the widget (with durations and more than 8 results), I have to use the GData YouTube feed. In the GData feed the duration information is inside a special YouTube namespace. For parsing elements that are within namespace to work across browsers you will need to make use of the Feed API's google.feeds.getElementsByTagNameNS. A slight down side to using the Google AJAX Feed API to fetch the feeds is that each feed is cached for about an hour.

JavaScript Pagination

After some JavaScript development of my own, using the Google Feed API, I was able to construct a widget that displayed video thumbnails, titles, durations, SWF urls and on clicking the thumbnail I could even invoke a flash player (Embedding the YouTube player using SWFObject). To add the final magic, I needed to paginate the video thumbnails like the official gadget does. I found a very nice JavaScript script for paginating tables and my paging mechanism is derived from that. My modified version works with any element type (not just tables) and it achieves this by making use of the the ultimate getElementsByClassName method.

See my modified paging mechanism in standalone here.


A very enjoyable experience (a couple of days messing about!). I hope this offers some insight into some of the problems you might run into when working with the Google APIs. The final widget comes pretty close to the functionality of the official gadget and I'm sure I could polish this further.

Friday, April 25, 2008

PartnerBar: Google's feed aggregating magic javascript widget thingy

I was browsing the Google AJAX Search API blog recently when I discovered they were using an interesting widget at the bottom of the page, it turned out to be PartnerBar. PartnerBar was created as an example for the Google AJAX Feed API. I have been very busy with family stuff recently so I managed to completely miss this when news of it first broke in December (Ajax Feed Partner Bar and The PartnerBar - Contextual Cross Linking). In my defence this is easy to miss, it is called "PartnerBar", which means nothing to anybody on this side of the pond and the launch article talks about "contextual cross linking" (meaningless technobabble!). Essentially, PartnerBar is a feed aggregating magic javascript widget thingy, if it had been launched as such I would have spotted it sooner.

Anyway I have been playing with it recently and have made some minor tweaks to improve it for my own usage, I'm very happy with the results. I can't decide whether to install the resulting widget permanently on my blog because I think people would find it annoying if I were to place it somewhere prominent.

My tweaks and modifications

Feed titles and links.

In the instance of PartnerBar on their blog, Google store an array of their blogs in a JavaScript file:

Incidentally, notice how it supports tags so that you can use the same array for multiple widgets with distinct content.

My first modification was to tweak the code so that I did not have to supply the feed name and link, this information is stored in the feed (why store it twice?). To achieve this I overrode the loadPartner method - I think the trick with overriding methods in JavaScript is to make sure the method you override is a small one!

My array of blogs looks like this: blogs.js

Alternating row colours

Secondly, I wanted the feed entries to display in alternating colours. Again, I found a small JavaScript method in PartnerBar (resetClassOnPartnerDom) and overrode it (AFAIK there is no "super" keyword in JavaScript, which is a shame). In order to do this I made use of a getElementsByClassName method that I found on the web.

Final result

You should be able to see the widget in action at the top of this blog entry. However, you can link out to see the final result in standalone glory here.

Here is my final JavaScript which my customised widget uses: partnerBar.js.


Although I suffer with CSS, I am not a graphic designer and so I tend to steal gather inspiration from elsewhere. I based my widget on the Infected table theme from the CSS table gallery.

Friday, April 18, 2008

Experiments and thoughts on Google App Engine

I have been experimenting with the Google App Engine (GAE) recently which is a Google hosted Python application server. I am not a native Python programmer (I've written about 5 scripts in my life). Even so, GAE is a great toy to play with.

Python on GAE

There are currently some weaknesses in the implementation of Python available in GAE (re: C libraries, XML parsers, XSLT engines) but this is a preview release and all things considered it is a very impressive offering. Why Python? We are told that "Google are huge fans of Python" and it is evidently used extensively inside Google. One of Google's other "hosted" platforms, Google's Mashup Editor, is a JavaScript application server running on top of Java, so Google obviously have experience in the Java application server area. As a Java programmer, I would get more excited if I could upload WAR files to GAE (even if I had to use proprietary Google Java APIs for some things) but you can't win them all. You never know, Java support could be just around the corner.

I'm new to writing Python web applications and so far it reminds me of writing Perl CGI code. This is not the best way to write web applications and I miss JSP at this point. There must be better ways of doing this using wsgiref and the Django framework which come pre-installed in GAE. These may help me get around the "scriptyness" of my python web application programming. My writing simple python web applications experience has surprised me somewhat as I had been drip fed with the idea that Python was a more efficient language (and hence better!) than Java/JSP (but I suspect I'm not doing it right!). So far it seems that you have to write a lot more Python code than the equivalent in JSP e.g. to get a parameter from an URL:

Python version

from cgi import parse_qs, escape

def hello_world(environ, start_response):
parameters = parse_qs(environ.get('QUERY_STRING', ''))
if 'subject' in parameters:
subject = escape(parameters['subject'][0])
subject = 'World'
start_response('200 OK', [('Content-Type', 'text/html')])
return ['''Hello %(subject)s

Hello %(subject)s!

''' % {'subject': subject}]

JSP version

Hello ${empty param.subject ?  'World' : param.subject}

To be absolutely honest, I have never liked the conditional expression tenary operator (e.g '?' above) syntax and in this case I would prefer something more verbose but this illustrates the potential efficiency inherent in contemporary JSP.

Datastore on GAE

In GAE you are given access to a datastore, although this is not a relational database but an instance of Google's BigTable database. To access this database there is an SQL-like language called GQL available. We are told that the Datastore API is modelled after Django's Database API.

Worries about using GAE as a commercial platform?

If I were a business with an interest in using GAE commercially, storing my valuable data in proprietary Google storage would worry me somewhat. There are open source versions of BigTable available (e.g. HBase, HyperTable) and these owe there existence to Google but as there is not yet a standard "GQL" mechanism, data exit strategies seem limited.

I would be greatly concerned that my applications are hosted by Google and that my Python developers were exposed to Google. GAE could be described as a lobster pot. Getting Google to host your business means that Google are in a prime position to absorb successful businesses, steal talented developers, learn more about your business quicker than you can yourself and learn from your mistakes.

Some fun stuff, an example Google App Engine app

Using GAE, Mark Pilgrim's Universal Feed Parser code and the JavaScript library I have cobbled together a fun example.

See how Google use the domain which is like, nice! I look forward to wikispot etc.

Access the source code here:
Main script:
Download the full app:

I am still having some caching problems, please help!