Wednesday, November 05, 2008

Re-inventing ORM, annotations re-invented for databases!

I am working on a web application where one of the requirements is that I use an in-house Java database access framework. Personally I would prefer to use something that complements Spring and that makes my life easier such as Hibernate Annotations, iBATIS or even Spring JDBCTemplates but requirements are requirements.

One of the problems I face is that I need to re-invent a lot of ORM wheels. The framework is very similar in essence to iBATIS.All the SQL is stored in external XML configuration files. The framework conducts the mappings conversion from ResultSets to lists of beans (using commons-beanutils). As such, like iBATIS there is plenty of scope to generate most of the repetitive Java code (interfaces, implementations, facades) and configuration files (SQL CRUD). I have previously experimented with the Middlegen code generation with iBATIS. Middlegen code is a little long in the tooth, as a project it seems to have stagnated back in 2005 (I suppose iBATOR and Hibernate Tools have long since superseded it). That said, because Middlegen is based on Velocity templates it is very flexible and perfect for this kind of thing. I was able to port my previous iBATIS code generating templates to produce suitable Java code for use with the in-house framework.

I have managed to get some quite sophisticated code generated. Apparently "Lazy Loading is easy", so I am attempting to implement that using CGLIB in my DAO. I have also produced a method that is similar to Hibernates's SaveOrUpdate method. A given entity may have two kinds of relationship; target one (OneToOne, ManyToOne) or target many (OneToMany). I have no ManyToMany relationships and this is deliberate!

How my SaveOrUpdate works is derived from some code I found in iBATIS in Action (By Clinton Begin, Brandon Goodin and Larry Meadors, Published by Manning). In section 5.3.2 Updating or deleting child records the following method is suggested:


public void saveOrder(SqlMapClient sqlMapClient, Order order)
throws SQLException {
if (null == order.getOrderId()) {
sqlMapClient.insert("Order.insert", order);
} else {
sqlMapClient.update("Order.update", order);
}
sqlMapClient.delete("Order.deleteDetails", order);
for(int i=0;i<order.getOrderItems().size();i++) {
OrderItem oi = (OrderItem) order.getOrderItems().get(i);
oi.setOrderId(order.getOrderId());
sqlMapClient.insert("OrderItem.insert", oi);
}
}

The above method addresses the target many relationships but does not address the target one relationships. My method works more like this:

  1. Iterate through all the target one entities and SaveOrUpdate these. Inserting any generated ids into the current entity.
  2. Save the current entity data fields
  3. Iterate through all the target many relationships and SaveOrUpdate these.

At this point I got a little stuck. I found that relatively often my target one relationships should not be the subject of further SaveOrUpdates. The characteristic of these relationships is when the data is fairly static and mostly provides the conent of drop-down lists. When you make a selection from a such a list you do not usually need to update the list itself!

Essentially, Middlegen generates everything based on the contents of the database. For these relationships I needed some way to designate the code generation behaviour for certain database columns.

In Java Persistence API implementations, like Hibernate Annotations, this kind of behaviour is very easily represented by using @Column(insertable=false, updatable=false). I needed a way for Middlegen to acquire this information from the database. Then I had a brainwave, in the most recent version of Middlegen (from the CVS) you can "getRemarks" for tables and columns. Database remarks can be acquired from DatabaseMetaData. The unfortunate thing is that I am using an Oracle database which AFAICT uses a non-standard treatment of "remarks". It turns out that the default behaviour of Oracle drivers is not to populate the DatabaseMetaData with remarks. In Oracle remarks are contained inside two special tables; user_tab_comments and user_col_comments. You can override the default behaviour of Oracle driver to populate remarks, this can be done at the BasicDataSource (using addConnectionProperty("remarksReporting", "true")) or on a OracleConnection (using setRemarksReporting(true)).

Armed with this information I can "annotate" database columns, so I can use something like this:


COMMENT ON COLUMN <table>.<column> IS 'insertable=false, updatable=false';

I tweaked MiddlegenPopulator to allow it to acquire remarks for Oracle from the DatabaseMetaData. The database now contains all the information I need to generate my DAO code. I can now tweak my DAO generation templates to miss out the SaveOrUpdate code for target one relationships which match the column comment of 'insertable=false, updatable=false'.

Simple huh? Or maybe just plain scary... annotations with databases!

Friday, September 19, 2008

My Google Developer Day 2008 at Wembley Stadium

I recently attended the Google Developer Day 2008, Tuesday 16th September at Wembley Stadium (London). I went to the following sessions:

  • Keynote - Tom Stocky, Mike Jennings (Android bit), Sumit Chandel (GWT bit)
  • The State of AJAX - Dion Almaer
  • A Deeper Look at Google App Engine - Mano Marks
  • What's New in Geo - Jean-Laurent Wotton (Google), Russell Middleton (Google) and Angela Relle (Met Office)
  • V8: the Chrome engine - Kevin Millikin

What follows are my rough notes from my memory - by that I mean brain memory not USB memory. Also there liable to be some flaws in my understanding!

Keynote

Tom Stocky (Google's Director of Product Management) began the day with his keynote speech. This touched upon many of Google's latest and greatest offerings; Chrome, App Engine, Android, GWT and Open Social. The first European demo of an Android device (HTC dream?) was apparently the most newsworthy. The salient points for me where concerning Chrome and GWT. Concerning Chrome - a little bit of hype, some justification. Some explanation of what makes Chrome different, multiprocess architecture (therefore increased stability), high performance V8 javascript engine and something about re-invigorating the browser arms race. I have been aware of GWT (apparently pronounced "GWIT" as in twit) but never used it - it could be described as a superior high performance AJAX applications development tool, producing obfuscated JavaScript for people happier debugging their JavaScript in Java IDEs!

The State of AJAX

Dion Almaer (of Ajaxian.com fame) gave a very entertaining and detailed talk concerning the current AJAX landscape (not just what Google are doing but more general). Some of the thing this presentation touched on included:

  • Cloud computing; Amazon [E2/S3 etc], Google App Engine and Aptana Cloud.
  • GWT, jQuery(+ UI), prototype(+ Scripticulous), dojo: The convergence of JavaScript libraries and how they are ALL great, use them!
  • Mozilla Prism (although no reference of Mozilla Prism's similarity to Google Chrome)
  • Various Mozilla Monkey projects, of particular interested was Screaming Monkey, a project which can install a Mozilla JavaScript engine into IE.
  • A fun demo of using a Wii controller with a browser based game (using AJAX, applet and wiiusej).
  • Brenden Eich says "Threads suck" so why not use Google Gear's worker pools for thread-like functionality?

A Deeper Look at Google App Engine

Mano Marks lead a guided tour of some the things Google App Engine has to offer and he encouraged audience interaction. He was somewhat cagey on the topics of SLAs and future language support despite receiving numerous audience queries on these subjects. The audience seemed to be very well acquainted with this technology already, for example there was no explanation given about what Sharding was and the audience didn't ask (I could only assume everybody knew already, which I was very impressed with!). After fending off the SLA and future language questions the remainder of the talk highlighted numerous Google App Engine shortcomings and some workarounds for these. Limitations highlighted:

  • Limitations of BigTable:
    • fast hashtable - access by key (up to 512 chars) should be really fast
    • little in the way of referential integrity can have equivalent to foreign keys
    • exercise all your queries in the developer environments to ensure the appropriates indexes are created for production
    • sharding, expensive to perform an object count after the fact
    • use memcache to reduce use of resources
  • File upload/download/storage limitation of 1Mb
  • FetchUrl limit of 4 seconds (there was something that had a 10 second limit but I forget what)

What's New in Geo

After lunch, there was another very detailed and interesting review of the recent Google Geo developments, including:

  • New ajax loading mechanism
  • Geocoding
  • directions
  • integration with local search
  • StreetView
  • Google Maps API for Flash®
  • Google static maps - useful for mobile apps
  • Google Earth plugin
  • MET office Outreach climate change example - with world temperature map changing over time

V8 - the Chrome engine

Probably the most deeply technical session of the day, scarily so in places, in fact several people left the talk quite early, I'm glad that I stayed. I didn't quite understand all of it. Some of the audience looked in a trance like state for this talk (it was late in the day!), I was very surprised that so many non-Googlers seemed so casually conversant with the finer details of creating optimised JavaScript compilers! The take home message that I extracted was something along the lines of:

  • JavaScript is compiled into native machine code and not interpreted
  • Up to 200% faster than previous engines but there are areas that mess this up (e.g. usage of eval() and with)
  • It works on Windows, Linux and Macs, it is Chrome itself that doesn't as yet!
  • JavaScript is an intrinsically dynamic language but most of the time developers use it just like a statically typed language. Therefore, optimisation technique from statically typed languages can be applied (at least most of the time)
  • Functions are only complied when they have been used
  • Functions used more that once are cached for future use, unless they "change"
  • Functions go through numerous states and their state determines the V8 caching approach.
    • unitialised
    • pre-monomorphic - the function has only been used once - don't do anything yet, it could be use one time init code
    • monomorphic - fast - 90% of the time this is the case - e.g. objects are used in a single way
    • megamorphic - blast the developer is actually using the more dynamic features of the JavaScript language resort to the slower uncached approach
  • JavaScript standard libraries are written in JavaScript! So any improvement in performance for the compiler will improve the performance across the board.
  • Currently, no improvement in DOM accessing technologies

Related links

Thursday, June 12, 2008

My Java App IE8 Readiness Toolkit!

Oh joy, another major HTML/CSS rendering engine for me to test my web applications against and I really cannot be bothered. IE8 may well turn out to be a really good thing but my early experiences of IE8 Beta are that it makes a right royal mess of my CSS/DHTML powered applications so until IE8 renders like IE7 and Firefox here is my servlet filter solution for my Java applications:


package edu.bristol;

import java.io.IOException;
import javax.servlet.*;
import javax.servlet.http.HttpServletResponse;

public class FixForIE8 implements Filter {

public void init(FilterConfig filterConfig)
throws ServletException {
}

public void doFilter(ServletRequest request,
ServletResponse response,
FilterChain chain)
throws IOException, ServletException {
((HttpServletResponse) response).setHeader("X-UA-Compatible", "IE=EmulateIE7");
chain.doFilter(request, response);
}

public void destroy() {
}
}

See also: Introducing IE=EmulateIE7

Thursday, May 15, 2008

Nested Menus in JavaScript: A new theme for jQuery UI Tabs

I accidentally discovered the jQuery UI Tabs widget for JavaScript rendering of tabs. I found this via the new uPortal 3 quickstart where the new skin uses the jQuery UI library extensively.

Originally, the tabs code was produced by Klaus Hartl and it is now an official part of the jQuery UI project. Something that I found particularly interesting is the nested tabs demo example. Although from one perspective this demonstrates nesting of tab menus, in the current example it doesn't look like a single nested tab menu navigation system. I thought I would attempt to theme this example to look more like the nested tab example from the navigation tabs section of the Yahoo! Design Pattern Library.

Click this link for a working example of jQuery UI nested tabs - Yahoo! style

To give the example a little more sophistication, I created another example using my new theme which makes use of the Google AJAX Feed API

Click this link for a working example of jQuery UI nested tabs - Yahoo! style accessing Google feeds

You are very welcome to download my example code jQueryNestedMenus.zip

A little fly in the ointment - Best use is DHTML/AJAX applications

The jQuery documentation says that Tabs supports loading tab content via Ajax in an unobtrusive manner and goes on to say that this degrades gracefully - the links, e.g. the content, will still be accessible with JavaScript disabled. You could argue that since the menu navigation will disappear if you clicked on a link when JavaScript is disabled then this is not a particularly graceful outcome!

Therefore, I think that using this widget for creating navigational menu systems should be confined to AJAX/DHTML JavaScript enabled applications. If you want a web application that still provides menus when JavaScript is disabled you should use a different approach (e.g. see my previous blog entry on Simple Nested Tab Menus for Java web applications).

Wednesday, April 30, 2008

YouTube widget: clone of the iGoogle gadget using Google AJAX APIs

I happened upon the Official YouTube Gadget for iGoogle the other day and thought it would be a fun project to replicate it using the myriad of Google API's that are now available. From the developer perspective, this mini application is entirely client side JavaScript - as the Google API's take care of any server side processing.

See my YouTube widget in standalone here

My workings...

I was able to extract some useful utility JavaScript code for formatting the video duration, truncating the titles, etc. from the gadget code itself:

http://www.google.com/ig/modules/youtube_videos.xml

That said, I needed to re-engineer the greater part of the widget using the available Google APIs.

Google AJAX Search API approach

I initially thought the widget would be driven primarily by Google AJAX Search API due to their support for YouTube search access (Direct Access to YouTube Channels). This support means that using a GvideoSearch for ytfeed:recently_featured would return a JSON object containing all the detail you need about that feed. However, the support for YouTube feeds is limited to variants of most_viewed, recently_featured and top_rated (and YouTube channels). The major reason that I chose to abandon this avenue was that the search API provides very limited results (GSearchgoogle.search.Search.LARGE_RESULTSET = 8 results!).

Clicking here you will see the JSON object returned by the Google Search API GvideoSearch for ytfeed:recently_featured.

Google AJAX Feed API approach

I turned my attention to the Google AJAX Feed API which does not have the same severe result set limitations as the Google AJAX Search API.

YouTube feed issues

There seem to be two variants of YouTube feeds knocking about. One in RSS 2.0 and one in Atom:

Clicking here you will see the JSON object returned by the Google Feed API result for http://youtube.com/rss/global/recently_featured.rss (RSS 2.0) using the combined JSON/XML result format [feed.setResultFormat(google.feeds.Feed.MIXED_FORMAT);].

Clicking here you will see the JSON object returned by the Google Feed API result for http://gdata.youtube.com/feeds/api/standardfeeds/recently_featured (Atom) using the combined JSON/XML result format [feed.setResultFormat(google.feeds.Feed.MIXED_FORMAT);].

Using this JSON pretty printer and changing the name of the callback in the GvideoSearch and Gfeed calls I was able to display a live version of the recently_featured feeds for side by side comparison. For some reason this only works properly in Firefox.

The problem with using the YouTube RSS 2.0 feed is that the supplied feed is not valid RSS 2.0. The duration attribute is not a supported part of RSS 2.0. Therefore, when you fetch the feed via the Google AJAX Feed API you get no duration information.

To get the complete behaviour required for the widget (with durations and more than 8 results), I have to use the GData YouTube feed. In the GData feed the duration information is inside a special YouTube namespace. For parsing elements that are within namespace to work across browsers you will need to make use of the Feed API's google.feeds.getElementsByTagNameNS. A slight down side to using the Google AJAX Feed API to fetch the feeds is that each feed is cached for about an hour.

JavaScript Pagination

After some JavaScript development of my own, using the Google Feed API, I was able to construct a widget that displayed video thumbnails, titles, durations, SWF urls and on clicking the thumbnail I could even invoke a flash player (Embedding the YouTube player using SWFObject). To add the final magic, I needed to paginate the video thumbnails like the official gadget does. I found a very nice JavaScript script for paginating tables and my paging mechanism is derived from that. My modified version works with any element type (not just tables) and it achieves this by making use of the the ultimate getElementsByClassName method.

See my modified paging mechanism in standalone here.

Conclusions

A very enjoyable experience (a couple of days messing about!). I hope this offers some insight into some of the problems you might run into when working with the Google APIs. The final widget comes pretty close to the functionality of the official gadget and I'm sure I could polish this further.

Friday, April 25, 2008

PartnerBar: Google's feed aggregating magic javascript widget thingy

I was browsing the Google AJAX Search API blog recently when I discovered they were using an interesting widget at the bottom of the page, it turned out to be PartnerBar. PartnerBar was created as an example for the Google AJAX Feed API. I have been very busy with family stuff recently so I managed to completely miss this when news of it first broke in December (Ajax Feed Partner Bar and The PartnerBar - Contextual Cross Linking). In my defence this is easy to miss, it is called "PartnerBar", which means nothing to anybody on this side of the pond and the launch article talks about "contextual cross linking" (meaningless technobabble!). Essentially, PartnerBar is a feed aggregating magic javascript widget thingy, if it had been launched as such I would have spotted it sooner.

Anyway I have been playing with it recently and have made some minor tweaks to improve it for my own usage, I'm very happy with the results. I can't decide whether to install the resulting widget permanently on my blog because I think people would find it annoying if I were to place it somewhere prominent.

My tweaks and modifications

Feed titles and links.

In the instance of PartnerBar on their blog, Google store an array of their blogs in a JavaScript file: http://www.google.com/uds/solutions/partnerbar/google-blogs.js.

Incidentally, notice how it supports tags so that you can use the same array for multiple widgets with distinct content.

My first modification was to tweak the code so that I did not have to supply the feed name and link, this information is stored in the feed (why store it twice?). To achieve this I overrode the loadPartner method - I think the trick with overriding methods in JavaScript is to make sure the method you override is a small one!

My array of blogs looks like this: blogs.js

Alternating row colours

Secondly, I wanted the feed entries to display in alternating colours. Again, I found a small JavaScript method in PartnerBar (resetClassOnPartnerDom) and overrode it (AFAIK there is no "super" keyword in JavaScript, which is a shame). In order to do this I made use of a getElementsByClassName method that I found on the web.

Final result

You should be able to see the widget in action at the top of this blog entry. However, you can link out to see the final result in standalone glory here.

Here is my final JavaScript which my customised widget uses: partnerBar.js.

Design

Although I suffer with CSS, I am not a graphic designer and so I tend to steal gather inspiration from elsewhere. I based my widget on the Infected table theme from the CSS table gallery.

Friday, April 18, 2008

Experiments and thoughts on Google App Engine

I have been experimenting with the Google App Engine (GAE) recently which is a Google hosted Python application server. I am not a native Python programmer (I've written about 5 scripts in my life). Even so, GAE is a great toy to play with.

Python on GAE

There are currently some weaknesses in the implementation of Python available in GAE (re: C libraries, XML parsers, XSLT engines) but this is a preview release and all things considered it is a very impressive offering. Why Python? We are told that "Google are huge fans of Python" and it is evidently used extensively inside Google. One of Google's other "hosted" platforms, Google's Mashup Editor, is a JavaScript application server running on top of Java, so Google obviously have experience in the Java application server area. As a Java programmer, I would get more excited if I could upload WAR files to GAE (even if I had to use proprietary Google Java APIs for some things) but you can't win them all. You never know, Java support could be just around the corner.

I'm new to writing Python web applications and so far it reminds me of writing Perl CGI code. This is not the best way to write web applications and I miss JSP at this point. There must be better ways of doing this using wsgiref and the Django framework which come pre-installed in GAE. These may help me get around the "scriptyness" of my python web application programming. My writing simple python web applications experience has surprised me somewhat as I had been drip fed with the idea that Python was a more efficient language (and hence better!) than Java/JSP (but I suspect I'm not doing it right!). So far it seems that you have to write a lot more Python code than the equivalent in JSP e.g. to get a parameter from an URL:

Python version

from cgi import parse_qs, escape

def hello_world(environ, start_response):
parameters = parse_qs(environ.get('QUERY_STRING', ''))
if 'subject' in parameters:
subject = escape(parameters['subject'][0])
else:
subject = 'World'
start_response('200 OK', [('Content-Type', 'text/html')])
return ['''Hello %(subject)s

Hello %(subject)s!

''' % {'subject': subject}]

JSP version

Hello ${empty param.subject ?  'World' : param.subject}

To be absolutely honest, I have never liked the conditional expression tenary operator (e.g '?' above) syntax and in this case I would prefer something more verbose but this illustrates the potential efficiency inherent in contemporary JSP.

Datastore on GAE

In GAE you are given access to a datastore, although this is not a relational database but an instance of Google's BigTable database. To access this database there is an SQL-like language called GQL available. We are told that the Datastore API is modelled after Django's Database API.

Worries about using GAE as a commercial platform?

If I were a business with an interest in using GAE commercially, storing my valuable data in proprietary Google storage would worry me somewhat. There are open source versions of BigTable available (e.g. HBase, HyperTable) and these owe there existence to Google but as there is not yet a standard "GQL" mechanism, data exit strategies seem limited.

I would be greatly concerned that my applications are hosted by Google and that my Python developers were exposed to Google. GAE could be described as a lobster pot. Getting Google to host your business means that Google are in a prime position to absorb successful businesses, steal talented developers, learn more about your business quicker than you can yourself and learn from your mistakes.

Some fun stuff, an example Google App Engine app

Using GAE, Mark Pilgrim's Universal Feed Parser code and the script.aculo.us JavaScript library I have cobbled together a fun example.

http://feed-ticker.appspot.com/

See how Google use the domain appspot.com which is like blogspot.com, nice! I look forward to wikispot etc.

Access the source code here:
Main script: home.py
Download the full app: feed-ticker.zip

I am still having some caching problems, please help!