Sunday, February 12, 2006

An XML to JSON webservice

On a blog entry I was reading somebody commented that all we need now is a general purpose XML to JSON webservice. I thought a syndicated feed XML to JSON service would be really easy to produce, so I thought I'd knock one up for fun! Essentially this is a guise of the proxy that is necessary in AJAX apps to get around the XMLHttpRequest security restrictions. This uses the On Demand JavaScript approach, advocated by the Yahoo JSON API and used by Google Maps, meaning it doesn't suffer the same server security constraint and could therefore be used by remote servers (the transport format could be XML or JSON but I've chosen to use JSON to satisfy my own curiosity).

I recently questioned the suggestion that JSON is easier to use than XML (especially since E4X has arrived on the scene). This outburst followed an article on XML.com that seemed to suggest that JSON was an essential part of a recipe that could be used to get around some of the short comings of AJAX. I hope I showed that XML could have equally well have been used in the place of JSON using the same approach.

I also pointed out that pretty much Yahoo were the only chaps to offer JSON as an output format. I have been playing with JSON recently and my position has softened towards it, I still don't think that is a ready made replacement for XML in all circumstances but it does have its place.

Looking at Yahoo's JSON API I thought that the easiest way to do this is with a plain old java servlet (I suppose it is called a POJS nowadays ;)). First we need someway of accessing multiple syndication formats. I'm going to use Rome and Rome's Fetcher subproject to retrieve the feeds. Rome can access multiple feed formats (including Atom) and converts them to a common internal object representation. Having obtained an object representation all we need to do is output this information in the JSON format. There is Java JSON API available to help with the JSON conversion. After a short amount of coding I have my basic XML to JSON servlet (okay there is still room for improvement but it does the job).


package xmltojson;

import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.fetcher.FeedFetcher;
import com.sun.syndication.fetcher.FetcherException;
import com.sun.syndication.fetcher.impl.FeedFetcherCache;
import com.sun.syndication.fetcher.impl.HashMapFeedInfoCache;
import com.sun.syndication.fetcher.impl.HttpURLFeedFetcher;
import com.sun.syndication.io.FeedException;
import java.io.IOException;
import java.io.PrintWriter;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Iterator;
import java.util.List;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

public class XMLtoJSONServlet extends HttpServlet {

FeedFetcherCache feedInfoCache;
FeedFetcher feedFetcher;

public void init()
throws ServletException {
feedInfoCache = HashMapFeedInfoCache.getInstance();
feedFetcher = new HttpURLFeedFetcher(feedInfoCache);
}

protected void doGet(HttpServletRequest req,
HttpServletResponse res)
throws ServletException,
IOException {
String callback = req.getParameter("callback");
String url = req.getParameter("url");
if (callback==null)
callback = "";
if (url==null)
url="http://content.mark-mclaren.info/rss.xml";
PrintWriter out = res.getWriter();
out.print(callback + "("+fetchFeed(url)+")");
}

private String fetchFeed(String url)
throws IOException {
try {
SyndFeed feed = feedFetcher.retrieveFeed(new URL(url));
return syndFeed2JSON(feed);
} catch (MalformedURLException e){
} catch (FeedException e){
} catch (FetcherException e){
}
return "Not working";
}

private String syndFeed2JSON(SyndFeed feed) {
JSONObject jsonFeed = new JSONObject();
try {
String title = feed.getTitle();
String link = feed.getLink();
jsonFeed.put("title", title);
jsonFeed.put("link", link);
JSONArray jsonFeedEntries = new JSONArray();
List entries = feed.getEntries();
Iterator i = entries.iterator();
while (i.hasNext()) {
SyndEntry entry = (SyndEntry) i.next();
String itemTitle = entry.getTitle();
String itemLink = entry.getLink();
JSONArray jsonFeedEntry = new JSONArray();
jsonFeedEntry.put(itemTitle);
jsonFeedEntry.put(itemLink);
jsonFeedEntries.put(jsonFeedEntry);
}
jsonFeed.put("entries",jsonFeedEntries);
return jsonFeed.toString();
} catch (JSONException e) {
}
return "";
}

}

You can download the above servlet source here. See this example of what the above servlet outputs.

Add a smattering of JavaScript and some CSS borrowed from Listutorial and we have a working example! If you haven't guessed already the RSS list at the top left of this blog entry is produced using the XML to JSON approach (if you don't have JavaScript enabled you won't see it).

For this blog entry I have taken a static copy of what the servlet would output as I don't want to provide the XML to JSON conversion gateway for the world.

There is no reason to stop at XML to JSON. Add a little Spring Framework magic and it could quite easily become a database => JSON or webservice => JSON gateway. Let the mashups begin!

6 comments:

Mark McLaren said...

Thanks for this snippet!
Note: Comment imported. Original by Anonymous at 2007-06-01 20:16

Mark McLaren said...

I wouldn't mind using this if I didn't have to go chasing for Rome's implementation of the fetcher etc. Good posting, useless without the Rome jars nonetheless.
Note: Comment imported. Original by Anonymous at 2007-09-09 19:53

Mark McLaren said...

Hi,

Thanks for great sample, can you please help me to understand how you are making the call to the server, I mean where in the code XMLHttpRequest is made?



thanks



Amy
Note: Comment imported. Original by Anonymous at 2007-12-11 00:05

Mark McLaren said...

Hi Amy,



The beauty of this approach is that it does *NOT* use XMLHttpRequest. There are security restrictions with XMLHttpRequest which mean that it cannot access data from any server except the server where your web page is hosted. So you cannot easily access remote data without a proxy. The On Demand JavaScript approach uses a sort of proxy but the advantage is that the data can then be accessed by any host.





Here is my explanation of what is happening:



Stage 1 - proxying the data and establishing the callback function name



First you need some kind of proxy. In the above case the proxy is a Java Servlet but it could be a script of some kind (e.g. ASP, Perl, PHP or whatever). This proxy fetches the data (in the above case my RSS feed) and puts it into a format that can be used later (e.g. XML or JSON). The important part of this is that the resulting data should include a callback function name.





So you end up with a script that outputs something like:





getData(<formatted data goes here>)





where "getData" is the name of your callback function. In the above example the output I am using is XMLtoJSON.txt. For this example this data is a static text file but it should be some server side dynamically generated output. Notice the callback function is called "getRss". Most of Yahoo's web services can output data in this format and you can change the name of the callback function dynamically. e.g.





http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Madonna&output=json&callback=getResults



Stage 2 - making use of the data



What the proxy actually generates is a small piece of JavaScript. Although XMLHttpRequest restricts the access of data to the domain where the hosted page is located, there is no such restriction about loading remote JavaScript files. So the next stage is to load the remote JavaScript code and make use of it by creating the callback function.



example 1 (from the example above):



<script type="text/javascript">

function getRss(data) { alert(data.title + "\n" + data.link);}

</script>

<script type="text/javascript" src="http://content.mark-mclaren.info/XMLtoJSON/XMLtoJSON.txt"></script>





for more advanced processing of the data see xml2json.js



example 2 (from Yahoo Web Services)



<script type="text/javascript">

function getResults(data) { alert(data.ResultSet.totalResultsAvailable); }

</script>

<script type="text/javascript" src="http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Madonna&output=json&callback=getResults"></script>





You will see that in your webpage you need to create a function with the same name as the callback function name you used. So what the proxy generates becomes a function call passing in the retrieved data. All that remains is that you need to learn how to access data in the returned format (e.g. JSON format is the most popular for this - although technically this approach could also work with XML).





For more information see Yahoo's documentation on this approach Using JSON with Yahoo! Web Services





I hope this helps, Mark


Note: Comment imported. Original by markmc website: http://content.mark-mclaren.info/ at 2007-12-11 17:55

Mark McLaren said...

hi, am doing something a bit similar, but while the stand-alone java program works...fetching my feeds properly, When i try it using servlets, tomcat gives me this error:

exception



javax.servlet.ServletException: Servlet execution threw an exception

root cause

java.lang.NoClassDefFoundError: com/sun/syndication/io/SyndFeedInput



can u advise please.
Note: Comment imported. Original by Roushdat website: http://www.roushdat.com at 2008-02-18 16:32

Mark McLaren said...

You need a copy of Rome and Rome Fetcher for this to work (I think you might need JDOM also). Get them here:





https://rome.dev.java.net/source/browse/rome/www/dist/


Note: Comment imported. Original by markmc website: http://content.mark-mclaren.info/ at 2008-02-18 22:09