Wednesday, February 14, 2007

Invalid XML Characters: when valid UTF8 does not mean valid XML

I was working on Java application recently when I got the following exception

org.xml.sax.SAXParseException: An invalid XML character (Unicode: 0x1a) was found in the element content of the document.

I was using Castor in an attempt to unmarshal (XML->Java Objects) an XML string. The XML originated from a bookmarks file that was uploaded into the application by the user, tidied up, transformed and then stored in a Oracle database. It was at the point at which the XML string was returned from the database and was being converted into Java objects that this error occurred.

I immediately assumed it to be some kind of character encoding conversion problem.

With some databases (e.g. MySQL) it is possible to set data to be passed as UTF8 by passing certain settings via the JDBC url. I do not use MySQL so I do not know whether this would fix my problem; although I suspect it would not.

I tried various methods to convert my XML string into valid UTF8 and I was pretty sure that I had achieved satisfactory UTF8 conversion but I still got the error.

This is when I discovered that not all valid UTF8 characters are valid XML characters, which probably makes sense (what with control characters and such) but I have never had to think about this before.

After spending several hours previously messing with numerous UTF8 conversion techniques I eventually found a solution. I found it in the Xalan mailing list. I am reproducing this solution here because it was not mentioned in the context of the "Unicode: 0x1a" error, if it was I would have found the solution more quickly. The XML standard specifies which UTF8 characters are valid in XML documents, so it is possible to take a UTF8 document and filter out all the invalid characters using a method like this:


/**
* This method ensures that the output String has only
* valid XML unicode characters as specified by the
* XML 1.0 standard. For reference, please see
* <a href="http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char">the
* standard</a>. This method will return an empty
* String if the input is null or empty.
*
* @param in The String whose non-valid characters we want to remove.
* @return The in String, stripped of non-valid characters.
*/

public String stripNonValidXMLCharacters(String in) {
StringBuffer out = new StringBuffer(); // Used to hold the output.
char current; // Used to reference the current character.

if (in == null || ("".equals(in))) return ""; // vacancy test.
for (int i = 0; i < in.length(); i++) {
current = in.charAt(i); // NOTE: No IndexOutOfBoundsException caught here; it should not happen.
if ((current == 0x9) ||
(current == 0xA) ||
(current == 0xD) ||
((current >= 0x20) && (current <= 0xD7FF)) ||
((current >= 0xE000) && (current <= 0xFFFD)) ||
((current >= 0x10000) && (current <= 0x10FFFF)))
out.append(current);
}
return out.toString();
}

I hope somebody else finds this useful (and it saves them a few hours of head scratching), alternatively if people reading this know of a better solution then please do let me know!

249 comments:

«Oldest   ‹Older   201 – 249 of 249   Newer›   Newest»
Radhika said...

Thanks for sharing this fantastic blog, really very informative. Your writing skill is very good, you must keep writing this type of blogs

Home Salon Dubai
wedding car hire gurgaon
wedding car hire banglore wedding car hire delhi
wedding car hire dehradun
wedding car hire noida

neha said...

I am very impressive to read this post thanks for sharing this very good information.
Events Planner In Delhi
Wedding Planner In Delhi
Download Free Newspaper WordPress Theme
Download Yoast SEO Premium WordPress Plugin

Rajesh said...

Nice infromation
Selenium Training In Chennai
Selenium course in chennai
Selenium Training
Selenium Training institute In Chennai
Best Selenium Training in chennai
Selenium Training In Chennai

Rajesh said...

Rpa Training in Chennai
Rpa Course in Chennai
Rpa training institute in Chennai
Best Rpa Course in Chennai
uipath Training in Chennai
Blue prism training in Chennai

Rajesh said...

Data Science Training In Chennai
Data Science Course In Chennai
Data Science Training institute In Chennai
Best Data Science Training In Chennai


Python Training In Chennai
Python course In Chennai
Protractor Training in Chennai
jmeter training in chennai
Loadrunner training in chennai

madin said...

From cool glitter trainers to decadent velvet sandals, Alexander McQueen shoes for new season ALEXANDER MCQUEEN Bags For Sale are fit for a dark historical romance. cheap ALEXANDER MCQUEEN boot womens We have selected the key styles of the season to fall for here.

Softgen Infotech said...

Such a great information for blogger iam a professional blogger thanks…

Get Best Dell Boomi Training in Bangalore from Real Time Industry Experts with 100% Placement Assistance in MNC Companies. Book your Free Demo with Softgen Infotech.

Customer relationship management said...

Its really helpful for the users of this site. I am also searching about these type of sites now a days. So your site really helps me for searching the new and great stuff.

sap s4 hana training in bangalore

sap simplefinance training in bangalore

sap training in bangalore

sap abap training in bangalore

sap basis training in bangalore

sap bi training in bangalore

sap dynpro training in bangalore

sap fico training in bangalore

Customer relationship management said...

This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information.

sap crm training in bangalore

sap ehs training in bangalore

sap bw training in bangalore

sap hana training in bangalore

sap hr training in bangalore

sap mm training in bangalore

sap pm training in bangalore

sap pp training in bangalore

Customer relationship management said...

It is very good and useful for students and developer.Learned a lot of new things from your post Good creation,thanks for give a good information.

sap ps training in bangalore

sap qm training in bangalore

sap scm training in bangalore

sap sd training in bangalore

sap srm training in bangalore

sap hybris training in bangalore

sap wm training in bangalore

sap ewm training in bangalore

Customer relationship management said...

I have to voice my passion for your kindness giving support to those people that should have guidance on this important matter.

sap solution manager training in bangalore

sap security training in bangalore

sap grc security training in bangalore

sap ui5 training in bangalore

sap bods training in bangalore

sap apo training in bangalore

sap gts training in bangalore

sap hana admin training in bangalore

Customer relationship management said...

Excellent post for the people who really need information for this technology.

sap idm training in bangalore

sap mdm training in bangalore

sap successfactor training in bangalore

sap fiori training in bangalore

sap bpc training in bangalore

sap testing training in bangalore

sap testing training in bangalore

sap simple logistics training in bangalore

meritstep Technology said...

Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
angular js online training

aj said...
This comment has been removed by the author.
Shopclues Winner List said...

Shopclues winner list here came up with a list of offers where you can win special shopclues prize list by just playing a game & win prizes.
Shopclues lucky customer
Shopclues lucky customer 2020
Winner list Shopclues
Shopclues Prize
Prize Shopclues

Praneeth M said...

It's very useful post and i had good experience with this salesforce training in bangalore who are offering good certification assistance. I would say salesforce training is a best way to get certified on crm.

Deepa said...

salesforce training in Noida is a best institute for training.

Ananad said...

I have been searching for a useful post like this on salesforce course details, it is highly helpful for me and I have a great expereince with this
Salesforce Training who are providing certificaiton and job asistance and see Salesforce Training in Hyderabad
Salesforce training in ameerpet
Salesforce training in kukatpally
Salesforce training in madhapur
Salesforce training in gachibowli  

high technologies solutions said...

Thanks for posting a blog. Very useful content shared by you

AutoCAD Training institute in Noida


AutoCAD Training institute in Delhi

Jack sparrow said...


I am really happy to say it’s an interesting post to read . I learn new information from your article , you are doing a great job . Keep it up and a i also want to share some information regarding online selenium training and selenium videos

Shananda DasGupta said...

Great post. salesforce training in Canada is a best institute for training.

madin said...

International Watch Company, or IWC, was first introduced to the world in 1868 when an American Engineer,replica watches sale Florentine Ariosto Jones, decided to open his own factory in Switzerland. replica IWC watches has earned its reputation as one of the world's finest manufacturers of Swiss luxury watches. IWC only manufactures automatic and manual timepieces which include features such as the chronograph and GMT dual time zone. IWC has definitely made its mark in the world of watches with its dedication to excellence in form and function. Using a range of supplies a large selection of IWC watches at the best sale prices online so buy yours today!

madin said...

A heritage brand with a cult following, Moncler makes cold weather effortless to maneuver. cheap moncler jackets moncler Super soft puffer jackets and packaway hooded styles offer a variety of choices that'll each prove indispensable throughout the cheap moncler short-outerwear seasons' increasingly unpredictable climes.

madin said...

replica cartier watches is the renowned French jeweller and watchmaker with a history of pioneering design and a drive for excellence. The replica calibre de cartier watchesfirst references to Cartier wristwatches date back to 1888. Late in the 19th century it was pocket and chatelaine watches that were most popular, but Louis Cartier saw a future in timepieces that could be worn on the wrist.

madin said...

replica patek philippe watches Men’s collection includes Calatrava, Nautilus, Aquanaut,replica watches and Gondolo - all available in a range from time-only styles to grand complications including perpetual calendars and minute repeaters.

Unknown said...


Thank you for sharing your post. It is awesome.

Arnsim
Best product and digital agency
Best web design and development company
Best digital marketing agency
Best customer support services
Best mobile app development company
Best design and development
Best design and development service provider
Arnsim - A web designand development company

Kranthi Kiran said...

Nice post I have been searching for a useful post like this on salesforce course details, it is highly helpful for me and I have a great experience with this   
Salesforce Training melbourne 

Rajesh said...

Nice infromation
Selenium Training In Chennai
Selenium course in chennai
Selenium Training
Selenium Training institute In Chennai
Best Selenium Training in chennai
Selenium Training In Chennai

Rajesh said...

Rpa Training in Chennai
Rpa Course in Chennai
Rpa training institute in Chennai
Best Rpa Course in Chennai
uipath Training in Chennai
Blue prism training in Chennai

Data Science Training In Chennai
Data Science Course In Chennai
Data Science Training institute In Chennai
Best Data Science Training In Chennai

Rajesh said...

Python Training In Chennai
Python course In Chennai
Protractor Training in Chennai
jmeter training in chennai
Loadrunner training in chennai

TestLeaf Software Training said...

Nice infromation
Selenium Training In Chennai
Selenium course in chennai
Selenium Training
Selenium Training institute In Chennai
Best Selenium Training in chennai
Selenium Training In Chennai

TestLeaf Software Training said...

Rpa Training in Chennai
Rpa Course in Chennai
Rpa training institute in Chennai
Best Rpa Course in Chennai
uipath Training in Chennai
Blue prism training in Chennai

TestLeaf Software Training said...

Data Science Training In Chennai
Data Science Course In Chennai
Data Science Training institute In Chennai
Best Data Science Training In Chennai


Python Training In Chennai
Python course In Chennai
Protractor Training in Chennai
jmeter training in chennai
Loadrunner training in chennai

shreekavi said...

Thanks for this wonderful blog it is really informative to all.keep update more information about this
Selenium Training in Chennai
Selenium Training in Bangalore
Selenium Training in Coimbatore
Selenium Course in Bangalore
Best Selenium Training in Bangalore
Selenium training in marathahalli
Selenium training in Btm
Ethical Hacking Course in Bangalore
Tally Course in Chennai

chasey marina said...

Your article was really impressive.Do keep post like this . You can also visit my site if you have time.Kindly visit our site for ERROR 0X50 for additional info...

Shopclues Winner List said...

Shopclues winner list here an offer you can win the Shopclues lucky draw. Here you can win the Shopclues winner list 2020 and Shopclues winner list. Shopclues lucky draw where you can win the prize just playing the game. You can also check Shopclues lucky draw and Shopclues winner name 2020.

lokesh said...

Thanks for sharing the valuable information. This blog contains various good concept and ideas.
DOT NET Training in Chennai
DOT NET Training in Bangalore
DOT NET Training Institutes in Bangalore
DOT NET Course in Bangalore
Best DOT NET Training Institutes in Bangalore
DOT NET Institute in Bangalore
Dot NET Training in Marathahalli
AWS Training in Bangalore
Data Science Courses in Bangalore
DevOps Training in Bangalore

Dharmendra Chidar said...

I have website Live Digital india for Related to Movies Entertenmaint
Download Movies
https://livedigitalindia.in

Thanks
Dharmendra

svrtechnologies said...

Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly software testing training online , but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post


I am sharing biography of top 10 businesses of india. Check my link for more information.

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post



I am sharing < biography about bill gates education

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post

I am sharing biography

about sheikh-mohammed-bin-rashid-al-maktoum Check my link for more information.

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post


I am sharing <biography
of top-10-young-entrepreneur-in-india Check my link for more information.

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post



I am sharing <biographyof TOP 10 HIGHEST PAYING JOB IN INDIA
Check my link for more information.

Daksh Khattar said...

Great post You are sharing much information keep it up ..awesome post


I am sharing <biography of top-10-richest-actor-in-india Check my link for more information.

Venkatesh CS said...

Excellent Blog. Thank you so much for sharing.
Artificial Intelligence Training in Chennai
Best Artificial Intelligence Training in Chennai
artificial intelligence training institutes in Chennai
artificial intelligence certification training in Chennai
artificial intelligence course in Chennai
artificial intelligence training course in Chennai
artificial intelligence certification course in Chennai
artificial intelligence course in Chennai with placement
artificial intelligence course fees in chennai
best artificial intelligence course in Chennai
AI training in chennai
artificial intelligence training in omr
artificial intelligence training in Velachery
artificial intelligence course in omr
artificial intelligence course in Velachery

Unknown said...

I blog quite often and I seriously appreciate your information. The article has truly peaked my interest. I'm going to book mark your website and keep checking for new details about once a week. I opted in for reviews your Feed too.

Infocampus said...

It’s hard to come by experienced people about this subject, but you seem like you know what you’re talking about! Thanks.
Java Training in Bangalore
Python Training In Bangalore

Infocampus said...

It’s hard to come by experienced people about this subject, but you seem like you know what you’re talking about! Thanks.
Java Training in Bangalore
Python Training In Bangalore

«Oldest ‹Older   201 – 249 of 249   Newer› Newest»