Pages

Tuesday, December 7, 2010

Parsing HTML in Java

I am working on a social engineering tool that requires me to download a webpage's content, parse HTML as DOM, and then replace certain parts of the page with custom code.

So I thought - no problem. HTML is a subset of XML, so I'll just use a standard SAX parser. The first stab at the code looked like this (where site is a string variable containing the HTML):


DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setValidating(false);
dbf.setNamespaceAware(true);
dbf.setIgnoringComments(false);
dbf.setIgnoringElementContentWhitespace(false);
dbf.setExpandEntityReferences(false);
DocumentBuilder db = dbf.newDocumentBuilder();
Document d = db.parse(new InputSource(new StringReader(site)));


Looks good? Wrong!!
Most HTML in websites is not well-formed XML, so the parser crashed after consuming 4 lines of HTML.

So I stumbled across a really neat Java library JTidy (http://jtidy.sourceforge.net/) which first beautifies the HTML by making it a well-formed XML, even if the website's author has been sloppy, and then parses the XML into DOM representation.
The resulting code is just as short as the original, except it actually works:


import org.w3c.tidy.Tidy;
final Tidy tidy = new Tidy();
tidy.setQuiet(true);
tidy.setShowWarnings(false);
tidy.setForceOutput(true);
Document d = tidy.parseDOM(new ByteArrayInputStream(site.getBytes("UTF-8")), null);

Reactions:

3 comments:

In contempo times there has been a billow of replica watches uk replica watches in the market.The new bearing professionals accept abundant money ability to adore the acceptable things in life.By cutting Tag Heuer replica watches they can accord in to their desires and yet not bake too abundant money.The acute user goes in for Tag Heuer Aquaracer Replicawatches because they wish to action abandoned the best.They cannot allow the prices of omega replica the absolute ones.Accept you anytime apparent any arch brilliant or sports personality cutting accepted brands? They do not!They accept watches like Rolex and Tag Heuer back it is a cachet symbol.Ordinary bodies accept no advantage afar from traveling in for replica Tag Heuer watches.If you wish to carbon the lifestyles of the affluent & famous,the easiest way is to don Tag Heuer replica watches of acclaimed brands like Rolex.Those who are anxious about the looks and superior of these replica Tag Heuer watches charge not be cartier replica worried.This is not alone accurate for Coach bags,but all artist handbags.If the bond and the seams are awkward again you can bet that the accomplishment on this bag is not real.I accept bought abounding replica artist handbags.The superior of these replica handbags is excellent.They are fabricated so exact the aforementioned with louis vuitton replica the absolute ones so that you can not acquisition that they are replicas.I abrasion my replica handbags to plan and go alfresco with my friends.None of my accompany and gucci replica colleagues begin that my artist handbags are replicas.Let me appearance you some of my replica artist handbags.

Georges Favre-Jacot on within 1929, by which occasion Zenith had accomplished watch-making history earning well-regarded cash incentives and also awards with regard to time keeping accuracy with global displays throughout Geneva, Paris, europe ,, france, Spain's capital, as well as Neuchatel. Throughout 1969, Peak cheap replica handbags made our planet's 1st computerized chronograph movement, faux montres omega the El Primero. Apart from this primary, the thing that makes this specific motion particular is that it oscillating replica chanel handbags with Thirty six, 000 alternations each hour, and so it will be able to calculate short replica Hublot watches period of time durations to a 10th of an second, a fake Breitling watches good unparalleled globe report in 1969. In 1998, Peak introduced a brand new technology involving ultra-thin movements, the particular Professional, which has been just lately selected as very best physical movements with the expert media. Zenith has been honored over 1, Five-hundred cash incentives as well as commendations : making Zenith the most granted Europe brand.

Post a Comment