Automatically extracts structured information from webpages
Usage no npm install needed!
<script type="module">
import raneWebAutoExtractor from 'https://cdn.skypack.dev/@rane/web-auto-extractor';
</script>
README
This is my fork of indix/web-auto-extractor where I've merged some of the PRs from the original repo and added some of my own fixes. Published to npm as @rane/web-auto-extractor.
Web Auto Extractor
Parse semantically structured information from any HTML webpage.
// IF CommonJS
var WAE = require('@rane/web-auto-extractor').default
// IF ES6
import WAE from '@rane/web-auto-extractor'
var parsed = WAE().parse(sampleHTML)
Let's use the following text as the sampleHTML in our example. It uses Schema.org vocabularies to structure a Product information and is encoded in microdata format.
Input
<div itemscope itemtype="http://schema.org/Product">
<span itemprop="brand">ACME</span>
<span itemprop="name">Executive Anvil</span>
<img itemprop="image" src="anvil_executive.jpg" alt="Executive Anvil logo" />
<span itemprop="description">Sleeker than ACME's Classic Anvil, the
Executive Anvil is perfect for the business traveler
looking for something to drop from a height.
</span>
Product #: <span itemprop="mpn">925872</span>
<span itemprop="aggregateRating" itemscope itemtype="http://schema.org/AggregateRating">
<span itemprop="ratingValue">4.4</span> stars, based on <span itemprop="reviewCount">89
</span> reviews
</span>
<span itemprop="offers" itemscope itemtype="http://schema.org/Offer">
Regular price: $179.99
<meta itemprop="priceCurrency" content="USD" />
{
"microdata": {
"Product": [
{
"@context": "http://schema.org/",
"@type": "Product",
"brand": "ACME",
"name": "Executive Anvil",
"image": "anvil_executive.jpg",
"description": "Sleeker than ACME's Classic Anvil, the\n Executive Anvil is perfect for the business traveler\n looking for something to drop from a height.",
"mpn": "925872",
"aggregateRating": {
"@context": "http://schema.org/",
"@type": "AggregateRating",
"ratingValue": "4.4",
"reviewCount": "89"
},
"offers": {
"@context": "http://schema.org/",
"@type": "Offer",
"priceCurrency": "USD",
"price": "119.99",
"priceValidUntil": "5 November!",
"seller": {
"@context": "http://schema.org/",
"@type": "Organization",
"name": "Executive Objects"
},
"itemCondition": "http://schema.org/UsedCondition",
"availability": "http://schema.org/InStock"
}
}
]
},
"rdfa": {},
"jsonld": {},
"metatags": {
"priceCurrency": [
"USD",
"USD"
]
}
}
The parsed object includes four objects - microdata, rdfa, jsonld and metatags. Since the above HTML does not have any information encoded in rdfa and jsonld, those two objects are empty.
Errors
There is no guarantee against malformed content when working with live pages on
the Internet. This is especially true when webmasters attempt to work with
JSON-LD. You can expect to parse a page with malformed content at some point.
In the case of JSON-LD, the parser will accumulate any parse errors encountered
in an errors array if at least one was encountered. Note, they will be full
Node.js error objects, so be prepared to pull off the data you need.
Clients can take advantage of checking for the presence of the errors property
and respond accordingly.
Caveat
I wouldn't call it a caveat but rather the parser is strict by design. It might not parse like expected if the HTML isn't encoded correctly, so one might assume the parser is broken.
The problem here is the itemprop - productionCompany which is of itemtype - Organization doesn't have any itemprop as its children, in this case - name.
The parser assumes every itemtype contains an itemprop, or every typeof contains a property in case of rdfa. So the "Black Rhino" information is lost.
It'll be nice to fix this by having a non-strict mode for parsing this information. PRs are welcome.