README

node-htmlcarve

Extract essential meta-informations from any web page, fast and dead simple. Do you need general informations from a given html-site, like the title, a summary, a favicon or a possible RSS-Feed? Just throw an url into this module, and it'll try to find that stuff for you.

Installation

Clone this repository, grab the single coffeescript/javascript-file, or simply use NPM:

npm install htmlcarve

Usage

use it from the command line:

htmlcarve http://venturebeat.com/

or from inside a script (I'll use coffeescript):

htmlcarve = require "htmlcarve"
htmlcarve.fromUrl "http://venturebeat.com/", (error, data) ->
  console.log data.result

Samples

{ source: 
   { html_meta: 
      { title: 'Ouch: HP is now promoting PCs running Windows 7 (because Windows 8 isn\'t doing so hot) | VentureBeat | Business | by Ricardo Bilton',
        summary: undefined,
        image: 'http://venturebeat.files.wordpress.com/2014/01/patrick-collison-headshot.jpg?w=311&h=150&crop=1',
        language: 'en',
        feed: 'http://feeds.venturebeat.com/VentureBeat',
        favicon: 'http://0.gravatar.com/blavatar/6a5449d7551fc1e8f149b0920ca4b6f6?s=16',
        keywords: undefined,
        author: undefined },
     open_graph: 
      { title: 'Ouch: HP is now promoting PCs running Windows 7 (because Windows 8 isn\'t doing so hot)',
        summary: 'HP\'s new Windows 7 promotion should tell you all you need to know about the state of its Windows 8 hardware. With its latest promotion, HP is heavily pushing PCs running Windows 7, which it says it...',
        image: 'http://venturebeat.files.wordpress.com/2014/01/hp-windows.png',
        language: undefined },
     twitter_card: 
      { title: undefined,
        summary: undefined,
        image: undefined,
        author: '@chernandburn' } },
  result: 
   { title: 'Ouch: HP is now promoting PCs running Windows 7 (because Windows 8 isn\'t doing so hot)',
     summary: 'HP\'s new Windows 7 promotion should tell you all you need to know about the state of its Windows 8 hardware. With its latest promotion, HP is heavily pushing PCs running Windows 7, which it says it...',
     image: 'http://venturebeat.files.wordpress.com/2014/01/hp-windows.png',
     author: '@chernandburn',
     language: 'en',
     feed: 'http://feeds.venturebeat.com/VentureBeat',
     favicon: 'http://0.gravatar.com/blavatar/6a5449d7551fc1e8f149b0920ca4b6f6?s=16',
     keywords: undefined },
  links: 
   { deep: 'http://venturebeat.com/2014/01/20/ouch-hp-is-now-promoting-pcs-running-windows-7-because-windows-8-isnt-doing-so-hot/',
     shallow: 'http://venturebeat.com/2014/01/20/ouch-hp-is-now-promoting-pcs-running-windows-7-because-windows-8-isnt-doing-so-hot/',
     base: 'http://venturebeat.com' } }

$ htmlcarve http://www.spin.com/articles/miserable-halloween-dream-stream/
{ source: 
   { html_meta: 
      { title: 'Stream Miserable\'s Cratering \'Halloween Dream\' | SPIN | SPIN Mix | Premieres',
        summary: 'Ex-Whirr singer preps solo EP for February 18 release',
        image: 'http://www.spin.com/sites/all/themes/zen_spin/assets/images/default-images/spin-logo.png',
        language: 'en',
        feed: 'http://www.spin.com/rss.xml',
        favicon: 'http://www.spin.com/favicon.ico',
        keywords: 'miserable',
        author: undefined },
     open_graph: 
      { title: 'Stream Miserable\'s Cratering \'Halloween Dream\'',
        summary: 'Ex-Whirr singer preps solo EP for February 18 release',
        image: 'http://www.spin.com/sites/all/files/140122-miserable.jpg',
        language: undefined },
     twitter_card: 
      { title: 'Stream Miserable\'s Cratering \'Halloween Dream\'',
        summary: 'Ex-Whirr singer preps solo EP for February 18 release',
        image: 'http://www.spin.com/sites/all/files/140122-miserable.jpg',
        author: undefined } },
  result: 
   { title: 'Stream Miserable\'s Cratering \'Halloween Dream\'',
     summary: 'Ex-Whirr singer preps solo EP for February 18 release',
     image: 'http://www.spin.com/sites/all/files/140122-miserable.jpg',
     author: undefined,
     language: 'en',
     feed: 'http://www.spin.com/rss.xml',
     favicon: 'http://www.spin.com/favicon.ico',
     keywords: 'miserable' },
  links: 
   { deep: 'http://www.spin.com/articles/miserable-halloween-dream-stream/',
     shallow: 'http://www.spin.com/articles/miserable-halloween-dream-stream/',
     base: 'http://www.spin.com' } }

How does this stuff work?

Htmlcarve will process several steps to gather all that informations.

Scan for OpenGraphProtocol (OGP) metadata, and use ist. Usually these informations (if present) are the most valuable and desireable ones.
Look for Twitter Card metadata. Append the found informations.
Go through general html metatags and extract informations from there.
Merge the results. If any information is present in more than one step above, use the information from the higher-priorized source. Priorization-order: OGP > TwitterCard > HtmlMetaTags.

ToDo/Roadmap:

summarize the html-content on the given page, if no further informations are found.
extract keywords if none are present
include the full protocols, not only this quick'n'dirty hack.
include schema.org
include microdata formats

License

MIT.

Usage no npm install needed!