ebri-scrap

EbriScrap is a tool that parse a HTML string, and return a JS object with all information that you need.

Usage no npm install needed!

<script type="module">
  import ebriScrap from 'https://cdn.skypack.dev/ebri-scrap';
</script>

README

EbriScrap

EbriScrap is a tool that parse a HTML string, and return a JS object with all information that you need.

npm version Build Status codecov Known Vulnerabilities Maintainability

Installing

With Yarn

yarn add ebri-scrap

With NPM

npm install ebri-scrap

Examples

Go to Examples to see EbriScrap in action.

Documentation

Configuration

Root EbriScrap configuration can be a string, an object, or an array.

Configuration Item

Field

In order to extract a single value, you have to use a field configuartion item. You have to specify one selector, up to one extractor and as many formators as you want !

Here is an example: "selector | extract:extractor1 | format:formator1 | format:formator2".

Selector

It should be a valid Cheerio / CSS selector. Example: h1.

Extractor
  • text (default): it calls .innerText on the HTML element matched by the selector.

    Example:

    const html = '<div>Hello world</div>';
    
    const config = 'div'; // or const config = 'div | extract:text';
    
    parse(html, config); // Output: "Hello world"
    
  • html: it returns the raw HTML of the HTML element matched by the selector.

    Example:

    const html = '<div>Hello world</div>';
    
    const config = 'div | extract:html';
    
    parse(html, config); // Output: "<div>Hello world</div>"
    
  • prop: it returns a property of the HTML element matched by the selector.

    Example:

    const html = '<a href="/unicorn-world">Hello world</div>';
    
    const config = 'a | extract:prop:href';
    
    parse(html, config); // Output: "/unicorn-world"
    
  • css: it returns a css of the HTML element matched by the selector. Warning: it only works with style property on the element !

    Example:

    const html = '<div style="font-size: 42px">Hello world</div>';
    
    const config = 'a | extract:css:font-size';
    
    parse(html, config); // Output: "42px"
    
Formattors
  • number: strip all no-digit characters and parse as float

    Example:

    const html = '<div>42</div>';
    
    const config = 'div | format:number';
    
    parse(html, config); // Output: 42
    
  • regex: find and replace with a text, using a regular expression. This formator needs two parameters: format:<THE_REGEX>:<REPLACEMENT_STRING>

    Example:

    const html = '<div>42</div>';
    
    const config = 'div | format:regex:4(.*):$1';
    
    parse(html, config); // Output: 2
    
  • url: add a base url if the path is relative This formator needs one parameter: format:<BASE_URL>

    Example:

    const html = '<a href="/unicorn-world">Hello world</div>';
    
    /* WARNING: as https://one-fake-domain.com contains colons, quotes (single or double) are mandatory ! */
    const config =
      "a | extract:prop:href | format:url:'https://one-fake-domain.com'";
    
    parse(html, config); // Output: "https://one-fake-domain.com/unicorn-world"
    
  • html-to-text: replace <br>, <p>, <div> with new lines, and then, returns text.

  • one-line-string: replace all new lines (\n), tabs (\t) and multi spaces with a single space

  • trim: remove leading and ending spaces

Group

A group configuration is a dictionary in which keys are the keys of the output object, and values are a piece of EbriScrapConfiguration (another group configuration, a field or an array configuration).

Example:

const html = `
    <section>
        <h1>What a wonderful world</h1>
        <p>Lorem Ipsum...</p>
    </section>`;

const config = {
    title: 'h1',
    content: 'p',
};

parse(html, config); // Output: { title: 'What a wonderful world': content: 'Lorem Ipsum...' }

Array

Array configuration are a bit more complicated. It is an array, with a single item with additional information:

  • containerSelector: the selector of the container (It should be a valid Cheerio / CSS selector.)
  • itemSelector: the selector on which you want to iterate (It should be a valid Cheerio / CSS selector.)
  • data: a Field/Group/Array configuration

Example:

const html = `
    <ul>
        <li>
            <p>Content 1</p>
        </li>
        <li>
            <p>Content 2</p>
        </li>
        <li>
            <p>Content 3</p>
        </li>
    </ul>`;

const config = [
    {
        containerSelector: 'ul',
        itemSelector: 'li',
        data: 'p',
    },
];

parse(html, config); // Output: ['Content 1', 'Content 2', 'Content 3']

Walkthrough example

Say we have the following HTML:

<html>
  <head>
    <title>EbriScrap</title>
  </head>
  <body>
    <table>
      <thead>
        <tr>
          <th>JS Library</th>
          <th>Link</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>Lodash</td>
          <td><a href="https://github.com/lodash/lodash">Github</a></td>
        </tr>
        <tr>
          <td>Cheerio</td>
          <td><a href="https://github.com/cheeriojs/cheerio">Github</a></td>
        </tr>
        <tr>
          <td>jQuery</td>
          <td><a href="https://github.com/jquery/jquery">Github</a></td>
        </tr>
      </tbody>
    </table>
  </body>
</html>

And we want to get the following object:

{
  name: 'EbriScrap',
  librairies: [
    {
      name: 'Lodash',
      link: 'https://github.com/lodash/lodash'
    },
    {
      name: 'Cheerio',
      link: 'https://github.com/cheeriojs/cheerio'
    },
    {
      name: 'jQuery',
      link: 'https://github.com/jquery/jquery'
    }
  ]
}

Here is what you have to do:

Name

It is a regular text field. We want to extract the text in <title>:

const nameConfig = 'title';

Library name

One again, it is a regular text field. We want to extract the text in the first <td>:

const libNameConfig = 'td:first-of-type';

Library link

This time, we want to get the link in the href property of the <a> in the second <td>:

const libLinkConfig = 'td:nth-of-type(2) a | extract:prop:href';

Library

Now that we have name and url, we want to create an object with two keys (name and url):

{
  name: /* Library name configuration item */,
  link: /* Library link configuration item */
}

// so we have:
{
  name: 'td:first-of-type',
  link: 'td:nth-of-type(2) a | extract:prop:href'
}

Libraries

We want to apply librairy configuration on every <tr> in <tbody>:

[
  {
    containerSelector: 'tbody',
    itemSelector: 'tr',
    data: /* Library configuration item */
  }
]

Resulting configuration

Here is the full configuration:

{
  name: 'title',
  librairies: [
      {
        containerSelector: 'tbody',
        itemSelector: 'tr',
        data: {
          name: 'td:first-of-type',
          link: 'td:nth-of-type(2) a | extract:prop:href'
        },
      }
  ]
}