Screen Scraping Connector for Loopback

NOTE: This is experimental!

This is a quick sample connector for Loopback using https://github.com/lapwinglabs/x-ray to screen scrape a website and return the data through a LB model.

Example App

An example app is provided here: https://github.ibm.com/apiconnect-solution-architecture/example-screenscrape-app

Usage

This connector requires two properties:

website: The URL of the page you want to scrape.

domTarget: The root DOM node from which to start the screen scrape.

Example:

{
    "dribbble": {
        "connector": "screenscrape",
        "name": "dribbble",
        "remotingEnabled": true,
        "website": "https://dribbble.com/search?q=mobile",
        "domScope": "li.group"
    }
}

For the model field definitions, you must tell the connector what DOM target this field will be bound to. e.g.

"properties": {
    "title": {
        "type": "string",
        "domTarget": ".dribbble-img strong"
    },
    "image": {
        "type": "string",
        "domTarget": ".dribbble-img [data-src]@data-src"
    },
    "comment": {
        "type": "string",
        "domTarget": ".dribbble-img .comment"
    },
    "date": {
        "type": "type",
        "domTarget": ".dribbble-img .timestamp"
    },
    "commentCount": {
        "type": "string",
        "domTarget": ".dribbble-shot .tools .cmnt span"
    },
    "viewsCount": {
        "type": "string",
        "domTarget": ".dribbble-shot .tools .views span"
    },
    "likes": {
        "type": "string",
        "domTarget": ".dribbble-shot .tools .fav a"
    }
},

Tests

If you want to try it out quickly, as a stand-alone test:

1. Download this repo and npm install
2. Run node example/model.js
3. You'll see results from https://dribbble.com/search?q=mobile in JSON format. Checkout example/model.js for a quick example.

To run the actual test(s): npm test