1clickcrawl

packaged configuration web crawler

Usage no npm install needed!

<script type="module">
  import clickcrawl from 'https://cdn.skypack.dev/1clickcrawl';
</script>

README

1ClickCrawler

Simple quick webcrawler for nodejs. Provide your own visit function and the crawler will do the rest.

Installation

$ npm install 1clickcrawler

Usage

var Crawler = require('1clickcrawler');
var crawler = new Crawler(delay, seeds, visit);
crawler.start()

Parameters

delay: milliseconds to wait between requests (ex: 200) seeds: array of urls to start the crawl (ex: ['https://example.com']) visit: a function that will process the data with params url, res, html (ex: function(url,res,html) {})