@achannarasappa/locust

Distributed serverless web crawling/web scraping with support for js execution

Usage no npm install needed!

<script type="module">
  import achannarasappaLocust from 'https://cdn.skypack.dev/@achannarasappa/locust';
</script>

README

Locust

Distributed web data discovery and collection framework

Quick Start

npm install @achannarasappa/locust

Features

Configuration driven jobs
Distributed execution model to support serverless architectures
Handle client-side JavaScript execution
Data extraction using CSS selectors
Depth-based stop condition along with support for custom stop conditions
Robust dev tooling with locust-cli to build and test jobs

Use Cases

Web indexing (i.e. web crawling)
Web data extraction (i.e. web scraping)

Reference