Distributed serverless web crawling/web scraping with support for js execution

Usage no npm install needed!

<script type="module">
  import achannarasappaLocust from 'https://cdn.skypack.dev/@achannarasappa/locust';


Build Status Coverage Status


Distributed web data discovery and collection framework

Quick Start

npm install @achannarasappa/locust


  • Configuration driven jobs
  • Distributed execution model to support serverless architectures
  • Handle client-side JavaScript execution
  • Data extraction using CSS selectors
  • Depth-based stop condition along with support for custom stop conditions
  • Robust dev tooling with locust-cli to build and test jobs

Use Cases

  • Web indexing (i.e. web crawling)
  • Web data extraction (i.e. web scraping)