simplecrawler
popularVery straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Updated by @cgiffard
webhead
An easy-to-use Node web crawler storing cookies, following redirects, traversing pages and submitting forms.
Updated by @pme-legend
robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Updated by @cakroyd
@zachleat/spider-pig
Get a list of local URL links from a root URL. Works with JavaScript generated content. Can also act as a live-DOM CSS search across multiple files (find all the templates that …
Updated by @zachleat
es6-crawler-detect
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
Updated by @jefferyhus
ppspider
web spider, support puppeteer, cheerio and so on, include task-queue and dispatcher
Updated by @xiyuan_fengyu
@waynechang65/ptt-crawler
A web crawler module designed to scarp data from Ptt.
Updated by @waynechang65
spider-detector
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
Updated by @michael.heuberger