express-robots-middleware

Middleware for Express to add a robots.txt

Usage no npm install needed!

<script type="module">
  import expressRobotsMiddleware from 'https://cdn.skypack.dev/express-robots-middleware';
</script>

README

express-robots-middleware

Middleware for generating robots.txt in Express

Installation

npm i express-robots-middleware --save

Features

Multiple User Agents
Multiple Allow/Disallow
Crawl-delay argument

Usage

Simple initialize and assign the middleware to a route and pass in your robots.txt options see mock.js for a working example.

const robotsMiddleware = expressRobotsMiddleware({
  UserAgent: '*',
  Disallow: ['/private'],
  Allow: '/',
  CrawlDelay: '5'
}]);

app.get('/robots.txt', robotsMiddleware);

License

MIT