express-seo-robots

Simple robots.txt file generation for express

Usage no npm install needed!

<script type="module">
  import expressSeoRobots from 'https://cdn.skypack.dev/express-seo-robots';
</script>

README

express-seo-robots

Simple robots.txt file generation for express

Install

npm install --save express-seo-robots

Usage

Add middleware to your express application before all other routes.

var express = require('express');
var app = express();
var robotsTxt = require('express-seo-robots');

// add middleware with robots.txt config
app.use(robotsTxt({ userAgent: '*', allow: '/', sitemap: 'https://yourdomain.com/sitemap.xml' }));

// standard express route
app.get('/', function(req, res) {
    res.send('Hello world!');
});

Based on the setup above, a requests to /robots.txt will now return:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

You can provide multiple entries by passing an array of values, the following:

// add robots.txt middleware with custom config
app.use(robotsTxt([
    { userAgent: 'Googlebot-news', allow: '/news', CrawlDelay: '5' },
    { userAgent: 'Googlebot', disallow: '/private' },
    { userAgent: 'Googlebot' disallow: '/*.xls

 }
]);

Produces:

User-agent: Googlebot-news
Allow: /
Crawl-delay: 5

User-agent: Googlebot
Disallow: /private

User-agent: Googlebot
Disallow: /*.xls$

Options

Property Type Description
userAgent string Name of a search engine/bot that the rule applies to
allow string URL relative to the root domain that should be crawled by the user agent
disallow string URL relative to the root domain that should not be crawled by the user agent
crawlDelay integer Time in seconds the search engine/bot should wait before sending the next request
sitemap Array List of fully-qualified SiteMap URLs

Contributing

Feel free to contribute, either by raising an issue or:

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -m 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request

History

For change-log, check releases.

License

Licensed under MIT License © Orca Scan, the Barcode Scanner app for iOS and Android.