express-seo-robots

Simple robots.txt file generation for express

Install

npm install --save express-seo-robots

Usage

Add middleware to your express application before all other routes.

var express = require('express');
var app = express();
var robotsTxt = require('express-seo-robots');

// add middleware with robots.txt config
app.use(robotsTxt({ userAgent: '*', allow: '/', sitemap: 'https://yourdomain.com/sitemap.xml' }));

// standard express route
app.get('/', function(req, res) {
    res.send('Hello world!');
});

Based on the setup above, a requests to /robots.txt will now return:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

You can provide multiple entries by passing an array of values, the following:

// add robots.txt middleware with custom config
app.use(robotsTxt([
    { userAgent: 'Googlebot-news', allow: '/news', CrawlDelay: '5' },
    { userAgent: 'Googlebot', disallow: '/private' },
    { userAgent: 'Googlebot' disallow: '/*.xls

 }
]);

Produces:

User-agent: Googlebot-news
Allow: /
Crawl-delay: 5

User-agent: Googlebot
Disallow: /private

User-agent: Googlebot
Disallow: /*.xls$

Options

Property	Type	Description
userAgent	string	Name of a search engine/bot that the rule applies to
allow	string	URL relative to the root domain that should be crawled by the user agent
disallow	string	URL relative to the root domain that should not be crawled by the user agent
crawlDelay	integer	Time in seconds the search engine/bot should wait before sending the next request
sitemap	Array	List of fully-qualified SiteMap URLs

Contributing

Feel free to contribute, either by raising an issue or:

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -m 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request

History

For change-log, check releases.

License

Licensed under MIT License © John Doherty