@deskeen/web-builder-create-robots-txt

Web Builder Module - Create robots.txt file

Usage no npm install needed!

<script type="module">
  import deskeenWebBuilderCreateRobotsTxt from 'https://cdn.skypack.dev/@deskeen/web-builder-create-robots-txt';
</script>

README

Web Builder Module - Create robots.txt file

This module allows the @deskeen/web-builder engine to create a robots.txt file.

Install

npm install @deskeen/web-builder
npm install @deskeen/web-builder-create-robots-txt

Usage

And add the module to the list of modules:

const builder = require('@deskeen/web-builder')
await builder.build({
  source: [
    // List of files or directories
  ],
  modules: [
    [
      '@deskeen/web-builder-create-robots-txt',
      {
        path: '/deploy/path', // Deploy path
        sitemapUrl: 'https://example.com/sitemap.xml', // Sitemap URL
        disallowedCrawlers: [
          // List of crawlers to exclude
          // Example:
          // 'Facebot'
        ],
        disallowedUrls: [
          // List of URLs to exclude
          // Example:
          // '/login'
        ],
      }
    ]
  ]
})

Popular Crawler Names

  • Apple: Applebot
  • Amazon: ia_archiver
  • Askdotcom: Teoma
  • Bing: bingbot
  • DuckDuckGo: DuckDuckBot
  • Facebook: Facebot
  • Google: Googlebot
  • Google Ads: AdsBot-Google, Mediapartners-Google
  • Google Images: Googlebot-Image
  • IBM: ScoutJet
  • MSN: msnbot
  • Twitter: Twitterbot
  • Yahoo: Slurp
  • Chinese search engine: baiduspider
  • China’s largest eCommerce site: EtaoSpider
  • Russian search engine: Yandex
  • Czech Republic search engine: seznambot

Contact

You can reach me at {my_firstname}@{my_name}.fr

Licence

MIT Licence - Copyright (c) Morgan Schmiedt