Robots.txt Generator

Create, validate, and download robots.txt files for your website. Configure user-agent rules, sitemaps, crawl-delay, and export instantly.

Quick Presets

Configuration

Rule Group 1

Preview

User-agent: *
Allow: /

Validation

No issues found. Your robots.txt looks good.

よくある質問

Q. robots.txtはSEOに影響しますか？

A. 直接的なランキング要因ではありませんが、クローラーのアクセスを制御することでインデックスされてほしくないページを除外できます。重複コンテンツ問題の解消にも役立ちます。

Q. robots.txtで全クローラーをブロックするには？

A. User-agent: * に Disallow: / を設定することで全クローラーのアクセスをブロックできます。ただし既にインデックスされたページは削除されません。

Q. robots.txtはどこに配置すればいいですか？

A. Webサイトのルートディレクトリに配置し、https://yourdomain.com/robots.txt でアクセスできる状態にする必要があります。サブディレクトリへの配置は無効です。

関連ツール

メタタグジェネレーターOGPプレビュー

What Is a robots.txt File?

A robots.txt file is a plain text file placed at the root of your website that tells web crawlers which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol and is the first file crawlers check before indexing your site. Every major search engine, including Google, Bing, and Yahoo, respects robots.txt directives.

robots.txt Syntax Explained

A robots.txt file consists of one or more rule groups. Each group starts with a User-agent directive followed by Allow and Disallow rules. Here is a basic example:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

The wildcard * matches all crawlers. You can create separate rule groups for specific bots like Googlebot or Bingbot. The Sitemap directive tells crawlers where to find your XML sitemap.

How to Use This Generator

  1. Choose a preset or start from scratch. Presets cover common scenarios like allowing all bots, blocking all bots, blocking AI crawlers, or a standard SEO setup.
  2. Add user-agent groups for each bot or group of bots you want to configure. Select from common bots or enter a custom name.
  3. Define Allow and Disallow rules for each user-agent. Add as many path rules as needed.
  4. Set optional parameters like Crawl-delay and Sitemap URL.
  5. Review the live preview with syntax highlighting and check validation warnings.
  6. Copy or download the generated robots.txt file and place it at the root of your website.

Where to Place robots.txt

The robots.txt file must be placed at the root of your domain. For example, if your site is https://example.com, the file should be accessible at https://example.com/robots.txt. It will not work if placed in a subdirectory. Each subdomain needs its own robots.txt file.

Common robots.txt Mistakes

  • Forgetting to include a wildcard (*) user-agent rule, leaving unmatched bots with no directives.
  • Using Disallow: / for all bots, which blocks your entire site from search engines.
  • Placing the file in the wrong directory or on a different subdomain.
  • Not including a Sitemap directive, which helps crawlers discover all your pages.
  • Using relative paths instead of paths starting with a forward slash.

Blocking AI Crawlers

With the rise of AI training crawlers, many site owners want to prevent their content from being used to train language models. Bots like GPTBot (OpenAI), Google-Extended (Google AI), CCBot (Common Crawl), and ChatGPT-User can be specifically blocked using dedicated user-agent rules. Use the "Block AI Bots" preset above for a quick configuration.