Search & AI Crawler Control

Robots.txt File Generator

Configure how Googlebot, Bingbot, and modern AI engines (*GPTBot, PerplexityBot, ClaudeBot*) crawl and index your domain.

Crawler Directives

Allow AI Crawlers (GEO Optimization)

Allows ChatGPT, Perplexity, and Claude to cite your site in AI search answers.

Disallow Sensitive Paths

Blocks search crawlers from scraping `/admin`, `/api`, `/dashboard`, and internal endpoints.

Explicit GooglebotExplicit Bingbot

Sitemap URL Directive

Crawl Delay

robots.txt Output

# Standard directives for search crawlers
User-agent: *
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /api/*
Disallow: /dashboard/*

# Googlebot explicit allowance
User-agent: Googlebot
Allow: /

# Bingbot explicit allowance
User-agent: Bingbot
Allow: /

# AI crawlers for GEO search engine optimization (ChatGPT, Perplexity, Claude)
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /

# Main XML sitemap indexing directive
Sitemap: https://tejrank.com/sitemap.xml

Instant search indexing with IndexNow

Don't wait weeks for crawlers to find your robots.txt or sitemaps. TejRank natively pings the IndexNow protocol the second you publish so Bing and Yandex index your URLs within minutes.

Explore TejRank Indexing