Free tools

Robots.txt Examples

Robots.txt file content for francebleu.fr.

Robot.txt file for: francebleu.fr

      User-agent: *
Sitemap: https://www.francebleu.fr/sitemap.xml
Sitemap: https://www.francebleu.fr/sitemap-news.xml

# Player Embed
Disallow: /player/export
Disallow: /player/embed
Disallow: /player/evenementiel

Disallow: /9j/

# Widgets
Disallow: /widgets
Disallow: /search-widget-results
Disallow: /bleustatic-production
Disallow: /cruiser-production
Disallow: /get-block-meteo
Disallow: /cruiser-static-production
Disallow: /embed/elections

# FTV
Disallow: /tv/

# Preview
Disallow: /preview/

# Other
Disallow: /newsletter/html
Disallow: /recherche?
Disallow: /sms
Disallow: /grid
Disallow: /musique/c-est-quoi-cette-chanson/list
Disallow: /amp/

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Linguee Bot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /