Free tools
Robots.txt Examples
Robots.txt file content for timesnownews.com.
Robot.txt file for: timesnownews.com
User-agent: * Disallow: /wapi/ Disallow: /?nbck=1 Disallow: /?nexthome=1 Disallow: /presite Disallow: /story/* Disallow: /test-section/ Disallow: /embed/ Disallow: /search-result/ #Sitemaps Sitemap: https://www.timesnownews.com/google-news-sitemap-en.xml Sitemap: https://www.timesnownews.com/staticsitemap/timesnow/sitemap-index.xml Sitemap: https://www.timesnownews.com/staticsitemap/timesnowvideo/sitemap-index.xml Sitemap: https://www.timesnownews.com/staticsitemap/timesnowslideshow/sitemap-index.xml #Baiduspider User-agent: Baiduspider Disallow: / #Yandex User-agent: Yandex Disallow: / #Facebook User-agent: facebookexternalhit/1.1 Disallow: User-agent: DataForSeoBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: SemrushBot Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: Meta-ExternalAgent Disallow: /