Free tools

Robots.txt Examples

Robots.txt file content for timesnownews.com.

Robot.txt file for: timesnownews.com

      User-agent: *
Disallow: /wapi/
Disallow: /?nbck=1
Disallow: /?nexthome=1
Disallow: /presite
Disallow: /story/*
Disallow: /test-section/
Disallow: /embed/
Disallow: /search-result/

#Sitemaps
Sitemap: https://www.timesnownews.com/google-news-sitemap-en.xml
Sitemap: https://www.timesnownews.com/staticsitemap/timesnow/sitemap-index.xml
Sitemap: https://www.timesnownews.com/staticsitemap/timesnowvideo/sitemap-index.xml
Sitemap: https://www.timesnownews.com/staticsitemap/timesnowslideshow/sitemap-index.xml

#Baiduspider
User-agent: Baiduspider
Disallow: /

#Yandex
User-agent: Yandex
Disallow: /


#Facebook
User-agent: facebookexternalhit/1.1
Disallow:

User-agent: DataForSeoBot
Disallow: /
 
User-agent: Screaming Frog SEO Spider
Disallow: /
 
User-agent: SemrushBot
Disallow: /

User-Agent: PerplexityBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /