Free tools
Robots.txt Examples
Robots.txt file content for jagran.com.
Robot.txt file for: jagran.com
User-agent: * Allow: / Disallow: *-undefined-* Disallow: /state/* Disallow: /local/* Disallow: /RelatedArticleDetail.html?* Disallow: /assets/js/* Disallow: /*-13276288.html Disallow: /search/* Disallow: /home.html Disallow: /home-new.html Disallow: /english/ Disallow: /error.html Disallow: /JCOM_PREFIX Disallow: /aspx?id Disallow: /news-sitemap-nd.xml Disallow: /web-sitemap-nd.xml Disallow: /jagran/*.page Disallow: /shayri/ Disallow: /iw-cc/ Disallow: /detail/ Disallow: /7176/ Disallow: /listing/interstitial_1x1 Disallow: /*300x250 Disallow: /*980x50 Disallow: /*728x90 Disallow: /*160x600 Disallow: /*980x50 Disallow: /mp4 Disallow: /*320x75 Disallow: /photogallery/ Disallow: /lite/shayri/ Disallow: /delhi/video/ Disallow: /*/video/ Disallow: /berger-paints.html Disallow: /berger/* Disallow: /article2/ Disallow: /article3/ Disallow: /article4/ Disallow: /article5/ Disallow: /article6/ Disallow: /article7/ Disallow: /*/wp-admin/ Disallow: /*/plt/ Disallow: /*-20848123.html Disallow: /podcast/* Disallow: /blogs/* Disallow: /english/ Disallow: /search/* Disallow: /topics/ Disallow: /exam-results.html Disallow: /berger-paints.html Disallow: /berger/* Disallow: /podcast/* Disallow: *?src=* Disallow: */sports40/* Disallow: */?s=* Disallow: */news-news-* Disallow: *state-news-hindi-page* Disallow: *?utm_source* Disallow: *?itm_source* Disallow: *?itm_medium* Disallow: */CategoryLanding/* Disallow: */-* Disallow: *-page* Disallow: */litetest/* Disallow: /pfp.html Disallow: /pfp.html/ Disallow: /pfp.html/* User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Bytespider Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Amazonbot Disallow: / Sitemap: https://www.jagran.com/all-news-sitemap.xml Sitemap: https://www.jagran.com/news-sitemap.xml Sitemap: https://www.jagran.com/sitemap.xml