Free tools

Robots.txt Examples

Robots.txt file content for jagran.com.

Robot.txt file for: jagran.com

      User-agent: *
Allow: /
Disallow: *-undefined-*
Disallow: /state/*
Disallow: /local/*
Disallow: /RelatedArticleDetail.html?*
Disallow: /assets/js/*
Disallow: /*-13276288.html
Disallow: /search/*
Disallow: /home.html
Disallow: /home-new.html
Disallow: /english/ 
Disallow: /error.html
Disallow: /JCOM_PREFIX
Disallow: /aspx?id
Disallow: /news-sitemap-nd.xml
Disallow: /web-sitemap-nd.xml
Disallow: /jagran/*.page
Disallow: /shayri/
Disallow: /iw-cc/
Disallow: /detail/
Disallow: /7176/
Disallow: /listing/interstitial_1x1
Disallow: /*300x250
Disallow: /*980x50
Disallow: /*728x90
Disallow: /*160x600
Disallow: /*980x50
Disallow: /mp4
Disallow: /*320x75
Disallow: /photogallery/
Disallow: /lite/shayri/
Disallow: /delhi/video/
Disallow: /*/video/
Disallow: /berger-paints.html
Disallow: /berger/*
Disallow: /article2/
Disallow: /article3/
Disallow: /article4/
Disallow: /article5/
Disallow: /article6/
Disallow: /article7/
Disallow: /*/wp-admin/
Disallow: /*/plt/
Disallow: /*-20848123.html
Disallow: /podcast/*
Disallow: /blogs/*
Disallow: /english/
Disallow: /search/*
Disallow: /topics/
Disallow: /exam-results.html
Disallow: /berger-paints.html
Disallow: /berger/*
Disallow: /podcast/*
Disallow: *?src=*
Disallow: */sports40/*
Disallow: */?s=*
Disallow: */news-news-*
Disallow: *state-news-hindi-page*
Disallow: *?utm_source*
Disallow: *?itm_source*
Disallow: *?itm_medium*
Disallow: */CategoryLanding/*
Disallow: */-*
Disallow: *-page*
Disallow: */litetest/*
Disallow: /pfp.html
Disallow: /pfp.html/
Disallow: /pfp.html/*

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ImagesiftBot 
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Amazonbot
Disallow: /


Sitemap: https://www.jagran.com/all-news-sitemap.xml
Sitemap: https://www.jagran.com/news-sitemap.xml
Sitemap: https://www.jagran.com/sitemap.xml