Free tools

Robots.txt Examples

Robots.txt file content for studocu.com.

Robot.txt file for: studocu.com

      User-agent: *
Disallow: /file
Disallow: /queue/receive
Disallow: /api/
Disallow: /app/
Disallow: /classification/
Disallow: /translation/
Disallow: /rbac/
Disallow: /*/submit/
Disallow: /document/queue/
Disallow: /authentication/
Disallow: /notifications/
Disallow: /notifications$
Disallow: /email/
Disallow: /ajax/
Disallow: */cms/
Disallow: /cms/
Disallow: /payment/
Disallow: /payment$
Disallow: /mixpanel/
Disallow: /*/invite/
Disallow: /*/trial/
Disallow: /*/request/
Disallow: */followers$
Disallow: */followers?
Disallow: */group$
Disallow: */group?
Disallow: /*/studylist/
Allow: /en-us/studylist/*
Disallow: /practice$
Disallow: /practice?
Disallow: /practice/
Disallow: /*/registration?
Disallow: /*/registration/
Disallow: /*/login?
Disallow: /*/document/upload?
Disallow: /*/document/upload/
Disallow: /*/upgrade$
Disallow: /*/upgrade?
Disallow: /*/search/*
Disallow: /*/search?*
Disallow: /*/search?*languages=
Disallow: /*/search/*languages=
Disallow: /en/search$
Disallow: /en/search/
Disallow: /*/app-store/
Disallow: /tiktoken_bg.wasm
Disallow: /rest-api/
Disallow: /*/notes?
Disallow: /27m703Hm/xhr/api/v2/collector
Disallow: /27m703Hm/xhr/b/s
Disallow: /27m703Hm/init.js
Disallow: /*?_pxhc=*

User-agent: GPTBot
User-agent: Google-Extended
User-agent: YandexBot
User-agent: Baiduspider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Applebot-extended
User-agent: CCBot
User-agent: anthropic-ai
User-agent: ClaudeBot
User-agent: Claude-web
User-agent: omgili
User-agent: cohere-ai
User-agent: Scrapy
User-agent: Diffbot
User-agent: ImagesiftBot
Disallow: /

Sitemap: https://www.studocu.com/sitemap/document_index.xml
Sitemap: https://www.studocu.com/sitemap/institution_index.xml
Sitemap: https://www.studocu.com/sitemap/course_index.xml
Sitemap: https://www.studocu.com/sitemap/qa_index.xml
Sitemap: https://www.studocu.com/sitemap/qa-queue_index.xml
Sitemap: https://www.studocu.com/sitemap/quiz_index.xml
Sitemap: https://www.studocu.com/sitemap/subject_index.xml
Sitemap: https://www.studocu.com/sitemap/book_index.xml
Sitemap: https://www.studocu.com/sitemap/high-school_index.xml
Sitemap: https://www.studocu.com/sitemap/high-school-course_index.xml
Sitemap: https://www.studocu.com/sitemap/high-school-degree_index.xml
Sitemap: https://www.studocu.com/sitemap/degree_index.xml
Sitemap: https://www.studocu.com/sitemap/degree-course_index.xml