Free tools

Robots.txt Examples

Robots.txt file content for stylecraze.com.

Robot.txt file for: stylecraze.com

      User-agent: *    # applies to all robots
Disallow: /inputfile1.txt
Disallow: /wp-dp.php
Disallow: /fork/stylecraze.html
Disallow: /feed-stylecrazev44.php
Disallow: /tag/
Disallow: /banners/
Disallow: /staging/
Disallow: /activity/
Disallow: /affiliates/
Disallow: /questions/tags/
Disallow: /disscussion-forums/
Disallow: /filter-products
Disallow: /xn/
Disallow: /pages/
Disallow: /page/
Disallow: /photo/
Disallow: /video/
Disallow: /blog/
Disallow: /profiles/
Disallow: /forum/
Disallow: /article/
Disallow: /groups/
Disallow: /comment-subscriptions
Disallow: /questions/
Disallow: /photos/
Disallow: /videos/
Disallow: /members/
Disallow: /brands/
Disallow: /add-blog-post/
Disallow: /create/
Disallow: /upload-video/
Disallow: /shop/*
Disallow: /blogs/
Disallow: */view/slideshow/
Disallow: /wp-admin/
Disallow: /venus*
Disallow: /fb-posts/
Disallow: /links/
Disallow: /mobile*
Disallow: /sponsored/
Disallow: /search-results/
Disallow: /fblivevid.html
Disallow: /members*
Disallow: /v.lkqd.net/*
Disallow: /*?pid=*
Disallow: /*aul_class
Disallow: /wp-login.php*
Disallow: /giveways/*
Disallow: /nootropics/*
Disallow: /slideshow/*
Disallow: /exclude-homepageads/*
Disallow: /sc-admin-login/*
Disallow: /errors/error403.htm
Disallow: /pn/*
Disallow: /find/*
Disallow: /search/*
Disallow: /trending/
Disallow: /collaboration/
Disallow: *.pdf
Disallow: *.ttf
Disallow: *.woff2
Disallow: *.js
Disallow: */page/*
Disallow: /feedback-form/*
Disallow: /performance-test/*

User-Agent: MJ12bot
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: Boomtrain-Content-Bot*
Disallow:
Allow: /*
Sitemap: https://www.stylecraze.com/sitemap_index.xml