Free tools
Robots.txt Examples
Robots.txt file content for instacart.com.
Robot.txt file for: instacart.com
### Any bot scraping or crawling this site must abide by ### Instacart's Terms of Service https://www.instacart.com/terms User-Agent: ChatGPT-User Allow: /store/partner_recipes User-Agent: Twitterbot Allow: /store/list/* User-Agent: Yandex Disallow: / User-Agent: facebookexternalhit Disallow: /assets User-Agent: * Disallow: /accounts/unlock Disallow: /api/ Disallow: /ex/ Disallow: /orders/ Disallow: /redeem_coupon_token Disallow: /store/ Disallow: /store/order_changes/ Disallow: /store/*/browse_departments* Disallow: /store/*/ng/l/* Disallow: /store/referrals* Disallow: /store/item* Disallow: /store/pickup/status/* Disallow: /store/*/buy_it_again* Disallow: /store/*/l/* Disallow: /widgets/* Disallow: /client-side-redirect/* Disallow: /track/* Disallow: /affiliate-program* Disallow: /family/invite/* Disallow: /family/accept/* Disallow: /rate/orders/* Disallow: /store_manager_admin Disallow: /rest/v8/product/542617319 Disallow: /rest/v8/product/258211234 Disallow: /rest/v8/product/345162378 Disallow: /rest/v8/product/183142388 Disallow: /login?next= Disallow: /*?*interaction_id= Disallow: /*?*item_detail_product_id= Allow: /store/categories/* Allow: /store/hub/* Allow: /store/s?k=* Allow: /store/list/* Allow: /store/brands/* Allow: /store/pages Allow: /store/*/storefront Allow: /store/*/s?k=* Allow: /store/*/collections/* Allow: /s?k=* Allow: /store/*/departments* Allow: /store/*/departments/*/aisles* Allow: /store/*/flyer Allow: /store/*/search_v3* Allow: /store/recipes/* Allow: /store/aldi/products/* Allow: /store/costco/products/* Allow: /store/kroger/products/* Allow: /store/publix/products/* Allow: /store/safeway/products/* Allow: /store/sprouts/products/* Allow: /store/walgreens/products/* ### If you're not a bot, we're hiring: https://instacart.careers/current-openings/