# podash robots.txt # Content pages (/, /podcast/, /latest, /trending, /category/*) stay # crawlable so search engines keep indexing us. We only disallow action/auth # endpoints (no SEO value, and a magnet for crawlers) and ask for a crawl delay # to keep the lean node from being hammered. Sitemap: https://www.podash.com/sitemap.xml User-agent: * Disallow: /podcast/*/subscribe Disallow: /podcast/*/unsubscribe Disallow: /subscriptions Disallow: /search Disallow: /login Disallow: /signup Disallow: /forgot-password Disallow: /reset/ Disallow: /verify/ Disallow: /auth/ Disallow: /logout Crawl-delay: 10 # Barkrowler (babbar.tech) — SEO backlink-graph crawler, no user value to us and # the single biggest crawl source. It honors robots.txt; ask it to stay out. User-agent: Barkrowler Disallow: /