User-agent: * Disallow: /preview-*/ # Allow AI crawlers on public content (effective only after ADVISORY_PUBLIC=true) User-agent: GPTBot Allow: / Disallow: /preview-*/ User-agent: ClaudeBot Allow: / Disallow: /preview-*/ User-agent: PerplexityBot Allow: / Disallow: /preview-*/ User-agent: Google-Extended Allow: / Disallow: /preview-*/