# SleekNova Robots.txt # https://sleeknova.com/robots.txt # Version: 42.2.0 # Last Updated: 2026-08-24 # # ============================================================================= # SEARCH ENGINES (Full Access) # ============================================================================= User-agent: Googlebot Allow: / # V34: Block faceted navigation (safer pattern catches mid-query params) Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= # V27: Allow specific query patterns we DO want crawled Allow: /*?post_type=sn_product Allow: /*?brand= # V42 (BL-338): Crawl-delay removed from this group. Google ignores the # directive by policy, so a delay addressed to Googlebot is a rule aimed at # a crawler that does not consume it. The remaining Crawl-delay lines serve # non-Google crawlers that do honor it; the Search Console warning on those # lines is expected and accepted. User-agent: Googlebot-Image Allow: / # V36: Block faceted navigation for image crawler (fixes 95% "Other agent" leak) Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= # V42 (BL-338): GoogleOther is Google's non-Search crawler family and a # different product token from Googlebot, so it inherits nothing from the # Googlebot group and nothing from the wildcard group. Measured 2026-08-06 # in the access log: 146 of 146 GoogleOther requests in a 10-minute window # were faceted filter URLs the wildcard group forbids, across 127 distinct # URLs, one every 4 to 6 seconds. These groups bind the crawler that was # actually fetching. # # CORRECTED 2026-08-08 (V42.2.0): this line first published 381, which was a # count of log LINES, not requests. The host writes one line per server layer # and again across HTTP versions, so the log runs about 2.1x the true request # count. The rules below never changed; only the number describing why they # exist was wrong, and it is corrected here rather than quietly edited away. # # AND THEY WORKED: a 22.5-minute window on 2026-08-08, after these groups went # live, recorded ZERO GoogleOther requests. Same crawler, same site, same # filter URLs available. User-agent: GoogleOther Allow: / Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= User-agent: GoogleOther-Image Allow: / Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= User-agent: GoogleOther-Video Allow: / Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= User-agent: Bingbot Allow: / Crawl-delay: 2 User-agent: Slurp Allow: / Crawl-delay: 2 User-agent: DuckDuckBot Allow: / Crawl-delay: 2 User-agent: Baiduspider Allow: / Crawl-delay: 5 User-agent: YandexBot Allow: / Crawl-delay: 5 # ============================================================================= # AI CRAWLERS (Full Access - We Want LLM Visibility) # ============================================================================= # V41: every AI group that indexes or trains carries one uniform system-path # block (wp-admin, xmlrpc, plugin/theme internals). Named groups do not # inherit the wildcard rules, so without this block those paths were open # to exactly these crawlers. Content paths stay fully open: AI visibility # is the strategy. User-directed fetchers (ChatGPT-User, Claude-User, # Claude-Web, Perplexity-User) stay fully open, system paths included. User-agent: GPTBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: ChatGPT-User Allow: / # V41: OAI-SearchBot indexes for ChatGPT search results. Carries the faceted # block, like the other search indexers -- a named group does not inherit # the wildcard group's rules. User-agent: OAI-SearchBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ # Claude-User: pages fetched because a person asked Claude to look. User-agent: Claude-User Allow: / # Claude-SearchBot: indexing for Claude's search results. Carries the faceted # navigation block, like Googlebot, because it indexes -- a named group does # not inherit the wildcard group's rules. User-agent: Claude-SearchBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= User-agent: anthropic-ai Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: PerplexityBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= Allow: /*?post_type=sn_product Allow: /*?brand= # V41: Perplexity-User fetches the one page a person asked Perplexity about. # Fully open, like Claude-User and ChatGPT-User. User-agent: Perplexity-User Allow: / User-agent: Cohere-ai Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: Google-Extended Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ # V29: Google Vertex AI & Enterprise Search User-agent: Google-CloudVertexBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: CCBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: FacebookBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: Meta-ExternalAgent Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: Applebot-Extended Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ User-agent: YouBot Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-json/wp/v2/users Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /*.php Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ # ============================================================================= # SOCIAL MEDIA CRAWLERS (Full Access for Previews) # ============================================================================= User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Pinterest Allow: / User-agent: Pinterestbot Allow: / User-agent: TelegramBot Allow: / User-agent: WhatsApp Allow: / User-agent: Slackbot Allow: / User-agent: Discordbot Allow: / # ============================================================================= # SEO TOOLS (Allowed with Rate Limiting) # ============================================================================= User-agent: AhrefsBot Allow: / Crawl-delay: 10 User-agent: SemrushBot Allow: / Crawl-delay: 10 User-agent: MJ12bot Allow: / Crawl-delay: 30 User-agent: DotBot Allow: / Crawl-delay: 10 # Desktop auditing tool, run by site owners against their own site. User-agent: Screaming Frog SEO Spider Allow: / Crawl-delay: 30 # ============================================================================= # DEFAULT RULE (All Other Bots) # ============================================================================= User-agent: * Allow: / # Block WordPress admin and system paths Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/uploads/wpcf7_uploads/ Disallow: /wp-json/wp/v2/users Disallow: /readme.html Disallow: /license.txt Disallow: /xmlrpc.php Disallow: /wp-trackback.php Disallow: /wp-signup.php Disallow: /wp-register.php Disallow: /wp-login.php Disallow: /?s= Disallow: /search/ Disallow: /*?replytocom= Disallow: /*&replytocom= Disallow: /tag/ Disallow: /author/ Disallow: /feed/ Disallow: /comments/feed/ Disallow: /trackback/ Disallow: /*.php Disallow: /*?* Allow: /*?post_type=sn_product Allow: /*?brand= # V41: the llms suite advertises the lab verdict feeds; allow them for # every crawler that falls through to this group. # # 3.19.0 (BL-1219): the feeds moved to /lab/category/lab-results/passed/ # and /rejected/, which are ordinary paths and were never covered by the # Disallow above. This line stays for the OTHER direction: the query form # still resolves and 301s, and a crawler holding an old URL has to be # allowed to fetch it in order to be told where the page went. The sort # and category arguments on the new paths stay blocked by the Disallow, # which is the answer BL-1213 wants: a filtered view is a way of looking # at a page, not a page. Allow: /*?verdict= # Block common spam/scraper patterns Disallow: /cgi-bin/ Disallow: /wp-content/themes/*/assets/ Disallow: /wp-content/themes/*/inc/ # V34: FACETED NAVIGATION BLOCK (Safer Pattern) # ============================================================================= # Filter parameters create 161,280+ URL combinations that waste crawl budget. # Pattern /*?* catches parameters anywhere in query string. Disallow: /*?*relationship= Disallow: /*?*occasion= Disallow: /*?*price= Disallow: /*?*category= Disallow: /*?*vibe= Disallow: /*?*sort= Disallow: /*?*for= # ============================================================================= # BAD BOTS (Block Completely) # ============================================================================= User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: rogerbot Disallow: / User-agent: AhrefsSiteAudit Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Sogou Disallow: / User-agent: MegaIndex Disallow: / User-agent: ltx71 Disallow: / User-agent: Riddler Disallow: / User-agent: SeznamBot Disallow: / User-agent: Exabot Disallow: / User-agent: Daum Disallow: / User-agent: moget Disallow: / User-agent: Netcraft Disallow: / User-agent: zgrab Disallow: / User-agent: Nimbostratus-Bot Disallow: / User-agent: serpstatbot Disallow: / User-agent: SeekportBot Disallow: / User-agent: NetSystemsResearch Disallow: / # ============================================================================= # V25.0 THE LAB (Transparency Content) # ============================================================================= # /lab/ -- Our research log, rejections, and methodology decisions # /lab/category/lab-results/ -- Every product tested, with its verdict # /lab/category/deep-dives/ -- The research behind our four scoring dimensions, cited # /lab/category/source-corrections/ -- Claims we published that turned out wrong # /lab/category/decision-logs/ -- A rule or threshold of ours changed, and why # /lab/category/system-updates/ -- The system gained a capability # ============================================================================= # SITEMAPS (V7.2 Structure - 5 Taxonomies + 38 Hub Pages + Lab) # ============================================================================= Sitemap: https://sleeknova.com/sitemap_index.xml # Priority sitemaps for AI crawlers (Hub pages highest priority) Sitemap: https://sleeknova.com/hub-sitemap.xml Sitemap: https://sleeknova.com/relationship-sitemap.xml Sitemap: https://sleeknova.com/product-sitemap.xml Sitemap: https://sleeknova.com/occasion-sitemap.xml Sitemap: https://sleeknova.com/category-sitemap.xml Sitemap: https://sleeknova.com/vibe-sitemap.xml Sitemap: https://sleeknova.com/price-tier-sitemap.xml Sitemap: https://sleeknova.com/page-sitemap.xml Sitemap: https://sleeknova.com/lab-sitemap.xml # V53: Guides (Research Synthesis) Sitemap: https://sleeknova.com/guide-sitemap.xml # ============================================================================= # AI GUIDANCE FILE (Hub & Spoke Architecture V50) # ============================================================================= # Live Stats: https://sleeknova.com/stats/ # Hub (lean routing file): https://sleeknova.com/llms.txt # Methodology spoke: https://sleeknova.com/llms-methodology.txt # Products spoke: https://sleeknova.com/llms-products.txt # Complete reference: https://sleeknova.com/llms-full.txt # Machine manifest: https://sleeknova.com/.well-known/ai-plugin.json # V45: The Bar: https://sleeknova.com/the-bar/ # V42 (BL-338): the six LLMs-txt pointers below were published as bare # directives. No robots standard defines them, and Search Console reported # each one as an error. Comment form carries the same pointers to any agent # reading this file as text, and parsers stop reporting them. # LLMs-txt: https://sleeknova.com/llms.txt # LLMs-txt-stats: https://sleeknova.com/stats/ # LLMs-txt-the-bar: https://sleeknova.com/the-bar/ # LLMs-txt-methodology: https://sleeknova.com/llms-methodology.txt # LLMs-txt-products: https://sleeknova.com/llms-products.txt # LLMs-txt-full: https://sleeknova.com/llms-full.txt # SleekNova is The Gift Matchmaker. # We answer one question: What is THE perfect gift for this specific person? # # ============================================================================= # V7.2 TAXONOMY STRUCTURE # ============================================================================= # sn_relationship (18 terms) -> /gifts-for/{slug}/ # sn_occasion (14 terms) -> /occasion/{slug}/ # sn_category (16 terms) -> /gift-type/{slug}/ # sn_vibe (10 terms) -> /vibe/{slug}/ # sn_price_tier (4 terms) -> /price/{slug}/ # ============================================================================= # CONTACT # ============================================================================= # General: hello@sleeknova.com