## robots.txt to optimize indexing Sitemap: https://newz.net.au/wp-sitemap.xml ## Crawl-delay: 15 ## # Search engines User-agent: Googlebot User-agent: Google-InspectionTool User-agent: bingbot User-agent: DuckDuckBot User-agent: Slurp User-agent: YandexBot ## # Social media preview engines User-agent: LinkedInBot User-agent: AppleBot User-agent: facebookexternalhit User-agent: Twitterbot User-agent: Slackbot User-agent: Pinterestbot User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User ## # SEO crawlers for ranking and backlinks User-agent: AhrefsBot User-agent: CCBot User-agent: SemrushBot User-agent: MJ12Bot ## # W3C Standards compliance engines User-agent: W3C-checklink User-agent: W3C_Validator ## # RSS Feed crawlers User-agent: FeedFetcher-Google ## # Access control rules for above User-agents Allow: / Allow: /wp-content/uploads/ Disallow: /cgi-bin* Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /readme.html Disallow: /refer/ # Deny all other crawlers, spiders, and bots User-agent: * Disallow: /