{"id":1338,"date":"2026-08-20T07:56:18","date_gmt":"2026-08-20T07:56:18","guid":{"rendered":"https:\/\/www.freethought.uk\/help\/locating-and-creating-robots-txt-in-scout-control-panel\/"},"modified":"2026-08-20T07:56:18","modified_gmt":"2026-08-20T07:56:18","slug":"locating-and-creating-robots-txt-in-scout-control-panel","status":"publish","type":"post","link":"https:\/\/www.freethought.uk\/help\/locating-and-creating-robots-txt-in-scout-control-panel\/","title":{"rendered":"Locating and Creating robots.txt in Scout Control Panel"},"content":{"rendered":"\n

What is robots.txt?<\/h2>\n\n\n\n

robots.txt is a plain text file that tells search engine crawlers (like Google, Bing, and others) which pages or sections of your website they are allowed to visit. It sits in the root of your website and is one of the first files a crawler looks for when indexing your site.<\/p>\n\n\n\n

Having a properly configured robots.txt helps you control how search engines interact with your content, prevent indexing of private or duplicate pages, and manage crawl budget on larger sites.<\/a><\/p>\n\n\n\n

Where to Find robots.txt in Scout<\/h2>\n\n\n\n

In Scout Control Panel, your website’s root directory is located at:<\/p>\n\n\n\n

\/home\/[username]\/site\/public_html\/<\/pre>\n\n\n\n
To access it:<\/p>\n\n\n\n
    \n
  1. Log in to Scout Control Panel<\/li>\n\n\n\n
  2. Select the site you want to manage<\/li>\n\n\n\n
  3. Open File Manager<\/strong> from the Files dashboard<\/li>\n\n\n\n
  4. Click on the Site Folder and navigate to the public_html<\/code> folder <\/li>\n<\/ol>\n\n\n\n
    If robots.txt exists, you’ll see it listed here. If it doesn’t appear, it hasn’t been created yet and you’ll need to create it manually (see below).<\/a><\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Creating robots.txt<\/h2>\n\n\n\n
    Scout does not generate a robots.txt file automatically. If one doesn’t exist, you’ll need to create it yourself:<\/p>\n\n\n\n
      \n
    1. In File Manager, navigate to public_html<\/code><\/li>\n\n\n\n
    2. Click New File<\/strong><\/li>\n\n\n\n
    3. Name the file exactly robots.txt<\/code> (all lowercase, no spaces)<\/li>\n\n\n\n
    4. Add your desired rules (see examples below)<\/li>\n\n\n\n
    5. Save the file<\/li>\n<\/ol>\n\n\n\n
      The file must be placed directly in public_html<\/code>, not inside a subfolder. Crawlers will only look for it at https:\/\/yourdomain.com\/robots.txt<\/code>.<\/a><\/p>\n\n\n\n
      Basic Examples<\/h2>\n\n\n\n
      Allow all crawlers to access everything (default behaviour if no file exists):<\/strong><\/p>\n\n\n\n
      User-agent: *\nAllow: \/<\/pre>\n\n\n\n
      Block all crawlers from your entire site:<\/strong><\/p>\n\n\n\n
      User-agent: *\nDisallow: \/<\/pre>\n\n\n\n
      Block a specific folder (e.g. an admin area or staging content):<\/strong><\/p>\n\n\n\n
      User-agent: *\nDisallow: \/admin\/\nDisallow: \/staging\/<\/pre>\n\n\n\n
      Block a specific crawler (e.g. a known AI scraper):<\/strong><\/p>\n\n\n\n
      User-agent: GPTBot\nDisallow: \/<\/pre>\n\n\n\n
      Point crawlers to your sitemap:<\/strong><\/p>\n\n\n\n
      User-agent: *\nAllow: \/\n\nSitemap: https:\/\/yourdomain.com\/sitemap.xml<\/pre>\n\n\n\n
      <\/p>\n\n\n\n
      Common Use Cases<\/h2>\n\n\n\n
        \n
      • Blocking AI scrapers<\/strong> from training on your content (e.g. GPTBot, CCBot)<\/li>\n\n\n\n
      • Hiding admin or login pages<\/strong> from search results<\/li>\n\n\n\n
      • Preventing duplicate content issues<\/strong> by blocking crawlers from parameter-heavy URLs<\/li>\n\n\n\n
      • Prioritising crawling<\/strong> by pointing crawlers toward your sitemap<\/li>\n\n\n\n
      • Protecting staging or development content<\/strong> from being indexed prematurely<\/li>\n<\/ul>\n\n\n\n
        <\/p>\n\n\n\n
        Important Notes<\/h2>\n\n\n\n
          \n
        • robots.txt is a request<\/strong>, not a security measure. Well-behaved crawlers will respect it, but malicious bots may ignore it entirely. Do not rely on it to protect sensitive data.<\/li>\n\n\n\n
        • Changes take effect immediately, but search engines may take time to re-crawl your site.<\/li>\n\n\n\n
        • If you’re using WordPress, some SEO plugins (like Yoast or Rank Math) generate a virtual robots.txt. If you place a physical file in public_html<\/code>, it will override the plugin’s version.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
          What is robots.txt? robots.txt is a plain text file that tells search engine crawlers (like Google, Bing, and others) which pages or sections of your website they are allowed to visit. It sits in the root of your website and is one of the first files a crawler looks for…<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,23,26],"tags":[],"class_list":["post-1338","post","type-post","status-publish","format-standard","hentry","category-our-control-panel","category-faq","category-wordpress"],"_links":{"self":[{"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/posts\/1338","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/comments?post=1338"}],"version-history":[{"count":1,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/posts\/1338\/revisions"}],"predecessor-version":[{"id":1340,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/posts\/1338\/revisions\/1340"}],"wp:attachment":[{"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/media?parent=1338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/categories?post=1338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.freethought.uk\/help\/wp-json\/wp\/v2\/tags?post=1338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}