Technical
Robots.txt
A text file that tells search engine crawlers which parts of your website they are allowed to access.
Robots.txt is a simple text file placed at the root of your website that gives instructions to search engine crawlers about which areas they may or may not visit. It is one of the first files a crawler checks when it arrives at your site.
It is handy for keeping bots out of admin areas, internal search results, or duplicate sections that would waste crawl resources. One important caution: blocking a page in robots.txt stops it being crawled but does not reliably stop it appearing in search. To keep a page out of results entirely, use a "noindex" instruction instead.
Related terms
Want this handled for you instead of just defined? Our team applies these concepts to real Sydney websites every day. Get a free SEO audit.
Keep Learning