Two small files sit behind most websites and cause a lot of confusion: the sitemap and robots.txt. They are easy to check and easy to get wrong. This article explains what each one does, whether you need it and the mistake to avoid.
What a sitemap is
A sitemap is a file that lists the pages, videos and other files on your site and shows how they relate. Google's documentation says it helps search engines crawl your site more efficiently by showing which items you consider important, along with details such as when a page was last updated.
A sitemap does not guarantee that a page will be crawled or indexed. It helps search engines discover your pages.
Do you need one?
Google says you may not need a sitemap if your site is small, with about 500 pages or fewer that you want in search results, and if all important pages can be reached by links from your home page. You are more likely to benefit if:
- Your site is large and some pages are hard to reach by links.
- Your site is new and has few links from other sites.
- You have many images or videos you want found, or you appear in Google News.
Google also says that in most cases a site benefits from having one. Many content management systems, WordPress included, can create a sitemap automatically, so it often costs you nothing.
What robots.txt is
A robots.txt file tells search engine crawlers which URLs they may access on your site. According to Google's introduction to robots.txt, it is mainly used to manage crawler traffic, for example to avoid overloading your server, and to avoid crawling unimportant or near-duplicate pages.
A basic file that allows everything and points to your sitemap looks like this:
User-agent: *
Disallow:
Sitemap: https://yourbusiness.lk/sitemap.xml
Use your own domain and the real address of your sitemap.
The mistake to avoid
robots.txt is not a way to keep a page out of Google. Google's documentation is clear on this:
- To keep a page out of search results, use
noindexor password protection. - A page blocked in robots.txt can still appear in results, without a description, if other sites link to it.
- Hiding a page by blocking it is explicitly discouraged.
- The file is not a security measure. Well-behaved crawlers follow it, but others may not, so private content needs a password.
Be careful with blocking scripts, styles and images. Google says to block these only if the page is not significantly affected without them, because blocking resources Google needs can hurt its understanding of the page.
How to check yours
- Type
yourbusiness.lk/sitemap.xmlinto your browser. You should see a list of addresses. WordPress sites may use a slightly different name, so check your SEO plugin or site settings if it is not found. - Type
yourbusiness.lk/robots.txt. You should see a short text file. If it containsDisallow: /, your whole site is blocked from crawlers, which can be left over from a test version of the site. - Open Google Search Console and submit your sitemap, which tells Google where it is. Search Console also shows when it was last read and how many pages it found.
Our own blog is an example: its sitemap lives at /blog/sitemap.xml.
For WordPress sites
WordPress sites often get a sitemap from the core software or from an SEO plugin. If you use an SEO plugin, keep only one sitemap active, and check that WordPress is not set to ask search engines to ignore the site. Our website launch checklist includes this check.
When your pages are not showing up
New pages can take time to appear. If a page is missing after a few weeks, check that:
- It is linked from other pages on your site.
- It is not blocked by robots.txt or marked
noindex. - It loads without errors. See website errors explained.
- Your domain points to the right place. See how to check that your domain points to your hosting.
FAQ
Is a sitemap the same as a site map page for visitors?
No. A sitemap file is written for search engines. Some sites also have a visitor-facing page that lists links, which is a separate thing.
Can I have more than one sitemap?
Yes. Large sites often split their sitemap into several files, listed in a sitemap index.
Does a sitemap improve my ranking?
Google describes a sitemap as a way to help it discover your pages, not as a way to change where they rank.
