Do I really need a sitemap? This question has been asked for a long time, and in mid 2005 Google made it even more complicated by introducing a sitemap format just for their spiders.
First of all there are two types of sitemaps. One sitemap for users and one for Search engines. A sitemap for users may be beneficial in that it allows you to put every page on your web site just one click away for end users. Fewer clicks tend to help conversions, and on large web sites they may provide for an easy way for users to find exactly what they are looking for with the fewest clicks.
Sitemaps for search engines are another matter entirely. Originally the xml sitemap, as proposed by Google, was designed to help their spider crawl complex URLs that might have been missed by their spiders. These URLs tended to be in shopping cart systems and database driven sites developed by programmers that had little regard for the real world needs of a modern web site. Urls like domain.com/id7734&ask?whyam-i:like&this made deep crawls by spiders nearly impossible.
Since then having an xml sitemap has been trotted out as the solution to a long list of problems with websites when it comes to them not ranking well, or not having as many pages in the Google index . And for the vast majority of sites this isn’t true.
One of the deep indexing factors that Google does rely on is linking. On Matt Cutt’s blog in 2006 a discussion was raging about the supplemental index and Matt mentioned that the number of links pointing toward a site was a factor in how often it was crawled and how deeply the site was indexed. No mention of the, by then, 1 year old sitemap.xml.
Google itself even says “We don’t guarantee that we’ll crawl or index all of your URLs1” and that the sitemap is just used to learn about your site’s structure1.
One of the major reasons that I recommend against xml sitemaps is that they require constant updating every time you update your website. And while there are tools available to help automate the process2, that in itself is reason enough not to do it, more work for questionable benefits. The only time an xml sitemap might be justified is when your site navigation is less than useful and is a challenge for search engine spiders to crawl. Unfortunately this may mean that your site navigation is a challenge to follow for end users as well, so it should be dealt with by fixing the navigation, not slapping a sitemap on it and calling it “fixed”.
Another reason that I don’t recommend xml sitemaps, in most cases, is that Google has had a long standing policy that we should build web sites for our users and not for the search engines. Unless their spider isn’t advanced enough to do the job, then webmasters are expected to step up and make things easier for them. Xml sitemaps and nofollow are just a couple of examples of building something for search engines only that has no benefit as all for the user viewing your website.
So to answer the question, Do I really need a sitemap?, no, you don’t. You need clean navigation on a well structured web site. However, if you have complex URLs on an incredibly large web site with lousy navigation, it can’t hurt, but it also probably won’t help much either.