Learn why you should use a robots.txt file, and how to add one to your site. Test your site's SEO for free at www.seositecheckup.com.

In this video, you’ll hear some of the following:

" In this video we’re going to show you how to add a robot.txt file to your site. This is an important step in the publication of any website, as this file tells the search engines what to index on your site, and what not to index, which pages to access, and which to ignore. This includes private content, or unnecessary elements that would otherwise suck up bandwidth and increase load time on your server for no good reason.

Not having a robots.txt file can also generate errors in your apache log when search engine spiders request a robots file, and don’t find one.

After we run our website on seositecheckup, we can see that we are missing our robots.txt file.
So when a search engine like Google sends a spider to index the contents of our site, it will try
to index everything.

Now you might say, what’s wrong with that, isn’t it better to have everything indexed? Well no, actually. It’s much, much better to optimize what the search engines are indexing, to have them only include high value content in relation to your target audience.

It’s also important for better security to leave some parts of your site unindexed, and there might be cases where you don’t want visitors finding certain pages unless they’ve performed a certain action on another page on your site first. "

Hear more by watching the full video!

#robotsfile #googlespider #seo