September 22, 2005
robots.txt question
[Note: This post is part of the Be Dumb in Public program, of which I am a lifetime member.]
There’s lots of good info on the Web about how to create a robots.txt file that will keep the major search engines from spidering your site. But I haven’t found instructions aimed at my precise level of ineptitutde. So, here goes…
Let’s say my “hideme.com”directory exists at root level. That is, my host won’t let me go any further down than that. I see hideme.com plus all the other directories I own. Let’s say I want to put in a robots.txt file to protect the contents of hideme.com, but I want to leave the rest of my directories open to search engines.
1. Is this the right robots.txt content:
User-agent: *
Disallow: /hideme.com/
I’m especially concerned about getting the slashes right.
2. Where exactly do I put the robots.txt file? At the same level as the hideme.com directory, where I can see all my directories? Or inside the hideme.com file? Or elsewhere? Thanks in advance. And have pity: I was a Humanities major.