Robots.txt File
Our indexing robots obey the rules of the robots.txt protocols.
If User-Agent: startShoppingBot is found, then anything listed under User-Agent: * will be ignored.
Disallow:
Our robots support the disallow: directive in the robots.txt file.
To stop our robots from indexing your web site add the following to your robots.txt file:
User-Agent: startShoppingBot
Disallow: /
To stop our robots from indexing a particular file:
For example the following will stop our robots from indexing the about.html file:
User-Agent: startShoppingBot
Disallow: /about.html
To stop our robots from indexing a particular directory:
For example the following will stop our robots from indexing any url that contains the images directory in it:
User-Agent: startShoppingBot
Disallow: /images/
Our robots support regular expression pattern matching using the wildcard * and end of string matching with $.
For example 1 the following will stop our robots from indexing any url that contains the phrase addToTrolley in it:
User-Agent: startShoppingBot
Disallow: *addToTrolley*
For example 2 the following will stop our robots from indexing any url that ends with .css:
User-Agent: startShoppingBot
Disallow: *.css$
Allow:
Our robots support the allow: directive in the robots.txt file.
For example the following will stop our robots from indexing the images directory, except for the new directory under images.
User-Agent: startShoppingBot
Disallow: /images/
Allow: /images/new/
Because of the incorrect use the 'allow: /' statement in many websites, the Start Shopping indexing robots actually ignore it so that your disallow statements will actually work correctly.
Also do not forget that with the robots.txt file, everything is allowed by default, unless it is disallowed.