called The Robots Exclusion Protocol.
It includes User-Agent and Disallow. Here is the meaning of these words as these are very important part of robots.txt file :
User-Agent : It is used for the search engine robots. It gives the permission to a particular or all robots to index the
pages. Here is the syntax :
User-Agent : * ("*" means all robots have permission to crawl all the files/pages)
Disallow : It denies access to robots to directories/files. Here is the syntax :
Disallow : /images/ (it denies access to images directory.)
A simple robots.txt file should be :
User-Agent: *
Disallow:
It means we give permission to all the robots to visit and the directories and files.
It is very easy to create this file. Open notepad editor. Insert these instructions and save it by name robots in text
format. Put it in top-level directory and upload it. Suppose your website name is http://www.abctest.com/ then your robots
file's path should be here http://www.abctest.com/robots.txt
You can check your robots.txt file here http://tool.motoricerca.info/robots-checker.phtml.
If you need help to create robots.txt, use robots.txt generator tool here : http://www.seochat.com/seo-tools/robots-generator/
Excellent and decent post. I found this much informative, as to what I was exactly searching for. Thanks for such post and please keep it up.
ReplyDeleteDigital Marketing Consultant in Delhi
Digital Marketing Consultant
This is a term that describes websites that are only accessible via the internet and in this case, it refers to those anonymous websites that many people end up signing up for Read more about my website: dark web
ReplyDeletepanu
ReplyDelete