What is Robot.txt ?
This file gives guidance about their site to web roots; this is called The Robots Exclusion Protocol. They unknown, relentless force that guideline the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. By defining a few rules in this text file.
Two Deliberation when using / robots.txt
- The / robots.txt file is a publicity file. Anyone can see what sections of your do not want robots to use.
- Robots can ignore your / robots.txt. Especially malware robots that scan the web for security vulnerabilities and e-mail address harvesters used by spammers will pay no attention.
Recognize the restrictions of robot.txt
Before you build your robot.txt, you must know the uncertainty of this URL blocking method.