-
Notifications
You must be signed in to change notification settings - Fork 53
TWA 0501
"No robots file found at: ${domain}/robots.txt"
In the message output the variable ${domain} is replaced by the checked domain. So ${domain}/robots.txt should be the URL of the web robot instruction file.
Each web site should contain a robot instruction file robots.txt to control the data collection of the web crawlers.
Malware web search robots will ignore the file, but the file can control the data included in the standard web search engines like google or bing. So the file controls which part of the web site/web application are easy to find with the standard web search engines.
Import: disallow some content for standard web search does not hide the content for a possible attacker.
A web site without a robots.txt file is not a security issue but it is not good practice.
Add a valid robots.txt file into the top-level directory of your web server.