TWA 0503

TWA-0503

Message

"robots.txt lists what looks like CGI scripts"

Explanation

The robot instruction file robots.txt contains an entry which could point to CGI scripts stored on the web server.

The robots.txt file controls the web crawlers of the standard web engines (e.g. google, bing). To disallow the indexing of script files is an obvious idea. But if a possible attacker scans the web site, he will read the content of the robots.txt file. So, a disallow of the CGI scripts is an assistance for the attacker to find the location of the script very quick.

Constrain: The twa script search for the text part cgi-bin. Hence if this text part is not part of the CGI script URL this message is a false warning.

Remediation

Remove any hints to the CGI scripts from the robots.txt file. But use another way to prevent the CGI scripts scanned by web crawlers. Never allow finding your CGI scripts by a simple google search.

Try to restrict the access to the CGI script directory.