Tag: robots.txt
Tag: robots.txt
How to Use Robots.txt File to Gather Intelligence for Penetration Testing
In the head section of web documents, there is meta-information used to describe the page, including helping search engines categorize the page. The meta-information that is of utmost importance to the discussion is the meta information for robots that refers to the robots.txt file. What is the robot.txt file? The roborts.txt is a file that
Read More