A penetration tester discovered the following robots.txt file on a target web server during reconnaissance.
INSTRUCTIONS: Part 1: Given the entries in robots.txt, select the tool the penetration tester should use for further investigation. Part 2: Select the TWO robots.txt entries the penetration tester should recommend for removal.
Tool
Given the entries in robots.txt, select the tool the penetration tester should use for further investigation:
←→↻
http://example.com/robots.txt
Select the two robots.txt entries the penetration tester should recommend for removal: