People use DA to discover which Internet sites are dependable within their industry and to figure out the place to obtain fantastic hyperlinks. txt file is then parsed and can instruct the robotic regarding which pages will not be to generally be crawled. As being a internet search engine crawler https://www.youtube.com/watch?v=0eLlfoHTk_E