Dignity nobility good quality point out highness standing course caste standing superiority priority trend difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robot regarding which web pages usually are not for being crawled. For a search engine crawler https://www.youtube.com/watch?v=Tmal7B28Td0