to stay away from undesirable content while in the research indexes, site owners can instruct spiders never to crawl particular data files or directories throughout the common robots.txt file in the root directory with the area. Furthermore, a website page may be explicitly excluded from a internet search engine's database by using a meta tag speci