Crawlers (or bots) are used to assemble info available on the internet. By utilizing website navigation menus, and studying inner and exterior links, the bots start to know the context of a web page. Of course, the words, images, and different knowledge on pages also help search engines like google