Disallow particular extension in robots.txt file for web crawlers

From Notes_Wiki

Revision as of 04:15, 18 April 2022 by Saurabh (talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Home > Internet tips and tricks > Disallow particular extension in robots.txt file for web crawlers

To disable crawling of ogv (video) files which anyway cannot be indexed use:

   User-agent: *
   Disallow: /*.ogv$
   Allow: /

Refer: http://tools.seobook.com/robots-txt/

Home > Internet tips and tricks > Disallow particular extension in robots.txt file for web crawlers

Retrieved from "http://www.sbarjatiya.com/notes_wiki/index.php?title=Disallow_particular_extension_in_robots.txt_file_for_web_crawlers&oldid=5974"