Disallow particular extension in robots.txt file for web crawlers

From Notes_Wiki

Revision as of 09:19, 29 March 2018 by Saurabh (talk | contribs) (Created page with "<yambe:breadcrumb>Internet_tips_and_tricks|Internet tips and tricks</yambe:breadcrumb> =Disallow particular extension in robots.txt file for web crawlers= To disable crawling...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

<yambe:breadcrumb>Internet_tips_and_tricks|Internet tips and tricks</yambe:breadcrumb>

Disallow particular extension in robots.txt file for web crawlers

To disable crawling of ogv (video) files which anyway cannot be indexed use:

   User-agent: *
   Disallow: /*.ogv$
   Allow: /

Refer: http://tools.seobook.com/robots-txt/

Retrieved from "http://www.sbarjatiya.com/notes_wiki/index.php?title=Disallow_particular_extension_in_robots.txt_file_for_web_crawlers&oldid=3040"