Disallow particular extension in robots.txt file for web crawlers
From Notes_Wiki
<yambe:breadcrumb>Internet_tips_and_tricks|Internet tips and tricks</yambe:breadcrumb>
Disallow particular extension in robots.txt file for web crawlers
To disable crawling of ogv (video) files which anyway cannot be indexed use:
User-agent: * Disallow: /*.ogv$ Allow: /