Difference between revisions of "Disallow particular extension in robots.txt file for web crawlers"
From Notes_Wiki
(Created page with "<yambe:breadcrumb>Internet_tips_and_tricks|Internet tips and tricks</yambe:breadcrumb> =Disallow particular extension in robots.txt file for web crawlers= To disable crawling...") |
m |
||
| Line 1: | Line 1: | ||
[[Main Page|Home]] > [[Internet tips and tricks]] > [[Disallow particular extension in robots.txt file for web crawlers]] | |||
To disable crawling of ogv (video) files which anyway cannot be indexed use: | To disable crawling of ogv (video) files which anyway cannot be indexed use: | ||
| Line 10: | Line 9: | ||
Refer: http://tools.seobook.com/robots-txt/ | Refer: http://tools.seobook.com/robots-txt/ | ||
[[Main Page|Home]] > [[Internet tips and tricks]] > [[Disallow particular extension in robots.txt file for web crawlers]] | |||
Latest revision as of 04:15, 18 April 2022
Home > Internet tips and tricks > Disallow particular extension in robots.txt file for web crawlers
To disable crawling of ogv (video) files which anyway cannot be indexed use:
User-agent: * Disallow: /*.ogv$ Allow: /
Refer: http://tools.seobook.com/robots-txt/
Home > Internet tips and tricks > Disallow particular extension in robots.txt file for web crawlers
