对于大量的不良垃圾蜘蛛抓取网站内容,浪费流量的行为,使用我司linux主机的用户可以在网站根目录下面建立一个 .htaccess文件对其进行屏蔽.

规则如下.

RewriteEngine on

RewriteCond %{HTTP_USER_AGENT} "Bingbot|MSNbot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu" [NC]
RewriteRule ^(.*)$ http://127.0.0.1/$1 [R=301,L]

此外,对于已经建立过.htaccess的客户,只需要添加以下的内容即可.

RewriteCond %{HTTP_USER_AGENT} "Bingbot|MSNbot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu" [NC]
RewriteRule ^(.*)$ http://127.0.0.1/$1 [R=301,L]