屏蔽DataForSeo等垃圾搜索引擎蜘蛛及爬虫的方法

网站不大，成天乱七八糟的爬虫bot挺多，单单一个狗皮膏药一样的DataForSeo就密密麻麻的。

查了一下这个dataforseo好像是一家境外的seo公司，想不通为啥来爬我呢？顺便看了下，垃圾爬虫bot还不少呢，顺手一起给他屏蔽了！

基于宝塔的操作如下：

首先进入宝塔面板，文件管理进入/www/server/nginx/conf目录，新建空白文件kill_bot.conf。然后将以下代码保存到当前文件中。

#禁止垃圾搜索引擎蜘蛛抓取

if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms|DataForSeoBot") {
 return 403;
 break;
}

#禁止扫描工具客户端
if ($http_user_agent ~* "crawl|curb|git|Wtrace|Scrapy" ) {
 return 403;
 break;
}