Ìwé Directory
Baidu Spider n ṣaja ati ṣe iwadii alaye ajeji: Kini o yẹ MO ṣe ti iho naa ba ka tabi kọ?
Ti o ba ro pe oju opo wẹẹbu rẹ ko ti ni itọka nipasẹ Baidu, o gbọdọ kọkọ ṣe ayẹwo iwadii alantakun kan lori pẹpẹ orisun orisun Baidu.
Kini o yẹ MO ṣe ti crawler Baidu ba kuna lati ra ọna asopọ aisan naa?
Ti ayẹwo wiwa crawler Baidu kuna ni ọpọlọpọ igba, ogiriina le ti dina eto crawler naa.
Platform orisun wiwa Baidu > Ayẹwo Rarawl > Alaye Alailowaya Ra: iho kika/ki aṣiṣe ▼
- Paapaa nigba lilo Cloudflare CDN, eyiti o dina nipasẹ aiyipada.
- Lori intanẹẹti, ọrọ wa ti fifi adiresi IP kan kun
xxx.xxx.xxx.xxx/24 - Sibẹsibẹ, gbiyanju pe ko si abajade.
Emi ko ṣe idiwọ awọn spiders Baidu lori olupin naa, nitorinaa iṣoro naa yẹ ki o jẹ WAF Cloudflare!
Wọle si Cloudflare → Aabo → WAF → Awọn Ofin ogiriina → Ṣẹda Ofin ogiriina
- Wiwa awọn ofin WAF ti o ni ibatan crawler lori Cloudflare ati rii aṣayan “Robot Crawler Ofin” ▼
- Lẹhin ṣiṣẹda awọn ofin ogiriina, duro fun awọn iṣẹju 10, lẹhinna mu ayẹwo naa, ati pe gbogbo wọn ni aṣeyọri mu!
Maapu Aaye crawler Baidu kuna lati ra, akoko asopọ mọ bi?
Ti adirẹsi faili maapu aaye naa ti wa ni ifisilẹ sori iru ẹrọ orisun wiwa Baidu, awọn iṣoro yoo wa ti ikuna jijo ati akoko asopọ ▼
Baidu crawler kuna lati ra ojuutu maapu maapu aaye
Wọle si Cloudflare → Aabo → WAF → Awọn Ofin ogiriina → Ṣẹda Awọn ofin ogiriina ▼
- aaye, yan Olumulo-Aṣoju
- oniṣẹ ẹrọ, yan "ni ninu"
- Ṣafikun aṣoju olumulo tuntun kan, tẹ “Tabi” ti o kẹhin
- iye, tẹ aṣoju olumulo Baidu Spider UA wọnyi ni atele:
-
Baiduspider/2.0 -
Baiduspider-image -
Baiduspider-render/2.0 -
http://www.baidu.com/search/spider.html -
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) -
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html）
Lẹhin ipari, ṣe idanwo ati mu lẹẹkansi, ati abajade pada akọle HTTP 200, ti o fihan pe wiwa naa ṣaṣeyọri▼
-
抓取诊断 > 抓取详情以下是百度Spider抓取结果及页面信息： -
提交网址： https://www.etufo.org/sitemap_baidu.xml -
抓取网址： https://www.etufo.org/sitemap_baidu.xml -
抓取UA： Mozilla/5.0 (compatible; Baiduspider/2.0; -
+http://www.baidu.com/search/spider.html) -
抓取时间： 2022-11-11 19:03:44 -
网站IP： 172.***.***.149 -
下载时长： 0.868秒 -
返回HTTP头：HTTP/2 200
Awọn aṣoju olumulo ti awọn spiders miiran ati awọn crawlers tun le wa fun ara wọn ni ọna kanna.
Ireti Chen Weiliang Blog ( https://www.chenweiliang.com/ ) pín "Baidu Spider crawling ikuna lati ṣe iwadii alaye ajeji iho kika ati kọ awọn aṣiṣe kini lati ṣe nipa akoko asopọ", eyiti o ṣe iranlọwọ fun ọ.
Kaabo lati pin ọna asopọ ti nkan yii:https://www.chenweiliang.com/cwl-29315.html