Tsab ntawv teev npe
Baidu Kab laug sab nkag thiab kuaj xyuas cov ntaub ntawv txawv txav: Kuv yuav ua li cas yog tias lub qhov (socket) nyeem lossis sau ntawv?
Piv txwv tias koj lub vev xaib tsis tau txheeb xyuas los ntawm Baidu, koj yuav tsum xub ua qhov kev kuaj kab laug sab nkag rau ntawm Baidu nrhiav chaw pabcuam.
Kuv yuav ua li cas yog tias Baidu crawler tsis nkag mus rau qhov sib txuas?
Yog tias Baidu crawler tsis nkag mus thiab kuaj xyuas ob peb zaug, lub firewall yuav tau thaiv tus crawler.
Baidu Search Resource Platform > Crawl Diagnosis > Crawl Abnormal Information: qhov (socket) nyeem / sau yuam kev ▼
- Tshwj xeeb tshaj yog thaum siv Cloudflare CDN, uas yog thaiv los ntawm lub neej ntawd.
- Hauv Is Taws Nem, muaj kev sib tham txog kev ntxiv IP chaw nyob
xxx.xxx.xxx.xxx/24 - Txawm li cas los xij, sim tsis muaj txiaj ntsig.
Kuv tsis thaiv Baidu kab laug sab ntawm lub server, yog li qhov teeb meem yuav tsum yog Cloudflare's WAF!
Nkag mus rau Cloudflare → Kev Ruaj Ntseg → WAF → Cov Cai Hluav Taws Xob → Tsim Cov Cai Hluav Taws Xob
- Nrhiav rau crawler-related WAF cov cai ntawm Cloudflare thiab pom qhov "Legal Robot Crawler" kev xaiv ▼
- Tom qab tsim cov kev cai firewall, tos 10 feeb, thiab tom qab ntawd lob qhov kev kuaj mob, thiab tag nrho lawv tau ua tiav!
Baidu crawler Sitemap ua tsis tau tejyam nkag mus, kev sib txuas ncua sij hawm?
Yog hais tias lub sitemap cov ntaub ntawv chaw nyob raug xa mus rau ntawm Baidu tshawb nrhiav chaw platform, yuav muaj teeb meem ntawm kev nkag mus tsis tau thiab kev sib txuas ncua sij hawm ▼
Baidu crawler ua tsis tau tejyam los nkag rau Sitemap daim ntawv qhia kev daws teeb meem
Kos npe rau hauv Cloudflare → Kev Ruaj Ntseg → WAF → Cov Cai Hluav Taws Xob → Tsim Cov Cai Hluav Taws Xob ▼
- teb, xaiv User-Agent
- tus neeg teb xov tooj, xaiv "muaj"
- Ntxiv tus neeg siv tshiab, nyem "Lossis" thaum kawg
- tus nqi, nkag mus rau hauv qab no Baidu Spider UA neeg siv tus neeg sawv cev raws li:
-
Baiduspider/2.0 -
Baiduspider-image -
Baiduspider-render/2.0 -
http://www.baidu.com/search/spider.html -
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) -
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html）
Tom qab ua tiav, sim thiab nqa rov qab, thiab cov txiaj ntsig rov qab HTTP header 200, qhia tias kev nqa tau ua tiav ▼
-
抓取诊断 > 抓取详情以下是百度Spider抓取结果及页面信息： -
提交网址： https://www.etufo.org/sitemap_baidu.xml -
抓取网址： https://www.etufo.org/sitemap_baidu.xml -
抓取UA： Mozilla/5.0 (compatible; Baiduspider/2.0; -
+http://www.baidu.com/search/spider.html) -
抓取时间： 2022-11-11 19:03:44 -
网站IP： 172.***.***.149 -
下载时长： 0.868秒 -
返回HTTP头：HTTP/2 200
Cov neeg siv ntawm lwm cov kab laug sab thiab cov crawlers tuaj yeem tshawb nrhiav lawv tus kheej ib yam nkaus.
Cia siab Chen Weiliang Blog ( https://www.chenweiliang.com/ ) tau qhia "Baidu kab laug sab nkag mus tsis ua haujlwm los kuaj xyuas cov ntaub ntawv tsis zoo ntawm lub qhov (socket) nyeem thiab sau ntawv yuam kev yuav ua li cas txog lub sijhawm sib txuas", uas yuav pab tau rau koj.
Zoo siab txais tos los qhia qhov txuas ntawm kab lus no:https://www.chenweiliang.com/cwl-29315.html
Zoo siab txais tos rau Telegram channel ntawm Chen Weiliang blog kom tau txais qhov hloov tshiab tshiab!
📚 Daim ntawv qhia no muaj nuj nqis loj, 🌟Qhov no yog lub sijhawm tsis tshua muaj, tsis txhob nco nws! ⏰⌛💨
Share thiab like yog tias koj nyiam!
Koj qhov kev sib koom thiab kev nyiam yog peb qhov kev txhawb nqa tas mus li!