Artikel Direktori
Baidu Spider njupuk informasi pangecualian diagnostik: Apa sing kudu ditindakake yen soket maca lan nulis salah?
Kanthi nganggep yen situs web sampeyan durung dilebokake dening Baidu, sampeyan kudu nindakake diagnosis spider crawling dhisik ing platform sumber daya panelusuran Baidu.
Apa sing kudu ditindakake yen Baidu crawler gagal nyusup pranala diagnostik?
Yen diagnosis crawler Baidu gagal kaping pirang-pirang, firewall bisa uga wis mblokir program crawler.
Platform Sumber Daya Telusuran Baidu > Diagnostik Crawling > Informasi Kesalahan Crawling: Kesalahan Soket Maca/Nulis ▼
- Utamané nggunakakecloudflare Nalika nggunakake CDN, iku diblokir kanthi standar.
- Ing Internet, ngandika nambah alamat IP
xxx.xxx.xxx.xxx/24 - Nanging, nyoba iku ora ana gunane.
Aku ora ngalangi laba-laba Baidu ing server, dadi masalah kudu WAF Cloudflare!
Mlebet Cloudflare → Keamanan → WAF → Aturan Firewall → Nggawe Aturan Firewall
- Temokake aturan WAF sing ana gandhengane karo crawler ing Cloudflare, lan nemokake pilihan "crawler robot sing sah" ▼
- Sawise nggawe aturan firewall, ngenteni 10 menit, banjur njupuk diagnosis, lan kabeh padha kasil dijupuk!
Apa sing salah karo Baidu crawler Sitemap crawling gagal lan sambungan wektu entek?
Yen sampeyan ngirim alamat file Peta Situs ing platform sumber daya telusuran Baidu, bakal ana masalah kayata gagal crawling lan wektu entek sambungan ▼
Solusi kanggo kegagalan crawler Baidu kanggo njupuk peta Sitemap
Mlebu menyang Cloudflare → Keamanan → WAF → Aturan Firewall → Nggawe Aturan Firewall ▼
- lapangan, pilih "User Agent"
- operator, pilih Contains
- Tambah agen panganggo anyar, klik pungkasan "Utawa"
- Nilai, ketik agen panganggo Baidu Spider UA ing ngisor iki:
-
Baiduspider/2.0 -
Baiduspider-image -
Baiduspider-render/2.0 -
http://www.baidu.com/search/spider.html -
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) -
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html）
Sawise rampung, nyoba njupuk maneh, lan asil ngasilake header HTTP 200, nuduhake yen njupuk wis sukses▼
-
抓取诊断 > 抓取详情以下是百度Spider抓取结果及页面信息： -
提交网址： https://www.etufo.org/sitemap_baidu.xml -
抓取网址： https://www.etufo.org/sitemap_baidu.xml -
抓取UA： Mozilla/5.0 (compatible; Baiduspider/2.0; -
+http://www.baidu.com/search/spider.html) -
抓取时间： 2022-11-11 19:03:44 -
网站IP： 172.***.***.149 -
下载时长： 0.868秒 -
返回HTTP头：HTTP/2 200
Agen pangguna laba-laba lan crawler liyane uga bisa nggoleki awake dhewe kanthi cara sing padha.
Muga-muga artikel "Cara Nangani Kesalahan Socket Read/Write lan Connection Timeout" sing dienggo bareng ing blog Chen Weiliang ( https://www.chenweiliang.com/ ) iki bisa migunani kanggo sampeyan.
Monggo dipunbagi pranala artikel punika: https://www.chenweiliang.com/cwl-29315.html