Baidu spider crawling failure diagnosis abnormal information socket read and write error connection timeout what to do

Baidu Spider captures diagnostic exception information: What should I do if the socket reads and writes incorrectly?

Assuming that your website has not been included by Baidu, you must first perform spider crawling diagnosis on the Baidu search resource platform.

What should I do if Baidu crawler fails to crawl diagnostic links?

If the Baidu crawler crawl diagnosis fails several times, the firewall may have blocked the crawler program.

Baidu Search Resource Platform > Crawl Diagnosis > Crawl Exception Information: Socket read and write errors ▼

• Especially when using Cloudflare CDN, it is blocked by default.
• On the Internet, it is said to add the IP address xxx.xxx.xxx.xxx/24
• However, tried that to no avail.

I didn't block Baidu spiders on the server, so the problem should be Cloudflare's WAF!

Login to Cloudflare → Security → WAF → Firewall Rules → Create Firewall Rule

• Find the WAF rules related to crawlers on Cloudflare, and found the option of "legitimate robot crawler" ▼

• • After creating the firewall rules, wait for 10 minutes, and then grab the diagnosis, and all of them were successfully captured!

What's wrong with Baidu crawler Sitemap crawling failure and connection timeout?

If you submit the address of the Sitemap file on the Baidu search resource platform, there will be problems such as crawling failure and connection timeout ▼

Solution to the failure of Baidu crawler to grab the Sitemap map

Login to Cloudflare → Security → WAF → Firewall Rules → Create Firewall Rules ▼

1. field, select "User Agent"
2. operator, select Contains
3. Add a new user agent, click the last "Or"
4. Value, respectively enter the following Baidu Spider UA user agent:

After the completion, test the fetching again, and the result returns HTTP header 200, indicating that the fetching is successful▼

The user agents of other spiders and crawlers can also search for themselves in the same way.