Imunify360 blocking Googlebot
Forum
  1. Forums
  2. Imunify360
  3. Imunify360 and Imunify Sensor
  1. Darryl
  2. Friday, 23 November 2018
  3.  Subscribe via email
Hi,

We've had an issue reported which is fairly major as it's affected a clients Google search engine listing, I'm attaching an image to avoid this getting spidered. I assume Googlebot IPs should be automatically detected and not go through captcha? Any idea on a resolution for this as the client is understandably annoyed at this and we don't want it affecting others.

Thanks
Attachments (1)
Rate this post:
  1. 27.11.2018 01:11:46
  2. # 1
Rushan Accepted Answer
Posts: 4
Joined: 25.04.2018
0
Votes
Undo
Hi Darryl

Google bot should be automatically whitelisted by Imunify360 according to this document: https://support.google.com/webmasters/answer/80553?hl=en

Can you please submit support ticket to https://cloudlinux.zendesk.com/ so we can check if there is any misconfiguration on your server?
  1. 27.11.2018 21:11:11
  2. # 2
Darryl Accepted Answer
Posts: 6
Joined: 25.04.2018
0
Votes
Undo
Hi,

Assuming this is the whitelist used it doesn't look correct:

cat /var/imunify360/files/whitelist/v2/google.txt
# Google
# IPv4
# https://www.lifewire.com/what-is-the-ip-address-of-google-818153
8.8.8.8
8.8.4.4
74.125.224.72
64.233.160.0/24
66.102.0.0/24
66.249.64.0/24
72.14.192.0/24
74.125.0.0/24
209.85.128.0/24
216.239.32.0/24
64.68.90.0/24
65.249.64.0/19
# IPv6
# https://developers.google.com/speed/public-dns/docs/using
# https://developers.google.com/speed/public-dns/docs/dns64
# https://www.wordfence.com/blog/2015/05/ #wordfence-launches-ipv6-support-announces-wordfence-6-and-passes-6-million-downloads/
# ipv6.google.com
# 2001:4860:4860::8888
# 2001:4860:4860::8844
# 2001:4860:4860::6464
# 2001:4860:4860::64
# 2607:f8b0:4003:c04::8a
# IPv6 /64
2607:f8b0:4003:c04::/64
2001:4860:4860::/64

Based on that site it lists: 66.249.64.0 – 66.249.95.255

So it should be 66.249.64.0/19 not 66.249.64.0/24

This may actually be a typo as I can see 65.249.64.0/19 is listed but that doesn't appear to be a google IP range it's a Cisco range.

After a bit of digging it looks like pretty much all the IPv4 CIDR ranges are wrong, I've corrected these here:

64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
72.14.192.0/18
74.125.0.0/16
209.85.128.0/17
216.239.32.0/18
64.68.88.0/21
  1. 28.11.2018 07:11:27
  2. # 3
Carlos Accepted Answer
Posts: 0
Joined: 26.03.2019
0
Votes
Undo
Hi,

Seriously, what's going on with QA in CloudLinux? For a few weeks now there have been very serious bugs, like that one that left us without a database connection for proactive defense, but this is already the last straw.

It is understood that the software can have unexpected bugs, but do wrong up to the white lists for Google ... it is necessary to improve the QA immediately.

Imunify360 is an expensive solution, and customers who pay something like that is because we need guarantees.

Greetings,

Translated with http://www.DeepL.com/Translator
  1. 28.11.2018 09:11:33
  2. # 4
Darryl Accepted Answer
Posts: 6
Joined: 25.04.2018
0
Votes
Undo
These errors do appear to be mirrored in the current live whitelist versions:

https://files.imunify360.com/static/whitelist/v2/google.txt from https://files.imunify360.com/static/whitelist/v2/

I'd suggest reviewing all the other whitelists to ensure they are correct.
  1. 28.11.2018 13:11:13
  2. # 5
Anton Accepted Answer
Hello, main source for bots are white domains: https://files.imunify360.com/static/white_domains/v1/white_domains.txt

and whitelist generated for resolved ips: https://files.imunify360.com/static/whitelist/v2/autodetected-search-bots.txt

google.txt is for fallback purposes only.
  1. 28.11.2018 14:11:20
  2. # 6
Darryl Accepted Answer
Posts: 6
Joined: 25.04.2018
0
Votes
Undo
I can see the changes that I made in the google.txt are reflected in the /etc/imunify360-webshield/webshield-http.conf.d/static-whitelist.conf - the https://files.imunify360.com/static/whitelist/v2/autodetected-search-bots.txt doesn't contain the IP that was being blocked - 66.249.66.219 so this wouldn't have been covered by the original google.txt or that file.

Perhaps https://files.imunify360.com/static/white_domains/v1/white_domains.txt would cover it if an rdns lookup is done on each request though then it shouldn't have been blocked. Where is this file stored locally so I can check if there is an issue with that file as I can't see it in /etc/imunify360, /etc/imunify360-webshield or /var/imunify360

Thanks
  1. 28.11.2018 14:11:55
  2. # 7
Igor Seletskiy Accepted Answer
Posts: 1195
Joined: 09.02.2010
0
Votes
Undo
This issue was caused by junk DNS responses from one of the DNS servers we used.
We are changing our algorithms to re-validate DNS responses based on 'previous' state, as well as a secondary DNS server.
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.