Error 403 in gitlab

describe

Some users suddenly appear 403 forbidden.
Server environment: docker
gitlab version: 8.7
View log:

192.161.11.20 - - [08/Jan/2018:17:01:32 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:01:49 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:09 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:20 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:20 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:22 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:24 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
192.161.11.20 - - [08/Jan/2018:17:02:25 +0800] "GET / HTTP/1.1" 403 10 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

Count the following times:

root@783977abd2ec:/home/git/gitlab# cat /var/log/gitlab/nginx/gitlab_access.log | grep 403 | grep "192.161.11.20" | wc -l
62
root@783977abd2ec:/home/git/gitlab#

Troubleshooting

Because other users are normal, and this user was normal before. That's certainly not a server system configuration problem.

Find the official discussion board:
https://gitlab.com/gitlab-org/gitlab-ce/issues/1171

I looked at the following possibilities next to each other:

  • Check session: redis cli keys' * '| grep' ^ [a-f0-9] \ {32 \} $'| WC - L, the result is 0.
  • Check nginx configuration: previously excluded.
  • Check that there is a piece of data in attack: redis cli keys' * '| grep' rack:: Attack '.
root@9152a066a2ba:/usr/bin#
root@9152a066a2ba:/usr/bin# redis-cli keys '*' | grep 'rack::attack'
cache:gitlab:rack::attack:allow2ban:ban:192.161.11.20
root@9152a066a2ba:/usr/bin# redis-cli -h
redis-cli 2.8.4

This ip is the one that cannot be accessed.

If we find the problem, it will be solved. My planned solution:

  1. Check the data and confirm whether the data in redis can be deleted directly.
  2. Search the redis cli command usage, and delete the key with this command.

Before the first step is finished, the data will be deleted automatically. embarrassed!!! It is found that the data can be deleted directly. The next step is to use redis cli to delete the specified key.
Step 2: delete the key specified in redis:

redis-cli keys '*' | grep 'rack::attack' | xargs redis-cli DEL

Why?

In the / home/git/gitlab/config/gitlab.yml file, there is a rack "attack configuration. If it is an internal network, set enabled to false directly. If it is an external network, adjust the maxretry according to the actual situation.

  rack_attack:
    git_basic_auth:
      # Rack Attack IP banning enabled
      enabled: true
      #
      # Whitelist requests from 127.0.0.1 for web proxies (NGINX/Apache) with incorrect headers
      ip_whitelist: [127.0.0.1]
      #
      # Limit the number of Git HTTP authentication attempts per IP
      maxretry: 10
      #
      # Reset the auth attempt counter per IP after 60 seconds
      findtime: 60
      #
      # Ban an IP for one hour (3600s) after too many auth attempts
      bantime: 3600

Tags: Redis GitLab Mac OS X

Posted on Sat, 02 May 2020 00:48:29 -0700 by itsmani1