GitLab Cluster
背景,当前单节点 gitlab 已经无法满足公司需求,gitlab 官网方案用的资源比较多,最好采用 3rails,gitaly,其中 gitaly 分片存储
可能发生的问题,其中某个 gitaly 节点宕机会导致当前节点的 project 无法访问
一, 准备工作
1 资源申请
| 名称 | 数量 | 端口 | 备注 |
|---|---|---|---|
| rails | 3 | 80(HTTP),443(TCP/HTTPS),2222(TCP) | 对外 gitlab 服务 |
| gitaly | 3 | 9999(TCP),9236(TCP) | gitlab 存储 |
| 负载均衡(F5) | 1 | 80(HTTP),443(TCP/HTTPS),2222(TCP) | rails 负载 |
| 存储(NFS) | 1 | (111,635,2049,4046)TCP/UDP | |
| plpgsql | 1 | 5432(TCP) | 版本>=12 |
| redis | 1 | 6379(TCP) | 版本>=5 |
1.1 系统调优
参考: [[Linux系统调优]]
2 证书准备
2.1 Gitaly
2.1.1 配置文件
2.1.1.1 ca.conf
[ req ]default_bits = 4096distinguished_name = req_distinguished_name
[ req_distinguished_name ]countryName = Country Name (2 letter code)countryName_default = CNstateOrProvinceName = State or Province Name (full name)stateOrProvinceName_default = BeiJinglocalityName = Locality Name (eg, city)localityName_default = BeiJIngorganizationName = Organization Name (eg, company)organizationName_default = ccopscommonName = Common Name (e.g. server FQDN or YOUR name)commonName_max = 64commonName_default = GitLab CA Test2.1.1.2 server.conf
[ req ]default_bits = 2048distinguished_name = req_distinguished_namereq_extensions = req_ext
[ req_distinguished_name ]countryName = Country Name (2 letter code)countryName_default = CNstateOrProvinceName = State or Province Name (full name)stateOrProvinceName_default = BeiJinglocalityName = Locality Name (eg, city)localityName_default = BeiJingorganizationName = Organization Name (eg, company)organizationName_default = ccopscommonName = Common Name (e.g. server FQDN or YOUR name)commonName_max = 64commonName_default = gitaly-2.ccops.cc
[ req_ext ]subjectAltName = @alt_names
[alt_names]DNS.1 = gitaly-1.ccops.ccDNS.2 = gitaly-2.ccops.ccDNS.3 = gitaly-3.ccops.cc2.1.2 制作证书
# 制作 ca 私钥openssl genrsa -out ca.key 4096# 制作 ca 请求文件openssl req -new -sha256 -out ca.csr -key ca.key -config ca.conf# 制作 ca 证书openssl x509 -req -days 3650 -in ca.csr -signkey ca.key -out ca.crt# 制作服务私钥openssl genrsa -out server.key 2048
# 节点1# 制作服务请求文件openssl req -new -sha256 -out server-1.csr -key server.key -config server.conf# 制作服务证书openssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-1.csr -out server-1.pem -extensions req_ext -extfile server.conf# 节点2openssl req -new -sha256 -out server-2.csr -key server.key -config server.confopenssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-2.csr -out server-2.pem -extensions req_ext -extfile server.conf# 节点3openssl req -new -sha256 -out server-3.csr -key server.key -config server.confopenssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-3.csr -out server-3.pem -extensions req_ext -extfile server.conf
# 查看证书openssl x509 -text -noout -in server-1.pem2.2 Rails
这里用的购买的证书,尽量别用自签证书,测试部署的时候各种 509 问题,实在不行用Let’s Encrypt,参考文档
3 架构图

二, 部署
1 Redis
[[RedisSentinel笔记]]
2 Plpgsql
生产环境是 DB 团队提供的,以下命令是测试环境用的,不建议生产使用
mkdir /data/postgres_data
docker run -it --name postgres --restart always -e POSTGRES_PASSWORD='password' -e ALLOW_IP_RANGE=0.0.0.0/0 -v /data/postgres_data:/var/lib/postgresql -p 5432:5432 -d postgres:12.6
3 Gitaly
3.1 创建目录
所有 gitaly 节点都操作
mkdir -p /data/gitlab/config /data/gitlab/config/ssl /data/gitlab/config/trusted-certs /data/gitlab/logs /data/gitlab/data /data/gitlab/git-data3.2 Copy 证书
分别将server-[1,2,3].pem 和 server.key copy 到 gitlab-[1,2,3]服务器的/data/gitlab/config/ssl分别将server-[1,2,3].pem copy 到 gitlab-[1,2,3]服务器的/data/gitlab/config/trusted-certs/配置权限:chmod 644 /data/gitlab/config/trusted-certs/*chmod 644 /data/gitlab/config/trusted-certs/*目录结构:tree├── docker-compose.yaml├── gitlab.rb├── ssl│ ├── server-2.pem│ └── server.key└── trusted-certs └── server-2.pem3.3 配置文件
要修改的:
- gitlab_rails[‘internal_api_url’]
- gitaly[‘certificate_path’]
- gitaly[‘auth_token’]
- gitlab_shell[‘secret_token’]
- git_data_dirs
postgresql['enable'] = falseredis['enable'] = falsenginx['enable'] = falsepuma['enable'] = falseunicorn['enable'] = falsesidekiq['enable'] = falsegitlab_workhorse['enable'] = falsegrafana['enable'] = false
# If you run a separate monitoring node you can disable these servicesalertmanager['enable'] = falseprometheus['enable'] = false
# Prevent database migrations from running on upgrade automaticallygitlab_rails['auto_migrate'] = false
# Enable only the Gitaly servicegitaly['enable'] = true
# Configure the gitlab-shell API callback URL. Without this, `git push` will# fail. This can be your 'front door' GitLab URL or an internal load# balancer.# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly servergitlab_rails['internal_api_url'] = '' # gitlab对外域名
# Make Gitaly accept connections on all network interfaces. You must use# firewalls to restrict access to this address/port.# Comment out following line if you only want to support TLS connectionsgitaly['tls_listen_addr'] = "0.0.0.0:9999"gitaly['certificate_path'] = "/etc/gitlab/ssl/server-1.pem"gitaly['key_path'] = "/etc/gitlab/ssl/server.key"
# Enable service discovery for Prometheusconsul['enable'] = falseconsul['monitoring_service_discovery'] = false
# Set the network addresses that the exporters will listen on for monitoringgitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
# Gitaly Auth Token# Should be the same as praefect_internal_tokengitaly['auth_token'] = 'LYIddgqhn91vykzAxxxxxxxDtLSTnO'gitlab_shell['secret_token'] = 'vIH3gfTSXbxxxxxxx3ZH7DDVX17YJ'
git_data_dirs({ 'default' => {'gitaly_address' => 'tls://gitaly-1.ccops.cc:9999','path' => '/mnt/gitlab/git-data'},})3.4 Docker-compose
version: '2.0'services: gitaly: restart: always image: gitlab-ce:14.8.5 hostname: gitaly-1.ccops.cc extra_hosts: - "gitlab-tst.ccops.cc:10.1.1.1" shm_size: '256m' ports: - "9999:9999" - "9236:9236" volumes: - "/data/gitlab/config:/etc/gitlab" - "/data/gitlab/logs:/var/log/gitlab" - "/data/gitlab/data:/var/opt/gitlab" - "/data/gitlab/git-data:/mnt/gitlab/git-data" - /etc/localtime:/etc/localtime:ro cap_add: - SYS_TIME restart: unless-stopped3.5 启动 Gitaly
docker-compose up -d
4 Rails
Gitlab rails 需要共享的目录:
- /etc/gitlab
- /var/opt/gitlab/.ssh
- /var/opt/gitlab/gitlab-rails/uploads
- /var/opt/gitlab/gitlab-rails/shared
- /var/opt/gitlab/gitlab-ci/builds
以上文件路径 1、2 必需使用非对象存储,如 nfs, gfs 等,官方推荐使用 nfs
3、4、5 可使用 nfs ,但是官网推荐使用对象存储(minio, s3)
4.1 将 NFS 挂载三台 Rails 上
yum install nfs-utils nfs-utils-libmkdir /gitlab-datamount -t nfs ip:/path /gitlab-data (请更新nfs信息到该命令中)4.2 创建目录
在其中一台rails下的nfs目录下创建如下目录mkdir -p /gitlab-data/config /gitlab-data/config/ssl /gitlab-data/config/trusted-certs /gitlab-data/.ssh /gitlab-data/gitlab-rails/uploads /gitlab-data/gitlab-rails/share /gitlab-data/gitlab-ci/builds在三台rails服务器上创建目录:mkdir -p /data/gitlab/logs /data/gitlab/data4.3 Copy 证书
copy gitlab域名证书到 /gitlab-data/config/sslcopy 三台gitaly中的证书server[1,2,3].pem 到 /gitlab-data/config/trusted-certs中目录结构tree├── docker-compose.yaml├── gitlab.rb├── ssl│ ├── gitlab-tst.ccops.cc.crt│ └── gitlab-tst.ccops.cc.key└── trusted-certs ├── server-1.pem ├── server-2.pem └── server-3.pem4.4 配置文件
更多功能配置参考[[GitLab单节点#1 编写 docker-compose.yml]]
这里的配置就比较复杂了,简要说是要修改的:
- external_url
- gitaly[‘auth_token’]
- gitlab_shell[‘secret_token’]
- git_data_dirs
- postgresql 配置段落
- redis 配置段落
如果需要其他功能,比如lfs,sso,请参考 [[GitLab单节点]]
external_url 'https://gitlab-tst.ccops.cc'self_signed_cert = 'true'gitlab_rails['time_zone'] = 'Beijing'runtime_dir = '/dev/shm'# Enable Prometheus metrics access to Praefect. You must use firewalls# to restrict access to this address/port.
### Email Settingsgitlab_rails['gitlab_email_enabled'] = falsegitlab_rails['gitlab_email_from'] = ''gitlab_rails['gitlab_email_display_name'] = ''
### Gravatar Settings# gitlab_rails['initial_root_password'] = 'qweasd123' # 配置root密码,不建议添加,部署完使用gitlab-rake "gitlab:password:reset[root]"修改密码gitlab_rails['gravatar_plain_url'] = 'http://gravatar.duoshuo.com/avatar/%{hash}?s=%{size}&d=identicon'
### Webhook Settings###! Number of seconds to wait for HTTP response after sending webhook HTTP POST###! request (default: 10)gitlab_rails['webhook_timeout'] = 20
### LDAP Settings###! Docs: https://docs.gitlab.com/omnibus/settings/ldap.html###! **Be careful not to break the indentation in the ldap_servers block. It is###! in yaml format and the spaces must be retained. Using tabs will not work.**gitlab_rails['ldap_enabled'] = false
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' # remember to close this block with 'EOS' below main: label: 'LDAP' host: '' port: 389 uid: '' method: 'plain' # "tls" or "ssl" or "plain" bind_dn: '' password: '' active_directory: true allow_username_or_email_login: false base: '' active_directory: true allow_username_or_email_login: false base: '' user_filter: '' group_base: '' admin_group: '' sync_ssh_keys: falseEOS
### Backup Settings###! Docs: https://docs.gitlab.com/omnibus/settings/backups.htmlgitlab_rails['manage_backup_path'] = truegitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
# Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests# # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.# # The following two values must be the same as their respective values# # of the Gitaly setupgitaly['auth_token'] = 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO'gitlab_shell['secret_token'] = 'vIH3gfTSXbb4XNR3LznY3ZH7DDVX17YJ'
# git_data_dirs get configured for the Praefect virtual storage# Address is Interal Load Balancer for Praefect# Token is praefect_external_tokengit_data_dirs({ 'default' => { 'gitaly_address' => 'tls://gitaly-1.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' }, 'storage1' => { 'gitaly_address' => 'tls://gitaly-2.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' }, 'storage2' => { 'gitaly_address' => 'tls://gitaly-3.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' },})## Disable components that will not be on the GitLab application serverroles ['application_role']gitaly['enable'] = falsenginx['enable'] = truenginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab-tst.ccops.ccm.crt"nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab-tst.ccops.cc.key"nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"nginx['listen_addresses'] = ['0.0.0.0']nginx['redirect_http_to_https'] = trueweb_server['external_users'] = ['nginx']nginx['client_max_body_size'] = '5000m'nginx['real_ip_trusted_addresses'] = [ '0.0.0.0/0' ]nginx['real_ip_header'] = 'X-Forwarded-For'nginx['real_ip_recursive'] = 'on'
gitlab_pages['enable'] = truegitlab_pages['external_http'] = ['0.0.0.0:8087']gitlab_pages['inplace_chroot'] = truegitlab_exporter['enable'] = truegitlab_exporter['listen_address'] = '0.0.0.0'gitlab_exporter['listen_port'] = '9168'
# PostgreSQL connection detailspostgresql['enable'] = falsegitlab_rails['db_adapter'] = "postgresql"gitlab_rails['db_host'] = '10.1.1.1'gitlab_rails['db_port'] = 5432gitlab_rails['db_username'] = "user"gitlab_rails['db_password'] = "passwd"gitlab_rails['db_database'] = "gitlab"# Prevent database migrations from running on upgrade automatically#gitlab_rails['auto_migrate'] = false
# Redis connection detailsredis['enable'] = falsegitlab_rails['redis_host'] = "10.1.1.1"gitlab_rails['redis_port'] = 6379gitlab_rails['redis_password'] = "passwd"gitlab_rails['redis_database'] = 0
# Set the network addresses that the exporters used for monitoring will listen ongitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'sidekiq['listen_address'] = "0.0.0.0"puma['listen'] = '0.0.0.0'puma['exporter_enabled'] = truepuma['exporter_address'] = "0.0.0.0"puma['worker_timeout'] = 600gitlab_rails['env'] = { 'GITLAB_RAILS_RACK_TIMEOUT' => 600 }# Set number of Sidekiq threads per queue process to the recommend number of 10sidekiq['max_concurrency'] = 25
# Add the monitoring node's IP address to the monitoring whitelist and allow it to# scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with# the address and/or subnets gathered from the monitoring nodegitlab_rails['monitoring_whitelist'] = ['10.0.0.0/8','127.0.0.0/8','172.0.0.0/8']#nginx['status']['options']['allow'] = ['10.0.0.0/8', '127.0.0.0/8','172.0.0.0/8']
# note the 'https' below#external_url "https://gitlab.xpaas.ccops.com"letsencrypt['enable'] = false
### GitLab Shell settings for GitLab# gitlab_rails['gitlab_shell_ssh_port'] = 22gitlab_rails['git_max_size'] = 209715200gitlab_rails['git_timeout'] = 30
### GitLab email server settings###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html###! **Use smtp instead of sendmail/postfix.**
gitlab_rails['smtp_enable'] = truegitlab_rails['smtp_address'] = ""gitlab_rails['smtp_port'] = 25# gitlab_rails['smtp_user_name'] = "smtp user"# gitlab_rails['smtp_password'] = "smtp password"# gitlab_rails['smtp_domain'] = "example.com"gitlab_rails['smtp_authentication'] = "plain"gitlab_rails['smtp_enable_starttls_auto'] = truegitlab_rails['smtp_tls'] = false###! **Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert'**###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.htmlgitlab_rails['smtp_openssl_verify_mode'] = 'none'
# user['git_user_name'] = "GitLab"user['git_user_email'] = ""
################################################################################## GitLab Logging###! Docs: https://docs.gitlab.com/omnibus/settings/logs.html################################################################################logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log datalogging['svlogd_num'] = 30 # keep 30 rotated log fileslogging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hourslogging['svlogd_filter'] = "gzip" # compress logs with gziplogging['svlogd_udp'] = nil # transmit log messages via UDPlogging['svlogd_prefix'] = nil # custom prefix for log messages#logging['logrotate_frequency'] = "daily" # rotate logs dailylogging['logrotate_size'] = "500M" # do not rotate by size by defaultlogging['logrotate_rotate'] = 30 # keep 30 rotated logslogging['logrotate_compress'] = "compress" # see 'man logrotate'logging['logrotate_method'] = "copytruncate" # see 'man logrotate'#logging['logrotate_postrotate'] = nil # no postrotate command by default# logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz################################################################################## Logrotate##! Docs: https://docs.gitlab.com/omnibus/settings/logs.html#logrotate##! You can disable built in logrotate feature.################################################################################logrotate['enable'] = true
gitlab_ci['gitlab_ci_email_from'] = ''gitlab_ci['gitlab_ci_support_email'] = ''gitlab_ci['gravatar_enabled'] = false4.5 Docker-compose
version: '2.0'services: gitlab-rails: restart: always image: registry-sy.ccops.cc/agile_tools/gitlab-ce:14.8.5 extra_hosts: - "gitlab-tst.ccops.cc:10.1.1.4" - "gitaly-1.ccops.cc:10.1.1.1" - "gitaly-2.ccops.cc:10.1.1.2" - "gitaly-3.ccops.cc:10.1.1.3" ports: - "443:443" - "80:80" - "2222:22" - "8080:8080" - "9168:9168" - "9229:9229" - "8082:8082" - "8060:8060" - "8083:8083" - "8087:8087" volumes: - "/gitlab-data/config:/etc/gitlab" - "/data/gitlab/logs:/var/log/gitlab" - "/data/gitlab/data:/var/opt/gitlab" - "/gitlab-data/.ssh:/var/opt/gitlab/.ssh" - "/gitlab-data/gitlab-rails/uploads:/var/opt/gitlab/gitlab-rails/uploads" - "/gitlab-data/gitlab-rails/shared:/var/opt/gitlab/gitlab-rails/shared" - "/gitlab-data/gitlab-ci/builds:/var/opt/gitlab/gitlab-ci/builds" - "/etc/localtime:/etc/localtime:ro"4.6 启动 Gitlab
docker-compose up -d
5 查看集群状态

5.1 如果状态不对
gitaly 容器执行
docker exec -it <name}/opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml
6 配置 Gitaly 权重
默认情况 default 权重是 100,创建的所有 project 都分配到 default,需要手动改下

三, 排错与遇到的问题
docker logs <name>
tailf /data/gitlab/logs/gitlab-rails/production.log
1 连接不到 Redis
2022-06-09T06:05:04.579Z: {:message=>"Failed to create / detach partition(s)", :table_name=>"batched_background_migration_job_transition_logs", :exception_class=>Redis::CommandError, :exception_message=>"ERR unknown command `sentinel`, with args beginning with: `get-master-addr-by-name`, `sentine10.1.142.110`,", :connection_name=>"main"}2 找不到证书
默认是根据 external_url 配置获取证书文件名
2022-06-09_06:05:03.37115 nginx: [emerg] cannot load certificate "/etc/gitlab/ssl/gitlab.example.com.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/gitlab.example.com.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)3 连不到 Pgsql
这里卡半天,一定要检测 pgsql 连接啊
gitlab Reconfigured!Checking for unmigrated data on legacy storageUpgrade failed. Could not check for unmigrated data on legacy storage.
If you would like to restart the instance without perfming this checkcheck, add the following to your docker command:4 删除旧日志
2022-06-09_03:49:19.43475 ts=2022-06-09T03:49:19.432Z caller=log.go:168 level=debug msg="Querying namespace" namespace=pg_stat_bgwriter*** buffer overflow detected ***: terminatedxargs: tail: terminated by signal 65 访问 Gitaly 问题
一般都是 gitaly 问题,通过
/opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml命令测试是否正常
tailf /data/gitlab/logs/gitlab-rails/production.logGRPC::Unavailable (14:failed to connect to all addresses. debug_error_string:{"created":"@1654766155.243770117","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1654766155.243766927","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}):6 证书问题
could not create GitLab API client:7 跳转
如果配置 external_url 是 https,那么访问无法通过 http 请求访问
8 Error while Dialing: Dial Tcp: Lookup gitaly-d.ccops on 10.1.1.1:53: no such Host
使用 dns 解析,没通过 hosts 解析,需要 dns 添加 gitaly-d.ccops 解析
9 org.eclipse.jgit.api.errors.TransportException: https://gitlab-tst.ccops.cc/tst.git: Cannot Open Git-upload-pack
加密算法问题,添加
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';文章分享
如果这篇文章对你有帮助,欢迎分享给更多人!