GitLab Cluster

2371 字
12 分钟
GitLab Cluster

背景,当前单节点 gitlab 已经无法满足公司需求,gitlab 官网方案用的资源比较多,最好采用 3rails,gitaly,其中 gitaly 分片存储

可能发生的问题,其中某个 gitaly 节点宕机会导致当前节点的 project 无法访问

一, 准备工作#

1 资源申请#

名称数量端口备注
rails380(HTTP),443(TCP/HTTPS),2222(TCP)对外 gitlab 服务
gitaly39999(TCP),9236(TCP)gitlab 存储
负载均衡(F5)180(HTTP),443(TCP/HTTPS),2222(TCP)rails 负载
存储(NFS)1(111,635,2049,4046)TCP/UDP
plpgsql15432(TCP)版本>=12
redis16379(TCP)版本>=5

1.1 系统调优#

参考: [[Linux系统调优]]

2 证书准备#

2.1 Gitaly#

2.1.1 配置文件#

2.1.1.1 ca.conf#
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = BeiJing
localityName = Locality Name (eg, city)
localityName_default = BeiJIng
organizationName = Organization Name (eg, company)
organizationName_default = ccops
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
commonName_default = GitLab CA Test
2.1.1.2 server.conf#
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = BeiJing
localityName = Locality Name (eg, city)
localityName_default = BeiJing
organizationName = Organization Name (eg, company)
organizationName_default = ccops
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
commonName_default = gitaly-2.ccops.cc
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = gitaly-1.ccops.cc
DNS.2 = gitaly-2.ccops.cc
DNS.3 = gitaly-3.ccops.cc

2.1.2 制作证书#

Terminal window
# 制作 ca 私钥
openssl genrsa -out ca.key 4096
# 制作 ca 请求文件
openssl req -new -sha256 -out ca.csr -key ca.key -config ca.conf
# 制作 ca 证书
openssl x509 -req -days 3650 -in ca.csr -signkey ca.key -out ca.crt
# 制作服务私钥
openssl genrsa -out server.key 2048
# 节点1
# 制作服务请求文件
openssl req -new -sha256 -out server-1.csr -key server.key -config server.conf
# 制作服务证书
openssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-1.csr -out server-1.pem -extensions req_ext -extfile server.conf
# 节点2
openssl req -new -sha256 -out server-2.csr -key server.key -config server.conf
openssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-2.csr -out server-2.pem -extensions req_ext -extfile server.conf
# 节点3
openssl req -new -sha256 -out server-3.csr -key server.key -config server.conf
openssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in server-3.csr -out server-3.pem -extensions req_ext -extfile server.conf
# 查看证书
openssl x509 -text -noout -in server-1.pem

2.2 Rails#

这里用的购买的证书,尽量别用自签证书,测试部署的时候各种 509 问题,实在不行用Let’s Encrypt,参考文档

3 架构图#

image-20220610171849739
image-20220610171849739

二, 部署#

1 Redis#

[[RedisSentinel笔记]]

2 Plpgsql#

生产环境是 DB 团队提供的,以下命令是测试环境用的,不建议生产使用

mkdir /data/postgres_data

docker run -it --name postgres --restart always -e POSTGRES_PASSWORD='password' -e ALLOW_IP_RANGE=0.0.0.0/0 -v /data/postgres_data:/var/lib/postgresql -p 5432:5432 -d postgres:12.6

3 Gitaly#

3.1 创建目录#

所有 gitaly 节点都操作

Terminal window
mkdir -p /data/gitlab/config /data/gitlab/config/ssl /data/gitlab/config/trusted-certs /data/gitlab/logs /data/gitlab/data /data/gitlab/git-data

3.2 Copy 证书#

Terminal window
分别将server-[1,2,3].pem server.key copy gitlab-[1,2,3]服务器的/data/gitlab/config/ssl
分别将server-[1,2,3].pem copy gitlab-[1,2,3]服务器的/data/gitlab/config/trusted-certs/
配置权限:
chmod 644 /data/gitlab/config/trusted-certs/*
chmod 644 /data/gitlab/config/trusted-certs/*
目录结构:
tree
├── docker-compose.yaml
├── gitlab.rb
├── ssl
├── server-2.pem
└── server.key
└── trusted-certs
└── server-2.pem

3.3 配置文件#

要修改的:

  • gitlab_rails[‘internal_api_url’]
  • gitaly[‘certificate_path’]
  • gitaly[‘auth_token’]
  • gitlab_shell[‘secret_token’]
  • git_data_dirs
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false
# If you run a separate monitoring node you can disable these services
alertmanager['enable'] = false
prometheus['enable'] = false
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
# Enable only the Gitaly service
gitaly['enable'] = true
# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load
# balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server
gitlab_rails['internal_api_url'] = '' # gitlab对外域名
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/gitlab/ssl/server-1.pem"
gitaly['key_path'] = "/etc/gitlab/ssl/server.key"
# Enable service discovery for Prometheus
consul['enable'] = false
consul['monitoring_service_discovery'] = false
# Set the network addresses that the exporters will listen on for monitoring
gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
# Gitaly Auth Token
# Should be the same as praefect_internal_token
gitaly['auth_token'] = 'LYIddgqhn91vykzAxxxxxxxDtLSTnO'
gitlab_shell['secret_token'] = 'vIH3gfTSXbxxxxxxx3ZH7DDVX17YJ'
git_data_dirs({
'default' => {'gitaly_address' => 'tls://gitaly-1.ccops.cc:9999','path' => '/mnt/gitlab/git-data'},
})

3.4 Docker-compose#

version: '2.0'
services:
gitaly:
restart: always
image: gitlab-ce:14.8.5
hostname: gitaly-1.ccops.cc
extra_hosts:
- "gitlab-tst.ccops.cc:10.1.1.1"
shm_size: '256m'
ports:
- "9999:9999"
- "9236:9236"
volumes:
- "/data/gitlab/config:/etc/gitlab"
- "/data/gitlab/logs:/var/log/gitlab"
- "/data/gitlab/data:/var/opt/gitlab"
- "/data/gitlab/git-data:/mnt/gitlab/git-data"
- /etc/localtime:/etc/localtime:ro
cap_add:
- SYS_TIME
restart: unless-stopped

3.5 启动 Gitaly#

docker-compose up -d

4 Rails#

Gitlab rails 需要共享的目录:

  1. /etc/gitlab
  2. /var/opt/gitlab/.ssh
  3. /var/opt/gitlab/gitlab-rails/uploads
  4. /var/opt/gitlab/gitlab-rails/shared
  5. /var/opt/gitlab/gitlab-ci/builds

以上文件路径 1、2 必需使用非对象存储,如 nfs, gfs 等,官方推荐使用 nfs

3、4、5 可使用 nfs ,但是官网推荐使用对象存储(minio, s3)

4.1 将 NFS 挂载三台 Rails 上#

Terminal window
yum install nfs-utils nfs-utils-lib
mkdir /gitlab-data
mount -t nfs ip:/path /gitlab-data (请更新nfs信息到该命令中)

4.2 创建目录#

Terminal window
在其中一台rails下的nfs目录下创建如下目录
mkdir -p /gitlab-data/config /gitlab-data/config/ssl /gitlab-data/config/trusted-certs /gitlab-data/.ssh /gitlab-data/gitlab-rails/uploads /gitlab-data/gitlab-rails/share /gitlab-data/gitlab-ci/builds
在三台rails服务器上创建目录:
mkdir -p /data/gitlab/logs /data/gitlab/data

4.3 Copy 证书#

Terminal window
copy gitlab域名证书到 /gitlab-data/config/ssl
copy 三台gitaly中的证书server[1,2,3].pem /gitlab-data/config/trusted-certs中
目录结构
tree
├── docker-compose.yaml
├── gitlab.rb
├── ssl
├── gitlab-tst.ccops.cc.crt
└── gitlab-tst.ccops.cc.key
└── trusted-certs
├── server-1.pem
├── server-2.pem
└── server-3.pem

4.4 配置文件#

更多功能配置参考[[GitLab单节点#1 编写 docker-compose.yml]]

这里的配置就比较复杂了,简要说是要修改的:

  • external_url
  • gitaly[‘auth_token’]
  • gitlab_shell[‘secret_token’]
  • git_data_dirs
  • postgresql 配置段落
  • redis 配置段落

如果需要其他功能,比如lfs,sso,请参考 [[GitLab单节点]]

external_url 'https://gitlab-tst.ccops.cc'
self_signed_cert = 'true'
gitlab_rails['time_zone'] = 'Beijing'
runtime_dir = '/dev/shm'
# Enable Prometheus metrics access to Praefect. You must use firewalls
# to restrict access to this address/port.
### Email Settings
gitlab_rails['gitlab_email_enabled'] = false
gitlab_rails['gitlab_email_from'] = ''
gitlab_rails['gitlab_email_display_name'] = ''
### Gravatar Settings
# gitlab_rails['initial_root_password'] = 'qweasd123' # 配置root密码,不建议添加,部署完使用gitlab-rake "gitlab:password:reset[root]"修改密码
gitlab_rails['gravatar_plain_url'] = 'http://gravatar.duoshuo.com/avatar/%{hash}?s=%{size}&d=identicon'
### Webhook Settings
###! Number of seconds to wait for HTTP response after sending webhook HTTP POST
###! request (default: 10)
gitlab_rails['webhook_timeout'] = 20
### LDAP Settings
###! Docs: https://docs.gitlab.com/omnibus/settings/ldap.html
###! **Be careful not to break the indentation in the ldap_servers block. It is
###! in yaml format and the spaces must be retained. Using tabs will not work.**
gitlab_rails['ldap_enabled'] = false
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' # remember to close this block with 'EOS' below
main:
label: 'LDAP'
host: ''
port: 389
uid: ''
method: 'plain' # "tls" or "ssl" or "plain"
bind_dn: ''
password: ''
active_directory: true
allow_username_or_email_login: false
base: ''
active_directory: true
allow_username_or_email_login: false
base: ''
user_filter: ''
group_base: ''
admin_group: ''
sync_ssh_keys: false
EOS
### Backup Settings
###! Docs: https://docs.gitlab.com/omnibus/settings/backups.html
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
# Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests
# # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.
# # The following two values must be the same as their respective values
# # of the Gitaly setup
gitaly['auth_token'] = 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO'
gitlab_shell['secret_token'] = 'vIH3gfTSXbb4XNR3LznY3ZH7DDVX17YJ'
# git_data_dirs get configured for the Praefect virtual storage
# Address is Interal Load Balancer for Praefect
# Token is praefect_external_token
git_data_dirs({
'default' => { 'gitaly_address' => 'tls://gitaly-1.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' },
'storage1' => { 'gitaly_address' => 'tls://gitaly-2.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' },
'storage2' => { 'gitaly_address' => 'tls://gitaly-3.ccops.cc:9999', 'gitaly_token' => 'LYIddgqhn91vykzA8O908MQ7GDtLSTnO' },
})
## Disable components that will not be on the GitLab application server
roles ['application_role']
gitaly['enable'] = false
nginx['enable'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab-tst.ccops.ccm.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab-tst.ccops.cc.key"
nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-S
HA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
nginx['listen_addresses'] = ['0.0.0.0']
nginx['redirect_http_to_https'] = true
web_server['external_users'] = ['nginx']
nginx['client_max_body_size'] = '5000m'
nginx['real_ip_trusted_addresses'] = [ '0.0.0.0/0' ]
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
gitlab_pages['enable'] = true
gitlab_pages['external_http'] = ['0.0.0.0:8087']
gitlab_pages['inplace_chroot'] = true
gitlab_exporter['enable'] = true
gitlab_exporter['listen_address'] = '0.0.0.0'
gitlab_exporter['listen_port'] = '9168'
# PostgreSQL connection details
postgresql['enable'] = false
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_host'] = '10.1.1.1'
gitlab_rails['db_port'] = 5432
gitlab_rails['db_username'] = "user"
gitlab_rails['db_password'] = "passwd"
gitlab_rails['db_database'] = "gitlab"
# Prevent database migrations from running on upgrade automatically
#gitlab_rails['auto_migrate'] = false
# Redis connection details
redis['enable'] = false
gitlab_rails['redis_host'] = "10.1.1.1"
gitlab_rails['redis_port'] = 6379
gitlab_rails['redis_password'] = "passwd"
gitlab_rails['redis_database'] = 0
# Set the network addresses that the exporters used for monitoring will listen on
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
sidekiq['listen_address'] = "0.0.0.0"
puma['listen'] = '0.0.0.0'
puma['exporter_enabled'] = true
puma['exporter_address'] = "0.0.0.0"
puma['worker_timeout'] = 600
gitlab_rails['env'] = {
'GITLAB_RAILS_RACK_TIMEOUT' => 600
}
# Set number of Sidekiq threads per queue process to the recommend number of 10
sidekiq['max_concurrency'] = 25
# Add the monitoring node's IP address to the monitoring whitelist and allow it to
# scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with
# the address and/or subnets gathered from the monitoring node
gitlab_rails['monitoring_whitelist'] = ['10.0.0.0/8','127.0.0.0/8','172.0.0.0/8']
#nginx['status']['options']['allow'] = ['10.0.0.0/8', '127.0.0.0/8','172.0.0.0/8']
# note the 'https' below
#external_url "https://gitlab.xpaas.ccops.com"
letsencrypt['enable'] = false
### GitLab Shell settings for GitLab
# gitlab_rails['gitlab_shell_ssh_port'] = 22
gitlab_rails['git_max_size'] = 209715200
gitlab_rails['git_timeout'] = 30
### GitLab email server settings
###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
###! **Use smtp instead of sendmail/postfix.**
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = ""
gitlab_rails['smtp_port'] = 25
# gitlab_rails['smtp_user_name'] = "smtp user"
# gitlab_rails['smtp_password'] = "smtp password"
# gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
###! **Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert'**
###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.html
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
# user['git_user_name'] = "GitLab"
user['git_user_email'] = ""
################################################################################
## GitLab Logging
###! Docs: https://docs.gitlab.com/omnibus/settings/logs.html
################################################################################
logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log data
logging['svlogd_num'] = 30 # keep 30 rotated log files
logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
logging['svlogd_filter'] = "gzip" # compress logs with gzip
logging['svlogd_udp'] = nil # transmit log messages via UDP
logging['svlogd_prefix'] = nil # custom prefix for log messages
#logging['logrotate_frequency'] = "daily" # rotate logs daily
logging['logrotate_size'] = "500M" # do not rotate by size by default
logging['logrotate_rotate'] = 30 # keep 30 rotated logs
logging['logrotate_compress'] = "compress" # see 'man logrotate'
logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
#logging['logrotate_postrotate'] = nil # no postrotate command by default
# logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz
################################################################################
## Logrotate
##! Docs: https://docs.gitlab.com/omnibus/settings/logs.html#logrotate
##! You can disable built in logrotate feature.
################################################################################
logrotate['enable'] = true
gitlab_ci['gitlab_ci_email_from'] = ''
gitlab_ci['gitlab_ci_support_email'] = ''
gitlab_ci['gravatar_enabled'] = false

4.5 Docker-compose#

version: '2.0'
services:
gitlab-rails:
restart: always
image: registry-sy.ccops.cc/agile_tools/gitlab-ce:14.8.5
extra_hosts:
- "gitlab-tst.ccops.cc:10.1.1.4"
- "gitaly-1.ccops.cc:10.1.1.1"
- "gitaly-2.ccops.cc:10.1.1.2"
- "gitaly-3.ccops.cc:10.1.1.3"
ports:
- "443:443"
- "80:80"
- "2222:22"
- "8080:8080"
- "9168:9168"
- "9229:9229"
- "8082:8082"
- "8060:8060"
- "8083:8083"
- "8087:8087"
volumes:
- "/gitlab-data/config:/etc/gitlab"
- "/data/gitlab/logs:/var/log/gitlab"
- "/data/gitlab/data:/var/opt/gitlab"
- "/gitlab-data/.ssh:/var/opt/gitlab/.ssh"
- "/gitlab-data/gitlab-rails/uploads:/var/opt/gitlab/gitlab-rails/uploads"
- "/gitlab-data/gitlab-rails/shared:/var/opt/gitlab/gitlab-rails/shared"
- "/gitlab-data/gitlab-ci/builds:/var/opt/gitlab/gitlab-ci/builds"
- "/etc/localtime:/etc/localtime:ro"

4.6 启动 Gitlab#

docker-compose up -d

5 查看集群状态#

image-20220610155640637
image-20220610155640637

5.1 如果状态不对#

gitaly 容器执行

docker exec -it <name}/opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml

6 配置 Gitaly 权重#

默认情况 default 权重是 100,创建的所有 project 都分配到 default,需要手动改下

image-20220610163738368
image-20220610163738368

三, 排错与遇到的问题#

docker logs <name>

tailf /data/gitlab/logs/gitlab-rails/production.log

1 连接不到 Redis#

Terminal window
2022-06-09T06:05:04.579Z: {:message=>"Failed to create / detach partition(s)", :table_name=>"batched_background_migration_job_transition_logs", :exception_class=>Redis::CommandError, :exception_message=>"ERR unknown command `sentinel`, with args beginning with: `get-master-addr-by-name`, `sentine10.1.142.110`,", :connection_name=>"main"}

2 找不到证书#

默认是根据 external_url 配置获取证书文件名

Terminal window
2022-06-09_06:05:03.37115 nginx: [emerg] cannot load certificate "/etc/gitlab/ssl/gitlab.example.com.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/gitlab.example.com.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

3 连不到 Pgsql#

这里卡半天,一定要检测 pgsql 连接啊

Terminal window
gitlab Reconfigured!
Checking for unmigrated data on legacy storage
Upgrade failed. Could not check for unmigrated data on legacy storage.
If you would like to restart the instance without perfming this check
check, add the following to your docker command:

4 删除旧日志#

Terminal window
2022-06-09_03:49:19.43475 ts=2022-06-09T03:49:19.432Z caller=log.go:168 level=debug msg="Querying namespace" namespace=pg_stat_bgwriter
*** buffer overflow detected ***: terminated
xargs: tail: terminated by signal 6

5 访问 Gitaly 问题#

一般都是 gitaly 问题,通过/opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml命令测试是否正常

Terminal window
tailf /data/gitlab/logs/gitlab-rails/production.log
GRPC::Unavailable (14:failed to connect to all addresses. debug_error_string:{"created":"@1654766155.243770117","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1654766155.243766927","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}):

6 证书问题#

Terminal window
could not create GitLab API client:

7 跳转#

如果配置 external_url 是 https,那么访问无法通过 http 请求访问

8 Error while Dialing: Dial Tcp: Lookup gitaly-d.ccops on 10.1.1.1:53: no such Host#

使用 dns 解析,没通过 hosts 解析,需要 dns 添加 gitaly-d.ccops 解析

9 org.eclipse.jgit.api.errors.TransportException: https://gitlab-tst.ccops.cc/tst.git: Cannot Open Git-upload-pack#

加密算法问题,添加

Terminal window
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';

文章分享

如果这篇文章对你有帮助,欢迎分享给更多人!

GitLab Cluster
https://ccops.cc/posts/8565eeaf/
作者
蹦蹦站长
发布于
2022-06-13
许可协议
CC BY-NC-SA 4.0

评论区

Profile Image of the Author
蹦蹦站长
一个养兔子的运维
公告
感谢你的来访!希望在这里能找到对你有用的内容!
音乐
封面

音乐

暂未播放

0:00 0:00
暂无歌词
分类
标签
站点统计
文章
81
分类
6
标签
45
总字数
88,456
运行时长
0
最后活动
0 天前
站点信息
构建平台
Local
博客版本
Firefly v6.12.1
文章许可
CC BY-NC-SA 4.0

文章目录