如何在Kubernetes里使用cephrdb作为存储

1 准备工作#

部署集群参考: [[RookCeph安装]]

1.1 创建一个 ceph pool 创建存储池#

1
ceph osd pool create rbd 128
2
ceph osd pool set-quota rbd max_bytes $((20 * 1024 * 1024 * 1024)) #20G的存储池
3
rbd pool init rbd

1.1.1 查看集群状态#

csi-config-map.yaml 会用到

1
ceph -s
2
  cluster:
3
    id:     xxxxxxx-67a0f564e0d6
4
    health: HEALTH_OK
5

6
  services:
7
    mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 7d)
8
    mgr: ceph01.pjvndt(active, since 7d), standbys: ceph02.injlkl, ceph03.sulrio
9
    mds: 1/1 daemons up, 2 standby
10
    osd: 8 osds: 8 up (since 7d), 8 in (since 7d)
11
    rgw: 3 daemons active (3 hosts, 1 zones)
12

13
  data:
14
    volumes: 1/1 healthy
15
    pools:   10 pools, 265 pgs
16
    objects: 269 objects, 8.3 KiB
17
    usage:   20 GiB used, 400 GiB / 420 GiB avail
18
    pgs:     265 active+clean

1.1.2 查看用户 key#

csi-rbd-secret.yaml 文件会用到

1
ceph auth get client.admin
2
[client.admin]
3
        key = xxxxxxxxMLkw==
4
        caps mds = "allow *"
5
        caps mgr = "allow *"
6
        caps mon = "allow *"
7
        caps osd = "allow *"
8
exported keyring for client.admin

或者自己创建存储池、用户以及用户 key

1
ceph osd pool create kubernetes
2
rbd pool init kubernetes
3
ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
4
[client.kubernetes]
5
        key = xxxxxxxpmg==

1.1.3 查看集群信息#

csi-config-map.yaml 会用到

1
ceph mon dump
2
epoch 3
3
fsid 2a8e37c8-a0ef-11ec-bfb9-67a0f564e0d6
4
last_changed 2022-03-11T03:57:44.615026+0000
5
created 2022-03-11T03:56:03.739853+0000
6
min_mon_release 16 (pacific)
7
election_strategy: 1
8
0: [v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0] mon.ceph01
9
1: [v2:10.1.1.2:3300/0,v1:10.1.1.2:6789/0] mon.ceph02
10
2: [v2:10.1.1.3:3300/0,v1:10.1.1.3:6789/0] mon.ceph03
11
dumped monmap epoch 3

这里有两种方案

所有操作都在 pod 里使用 rbd 命令,只能在快照卷和克隆卷用 ReadOnlyMany,但是部署方便

node 节点上安装 ceph-common,使用 ceph-common 挂载无限制,但是部署不方便

2 第一种方法(功能多,推荐)#

2.1 生成 ceph-csi 的 kubernetes configmap#

下载此目录所有文件可以参考官网的,需要自己改镜像

1
cat csi-config-map.yaml
2
---
3
apiVersion: v1
4
kind: ConfigMap
5
metadata:
6
  name: "ceph-csi-config"
7
  namespace: rbd
8
data:
9
  config.json: |-
10
    [
11
      {
12
        "clusterID": "2a8e37c8-a0ef-11ec-bfb9-67a0f564e0d6",
13
        "monitors": [
14
          "10.1.1.1:6789",
15
          "10.1.1.2:6789",
16
          "10.1.1.3:6789"
17
        ]
18
      }
19
    ]

2.1.1 修改 kubectl 路径#

1
grep plugins_registry *
2
csi-nodeplugin-psp.yaml:            path: /var/lib/kubelet/plugins_registry/
3
csi-rbdplugin.yaml:    - pathPrefix: '/var/lib/kubelet/plugins_registry'

sed -i "s/var\/lib/data/g" *

1
grep plugins_registry *
2
csi-nodeplugin-psp.yaml:    - pathPrefix: '/data/kubelet/plugins_registry'
3
csi-rbdplugin.yaml:            path: /data/kubelet/plugins_registry/

2.1.2 生成 ceph-csi cephx 的 secret#

1
cat <<EOF > csi-rbd-secret.yaml
2
apiVersion: v1
3
kind: Secret
4
metadata:
5
  name: csi-rbd-secret
6
  namespace: rbd
7
stringData:
8
  userID: admin
9
  userKey: AQBTyCpiHHhbARAAfHqI0X9iMd3rnzJQHaMLkw==
10
EOF

2.1.3 部署 csi#

kubectl apply -f . -n rbd

2.2 使用 ceph 块儿设备#

2.2.1 创建 storageclass,注意 secret-name 要与上面 Secret 名字一致#

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
   name: csi-rbd-sc
5
provisioner: rbd.csi.ceph.com
6
parameters:
7
   clusterID: 2a8e37c8-a0ef-11ec-bfb9-67a0f564e0d6
8
   pool: rbd
9
   imageFeatures: layering
10
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
11
   csi.storage.k8s.io/provisioner-secret-namespace: rbd
12
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
13
   csi.storage.k8s.io/controller-expand-secret-namespace: rbd
14
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
15
   csi.storage.k8s.io/node-stage-secret-namespace: rbd
16
   csi.storage.k8s.io/fstype: ext4
17
reclaimPolicy: Delete
18
allowVolumeExpansion: true
19
mountOptions:
20
   - discard

clusterID 对应之前的步骤中的 fsid
imageFeatures，这个是用来确定创建的 image 的特征的
allowVolumeExpansion: true 是否开启在线扩容

kubectl apply -f sc.yaml -n rbd

2.2.2 查看 storageclass#

1
kubectl get StorageClass -n rbd
2
NAME            PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
3
csi-rbd-sc      rbd.csi.ceph.com      Delete          Immediate           true                   23m

2.2.3 创建 PVC#

1
cat pvc.yaml
2
apiVersion: v1
3
kind: PersistentVolumeClaim
4
metadata:
5
  name: rbd-pvc
6
spec:
7
  accessModes:
8
    - ReadWriteOnce
9
  resources:
10
    requests:
11
      storage: 1Gi
12
  storageClassName: csi-rbd-sc

kubectl aaply -f pvc.yaml

2.2.4 查看 pvc#

1
kubectl get pvc -n rbd
2
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
3
rbd-pvc   Bound    pvc-d98c1d23-6e67-4185-82c5-d862b9768267   1Gi        RWO            csi-rbd-sc     2d15h

2.2.5 测试挂载 pod 写入#

1
cat <<EOF > pod.yaml
2
kind: Pod
3
apiVersion: v1
4
metadata:
5
  name: rbd-pod
6
  namespace: rbd
7
spec:
8
  containers:
9
  - name: rbd-pod
10
    image:  nginx
11
    volumeMounts:
12
      - name: pvc
13
        mountPath: "/mnt"
14
  volumes:
15
    - name: pvc
16
      persistentVolumeClaim:
17
        claimName: rbd-pvc
18
EOF

1
kubectl apply -f pod.yaml -n rbd
2
kubectl exec -it rbd-pod -n rbd -- sh
3
echo "hello" > /mnt/1
4
cat /mnt/1

2.3 快照的使用#

2.3.1 安装 snapshot 驱动#

1
./install-snapshot.sh install

2.3.2 创建 VolumeSnapshotClass#

1
apiVersion: snapshot.storage.k8s.io/v1
2
kind: VolumeSnapshotClass
3
metadata:
4
  name: csi-rbdplugin-snapclass
5
driver: rbd.csi.ceph.com
6
parameters:
7
  clusterID: 2a8e37c8-a0ef-11ec-bfb9-67a0f564e0d6
8
  snapshotNamePrefix: "test-" # 命名RBD快照的前缀
9
  csi.storage.k8s.io/snapshotter-secret-name: csi-rbd-secret
10
  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system
11
deletionPolicy: Delete

2.3.3 创建 VolumeSnapshot#

要与 pvc 在同一个命名空间

1
apiVersion: snapshot.storage.k8s.io/v1
2
kind: VolumeSnapshot
3
metadata:
4
  name: rbd-pvc-snapshot
5
spec:
6
  volumeSnapshotClassName: csi-rbdplugin-snapclass # VolumeSnapshotClass名称
7
  source:
8
    persistentVolumeClaimName: rbd-pvc # pvc名称

2.3.4 查看快照#

kubectl describe volumesnapshot rbd-pvc-snapshot

2.3.5 还原快照#

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: rbd-pvc-restore
5
spec:
6
  storageClassName: csi-rbd-sc
7
  dataSource:
8
    name: rbd-pvc-snapshot
9
    kind: VolumeSnapshot
10
    apiGroup: snapshot.storage.k8s.io
11
  accessModes:
12
    - ReadWriteOnce
13
  resources:
14
    requests:
15
      storage: 1Gi

3 第二种方法#

3.1 安装 ceph 命令工具#

每个节点都需要安装

ceph-common,不装会报没有挂载类型错误

ceph-fuse,不装会报找不到配置文件

1
rpm -ivh https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ceph-release-1-1.el7.noarch.rpm
2
yum install -y yum-utils && \
3
yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && \
4
yum install --nogpgcheck -y epel-release && \
5
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && \
6
rm -f /etc/yum.repos.d/dl.fedoraproject.org*
7
yum install ceph-common ceph-fuse

3.2 创建 storageclass#

1
cat <<EOF > sc.yaml
2
apiVersion: storage.k8s.io/v1
3
kind: StorageClass
4
metadata:
5
  name: intree-rbd
6
provisioner: kubernetes.io/rbd
7
parameters:
8
  monitors: 10.1.1.1:6789,10.1.1.2:6789,10.1.1.3:6789
9
  adminId: admin
10
  adminSecretName: ceph-secret
11
  adminSecretNamespace: kube-system
12
  pool: rbd
13
  userId: admin
14
  userSecretName: ceph-secret
15
  userSecretNamespace: kube-system
16
  fsType: ext4
17
  imageFormat: "2"
18
  imageFeatures: "layering"
19
  kubectl apply -f sc.yaml -n rbd

1
cat <<EOF > csi-rbd-secret.yaml
2
apiVersion: v1
3
kind: Secret
4
metadata:
5
  name: csi-rbd-secret
6
  namespace: rbd
7
type: kubernetes.io/rbd
8
stringData:
9
  userID: admin
10
  userKey: xxxx0X9iMd3rnzJQHaMLkw==
11
EOF
12
kubectl apply -f csi-rbd-secret.yaml -n rbd

3.3 使用 ceph 块儿设备#

3.3.1 创建 PVC#

1
cat pvc.yaml
2
apiVersion: v1
3
kind: PersistentVolumeClaim
4
metadata:
5
  name: rbd-pvc
6
spec:
7
  accessModes:
8
    - ReadWriteOnce
9
  resources:
10
    requests:
11
      storage: 1Gi
12
  storageClassName: intree-rbd

kubectl aaply -f pvc.yaml

3.3.2 查看 pvc#

1
kubectl get pvc -n rbd
2
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
3
rbd-pvc   Bound    pvc-d98c1d23-6e67-4185-82c5-d862b9768267   1Gi        RWO            csi-rbd-sc     2d15h

4 排错#

4.1 查看日志#

kubectl logs deployment.apps/csi-rbdplugin-provisioner -n rbd --all-containers=true --max-log-requests=7

4.2 查看 pvc 状态#

kubectl describe pvc rbd-pvc -n rbd

4.3 报错找不到配置文件#

1
 persistentvolume-controller  Failed to provision volume with StorageClass "intree-rbd": failed to create rbd image: exit status 2, command output: did not load config file, using default settings.
2
2022-04-24 15:12:36.652 7fa722bb6c80 -1 Errors while parsing config file!
3
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
4
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
5
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
6
2022-04-24 15:12:36.652 7fa722bb6c80 -1 Errors while parsing config file!
7
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
8
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
9
2022-04-24 15:12:36.652 7fa722bb6c80 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
10
2022-04-24 15:12:36.680 7fa722bb6c80 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

检查 key 与地址是否对,ceph auth get client.admin,ceph mon dump

检查 ceph 是否有这个 pull,rados lspools

1 准备工作#

1.1 创建一个 ceph pool 创建存储池#

1.1.1 查看集群状态#

1.1.2 查看用户 key#

1.1.3 查看集群信息#

2 第一种方法(功能多,推荐)#

2.1 生成 ceph-csi 的 kubernetes configmap#

2.1.1 修改 kubectl 路径#

2.1.2 生成 ceph-csi cephx 的 secret#

2.1.3 部署 csi#

2.2 使用 ceph 块儿设备#

2.2.1 创建 storageclass,注意 secret-name 要与上面 Secret 名字一致#

2.2.2 查看 storageclass#

2.2.3 创建 PVC#

2.2.4 查看 pvc#

2.2.5 测试挂载 pod 写入#

2.3 快照的使用#

2.3.1 安装 snapshot 驱动#

2.3.2 创建 VolumeSnapshotClass#

2.3.3 创建 VolumeSnapshot#

2.3.4 查看快照#

2.3.5 还原快照#

3 第二种方法#

3.1 安装 ceph 命令工具#

3.2 创建 storageclass#

3.3 使用 ceph 块儿设备#

3.3.1 创建 PVC#

3.3.2 查看 pvc#

4 排错#

4.1 查看日志#

4.2 查看 pvc 状态#

4.3 报错找不到配置文件#

文章分享

评论区

音乐

文章目录