一, 安装#

安装参考文档:

配置参考

1 下载代码#

1
git clone --single-branch --branch v1.10.0 https://github.com/rook/rook.git

2 修改配置文件#

2.1 operator.yaml#

1
# 修改镜像地址
2
  ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.2"
3
  ROOK_CSI_REGISTRAR_IMAGE: "ccr.ccs.tencentyun.com/ccops/csi-node-driver-registrar:v2.5.1"
4
  ROOK_CSI_RESIZER_IMAGE: "ccr.ccs.tencentyun.com/ccops/csi-resizer:v1.4.0"
5
  ROOK_CSI_PROVISIONER_IMAGE: "ccr.ccs.tencentyun.com/ccops/csi-provisioner:v3.1.1"
6
  ROOK_CSI_SNAPSHOTTER_IMAGE: "ccr.ccs.tencentyun.com/ccops/csi-snapshotter:v6.0.1"
7
  ROOK_CSI_ATTACHER_IMAGE: "ccr.ccs.tencentyun.com/ccops/csi-attacher:v3.4.0"
8
  ROOK_CSI_NFS_IMAGE: "ccr.ccs.tencentyun.com/ccops/nfsplugin:v4.0.0"

2.2 cluster.yaml#

这里只做了简单配置,详细配置参考文档

1
# 添加Ceph配置
2
kind: ConfigMap
3
apiVersion: v1
4
metadata:
5
  name: rook-config-override
6
  namespace: rook-ceph # namespace:cluster
7
data:
8
  config: |
9
    [global]
10
    osd_pool_default_size = 1 # 一般为3或1,测试1,生产为3
11
    mon_warn_on_pool_no_redundancy = false # 关闭强制配置副本
12
    bdev_flock_retry = 20
13
    bluefs_buffered_io = false
14
---
15
# 监控配置, 安装监控后启用,要不然报错
16
  monitoring:
17
    enabled: false
18
# 开启
19
  mgr:
20
    count: 2
21
    allowMultiplePerNode: false
22
    modules:
23
      - name: pg_autoscaler
24
        enabled: true
25
# 配置 dashboard
26
  dashboard:
27
    enabled: true
28
    port: 8080 # 指定端口
29
    ssl: false # 关闭https
30
# 开启host network, 注意端口占用
31
    provider: host
32
# 配置存储
33
    nodes:
34
      - name: "10.1.1.1"
35
        devices:
36
          - name: "sdb"
37
      - name: "10.1.1.2"
38
        devices:
39
          - name: "sdb"
40
      - name: "10.1.1.3"
41
        devices:
42
          - name: "sdb"
43
# 配置调度节点(需提前打好标签)
44
  placement:
45
    mgr:
46
      nodeAffinity:
47
        requiredDuringSchedulingIgnoredDuringExecution:
48
          nodeSelectorTerms:
49
          - matchExpressions:
50
            - key: role
51
              operator: In
52
              values:
53
              - rook-mgr-node
54
    osd:
55
      nodeAffinity:
56
        requiredDuringSchedulingIgnoredDuringExecution:
57
          nodeSelectorTerms:
58
          - matchExpressions:
59
            - key: role
60
              operator: In
61
              values:
62
              - rook-osd-node

3 部署#

3.1 rook-operator#

1
# cd rook-1.9.10/deploy/examples
2
# kubectl create -f crds.yaml -f common.yaml -f operator.yaml
3
# kubectl -n rook-ceph get pod
4
NAME                                                     READY   STATUS      RESTARTS   AGE
5
rook-ceph-operator-6497755589-vs5rt                      1/1     Running     0          2d23h### CephCluster

3.2 监控#

参考
因为我们有自己的 Prometheus Operator, 而 rook ceph 也有 Operator 不能共存,所以无法集成到自己的 Prometheus , 采用 root ceph 单独部署 Prometheus 方案

3.3 部署 Prometheus#

1
wget https://raw.githubusercontent.com/coreos/prometheus-operator/v0.40.0/bundle.yaml
2
# 修改
3
subjects:
4
- kind: ServiceAccount
5
  name: prometheus-operator
6
  namespace: kube-system # 要部署的命名空间,这个很重要,默认default

1
cd rook/deploy/examples/monitoring
2
kubectl create -f service-monitor.yaml
3
kubectl create -f prometheus.yaml
4
kubectl create -f prometheus-service.yaml

3.3.1 配置 Grafana#

模板地址,全都加上,监控不怕多

3.3.1.1 安装插件#

1
grafana-cli plugins install grafana-piechart-panel
2
grafana-cli plugins install vonage-status-panel

3.3.1.2 配置 Grafana#

defaults.ini,grafana.ini 都要修改

1
[alerting]
2
execute_alerts = true
3
allow_embedding = true
4

5
[auth.anonymous]
6
enabled = true
7
org_name = Main Org.
8
org_role = Viewer

3.3.1.3 配置数据源#

名字不要修改

3.3.2 Dashboard 集成 Grafana#

1
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
2
# grafana url: https://xx.com:3000/
3
ceph dashboard set-grafana-api-url ${grafana url}
4
# 允许自签名证书
5
ceph dashboard set-grafana-api-ssl-verify False

3.3.3 查看#

这部是后面操作完查看的

3.4 CephCluster#

以下 pod 为正常

1
# kubectl create -f cluster.yaml
2
# kubectl -n rook-ceph get pod
3
NAME                                                     READY   STATUS      RESTARTS   AGE
4
csi-cephfsplugin-7tqw8                                   2/2     Running     0          2d23h
5
csi-cephfsplugin-k22lg                                   2/2     Running     0          2d23h
6
csi-cephfsplugin-m2crv                                   2/2     Running     0          2d23h
7
csi-cephfsplugin-provisioner-5dbbcdf76-99tx5             5/5     Running     0          2d23h
8
csi-cephfsplugin-provisioner-5dbbcdf76-bqmz9             5/5     Running     0          2d23h
9
csi-rbdplugin-bdlwq                                      2/2     Running     0          2d23h
10
csi-rbdplugin-dsk5w                                      2/2     Running     0          2d23h
11
csi-rbdplugin-provisioner-7c5c5b699-9bzcq                5/5     Running     0          2d23h
12
csi-rbdplugin-provisioner-7c5c5b699-jwsm9                5/5     Running     0          2d23h
13
csi-rbdplugin-tljkh                                      2/2     Running     0          2d23h
14
rook-ceph-crashcollector-k8s-ceph-100-6b966fbfdd-hq2cf   1/1     Running     0          2d23h
15
rook-ceph-crashcollector-k8s-ceph-136-6b76f4c697-trxlf   1/1     Running     0          2d23h
16
rook-ceph-crashcollector-k8s-ceph-142-696bcc5974-8tvjf   1/1     Running     0          2d23h
17
rook-ceph-mgr-a-7cd79f499c-lqfgc                         2/2     Running     0          2d17h
18
rook-ceph-mgr-b-5b5746f69f-wfd22                         2/2     Running     0          2d17h
19
rook-ceph-mon-a-845779bff7-bz8gz                         1/1     Running     0          2d23h
20
rook-ceph-mon-b-54dbc6c98d-m7psh                         1/1     Running     0          2d23h
21
rook-ceph-mon-c-b7bcb5764-cjhvc                          1/1     Running     0          2d23h
22
rook-ceph-operator-6497755589-vs5rt                      1/1     Running     0          2d23h
23
rook-ceph-osd-0-5c69ff6cb4-g428b                         1/1     Running     0          2d23h
24
rook-ceph-osd-1-ff44fddf7-pczt4                          1/1     Running     0          2d23h
25
rook-ceph-osd-2-5868fd8dbd-jq4d4                         1/1     Running     0          2d23h
26
rook-ceph-osd-prepare-k8s-ceph-100-kqtg8                 0/1     Completed   0          2d19h
27
rook-ceph-osd-prepare-k8s-ceph-136-2w9lh                 0/1     Completed   0          2d19h
28
rook-ceph-osd-prepare-k8s-ceph-142-8ljp5                 0/1     Completed   0          2d19h
29
rook-ceph-tools-556cf9bd64-2h5kr                         1/1     Running     0          2d23h

3.5 查看详情#

1
kubectl describe CephCluster -n rook-ceph rook-ceph

4 工具#

1
kubectl create -f toolbox.yaml
2
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
3
ceph status
4
  cluster:
5
    id:     6aa56874-3c30-4c29-abd2-5e4a1363f03c
6
    health: HEALTH_OK
7

8
  services:
9
    mon: 3 daemons, quorum a,b,c (age 2d)
10
    mgr: a(active, since 2d), standbys: b
11
    osd: 3 osds: 3 up (since 2d), 3 in (since 2d)
12

13
  data:
14
    pools:   3 pools, 65 pgs
15
    objects: 17 objects, 86 B
16
    usage:   80 MiB used, 600 GiB / 600 GiB avail
17
    pgs:     65 active+clean

5 dashboard#

1
# ls dashboard-*
2
dashboard-external-https.yaml  dashboard-external-http.yaml  dashboard-ingress-https.yaml  dashboard-loadbalancer.yaml
3
# 根据需求选择,这里用 dashboard-external-http.yaml,注意修改端口,我改成8080了
4
# kubectl apply -f dashboard-external-http.yaml
5
# kubectl -n rook-ceph get svc
6
NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
7
rook-ceph-mgr                            ClusterIP   10.43.145.9     <none>        9283/TCP         2d23h
8
rook-ceph-mgr-dashboard                  ClusterIP   10.43.137.127   <none>        7000/TCP         2d17h
9
rook-ceph-mgr-dashboard-external-http    NodePort    10.43.249.105   <none>        7000:32041/TCP   2d19h

5.1 访问#

用户名 admin 密码: kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

二, 使用#

1 块存储: RBD#

1.1 创建 pool#

1
apiVersion: ceph.rook.io/v1
2
kind: CephBlockPool
3
metadata:
4
  name: ccpool
5
  namespace: rook-ceph # namespace:cluster
6
spec:
7
  enableRBDStats: true
8
  failureDomain: osd
9
  replicated:
10
    size: 3 # 三个副本,后面测试高可用需要
11
# kubectl apply -f pool.yaml
12
# kubectl get CephBlockPool -n rook-ceph
13
NAME     PHASE
14
ccpool   Ready

1.2 查看#

1
# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
2
[rook@rook-ceph-tools-556cf9bd64-2h5kr /]$ ceph df
3
--- RAW STORAGE ---
4
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
5
hdd    600 GiB  600 GiB  82 MiB    82 MiB       0.01
6
TOTAL  600 GiB  600 GiB  82 MiB    82 MiB       0.01
7

8
--- POOLS ---
9
POOL                   ID  PGS  STORED  OBJECTS    USED  %USED  MAX AVAIL
10
ccpool                  6   32    19 B        1   4 KiB      0    570 GiB

1.3 外部集群使用#

CSI 部署与快照的使用参考:

如何在 Kubernetes 里使用 cephrbd 作为存储

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: csi-rbd-secret
5
  namespace: kube-system
6
stringData:
7
  userID: admin
8
  userKey: AQA3exFjJIhtBBAAujNu5Xvc4o4DH7vY7JO9lg==
9
---
10
apiVersion: storage.k8s.io/v1
11
kind: StorageClass
12
metadata:
13
   name: csi-rbd-sc
14
provisioner: rbd.csi.ceph.com
15
parameters:
16
   clusterID: 6aa56874-3c30-4c29-abd2-5e4a1363f03c
17
   pool: ccpool
18
   imageFeatures: layering
19
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
20
   csi.storage.k8s.io/provisioner-secret-namespace: kube-system
21
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
22
   csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
23
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
24
   csi.storage.k8s.io/node-stage-secret-namespace: kube-system
25
   csi.storage.k8s.io/fstype: ext4
26
reclaimPolicy: Delete
27
volumeBindingMode: Immediate
28
allowVolumeExpansion: true
29
mountOptions:
30
   - discard

1
# cat pvc.yaml
2
apiVersion: v1
3
kind: PersistentVolumeClaim
4
metadata:
5
  name: rbd-pvc
6
spec:
7
  accessModes:
8
    - ReadWriteOnce
9
  resources:
10
    requests:
11
      storage: 1Gi
12
  storageClassName: csi-rbd-sc
13
# kubectl apply -f pvc.yaml
14
persistentvolumeclaim/rbd-pvc created
15
# kubectl get pvc
16
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
17
rbd-pvc                  Bound    pvc-5b72e56c-8f06-47dc-a00e-4a67b0c1fbd0   1Gi        RWO            csi-rbd-sc                    11s

1.4 服务端查看#

1
# rbd ls -p ccpool
2
csi-vol-d6ff6220-2cc8-11ed-a287-0a7be4a0ba4b

1.5 手动挂载测试#

1
rbd feature disable ccpool/csi-vol-d6ff6220-2cc8-11ed-a287-0a7be4a0ba4b object-map fast-diff deep-flatten

2 文件存储: CephFS#

2.1 创建 pool#

1
apiVersion: ceph.rook.io/v1
2
kind: CephFilesystem
3
metadata:
4
  name: ccfs
5
  namespace: rook-ceph
6
spec:
7
  metadataPool: # 3副本元数据池
8
    replicated:
9
      size: 3
10
  dataPools: # 3副本复制池
11
    - name: replicated
12
      replicated:
13
        size: 3
14
  preserveFilesystemOnDelete: true
15
  metadataServer:
16
    activeCount: 1
17
    activeStandby: true

2.2 查看#

1
# kubectl get CephFilesystem -n rook-ceph
2
NAME   ACTIVEMDS   AGE   PHASE
3
ccfs   1           62s   Ready
4
# ceph df
5
--- RAW STORAGE ---
6
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
7
hdd    600 GiB  600 GiB  162 MiB   162 MiB       0.03
8
TOTAL  600 GiB  600 GiB  162 MiB   162 MiB       0.03
9

10
--- POOLS ---
11
POOL                   ID  PGS  STORED  OBJECTS     USED  %USED  MAX AVAIL
12
ccfs-metadata           7   32  90 KiB       24  372 KiB      0    190 GiB
13
ccfs-replicated         8   32     0 B        0      0 B      0    190 GiB
14

15
# 查看信息
16
ceph fs ls
17
name: ccfs, metadata pool: ccfs-metadata, data pools: [ccfs-replicated ]
18
# 详细信息
19
ceph fs dump
20
e8
21
enable_multiple, ever_enabled_multiple: 1,1
22
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag i
23
s stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
24
legacy client fscid: 1
25

26
Filesystem 'ccfs' (1)
27
fs_nameccfs
28
epoch8
29
flags32 joinable allow_snaps allow_multimds_snaps allow_standby_replay
30
created2023-06-09T05:37:28.081181+0000
31
modified2023-06-09T05:37:30.036108+0000
32
tableserver0
33
root0
34
session_timeout60
35
session_autoclose300
36
max_file_size1099511627776
37
required_client_features{}
38
last_failure0
39
last_failure_osd_epoch0
40
compatcompat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored
41
 in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
42
max_mds1
43
in0
44
up{0=15684943}
45
failed
46
damaged
47
stopped
48
data_pools[5]
49
metadata_pool4
50
inline_datadisabled
51
balancer
52
standby_count_wanted1

2.3 外部集群使用#

CSI 部署参考:

如何在 Kubernetes 里使用 cephfs 作为存储

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: csi-cephfs-secret
5
  namespace: cephfs
6
stringData:
7
  adminID: admin
8
  adminKey: AQA3exFjJIhtBBAAujNu5Xvc4o4DH7vY7JO9lg==
9
---
10
apiVersion: storage.k8s.io/v1
11
kind: StorageClass
12
metadata:
13
   name: csi-cephfs-sc
14
provisioner: cephfs.csi.ceph.com
15
parameters:
16
    # ceph 集群的clusterID
17
   clusterID: 6aa56874-3c30-4c29-abd2-5e4a1363f03c
18
    # ceph 用来对接K8S的pool
19
   pool: ccfs-replicated
20
    # ceph 对接K8S的文件系统
21
   fsName: ccfs # 上面的name
22
   csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
23
   csi.storage.k8s.io/provisioner-secret-namespace: cephfs
24
   csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
25
   csi.storage.k8s.io/controller-expand-secret-namespace: cephfs
26
   csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
27
   csi.storage.k8s.io/node-stage-secret-namespace: cephfs
28
reclaimPolicy: Delete
29
mountOptions:
30
   - discard

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: csi-cephfs-pvc
5
spec:
6
  accessModes:
7
    - ReadWriteOnce
8
  resources:
9
    requests:
10
      storage: 1M
11
  storageClassName: csi-cephfs-sc

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: csi-cephfs-pvc
5
spec:
6
  accessModes:
7
    - ReadWriteOnce
8
  resources:
9
    requests:
10
      storage: 1M
11
  storageClassName: csi-cephfs-sc
12

13
# kubectl get pvc
14
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
15
csi-cephfs-pvc   Bound    pvc-4c5eebcf-a9d6-4ea8-8251-ea0f41b8aad3   1Mi        RWO            csi-cephfs-sc   1s

2.4 服务端查看#

2.5 手动挂载测试#

1
# mount -t ceph 10.1.1.1:6789,10.1.1.2:6789,10.1.1.3:6789:/volumes/csi -o name=admin,secret=xxxxxx9iMd3rnzJQHaMLkw== /tmp/test
2
# cd /tmp/test2/
3
# ls
4
csi-vol-0715c282-2ce5-11ed-97df-ea43b6d16965

3 对象存储: S3#

注意节点标签

3.1 创建 zone#

1
apiVersion: ceph.rook.io/v1
2
kind: CephObjectRealm
3
metadata:
4
  name: realm-cc
5
  namespace: rook-ceph # namespace:cluster
6
---
7
apiVersion: ceph.rook.io/v1
8
kind: CephObjectZoneGroup
9
metadata:
10
  name: zonegroup-cc
11
  namespace: rook-ceph # namespace:cluster
12
spec:
13
  realm: realm-cc
14
---
15
apiVersion: ceph.rook.io/v1
16
kind: CephObjectZone
17
metadata:
18
  name: zone-cc
19
  namespace: rook-ceph # namespace:cluster
20
spec:
21
  zoneGroup: zonegroup-cc
22
  metadataPool:
23
    failureDomain: host
24
    replicated:
25
      size: 3
26
      requireSafeReplicaSize: true
27
  dataPool:
28
    failureDomain: host
29
    replicated:
30
      size: 3
31
      requireSafeReplicaSize: true
32
    parameters:
33
      compression_mode: nones
34
      compression_mode: none

3.2 创建 pool#

1
apiVersion: ceph.rook.io/v1
2
kind: CephObjectStore
3
metadata:
4
  name: ccs3-store
5
  namespace: rook-ceph
6
spec:
7
  metadataPool:
8
    failureDomain: host
9
    replicated:
10
      size: 3
11
  dataPool:
12
    failureDomain: host
13
    erasureCoded:
14
      dataChunks: 2
15
      codingChunks: 1
16
  preservePoolsOnDelete: true
17
  gateway:
18
    port: 888 # 暴露端口
19
    hostNetwork: true # 主机网络
20
    instances: 1 # 节点数
21
    annotations:
22
    placement: # 亲和性
23
      nodeAffinity:
24
        requiredDuringSchedulingIgnoredDuringExecution:
25
          nodeSelectorTerms:
26
          - matchExpressions:
27
            - key: role
28
              operator: In
29
              values:
30
              - rgw-node
31
    #  tolerations:
32
    #  - key: rgw-node
33
    #    operator: Exists
34
    #  podAffinity:
35
    #  podAntiAffinity:
36
    #  topologySpreadConstraints:
37
    resources:
38
      limits:
39
        cpu: "500m"
40
        memory: "1024Mi"
41
      requests:
42
        cpu: "500m"
43
        memory: "1024Mi"
44
  zone:
45
    name: zone-cc

3.3 账户#

1
apiVersion: ceph.rook.io/v1
2
kind: CephObjectStoreUser
3
metadata:
4
  name: cc-user # 用户名称
5
  namespace: rook-ceph
6
spec:
7
  store: ccs3-store # 与对象库 CRD 的名称匹配
8
  displayName: cc-display-name # 显示名称
9
  quotas:
10
    maxBuckets: 100 # 最大存储桶限制
11
    maxSize: 10G # 大小
12
    maxObjects: 10000 # 桶中的最大对象数
13
  capabilities:
14
    user: "*"
15
    bucket: "*"

3.4 查看#

1
# kubectl get CephObjectZone -A
2
NAMESPACE   NAME      PHASE
3
rook-ceph   zone-cc   Ready
4
# kubectl get CephObjectStore -A
5
NAMESPACE   NAME         PHASE
6
rook-ceph   ccs3-store   Connected
7
# kubectl get cephobjectstoreuser -A
8
NAMESPACE   NAME      PHASE
9
rook-ceph   cc-user   Ready
10
# kubectl -n rook-ceph get secret |grep ccs3-store-cc-user
11
# 如果没有说明上面账户有问题
12
rook-ceph-object-user-ccs3-store-cc-user   kubernetes.io/rook   3      4m55s
13
# 上部查到的结果
14
# AccessKey
15
kubectl -n rook-ceph get secret rook-ceph-object-user-ccs3-store-cc-user -o jsonpath='{.data.AccessKey}' | base64 --decode
16
# SecretKey
17
kubectl -n rook-ceph get secret rook-ceph-object-user-ccs3-store-cc-user -o jsonpath='{.data.SecretKey}' | base64 --decode

3.5 外部集群使用#

CSI 部署参考:

如何在 Kubernetes 里使用 Minio 作为存储

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  namespace: minio
5
  name: csi-s3-secret
6
stringData:
7
  accessKeyID: "CXJ2R8NNGQE6A2Y9BQXB"
8
  secretAccessKey: "oPcVD6XOVtBmhy6WxeAEZAwlRdt44Pv1C7YpyBDL"
9
  endpoint: http://10.1.1.1:888 # 因为用的主机网络,节点加端口就行

3.5.1 查看#

1
kubectl get pvc
2
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
3
csi-s3-pvc       Bound    pvc-e2c83416-0c15-4a6f-88ea-8352f4ab77fc   5Gi        RWO            csi-s3          4m1s

服务端已经有以 pv 名字的 bucket,也就是说这个账户只能创建 100 个 pvc

3.6 手动创建测试#

3.6.1 创建 backup#

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
   name: rook-ceph-bucket
5
provisioner: rook-ceph.ceph.rook.io/bucket
6
reclaimPolicy: Delete
7
parameters:
8
  objectStoreName: ccs3-store # 与对象库 CRD 的名称匹配
9
  objectStoreNamespace: rook-ceph
10
---
11
apiVersion: objectbucket.io/v1alpha1
12
kind: ObjectBucketClaim
13
metadata:
14
  name: ceph-bucket
15
spec:
16
  generateBucketName: ceph-bkt
17
  storageClassName: rook-ceph-bucket

3.6.2 查看#

-o yaml 可以查看状态,如果 configmap 没有自动创建一般是 Pending 了

1
# kubectl get ObjectBucketClaim
2
NAME          AGE
3
cc-bucket     48s
4

5
# bucket 名字
6
kubectl -n default get cm cc-bucket -o jsonpath='{.data.BUCKET_NAME}'
7
# AccessKey
8
kubectl -n default get secret cc-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode
9
# SecretKey
10
kubectl -n default get secret cc-bucket -o jsonpath='{.data.SECRET_ACCESS_KEY}' | base64 --decode

3.6.3 访问测试#

4 NFS#

注意节点标签

4.1 创建 pool#

1
apiVersion: ceph.rook.io/v1
2
kind: CephNFS
3
metadata:
4
  name: cc-nfs
5
  namespace: rook-ceph
6
spec:
7
  server:
8
    active: 1 # 多个节点并不高可用
9
    placement: # 亲和性
10
      nodeAffinity:
11
        requiredDuringSchedulingIgnoredDuringExecution:
12
          nodeSelectorTerms:
13
          - matchExpressions:
14
            - key: role
15
              operator: In
16
              values:
17
              - nfs-node
18
      topologySpreadConstraints:
19
      tolerations:
20
      - key: nfs-node
21
        operator: Exists
22
      podAffinity:
23
      podAntiAffinity:
24
    annotations:
25
      cc-annotation: something
26
    labels:
27
      cc-label: something
28
    resources:
29
      limits:
30
        cpu: "2"
31
        memory: "4Gi"
32
      requests:
33
        cpu: "2"
34
        memory: "4Gi"
35
    priorityClassName: ""
36
    logLevel: NIV_INFO # 日志级别
37
---
38
apiVersion: ceph.rook.io/v1
39
kind: CephBlockPool
40
metadata:
41
  name: cc-nfs
42
  namespace: rook-ceph # namespace:cluster
43
spec:
44
  name: .nfs
45
  failureDomain: osd
46
  replicated:
47
    size: 1
48
    requireSafeReplicaSize: false

4.2 对外暴露#

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: rook-ceph-nfs-my-nfs-load-balancer
5
  namespace: rook-ceph
6
spec:
7
  ports:
8
    - name: nfs
9
      port: 2049
10
  externalIPs:
11
  - 10.1.1.2 # 节点ip
12
  type: ClusterIP
13
  selector:
14
    app: rook-ceph-nfs
15
    ceph_nfs: cc-nfs
16
    instance: a

4.3 启用 Ceph 协调程序#

1
ceph mgr module enable rook
2
ceph mgr module enable nfs
3
ceph orch set backend rook

4.4 手动创建目录#

1
# ceph nfs export create cephfs cc-nfs /test ccfs
2
{
3
    "bind": "/test",
4
    "fs": "ccfs",
5
    "path": "/",
6
    "cluster": "cc-nfs",
7
    "mode": "RW"
8
}
9
# ceph nfs export ls cc-nfs
10
[
11
  "/test"
12
]

4.4.1 关闭 Ceph 协调程序#

1
ceph orch set backend ""
2
ceph mgr module disable rook

4.5 查看#

仅支持通过 TCP 的 V4 版本 NFS 协议

1
ceph nfs export info cc-nfs /test
2
{
3
  "export_id": 1,
4
  "path": "/",
5
  "cluster_id": "cc-nfs",
6
  "pseudo": "/test",
7
  "access_type": "RW",
8
  "squash": "none",
9
  "security_label": true,
10
  "protocols": [
11
    4
12
  ],
13
  "transports": [
14
    "TCP"
15
  ],
16
  "fsal": {
17
    "name": "CEPH",
18
    "user_id": "nfs.cc-nfs.1",
19
    "fs_name": "ccfs"
20
  },
21
  "clients": []
22
}

4.6 外部集群使用#

CSI 部署参考:

如何在 Kubernetes 里使用 NFS 作为存储

1
......
2
    spec:
3
      serviceAccountName: nfs-client-provisioner
4
      containers:
5
        - name: nfs-client-provisioner
6
          image: willdockerhub/nfs-subdir-external-provisioner:v4.0.2
7
          volumeMounts:
8
            - name: nfs-client-root
9
              mountPath: /persistentvolumes
10
          env:
11
            - name: PROVISIONER_NAME
12
              value: k8s-sigs.io/nfs-subdir-external-provisioner
13
            - name: NFS_SERVER
14
              value: 10.1.148.142 # 刚才对外暴露地址
15
            - name: NFS_PATH
16
              value: /test
17
      volumes:
18
        - name: nfs-client-root
19
          nfs:
20
            server: 10.1.148.142 # 刚才对外暴露地址
21
            path: /test # 刚才创建的路径

4.6.1 查看#

4.6.1.1 pvc#

1
kubectl get pvc nfs-pvc
2
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
3
nfs-pvc   Bound    pvc-1d2b8446-974e-4fff-9e85-387fecd7c564   1Mi        RWX            nfs-client     5s

4.6.1.2 服务端#

1
# cd /tmp/test2/
2
# ll
3
total 1
4
drwxrwxrwx 2 nobody nobody   0 Sep  6 13:22 default-nfs-pvc-pvc-1d2b8446-974e-4fff-9e85-387fecd7c564
5
drwxr-xr-x 4 nobody nobody 158 Sep  5 14:36 volumes

4.7 手动挂载测试#

1
# mount -t nfs 10.1.148.142:/test /tmp/test/
2
# df -h |grep 142
3
10.1.148.142:/test     190G     0  190G   0% /tmp/test

三, 总结#

此文章简单演示了通过 rook 部署 ceph 集群与管理
并且如何使用块存储,文件存储,对象存储,NFS
以及外部集群使用此集群存储

1 整体查看下 pod 挂载#

1
# kubectl get pod
2
NAME         READY   STATUS    RESTARTS   AGE
3
cephfs-pod   1/1     Running   0          22h
4
nfs-pod      1/1     Running   0          3m6s
5
rbd-pod      1/1     Running   0          23h
6
s3-pod       1/1     Running   0          12m

RookCeph安装

一, 安装#

1 下载代码#

2 修改配置文件#

2.1 operator.yaml#

2.2 cluster.yaml#

3 部署#

3.1 rook-operator#

3.2 监控#

3.3 部署 Prometheus#

3.3.1 配置 Grafana#

3.3.1.1 安装插件#

3.3.1.2 配置 Grafana#

3.3.1.3 配置数据源#

3.3.2 Dashboard 集成 Grafana#

3.3.3 查看#

3.4 CephCluster#

3.5 查看详情#

4 工具#

5 dashboard#

5.1 访问#

二, 使用#

1 块存储: RBD#

1.1 创建 pool#

1.2 查看#

1.3 外部集群使用#

1.4 服务端查看#

1.5 手动挂载测试#

2 文件存储: CephFS#

2.1 创建 pool#

2.2 查看#

2.3 外部集群使用#

2.4 服务端查看#

2.5 手动挂载测试#

3 对象存储: S3#

3.1 创建 zone#

3.2 创建 pool#

3.3 账户#

3.4 查看#

3.5 外部集群使用#

3.5.1 查看#

3.6 手动创建测试#

3.6.1 创建 backup#

3.6.2 查看#

3.6.3 访问测试#

4 NFS#

4.1 创建 pool#

4.2 对外暴露#

4.3 启用 Ceph 协调程序#

4.4 手动创建目录#

4.4.1 关闭 Ceph 协调程序#

4.5 查看#

4.6 外部集群使用#

4.6.1 查看#

4.6.1.1 pvc#

4.6.1.2 服务端#

4.7 手动挂载测试#

三, 总结#

1 整体查看下 pod 挂载#

文章分享

评论区

音乐

文章目录