如何在Kubernetes里使用Glusterfs存储

部署 Glusterfs 参考: [[glusterfs]]
Kubernetes 通过 StorageClass 访问 heketi,heketi 负责管理 Glusterfs 集群

1 heketi#

1.1 安装#

1.1.1 配置 yum 源并安装#

1
cat /etc/yum.repos.d/glusterfs.repo
2
[glusterfs]
3
name=glusterfs
4
baseurl=https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-9/
5
enabled=1
6
gpgcheck=0
7
yum install -y heketi heketi-client

1.1.2 配置文件#

这个一般只需要修改用户名与 key

1
cat /etc/heketi/heketi.json
2

3
{
4
  "_port_comment": "Heketi Server Port Number",
5
  "port": "8080", # 对外暴露端口号
6

7
  "_use_auth": "Enable JWT authorization. Please enable for deployment",
8
  "use_auth": true,
9

10
  "_jwt": "Private keys for access",
11
  "jwt": {
12
    "_admin": "Admin has access to all APIs",
13
    "admin": {
14
      "key": "admin" # admin key
15
    },
16
    "_user": "User only has access to /volumes endpoint",
17
    "user": {
18
      "key": "user"
19
    }
20
  },
21

22
  "_glusterfs_comment": "GlusterFS Configuration",
23
  "glusterfs": {
24
    "_executor_comment": [
25
      "Execute plugin. Possible choices: mock, ssh",
26
      "mock: This setting is used for testing and development.",
27
      "      It will not send commands to any node.",
28
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
29
      "      It will need the values in sshexec to be configured.",
30
      "kubernetes: Communicate with GlusterFS containers over",
31
      "            Kubernetes exec api."
32
    ],
33
    "executor": "ssh",
34

35
    "_sshexec_comment": "SSH username and private key file information",
36
    "sshexec": {
37
      "keyfile": "/etc/heketi/heketi_key",
38
      "user": "ops", # 远程用户名
39
      "port": "22", # 远程端口
40
      "sudo": true, # 是否sudo
41
      "fstab": "/etc/fstab"
42
    },
43

44
    "_db_comment": "Database file name",
45
    "db": "/var/lib/heketi/heketi.db",
46

47
    "_loglevel_comment": [
48
      "Set log level. Choices are:",
49
      "  none, critical, error, warning, info, debug",
50
      "Default is warning"
51
    ],
52
    "loglevel" : "info" # 日志级别
53
  }
54
}

1.1.3 添加免密登录#

1
cd /etc/heketi/
2
ssh-keygen -t rsa -q -f /etc/heketi/heketi_key -N ""
3
ssh-copy-id -i /etc/heketi/heketi_key.pub -p 9999 ops@10.1.1.1
4
ssh-copy-id -i /etc/heketi/heketi_key.pub -p 9999 ops@10.1.1.2
5
ssh-copy-id -i /etc/heketi/heketi_key.pub -p 9999 ops@10.1.1.3

1.1.4 启动#

1
systemctl enable heketi
2
systemctl start heketi

1.1.5 查看与测试#

1
systemctl status heketi
2
curl 10.1.1.1:8080/hello
3
Hello from Heketi

1.2 添加 Glusterfs 集群#

1.2.1 添加 heketi 专用磁盘#

不需要格式化

1
dd if=/dev/zero of=gfs-data.img bs=1M count=10240
2
losetup -d /dev/loop10
3
losetup  /dev/loop10 gfs-data.img

查看刚才创建的虚拟磁盘

1
lsblk
2
NAME                                                                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
3
loop10                                                                              7:10   0   10G  0 loop

1.2.2 配置 Glusterfs 集群#

磁盘一定要与上面查询到的一样

可以使用主机名,我为了方便直接使用 ip 地址了

1
cat /etc/heketi/topology.json
2

3
{
4
    "clusters": [
5
      {
6
        "nodes": [
7
          {
8
              "node": {
9
                "hostnames": {
10
                  "manage": [
11
                    "10.1.1.1"
12
                  ],
13
                  "storage": [
14
                    "10.1.1.1"
15
                  ]
16
                },
17
                "zone": 1
18
              },
19
              "devices": [
20
                "/dev/loop10"
21
              ]
22
            },
23
          {
24
            "node": {
25
              "hostnames": {
26
                "manage": [
27
                  "10.1.1.2"
28
                ],
29
                "storage": [
30
                  "10.1.1.2"
31
                ]
32
              },
33
              "zone": 1
34
            },
35
            "devices": [
36
              "/dev/loop10"
37
            ]
38
          },
39
          {
40
            "node": {
41
              "hostnames": {
42
                "manage": [
43
                  "10.1.1.3"
44
                ],
45
                "storage": [
46
                  "10.1.1.3"
47
                ]
48
              },
49
              "zone": 1
50
            },
51
            "devices": [
52
              "/dev/loop10"
53
            ]
54
          },
55
          {
56
            "node": {
57
              "hostnames": {
58
                "manage": [
59
                  "10.1.1.4"
60
                ],
61
                "storage": [
62
                  "10.1.1.4"
63
                ]
64
              },
65
              "zone": 1
66
            },
67
            "devices": [
68
              "/dev/loop10"
69
            ]
70
          }
71
        ]
72
      }
73
    ]
74
  }

1.2.3 连接集群#

1
heketi-cli --user admin --secret admin topology load --json=/etc/heketi/topology.json
2
        Found node 10.1.1.1 on cluster bd307c566431464e6d91578ff88724f8
3
                Adding device /dev/loop10 ... OK
4
        Found node 10.1.1.2 on cluster bd307c566431464e6d91578ff88724f8
5
                Adding device /dev/loop10 ... OK
6
        Found node 10.1.1.3 on cluster bd307c566431464e6d91578ff88724f8
7
                Adding device /dev/loop10 ... OK
8
        Found node 10.1.1.4 on cluster bd307c566431464e6d91578ff88724f8
9
                Adding device /dev/loop10 ... OK

1.2.4 创建测试存储卷#

1
heketi-cli --user admin --secret admin volume create --size=5
2

3
Name: vol_399d09a026f7e9faa4c112fa01c01f97
4
Size: 3
5
Volume Id: 399d09a026f7e9faa4c112fa01c01f97
6
Cluster Id: bd307c566431464e6d91578ff88724f8
7
Mount: 10.1.1.1:vol_399d09a026f7e9faa4c112fa01c01f97
8
Mount Options: backup-volfile-servers=10.1.1.2,10.1.1.3,10.1.1.4
9
Block: false
10
Free Size: 0
11
Reserved Size: 0
12
Block Hosting Restriction: (none)
13
Block Volumes: []
14
Durability Type: replicate
15
Distribute Count: 1
16
Replica Count: 3

1.2.5 查看#

1.2.5.1 glusterfs#

1
gluster volume info
2

3
Volume Name: vol_399d09a026f7e9faa4c112fa01c01f97
4
Type: Replicate
5
Volume ID: 26c8ae49-00e4-4f1b-9388-e2e3ec5eabf9
6
Status: Started
7
Snapshot Count: 0
8
Number of Bricks: 1 x 3 = 3
9
Transport-type: tcp
10
Bricks:
11
Brick1: 10.1.1.3:/var/lib/heketi/mounts/vg_61f345a18c1093cac5abfc6a373a8716/brick_0826c144799419a265dae9759214bfa2/brick
12
Brick2: 10.1.1.1:/var/lib/heketi/mounts/vg_e5b0b51220b519ed7bfb553ac1073a1f/brick_163d78579056f4e38ca6e8a0f8467f03/brick
13
Brick3: 10.1.1.4:/var/lib/heketi/mounts/vg_034008384665cab8fb2fdb1477893424/brick_7266a0b5b52f60e8c363a2e17f3a6915/brick
14
Options Reconfigured:
15
user.heketi.id: 399d09a026f7e9faa4c112fa01c01f97
16
cluster.granular-entry-heal: on
17
storage.fips-mode-rchecksum: on
18
transport.address-family: inet
19
nfs.disable: on
20
performance.client-io-threads: off

能看到说明以前正常,如果看不到看看日志,应该是免密没有做好

1.2.6 删除#

1
heketi-cli --user admin --secret admin volume delete ${Volume Id}

2 Kubernetes#

2.1 安装 glusterfs 客户端#

所有节点最好都执行

yum install glusterfs-fuse -y

2.2 StorageClass#

2.2.1 查看 heketi 的 Cluster Id#

1
heketi-cli --user admin --secret admin topology info
2
Cluster Id: b122b2fa65f1e02648d0bc434deefc5e
3

4
    File:  true
5
    Block: true
6

7
    Volumes:  ...........................

2.2.2 StorageClass 配置文件#

1
cat gfs.yaml
2
apiVersion: storage.k8s.io/v1
3
kind: StorageClass
4
metadata:
5
  name: gluster-heketi-storageclass
6
provisioner: kubernetes.io/glusterfs
7
parameters:
8
  resturl: "http://10.1.1.1:8080" # heketi服务端地址
9
  clusterid: "b122b2fa65f1e02648d0bc434deefc5e"  # 刚才查到的Cluster Id
10
  restauthenabled: "true"
11
  restuser: "admin"
12
  secretNamespace: "gfs"
13
  secretName: "heketi-secret"
14
  gidMin: "40000"
15
  gidMax: "50000"
16
  volumetype: "replicate:2"
17
---
18
apiVersion: v1
19
kind: Secret
20
metadata:
21
  name: heketi-secret
22
data:
23
  key: YWRtaW4= # heketi.json里配置的admin key base64加密
24
type: kubernetes.io/glusterfs

添加

1
kubectl apply -f gfs.yaml  -n gfs

2.3 pvc#

1
cat pvc.yaml
2
kind: PersistentVolumeClaim
3
apiVersion: v1
4
metadata:
5
  name: gfs-pvc
6
spec:
7
  storageClassName: gluster-heketi-storageclass
8
  accessModes:
9
    - ReadWriteMany
10
  resources:
11
    requests:
12
      storage: 1G
13

14
kubectl apply -f pvc.yaml  -n gfs

2.3.1 查看#

1
kubectl get pvc -n gfs
2
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
3
gfs-pvc   Bound    pvc-e3e5cfb6-25b1-429c-8e85-2bf93a5ae61f   1Gi        RWX            gluster-heketi-storageclass   4d17h

2.4 pod#

1
cat pod.yaml
2
kind: Pod
3
apiVersion: v1
4
metadata:
5
  name: gfs-pod
6
  namespace: gfs
7
spec:
8
  containers:
9
  - name: gfs-pod
10
    image:  nginx
11
    volumeMounts:
12
      - name: pvc
13
        mountPath: "/mnt"
14
  volumes:
15
    - name: pvc
16
      persistentVolumeClaim:
17
        claimName: gfs-pvc

2.5 查看#

2.5.1 pod#

1
kubectl get pod -n gfs
2
NAME      READY   STATUS    RESTARTS   AGE
3
gfs-pod   1/1     Running   0          4d17h

2.5.2 Glusterfs#

1
gluster volume info
2

3
Volume Name: vol_3373ccb6e612ae94f39d56a8b55c5068
4
Type: Replicate
5
Volume ID: 751b7641-443f-4207-875a-4d42f13abfef
6
Status: Started
7
Snapshot Count: 0
8
Number of Bricks: 1 x 2 = 2
9
............

3 静态绑定#

3.1 glusterfs 创建 volume#

gluster volume create kube_vol 10.1.1.1:/data/gfs2_data 10.1.1.2:/data/gfs2_data 10.1.1.3:/data/gfs2_data

3.1.1 启动 volume#

gluster volume start kube_vol

3.2 配置 kubernetes 访问 glusterfs 集群#

1
 cat endpoints.yaml
2

3
apiVersion: v1
4
kind: Endpoints
5
metadata:
6
  name: glusterfs-cluster
7
subsets:
8
- addresses:
9
  - ip: 10.1.1.1
10
  ports:
11
  - port: 20
12
- addresses:
13
  - ip: 10.1.1.2
14
  ports:
15
  - port: 20
16
- addresses:
17
  - ip: 10.1.1.3
18
  ports:
19
  - port: 20
20
- addresses:
21
  - ip: 10.1.1.4
22
  ports:
23
  - port: 20
24
---
25
apiVersion: v1
26
kind: Service
27
metadata:
28
  name: glusterfs-cluster
29
spec:
30
  ports:
31
  - port: 20

3.3 创建 pv#

1
apiVersion: v1
2
kind: PersistentVolume
3
metadata:
4
  name: gfs-pv
5
spec:
6
  accessModes:
7
  - ReadWriteOnce
8
  capacity:
9
    storage: 1Gi
10
  glusterfs:
11
    endpoints: glusterfs-cluster
12
    path: kube_vol
13
    readOnly: true

3.4 创建 pvc 绑定 pv#

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: gfs-pvc2
5
spec:
6
  volumeName: gfs-pv
7
  accessModes: ["ReadWriteOnce"]
8
  resources:
9
    requests:
10
      storage: 1Gi

3.5 查看#

1
kubectl get pvc gfs-pvc2 -n gfs
2
NAME       STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
3
gfs-pvc2   Bound    gfs-pv   1Gi        RWO                           19m

4 问题#

4.1 mount: unknown filesystem type ‘glusterfs’#

安装 glusterfs 客户端

yum install glusterfs-fuse -y

4.2 添加集群初始化失败#

硬盘不需要格式化

4.3 pod 挂载不上 pvc#

查看 glusterfs 集群 volume 是否正常,可能没启动,gluster volume status

如何在Kubernetes里使用Glusterfs存储

1 heketi#

1.1 安装#

1.1.1 配置 yum 源并安装#

1.1.2 配置文件#

1.1.3 添加免密登录#

1.1.4 启动#

1.1.5 查看与测试#

1.2 添加 Glusterfs 集群#

1.2.1 添加 heketi 专用磁盘#

1.2.2 配置 Glusterfs 集群#

1.2.3 连接集群#

1.2.4 创建测试存储卷#

1.2.5 查看#

1.2.5.1 glusterfs#

1.2.6 删除#

2 Kubernetes#

2.1 安装 glusterfs 客户端#

2.2 StorageClass#

2.2.1 查看 heketi 的 Cluster Id#

2.2.2 StorageClass 配置文件#

2.3 pvc#

2.3.1 查看#

2.4 pod#

2.5 查看#

2.5.1 pod#

2.5.2 Glusterfs#

3 静态绑定#

3.1 glusterfs 创建 volume#

3.1.1 启动 volume#

3.2 配置 kubernetes 访问 glusterfs 集群#

3.3 创建 pv#

3.4 创建 pvc 绑定 pv#

3.5 查看#

4 问题#

4.1 mount: unknown filesystem type ‘glusterfs’#

4.2 添加集群初始化失败#

4.3 pod 挂载不上 pvc#

文章分享

评论区

音乐

文章目录