health: HEALTH

health: HEALTH_WARN

too few PGs per OSD 错误

ceph -scluster:id:da54ea6a-111a-434c-b78e-adad1ac66abbhealth: HEALTH_WARNtoo few PGs per OSD (10 < min 30)services:mon: 3 daemons, quorum master1,master2,master3mgr: master1(active), standbys: master2osd: 3 osds: 3 up, 3 indata:pools: 1 pools, 10 pgsobjects: 17 objects, 24 MiBusage: 3.0 GiB used, 24 GiB / 27 GiB availpgs:10 active+clean

从上面可以看到，提示说每个osd上的pg数量小于最小的数目30个。pgs为10，因为是2副本的配置，所以当有3个osd的时候，每个osd上均分了10/3 *2=6个pgs,也就是出现了如上的错误小于最小配置30个。

集群这种状态如果进行数据的存储和操作，会发现集群卡死，无法响应io，同时会导致大面积的osd down。

修改默认pool rbd的pgs

ceph osd pool set rbd pg_num 50

此时，ceph -s 查看会提示pg_num 大于 pgp_num，所以还需要修改pgp_num

ceph osd pool set rbd pgp_num 50

再次查看：

ceph -scluster:id:da54ea6a-111a-434c-b78e-adad1ac66abbhealth: HEALTH_WARNapplication not enabled on 1 pool(s)

提示 application not enabled

[root@master1 ~]# ceph health detailHEALTH_WARN application not enabled on 1 pool(s)POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)application not enabled on pool 'rbd'use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.[root@master1 ~]# ceph osd pool application enable rbd rbdenabled application 'rbd' on pool 'rbd'

ceph health detail

HEALTH_WARN application not enabled on 1 pool(s)

POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)

application not enabled on pool 'rbd'

use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

[root@master1 ~]# ceph osd pool application enable rbd rbd

enabled application 'rbd' on pool 'rbd'

根据上面的提示，选择存储方式：cephfs ，rbd ，rgw 。此处我是rbd块存储。

查看：

root@master1 ~]# ceph -scluster:id:da54ea6a-111a-434c-b78e-adad1ac66abbhealth: HEALTH_OKservices:mon: 3 daemons, quorum master1,master2,master3mgr: master1(active), standbys: master2osd: 3 osds: 3 up, 3 indata:pools: 1 pools, 50 pgsobjects: 17 objects, 24 MiBusage: 3.0 GiB used, 24 GiB / 27 GiB availpgs:50 active+clean

HEALTH_WARN application not enabled on pool '.rgw.root'

在创建rgw对象存储时，报错：

root@master1 ceph-cluster]# ceph health detailHEALTH_WARN application not enabled on 1 pool(s)POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)application not enabled on pool '.rgw.root'use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.[root@master1 ceph-cluster]# ceph osd pool application enable .rgw.root rgwenabled application 'rgw' on pool '.rgw.root'ceph -s 查看已恢复正常

too many PGs per OSD告警的处理

ceph自L版本后，mon_max_pg_per_osd,默认值也从300变更为200

修改ceph.conf文件

[root@master1 ceph-cluster]# cat ceph.conf [global]......mon_max_pg_per_osd = 1000在global下面追加一行。推送到各节点：ceph-deploy --overwrite-conf config push master1 master2 master3重启mgr和各节点monsystemctl restart ceph-mgr@master1systemctl restart ceph-mon@master1systemctl restart ceph-mon@master2systemctl restart ceph-mon@master3ceph -s 查看已恢复正常

一般ceph.conf在不修改配置文件的情况下，好的参数都使用的默认配置，此时创建rgw用户时也会报错：

rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)[root@master1 ceph-cluster]# radosgw-admin user create --uid=radosgw --display-name='radosgw'{"user_id": "radosgw","display_name": "radosgw","email": "","suspended": 0,"max_buckets": 1000,#最大值1000已生效"auid": 0,"subusers": [],......

解决办法同上，修改默认配置文件后重启即可。