After updating\/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers. Ceph reports a non-zero exit status from zabbix_sender.<\/p>\n
Check to install zabbix-server.<\/p>\n
\napt install zabbix-sender\n<\/pre>\nInfo:<\/p>\n
\n# ceph mon versions\n<\/pre>\nOutput:<\/p>\n
\n{\n "ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable)": 4\n}\n<\/pre>\nCheck Debian version<\/p>\n
# cat \/etc\/debian_version\n11.2\n<\/pre>\n\n# lsb_release -a\nNo LSB modules are available.\nDistributor ID: Debian\nDescription: Debian GNU\/Linux 11 (bullseye)\nRelease: 11\nCodename: bullseye\n<\/pre>\n\nceph -s\n<\/pre>\n\n cluster:\n id: 20a806df-b3f4-4032-ab8e-ae15aef1e836\n health: HEALTH_WARN\n Failed to send data to Zabbix\n mon px1 is low on available space\n\n services:\n mon: 3 daemons, quorum px1,px2,px3 (age 8d)\n mgr: px1(active, since 8d), standbys: px2, px3\n osd: 32 osds: 32 up (since 8d), 32 in (since 11d)\n\n data:\n pools: 7 pools, 640 pgs\n objects: 3.52M objects, 13 TiB\n usage: 53 TiB used, 124 TiB \/ 176 TiB avail\n pgs: 639 active+clean\n 1 active+clean+scrubbing+deep\n\n io:\n client: 0 B\/s rd, 23 MiB\/s wr, 0 op\/s rd, 34 op\/s wr\n\n<\/pre>\n\nceph health detail\nHEALTH_WARN Failed to send data to Zabbix; mon px1 is low on available space\nMGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix\n \/usr\/bin\/zabbix_sender exited non-zero: zabbix_sender [4088351]: DEBUG: answer []\nzabbix_sender [4088351]: Warning: incorrect answer from server []\n<\/pre>\nSetting “debug mgr = 20” yields no additional information. Zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4:<\/p>\n
# ceph zabbix config-show\n{"zabbix_port": 10050, "zabbix_host": "stats.myzabbix.host", "identifier": "px1.cephserver.host", "zabbix_sender": "\/usr\/bin\/zabbix_sender", "interval": 60}<\/pre>\nSo, how to reconfigure it.
\nSOME DETAILS:<\/p>\nZabbix plugin
\nThe Zabbix plugin actively sends information to a Zabbix server like:
\nCeph status
\nI\/O operations
\nI\/O bandwidth
\nOSD status
\nStorage utilization
\nRequirements<\/p>\nThe plugin requires that the zabbix_sender executable is present on all machines running ceph-mgr. It can be installed on most distributions using the package manager.
\nDependencies.<\/p>\nInstalling zabbix_sender can be done under Ubuntu or CentOS using either apt or dnf.<\/p>\n
On Ubuntu Xenial:<\/p>\n
\n# apt install zabbix-agent\n<\/pre>\nOn Fedora:<\/p>\n
\n# dnf install zabbix-sender\n<\/pre>\nEnabling<\/p>\n
You can enable the zabbix module with:<\/p>\n
\n# ceph mgr module enable zabbix\n<\/pre>\nConfiguration<\/p>\n
Two configuration keys are vital for the module to work:<\/p>\n
zabbix_host
\nidentifier (optional)<\/p>\nThe parameter zabbix_host controls the hostname of the Zabbix server to which zabbix_sender will send the items. This can be a IP-Address if required by your installation. The identifier parameter controls the identifier\/hostname to use as source when sending items to Zabbix. This should match the name of the Host in your Zabbix server. When the identifier parameter is not configured the ceph- of the cluster will be used when sending data to Zabbix.<\/p>\n
This would for example be ceph-c4d32a99-9e80-490f-bd3a-1d22d8a7d354<\/p>\n
Additional configuration keys which can be configured and their default values:<\/p>\n
zabbix_port: 10051
\nzabbix_sender: \/usr\/bin\/zabbix_sender
\ninterval: 60
\nConfiguration keys
\nConfiguration keys can be set on any machine with the proper cephx credentials, these are usually Monitors where the client.admin key is present.
\nceph zabbix config-set
\nFor example:<\/p>\n\n# ceph zabbix config-set zabbix_host zabbix.localdomain\n# ceph zabbix config-set identifier ceph.eu-ams02.local\n<\/pre>\nThe current configuration of the module can also be shown:<\/p>\n
\n# ceph zabbix config-show\n\n<\/pre>\nManually sending data<\/p>\n
If needed the module can be asked to send data immediately instead of waiting for the interval.<\/p>\n
This can be done with this command:<\/p>\n
# ceph zabbix send\n<\/pre>\nThe module will now send its latest data to the Zabbix server.
\nDebugging<\/p>\nShould you want to debug the Zabbix module increase the logging level for ceph-mgr and check the logs.<\/p>\n
\n[mgr]\n debug mgr = 20\n\n<\/pre>\nWith logging set to debug for the manager the plugin will print various logging lines prefixed with mgr[zabbix] for easy filtering.<\/p>\n","protected":false},"excerpt":{"rendered":"
After updating\/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers. Ceph reports a non-zero exit status from zabbix_sender. Check to install zabbix-server. apt install zabbix-sender Info: # ceph mon versions Output: { "ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable)": 4 } Check Debian version # cat \/etc\/debian_version 11.2 … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[90],"tags":[],"class_list":["post-5263","post","type-post","status-publish","format-standard","hentry","category-proxmox"],"_links":{"self":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/5263","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/comments?post=5263"}],"version-history":[{"count":0,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/5263\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/media?parent=5263"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/categories?post=5263"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/tags?post=5263"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}