A set of zabbix UserParameter scripts and production ready monitoring templates for linux systems.

• this project ist originated on: https://github.com/scoopex/zabbix-agent-extensions
• the ci is implmented with github actions
• docker images of a zabbix agent can be obtained at https://hub.docker.com/repository/docker/scoopex666/zabbix-agent-with-agent-extensions

This package provides the following capabilities:

• linux standard monitoring using the standard items provided by the zabbix-agent
  • monitor memory behavior
  • monitor important servcies: smtp, ssh, cron
  • swap usage
  • 5min system load
  • monitor dmesg for bad behavior of the system
  • monitor the maximum and minimum of processes
  • automatic discovery
    • filesystems: inode and space measures
      (the amount of discovered filesystems can be limited by a configuration file on the monitored host)
    • network interfaces: packets and transferrates per second
    • storage devices: operations per second (the amount of discovered devices can be limited by a configuration file on the monitored host)
    • network interfaces: packets and transferrates per second
  • number of processes
• monitor ICMP ping
• monitor the MTA mailqueue
• monitor NFS operations/retransmits
• apache monitoring: (enable /server-status for localhost clients, https://httpd.apache.org/docs/2.4/mod/mod_status.html)
  • Check Apache Server-Status (Readers, Writers - alert if to many slots are in use)
  • Loadbalancer check
  • Monitor Mod JK backend status
• elasticsearch node and cluster monitoring (needs elasticsearch and es_stats_zabbix Python modules)
• redis Monitoring (needs redis Python module)
• NGINX Monitoring (enable /basic_status only for localhost clients, https://nginx.org/en/docs/http/ngx_http_stub_status_module.html)
• generic discovery (put json snippets to /var/run/zabbix-generic-discovery/ITEMNAME.json i.e. with puppet and get a combined discovery value)
• monitor puppet state (execution statistics, disabled state, ...)
• linux software raid state
• check quality of ntp sync behavior
• check for required reboot
• monitor open file descriptors
• Configurable autodiscovery of block devices
• Monitor performance havior of discovered block devices
• Configurable autodisovery of real disk devices
• Check smart state of disks and gather statistics
• Discover network devices and monitor performance and error behavior
• Zabbix agent version
• Monitor springboot servers
  • beta, monitoring is still rudimentary
  • contributions are very welcome

Almost all measures are integrated in graphs which are displayed on the various host screens.

The project inlcudes various zabbix templates for relase 5.2.

Note: There are also 2.2 and 3.0 templates, but they are outdated because we do not support them anymore. Probably these outdated templates might be a good starting point for your own environment.

A quick is provided by the following files:

• custom-os-linux
• custom-os-linux-hardware
• custom-os-puppet
• custom-service-apache
• custom-service-cups
• custom-service-elasticsearch-cluster
• custom-service-elasticsearch-node
• custom-service-nginx
• custom-service-nginx-logstats
• custom-service-redis
• custom-service-varnish
• custom-service-springboot
• custom-os-kubernetes-node

For development, debugging and contribution, please have a look for the development document.

• Install the zabbix-agent according to the description at zabbix download page
• Download a package suitable for your distribution using the releases section
  (or build your own package as described in development document)
• Configure the most important configuration values for your zabbix agent
  • "Server=..." and/or "ServerActive=..."
  • "Hostname=" - your hostname
• Install the distribution package of the zabbix-agent-extensions using the package management utils or a configuration management systems

  rpm -Uvh zabbix-agent-extensions-<release number>.noarch.rpm
  dpkg -i zabbix-agent-extensions-<release number>.noarch.rpm
  

  This adds a include statement for /usr/share/zabbix-agent-extensions/include.d and restart the zabbix agent automatically.

This procedure deploy zabbix agents on all worker nodes of your kubernetes cluster and provides autodiscovery for all nodes

• Make zabbix available to the deployed agent, i.e. by deploying a zabbix proxy in k8s
• Import template custom-os-kubernetes-node.xml
• Define a automatic discovery:
  • Configuration -> Actions -> "Autoregistration actions"
  • Add conditions:
    • "Host metadata contains kubernetes-node"
    • "Proxy equals infra-app-dev"
  • Add operations:
    • "Add host"
    • "Add to host groups: [DEVELOPMENT]"
    • "Link to templates: Custom - OS - Kubernetes Node"
• Configure deployment
  • Download kubernetes yaml file
  • Adapt the file to your own needs
  • Configure ZBX_ACTIVESERVER
  • Configure ZBX_PASSIVESERVERS
  • Configure the version of the image check dockerhub
  • Apply deployment

    kubectl apply -f zabbix-agent-daemonset-kubernetes.yaml
    kubectl describe -n infra-zabbix-agent daemonsets.apps zabbix-agent
    kubectl logs -n infra-zabbix-agent zabbix-agent-8n4ss -f
    

• Open Zabbix web frontend
• Create a host
  • Configuration -> Hosts -> Create host
  • Add the name ds "Host name" efined in the first step of this section
  • Define a agent interface with the correct fully qualified hostname and the ip address
• Open "Configuration" => "Templates"
  • Press button "Import"
  • Activate Linux template
    • Load "zabbix_templates/5.2/Custom - OS - Linux.xml" (also works for higher versions)
    • Open template "Custom - OS - Linux" and modify the default values defined in the macros (if neccessary)
      • Devices
        • {$DISK_HIGH_READ_IOPS_LIMIT} : alert limit for read iops/sec
        • {$DISK_HIGH_WRITE_IOPS_LIMIT} : alert limit for read iops/sec
        • {$DISK_IOPS_LIMIT_MEASURES} : how many measures need to be above the configured limits before alerting
      • Filesystems
        • {$DISK_USAGE_ABOVE_1TB_MINFREE_GBYTES} : how many gigagbytes should be free for filesystem above 1TB
        • {$DISK_USAGE_PERCENT_ALARM} : send alarm for filesystems below 1 TB on prcentage
        • {$DISK_USAGE_PERCENT_WARN} : send warning for filesystems below 1 TB on prcentage
      • {$MAXIMUM_NUMBER_RETRANSMISSIONS} : how many nfs retransmissions are ok on every measurement cycle
      • NTP
        • {$MAX_NTP_OFFSET_MS} : the maximum offset limit in milliseconds
        • {$MIN_NTP_SERVER_COUNT} : how many good ntp sources should be avaiilable, it is istrongly recommended to change the default to 2
      • {$MONITOR_LOAD_WARNING_MULT} : a multiplicator with the number of cpus for load monitoring
      • {$MONITOR_TIMEOUT} : amount of time to complain if hosts does not provide values anymore
    • Assign template "Custom - OS - Linux" to the desired hosts and modify the default values to host specific settings
  • Activate Apache template
    • Load "zabbix_templates/*/Custom - Service - Apache.xml"
    • Open template "Custom - Service - Apache" and modify the default values defined in the macros
    • Assign template "Custom - Service - Apache" to the desired hosts and modify the default values to host specific settings
  • Activate and configure other templates
• Check the function
  • Have a look at Monitoring -> Hosts -> Latest data
    • Check: "Show items without data"
    • Check: "Show details"
    • Press "Apply"
  • Check dashboards at Monitoring -> Hosts -> Dashboards

• Configure disk device discovery
  • Create files: (if the file does not exist, the default is used)
    • /etc/zabbix/item_zabbix_device_discovery.json
      
    • /etc/zabbix/item_zabbix_discovery_filesystems.json
      
  • Add content to include/exclude devices
    (what it does: include all devices and hardware models, after that filter out all devices and models which match to one of the python regexes)

    /usr/bin/zabbix_discovery_devices --help
    /usr/bin/zabbix_discovery_filesystems --help
    

  • Test

    zabbix_agentd -t "vfs.dev.discovery"
    /usr/bin/zabbix_discovery_devices --config /etc/zabbix/item_zabbix_discovery_devices.json --debug
    
    zabbix_agentd -t "vfs.fs.discovery"
    /usr/bin/zabbix_discovery_filesystems --config /etc/zabbix/item_zabbix_discovery_filesystems.json --debug
    

• Configure generic discovery
  • Decide to use a appname consiting of the following characters: a-zA-Z0-9. (Alphanumeric and dot characters)
  • Add file snippes to /var/run/zabbix-generic-discovery/ which look like -.json

    {
       "{#FOOOO}":"fooservice",
       "{#BAR}":"footype"
    },
    

  • Test the disovery by

    zabbix_agentd -t "generic.discovery[appname]"
    

Additional authors are very welcome - just submit your patches as pull requests.

• Marc Schoechlin ms@256bit.org
• Marc Schoechlin marc.schoechlin@vico-research.com (not active anymore)
• Marc Schoechlin marc.schoechlin@breuninger.de (not active anymore)

This software is licensed by GPLv2 - review file "LICENSE"