{"id":36164,"date":"2022-05-17T14:56:41","date_gmt":"2022-05-17T11:56:41","guid":{"rendered":"https:\/\/orbitsoft.com\/blog\/?p=36164"},"modified":"2023-06-26T13:07:57","modified_gmt":"2023-06-26T10:07:57","slug":"zabbix-and-prometheus","status":"publish","type":"post","link":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/","title":{"rendered":"Tasks we solve with Zabbix and Prometheus"},"content":{"rendered":"
\n
\n

\n In brief <\/h3>\n
    \n \n
  • \n \n 01<\/span>\n
    \n
    \n Project<\/span>\n <\/div>\n

    POSiFLORA Sales Accounting System<\/p> <\/div>\n <\/li>\n \n

  • \n \n 02<\/span>\n
    \n
    \n Tasks<\/span>\n <\/div>\n
      \n
    • Monitor system performance, uncover errors<\/li>\n
    • Notify specialists on Telegram regarding errors<\/li>\n
    • Ensure system stability during peak usage<\/li>\n<\/ul> <\/div>\n <\/li>\n \n
    • \n \n 03<\/span>\n
      \n
      \n Tools<\/span>\n <\/div>\n
        \n
      • Zabbix monitoring system<\/li>\n
      • Monitoring based on Prometheus and Grafana<\/li>\n<\/ul> <\/div>\n <\/li>\n \n
      • \n \n 04<\/span>\n
        \n
        \n Results<\/span>\n <\/div>\n
          \n
        • Monitoring system successfully implemented: operation of the system is now constantly monitored<\/li>\n
        • When errors are found, the system immediately reports via Telegram<\/li>\n
        • POSiFLORA has 99.95% uptime, with possible emergency shutdown time of less than one hour a year<\/li>\n<\/ul> <\/div>\n <\/li>\n <\/ul>\n <\/div>\n <\/div><\/div>\n\n

          Project: POSiFLORA CRM system for flower business<\/h3><\/div>\n\n\n

          POSiFLORA CRM system helps the flower business keep work under control: selling bouquets, considering the balance and hours of work of employees, and planning purchases. All processes are accessed from one interface. It\u2019s exceedingly convenient to work with POSiFLORA.<\/p>\n\n\n\n

          More than 8,500 users already work with the system, from small studios to large flower chains with multiple locations.<\/p>\n\n\n

          Tasks: to monitor operation of the system and prevent breakdowns<\/h3><\/div>\n\n\n

          POSiFLORA consists of two parts: the client and the server. The client part is the outer shell, the interface used by executives and florists. With the help of the client part, clients give orders to the system regarding the formation of bouquets, sales, and accounting for balances. The server part is the \u201cbrains\u201d of the system, everything that\u2019s hidden under the hood, such as accounting rules, databases, etc. Florists and managers don\u2019t see the back end, but if it breaks, the system won\u2019t work.<\/strong><\/p>\n\n\n\n

          To control the operation of both the server and client parts, OrbitSoft has created a monitoring system. Simply put, its task is to monitor the smooth operation of both parts of the system, as well as prevent breakdowns. To do this, we use the monitoring systems known as Zabbix and Prometheus. Let\u2019s talk about this in more detail\u2026<\/p>\n\n\n

          What tasks does the Zabbix monitoring system solve?<\/h3><\/div>\n\n\n

          The Zabbix monitoring system is a universal tool for monitoring the work of both the server and client parts of the system. It collects statistics on the operation of all parts of the system, analyzes this data, finds errors, and sends notifications regarding any errors found.<\/p>\n\n\n\n

          The principle of operation is as follows: We create a request to check the system indicator we need, and then Zabbix collects data and builds visual graphs. Here are the tasks it solves:<\/p>\n\n\n\n

          Checking the performance of system indicators.<\/strong> We check the number of active devices, download devices, download speed, server memory usage, CPU load, traffic, and more. The system monitors health through queries, and then stores this data.<\/p>\n\n\n\n

          Visualization of system operation.<\/strong> To make it more convenient for specialists to monitor performance, the system visualizes data in the form of graphs on the dashboard. This helps to reduce the time spent on analysis. A specialist immediately sees that there is a problem, and can begin solving it right away.<\/p>\n\n\n\n

          Finding errors and dangerous situations.<\/strong> For the system to be able to find errors and track situations that are dangerous for system operation, let\u2019s say, for example, an imminent server overload, the DevOps engineer creates \u201ctriggers\u201d. That is, he or she sets the value of the indicators, and describes the conditions by which Zabbix understands that something has gone wrong.<\/p>\n\n\n\n

          Hazard Warning.<\/strong> If Zabbix finds a dangerous situation, it immediately sends a notification about it through Telegram. The DevOps engineer sees this, and fixes the issue before it manifests as a real problem.<\/p>\n\n\n

          \n
          \n \"\"\n <\/div>\n
          An example of a request from Zabbix server to a Zabbix agent about system operation.<\/figcaption>\n <\/figure><\/div>\n\n
          \n
          \n \"\"\n <\/div>\n
          An example of visualization of data on the operation of the system in the form of graphs<\/figcaption>\n <\/figure><\/div>\n\n
          \n
          \n \"\"\n <\/div>\n <\/figure><\/div>\n\n\n

          Trigger conditions description example. It fires if the average 32-core CPU usage exceeds 100% over 5 minutes.<\/p>\n\n\n

          \n
          \n \"\"\n <\/div>\n
          Telegram notification example.<\/figcaption>\n <\/figure><\/div>\n\n
          \n
          \n \"\"\n <\/div>\n
          List of triggers for POSiFLORA<\/figcaption>\n <\/figure><\/div>\n\n

          What tasks does the Prometheus monitoring system solve?<\/h3><\/div>\n\n\n

          During the holidays, the number of sales increases, and the load on the server and client parts of the POSiFLORA system also increase. The Zabbix monitoring system can barely cope with the load. Because of this, the risk that the servers will become overloaded and the system will stop working increases.<\/p>\n\n\n\n

          To offload the monitoring system, we rebuilt it on Prometheus. Prometheus itself is just a database that stores time-based information. To get a full-fledged monitoring system, it\u2019s supplemented by exporters. Exporters are Prometheus agents that are on the target server. They collect the necessary metrics, translate them into a language understandable to the database, and send them by request. In our case, Node Exporter and MySQL Exporter send data approximately every 10 seconds:<\/p>\n\n\n\n

            \n
          • Node Exporter collects system metrics: data on RAM usage, CPU load, and hundreds of other indicators<\/li>\n\n\n\n
          • MySQL Exporter collects MySQL database server metrics<\/li>\n<\/ul>\n\n\n\n

            Exporter metrics are collected in the Prometheus database. To make it convenient to work with, Grafana, a visualization tool, is connected to it. It builds informative graphs, and sends error notifications by Telegram. <\/p>\n\n\n

            \n
            \n \"\"\n <\/div>\n
            This is what graphs built using MySQL Exporter metrics look like.<\/figcaption>\n <\/figure><\/div>\n\n\n

            Detection of problems in the operation of the Zabbix monitoring system. <\/strong>When Prometheus took over server monitoring, the number of network connection dropped significantly, and the risk of congestion was eliminated.<\/p>\n\n\n\n

            We found that the problem of extra connections of the Zabbix agent was related to its reaction to restarting Docker containers. Containers are safe \u201cpackaging\u201d for program code. Using them, the code is loaded onto the server and executed without conflicting with other system components. POSiFLORA is built using separate microservices, each of which runs in a separate container.<\/p>\n\n\n\n

            In our case, the containers are not just restarted, but re-created. Each time Zabbix recognized one as a new monitoring object, it added it to the check list, which created a lot of unnecessary network connections. Because of this, the overload increased.<\/p>\n\n\n\n

            Load distribution between monitoring systems.<\/strong> First, we reduced the container data storage period to one day, and figured out how to restart them, so the load on Zabbix decreased significantly.<\/p>\n\n\n\n

            However, we liked how the Prometheus-based system performed in server monitoring. For example, regarding containers, Prometheus did better because, unlike Zabbix, it receives and sends all metrics in one connection. Zabbix is \u200b\u200bgood at monitoring various web services such as POSiFLORA.<\/p>\n\n\n\n

            We decided to split the monitoring between them:<\/p>\n\n\n\n

              \n
            1. The state of the server, including containers, is monitored by Prometheus in conjunction with Grafana, and the Node Exporter and MySQL Exporter.<\/li>\n\n\n\n
            2. The work of POSiFLORA components continues to be monitored by Zabbix.<\/li>\n<\/ol>\n\n\n\n

              Results: Distributed the load and ensured smooth operation of the system.<\/strong><\/p>\n\n\n

              \n
              \n

              We\u00a0release projects for online and offline business<\/h2>\n
              \n

              We\u00a0work with businesses of\u00a0any scale.<\/strong>\u00a0Our clients include international companies, large and medium-sized businesses, and startups.<\/p>\n

              Telecom<\/strong>: cellular operators, Internet service providers.<\/p>\n

              Finance<\/strong>: electronic payment systems, insurance companies and brokers.<\/p>\n

              Retail<\/strong>: stores, trade networks.<\/p>\n

              Internet advertising<\/strong>: advertising agencies, advertising networks.<\/p>\n

              Medicine<\/strong>: clinics, medical services.<\/p> <\/div>\n

              \n Discuss or order development <\/div>\n <\/div>\n
              \n \"\"\n <\/div>\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"

              POSiFLORA CRM system helps the flower business keep work under control: selling bouquets, considering the balance and hours of work of employees, and planning purchases. All processes are accessed from one interface. It\u2019s exceedingly convenient to work with POSiFLORA. More than 8,500 users already work with the system, from small studios to large flower chains […]<\/p>\n","protected":false},"author":1,"featured_media":36165,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[196],"yoast_head":"\nTasks we solve with Zabbix and Prometheus - OrbitSoft Blog<\/title>\n<meta name=\"description\" content=\"POSiFLORA has 99.95% uptime due to Zabbix and Prometheus monitor system performance and notify specialists regarding errors\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Tasks we solve with Zabbix and Prometheus - OrbitSoft Blog\" \/>\n<meta name=\"twitter:description\" content=\"POSiFLORA has 99.95% uptime due to Zabbix and Prometheus monitor system performance and notify specialists regarding errors\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/orbitsoft.com\/blog\/wp-content\/uploads\/devops-\u043a\u0432\u0430\u0434\u0440\u0430\u0442-2048x2048-1.png\" \/>\n<meta name=\"twitter:creator\" content=\"@orbitsoft\" \/>\n<meta name=\"twitter:site\" content=\"@orbitsoft\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Tasks we solve with Zabbix and Prometheus - OrbitSoft Blog","description":"POSiFLORA has 99.95% uptime due to Zabbix and Prometheus monitor system performance and notify specialists regarding errors","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/","twitter_card":"summary_large_image","twitter_title":"Tasks we solve with Zabbix and Prometheus - OrbitSoft Blog","twitter_description":"POSiFLORA has 99.95% uptime due to Zabbix and Prometheus monitor system performance and notify specialists regarding errors","twitter_image":"https:\/\/orbitsoft.com\/blog\/wp-content\/uploads\/devops-\u043a\u0432\u0430\u0434\u0440\u0430\u0442-2048x2048-1.png","twitter_creator":"@orbitsoft","twitter_site":"@orbitsoft","twitter_misc":{"Written by":"admin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/","url":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/","name":"Tasks we solve with Zabbix and Prometheus - OrbitSoft Blog","isPartOf":{"@id":"https:\/\/orbitsoft.com\/blog\/#website"},"datePublished":"2022-05-17T11:56:41+00:00","dateModified":"2023-06-26T10:07:57+00:00","author":{"@id":"https:\/\/orbitsoft.com\/blog\/#\/schema\/person\/e515b3fa91e283750477594c4f028b7b"},"description":"POSiFLORA has 99.95% uptime due to Zabbix and Prometheus monitor system performance and notify specialists regarding errors","breadcrumb":{"@id":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/orbitsoft.com\/blog\/zabbix-and-prometheus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/orbitsoft.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Tasks we solve with Zabbix and Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/orbitsoft.com\/blog\/#website","url":"https:\/\/orbitsoft.com\/blog\/","name":"OrbitSoft Blog","description":"Discover the latest in news and resources for OrbitSoft","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/orbitsoft.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/orbitsoft.com\/blog\/#\/schema\/person\/e515b3fa91e283750477594c4f028b7b","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/orbitsoft.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b1b269c579caf059f82b6d114c63fc49?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b1b269c579caf059f82b6d114c63fc49?s=96&d=mm&r=g","caption":"admin"},"url":"https:\/\/orbitsoft.com\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/posts\/36164"}],"collection":[{"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/comments?post=36164"}],"version-history":[{"count":8,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/posts\/36164\/revisions"}],"predecessor-version":[{"id":36565,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/posts\/36164\/revisions\/36565"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/media\/36165"}],"wp:attachment":[{"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/media?parent=36164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/categories?post=36164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/orbitsoft.com\/blog\/wp-json\/wp\/v2\/tags?post=36164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}