Kubernetes Monitoring

Kubernetes monitoring is a form of reporting that helps with proactive management of clusters. Monitoring a Kubernetes cluster eases management of containerized infrastructure by tracking utilization of cluster resources including memory, CPU, and storage. Cluster operators can monitor and receive alerts if the desired number of pods are not running, if resource utilization is approaching critical limits, or when failures or misconfiguration cause pods or nodes to become unable to participate in the cluster

measurable metrics:

o Node resource utilization – there are many metrics in this area, all related to resource utilization. Network bandwidth, disk utilization, CPU, and memory utilization are examples of this. Using these metrics, one can find out whether or not to increase or decrease the number and size of nodes in the cluster.
o The number of nodes – the number of nodes available is an important metric to follow. This allows you to figure out what you are paying for (if you are using cloud providers), and to discover what the cluster is being used for.
o Running pods – the number of pods running will show you if the number of nodes available is sufficient and if they will be able to handle the entire workload in case a node fails.

Kubernetes Pod Monitoring

The act of monitoring a pod can be separated into three categories:
(1) Kubernetes metrics,
(2) pod container metrics, and
(3) application metrics.

Using Kubernetes metrics, we can monitor how a specific pod and its deployment are being handled by the orchestrator. The following information can be monitored: the number of instances a pod has at the moment and how many were expected (if the number is low, your cluster may be out of resources), how the on-progress deployment is going (how many instances were changed from an older version to a new one), health checks, and some network data available through network services. Container metrics are available mostly through cAdvisor and exposed by Heapster, which queries every node about the running containers. In this case, metrics like CPU, network, and memory usage compared with the maximum allowed are the highlights. Finally, there are the application specific metrics. These metrics are developed by the application itself and are related to the business rules it addresses.
For example, a database application will probably expose metrics related to an indices’ state and statistics concerning tables and relationships. An e-commerce application would expose data concerning the number of users online and how much money the software made in the last hour, for example. The metrics described in the latter type are commonly exposed directly by the application: if you want to keep closer track you should connect the application to a monitoring application.

Heapster, InfluxDB, and Grafana with Kubernetes

Prometheus and Grafana with Kubernetes Heapster and ELK Stack with Kubernetes.

Datadog

Proprietary APM solutions like Datadog aim to simplify monitoring by enabling organizations to monitor their applications and infrastructure more easily. By definition, these systems are designed so that setting up a data pipeline is as effortless as possible, a simplicity that will often come at a cost.

Dynatrace

Dynatrace is another APM solution that provides an agent that will run as a DaemonSet in your cluster. The steps to have a Dynatrace agent deployed are less straightforward since you have to collect data from more than one page until you have a proper deployment file.
Using ELK.

Elasticsearch

Elasticsearch provides real-time search and analytics for all data types, whether structured, unstructured, or numerical. It can efficiently store and index data in a way that enhances quick search and retrieval. It can also aggregate data to discover trends and patterns as the data and query volume grow. Because of its speed, Elasticsearch is a tremendous help in many instances, such as searching for a website or analyzing security events.

Logstash

Logstash collects data from multiple sources, transforms it, and then sends it to Elasticsearch for analysis. In short, it receives inputs and filters them down into helpful outputs for the other engines. Inputs primarily come from files, syslogs, and some lightweight log shippers called beats. Beats are small packages that are installed on target devices to feed information to Logstash. They may be in the form of Filebeats (logs and files), Packetbeats (network packet), Winlogbeats (Windows event logs), Metricbeats (system and service statistics), and others. Logstash receives the files, filters the data, puts the files in a supported format, and then outputs them to Elasticsearch. It can also generate output to a file, graphic, or several other formats.

Kibana

Kibana is responsible for visualizing the data stored in Elasticsearch. Kibana allows you to explore the data as well as manage and monitor the entire ELK Stack. Kibana gives shape to your data and provides the means to navigate the ELK Stack. Kibana helps you search for hidden insights, then visualize what you find in charts, gauges, maps, and more. You can then combine this information into a dashboard. Kibana also monitors the ELK Stack's health. Finally, it controls users and their level of access in the ecosystem.
New Relic monitoring New Relic is a Software as a Service offering that focuses on performance and availability monitoring. It uses a standardized Apdex (application performance index) score to set and rate application performance across the environment in a unified manner. Managing modern web applications requires attention to small details as it is often these little things that come together to adversely impact the user experience. The New Relic application monitoring tool reveals these parameters so that developers and site owners can take appropriate and timely corrective actions required to improve the performance of WordPress sites.