그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그 그

4 minute read

Monitoring

etcd-druid uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging of compaction jobs.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

The naming of metrics follows the suggested Prometheus best practices. All compaction related metrics are put under namespace etcddruid and the respective subsystems.

Snapshot Compaction

These metrics provide information about the compaction jobs that run after some interval in shoot control planes. Studying the metrics, we can deduce how many compaction job ran successfully, how many failed, how many delta events compacted etc.

Name	Description	Type
etcddruid_compaction_jobs_total	Total number of compaction jobs initiated by compaction controller.	Counter
etcddruid_compaction_jobs_current	Number of currently running compaction job.	Gauge
etcddruid_compaction_job_duration_seconds	Total time taken in seconds to finish a running compaction job.	Histogram
etcddruid_compaction_num_delta_events	Total number of etcd events to be compacted by a compaction job.	Gauge

There are two labels for etcddruid_compaction_jobs_total metrics. The label succeeded shows how many of the compaction jobs are succeeded and label failed shows how many of compaction jobs are failed.

There are two labels for etcddruid_compaction_job_duration_seconds metrics. The label succeeded shows how much time taken by a successful job to complete and label failed shows how much time taken by a failed compaction job.

etcddruid_compaction_jobs_current metric comes with label etcd_namespace that indicates the namespace of the Etcd running in the control plane of a shoot cluster..

Etcd

These metrics are exposed by the etcd process that runs in each etcd pod.

The following list metrics is applicable to clustering of a multi-node etcd cluster. The full list of metrics exposed by etcd is available here.

No.	Metrics Name	Description	Comments
1	etcd_disk_wal_fsync_duration_seconds	latency distributions of fsync called by WAL.	High disk operation latencies indicate disk issues.
2	etcd_disk_backend_commit_duration_seconds	latency distributions of commit called by backend.	High disk operation latencies indicate disk issues.
3	etcd_server_has_leader	whether or not a leader exists. 1: leader exists, 0: leader not exists.	To capture quorum loss or to check the availability of etcd cluster.
4	etcd_server_is_leader	whether or not this member is a leader. 1 if it is, 0 otherwise.
5	etcd_server_leader_changes_seen_total	number of leader changes seen.	Helpful in fine tuning the zonal cluster like etcd-heartbeat time etc, it can also indicates the etcd load and network issues.
6	etcd_server_is_learner	whether or not this member is a learner. 1 if it is, 0 otherwise.
7	etcd_server_learner_promote_successes	total number of successful learner promotions while this member is leader.	Might be helpful in checking the success of API calls called by backup-restore.
8	etcd_network_client_grpc_received_bytes_total	total number of bytes received from grpc clients.	Client Traffic In.
9	etcd_network_client_grpc_sent_bytes_total	total number of bytes sent to grpc clients.	Client Traffic Out.
10	etcd_network_peer_sent_bytes_total	total number of bytes sent to peers.	Useful for network usage.
11	etcd_network_peer_received_bytes_total	total number of bytes received from peers.	Useful for network usage.
12	etcd_network_active_peers	current number of active peer connections.	Might be useful in detecting issues like network partition.
13	etcd_server_proposals_committed_total	total number of consensus proposals committed.	A consistently large lag between a single member and its leader indicates that member is slow or unhealthy.
14	etcd_server_proposals_pending	current number of pending proposals to commit.	Pending proposals suggests there is a high client load or the member cannot commit proposals.
15	etcd_server_proposals_failed_total	total number of failed proposals seen.	Might indicates downtime caused by a loss of quorum.
16	etcd_server_proposals_applied_total	total number of consensus proposals applied.	Difference between etcd_server_proposals_committed_total and etcd_server_proposals_applied_total should usually be small.
17	etcd_mvcc_db_total_size_in_bytes	total size of the underlying database physically allocated in bytes.
18	etcd_server_heartbeat_send_failures_total	total number of leader heartbeat send failures.	Might be helpful in fine-tuning the cluster or detecting slow disk or any network issues.
19	etcd_network_peer_round_trip_time_seconds	round-trip-time histogram between peers.	Might be helpful in fine-tuning network usage specially for zonal etcd cluster.
20	etcd_server_slow_apply_total	total number of slow apply requests.	Might indicate overloaded from slow disk.
21	etcd_server_slow_read_indexes_total	total number of pending read indexes not in sync with leader’s or timed out read index requests.

The full list of metrics is available here.

Etcd-Backup-Restore

These metrics are exposed by the etcd-backup-restore container in each etcd pod.

The following list metrics is applicable to clustering of a multi-node etcd cluster. The full list of metrics exposed by etcd-backup-restore is available here.

No.	Metrics Name	Description
1.	etcdbr_cluster_size	to capture the scale-up/scale-down scenarios.
2.	etcdbr_is_learner	whether or not this member is a learner. 1 if it is, 0 otherwise.
3.	etcdbr_is_learner_count_total	total number times member added as the learner.
4.	etcdbr_restoration_duration_seconds	total latency distribution required to restore the etcd member.
5.	etcdbr_add_learner_duration_seconds	total latency distribution of adding the etcd member as a learner to the cluster.
6.	etcdbr_member_remove_duration_seconds	total latency distribution removing the etcd member from the cluster.
7.	etcdbr_member_promote_duration_seconds	total latency distribution of promoting the learner to the voting member.
8.	etcdbr_defragmentation_duration_seconds	total latency distribution of defragmentation of each etcd cluster member.

Prometheus supplied metrics

The Prometheus client library provides a number of metrics under the go and process namespaces.