watcher logo

Description

A watch which alerts if the time spent by a hosts CPU in IOWait, has increased by more than than N% in the last Y mins. N defaults to 5%, Y to 2 mins.

The watch searches across the last X minutes (default 4m), aggregating by hostname. A date histogram is constructed per host using an interval of Y (default 2m) – to ensure two buckets are present to calculate ‘change’ (see below). For each interval a metric script aggregation calculates the percentage of time spent in IOWait. A derivative pipeline aggregation in turn calculates the ‘change’ in IOWait between the intervals. If the ‘change’ for any host exceeds the configured threshold N, an alert is raised.

This watch assumes the data has been collected with Metricbeat.

Mapping Assumptions

A mapping is provided in mapping.json. This provides a subset of the mapping provided with Metricbeat. Watches require data producing the following fields:

  • @timestamp – authoritative date field for each log message
  • beat.hostname (string not_analyzed) – The host for which the document represents.

CPU pct statistics configured as scaled_float (with doc values) as produced by Metricbeat:

  • system.cpu.iowait.pct
  • system.cpu.user.pct
  • system.cpu.nice.pct
  • system.cpu.system.pct
  • system.cpu.idle.pct
  • system.cpu.irq.pct
  • system.cpu.softirq.pct
  • system.cpu.steal.pct

Data Assumptions

The Watch assumes each document represents the CPU state for a specific host at any moment in time. The watch assumes data is indexed into an index prefixed by “metricbeat” with type “doc”.

Other Assumptions

  • The watch assumes the window period X is twice that of the interval Y i.e. by default 4 and 2m respectively.
  • The watch assumes the schedule interval is equal to the interval Y i.e. 2m, to ensure no periods are “missed”.

Configuration

  • The watch is scheduled to execute every 2 minutes. This can be adjusted but should be equal to the “interval” parameter below.
  • The “interval” Y is the period over which IOWait is measured. This should be equal to the schedule and normally half the window. Defaults to 2m.
  • The “window” X over which the watch is executed. Allows the wait time to be calculated for the previous 2 intervals and thus a derivative to be used as the threshold i.e. change in IOwait. Defaults to 4m and will typically be twice the interval.
  • The threshold N. The amount of time IOWait must increase by on a specific host, for an alert to be produced. A % value. Defaults to 5.
Tested versions
ECS compliant

You must log in to submit a review.

Related downloads

Move to next ILM phase Watcher

This watcher job is moving specific indices based on e.g. disc usage into the next ILM phase.

Watch for changes in IOWaits

A watch which alerts if the time spent by a hosts CPU in IOWait, has increased by more than than N% in the last Y mins.

Watcher History Dashboard

This dashboard shows the history of executed watcher jobs.

Uptime watch using Heartbeat data

This watch checks the availability of your Heartbeat observed services. It will trigger an alert whenever at least one of your services is down.

Vega Compound Gauge

This is a compund gauge visualization made with Vega. Its very helpful for visualization of percentage values.

Watcher job to integrate ChatGPT in Elasticsearch

Watcher job to integrate ChatGPT API from OpenAI in Elasticsearch. Helpful to find solutions for error messages very quick.

These downloads could be also interesting for you

Watch to detect large shards

This watch is getting data from the Elasticsearch shards API directly and checking for large shards.

Cloudflare Kibana dashboards

Cloudflare dashboards and ingest pipelines to visualize cloudflare logs

Office 365 dashboards

A collection of Kibana dashboards to provide a holistic view of Microsoft Office 365 environments

Terraform Elasticsearch environments

Terraform example scripts to deploy Elastic Cloud Clusters + all necessary components in AWS and GCP

Watch for changes in IOWaits

A watch which alerts if the time spent by a hosts CPU in IOWait, has increased by more than than N% in the last Y mins.

Kibana alerting enhancement

This bundle enhances the Kibana alerting experience. Storing all relevant information in indices and visualize the data in dashboards.