Kibana Canvas GCP

Description

This Google Cloud Log Ingestion dashboard is visualizing many different metrics in a single architectural view. Its using the Data Flow Canvas Template as base. The target is to show the log events flow from Google Cloud Log Router via Pub / Sub topic and subscription to the dataflow job consumer into Elastic.

This canvas board shows the stream for Audit, Firewall, VPC Flow and DNS log data in Google Cloud. It is using global variables in order to set it up. For the audit stream it e.g. looks for objects that include the word “audit”. Which means that the log sink, the topic, the subscription and the dataflow job need to have this in the name.
There are also global variables to change the index of the data. It defaults to metric* and log* at the moment. If you are using different names in your Log Collection Setup or for your indices you need to change this variables within the Kibana Canvas Board.

I made all expressions aware of missing data. However it still shows error messages when the index pattern does not find any matching index.

To get data for all the services that get displayed here you need to configure the GCP Metric module with two generic metric data collectors.

- module: gcp
  metricsets:
    - metrics
  project_id: "xxx"
  credentials_file_path: "metricbeat-service-account.json"
  exclude_labels: false
  period: 1m
  metrics:
    - aligner: ALIGN_NONE
      service: dataflow
      metric_types: 
        - "job/backlog_elements"
        - "job/status"
        - "job/per_stage_system_lag"
        - "job/pubsub/read_count"
        - "job/pubsub/write_count"
- module: gcp
  metricsets:
    - metrics
  project_id: "xxx"
  credentials_file_path: "metricbeat-service-account.json"
  exclude_labels: false
  period: 1m
  metrics:
    - aligner: ALIGN_NONE
      service: logging
      metric_types: 
        - "exports/log_entry_count"
        - "exports/error_count"
        - "exports/byte_count"
Tested versions 7.x, 8.x
ECS compliant Yes

You must log in to submit a review.

Related downloads

Elastic Cloud Monitoring dashboard

Kibana dashboard that uses the Elastic Cloud monitoring data to provide better insights into what’s happening in your cloud environment.

Terraform Elasticsearch environments

Terraform example scripts to deploy Elastic Cloud Clusters + all necessary components in AWS and GCP

Google Cloud Log Ingestion dashboard

Canvas Board to analyze the log data collection of Google Cloud via Dataflow using the Google Cloud Metric module data

Google Cloud monitoring dashboard

Dashboard to monitor GCP resources using different metrics and logs.

These downloads could be also interesting for you

Buttons for Kibana dashboards

Link to every content you want within your Kibana dashboards. This example is using links to cloud providers.

Coffee Canvas Example

Kibana Canvas example infographic to show coffee consumption on a wallboard

Playable Pacman

This is a playable version of pacman made with Vega.

Sankey visualization example

This is an example of how to build an sankey visualization using the vega visualization in Kibana.

Vega Clock UTC

This is a working clock visualization in UTC time.

Resource Optimization Dashboard

Elastic Resource Optimization Dashboard to seamlessly integrate APM insights with cloud cost data for actionable resource management and cost-saving strategies