Surfacer Config

# This name is used for logging. If not defined, it's derived from the type. # Note that this field is required for the USER_DEFINED surfacer type and # should match with the name that you used while registering the user defined # surfacer.

# How many metrics entries (EventMetrics) to buffer. This is the buffer # between incoming metrics and the metrics that are being processed. Default # value should work in most cases. You may need to increase it on a busy # system, but that's usually a sign that you metrics processing pipeline is # slow for some reason, e.g. slow writes to a remote file. # Note: Only file and pubsub surfacer supports this option right now.

# If specified, only allow metrics that match any of these label filters. # Example: # allow_metrics_with_label { # key: "probe", # value: "check_homepage", # }

# Ignore metrics that match any of these label filters. Ignore has precedence # over allow filters. # Example: # ignore_metrics_with_label { # key: "probe", # value: "sysvars", # }

# Allow and ignore metrics based on their names. You can specify regexes # here. Ignore has precendence over allow. # Examples: # ignore_metrics_with_name: "validation_failure" # allow_metrics_with_name: "(total|success|latency)"

# Whether to add failure metric or not. This option is enabled by default.

# If set to true, cloudprober will export all metrics as gauge metrics. Note # that cloudprober inherently generates only cumulative metrics. To create # gauge metrics from cumulative metrics, we keep a copy of the old metrics # and subtract new metrics from the previous metrics. This transformation in # metrics has an increased memory-overhead because extra copies required. # However, it should not be noticeable unless you're producing large number # of metrics (say > 10000 metrics per second).

# Latency metric name pattern, used to identify latency metrics, and add # EventMetric's LatencyUnit to it.

# Environment variable containing additional labels to be added to all # metrics exported by this surfacer. # e.g. "CLOUDPROBER_ADDITIONAL_LABELS=env=prod,app=identity-service" # You can disable this feature by setting this field to an empty string. # Note: These additional labels have no effect if metrics already have the # same label.

# Matching surfacer specific configuration (one for each type in the above # enum)

# It represents the bigquery table columns. # bigquery_columns { # label: "id", # column_name: "id", # column_type: "string", # }

# It represents bigquery client timeout in seconds. So, if bigquery insertion # is not completed within this time period then the request will fail and the # failed rows will be retried later.

# This denotes the time interval after which data will be inserted in # bigquery. Default is 10 seconds. So after every 10 seconds all the em in # current will be inserted in bigquery in a default batch size of 1000

# Column name for metrics name, value and timestamp

# The cloudwatch metric namespace

# The cloudwatch resolution value, lowering this below 60 will incur # additional charges as the metrics will be charged at a high resolution # rate.

# The AWS Region, used to create a CloudWatch session. # The order of fallback for evaluating the AWS Region: # 1. This config value. # 2. EC2 metadata endpoint, via cloudprober sysvars. # 3. AWS_REGION environment value. # 4. AWS_DEFAULT_REGION environment value, if AWS_SDK_LOAD_CONFIG is set. # https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/

# The maximum number of metrics that will be published at one # time. Metrics will be stored locally in a cache until this # limit is reached. 1000 is the maximum number of metrics # supported by the Cloudwatch PutMetricData API. # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# Prefix to add to all metrics.

# Datadog API key. If not set, DD_API_KEY env variable is used.

# Datadog APP key. If not set, DD_APP_KEY env variable is used.

# Datadog server, default: "api.datadoghq.com"

# The maximum number of metrics that will be published at one # time. Metrics will be stored locally in a cache until this # limit is reached. Datadog's SubmitMetric API has a maximum payload # size of 500 kilobytes (512000 bytes). Compressed payloads must have a # decompressed size of less than 5 megabytes (5242880 bytes). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# Disable gzip compression of metric payload, when sending metrics to Datadog. # Compression is enabled by default.

# Where to write the results. If left unset, file surfacer writes to the # standard output.

# Compress data before writing to the file.

# If no URL is provided, OpenTelemetry SDK will use the environment variable # OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT in that # preference order.

# HTTP request headers. These can also be set using environment variables.

# Compression algorithm to use for gRPC requests.

# Whether to use insecure gRPC connection.

# If no URL is provided, OpenTelemetry SDK will use the environment variable # OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT in that # preference order.

# HTTP request headers. These can also be set using environment variables.

# Compression algorithm to use for HTTP requests.

# How often metrics will be exported. Note that metrics are accumulated # internally and exported at this interval. Increasing this value will # increase the memory usage.

# Prefix to use for metrics. Defaults to "cloudprober_".

# Label name

# Column to map this label to:

# Postgres connection string. # Example: # "postgresql://root:${PASSWORD}@localhost/cloudprober?sslmode=disable"

# Metrics table name. # To create table (when storing all labels in single column in JSON format): # CREATE TABLE metrics ( # time timestamp, metric_name varchar(80), value float8, labels jsonb # )

# Adding label_to_column fields changes how labels are stored in a Postgres # table. If this field is not specified at all, all the labels are stored as # jsonb values as the 'labels' column (this mode impacts performance # negatively). If label_to_colum entries are specified for some labels, # those labels are stored in their dedicated columns; all the labels that # don't have a mapping will be dropped.

# The maximum number of metric events will be commited in one transaction at one # time. Metrics will be stored locally until this limit is reached. Metrics will # be commited to postgres when the timer expires, or the buffer is full, whichever # happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be commited to postgres when the timer expires, or the buffer is full, # whichever happens first.

# default 60s

# Number of points in each timeseries. This field dictates how far back # can you go up to (resolution_sec * timeseries_size). Note that higher # this number, more memory you'll use.

# Max targets per probe.

# ProbeStatus URL # Note that older default URL /probestatus forwards to this URL to avoid # breaking older default setups.

# Page cache time

# Probestatus surfacer is enabled by default. To disable it, set this # option.

# How many metrics entries (EventMetrics) to buffer. Incoming metrics # processing is paused while serving data to prometheus. This buffer is to # make writes to prometheus surfacer non-blocking. # NOTE: This field is confusing for users and will be removed from the config # after v0.10.3.

# Whether to include timestamps in metrics. If enabled (default) each metric # string includes the metric timestamp as recorded in the EventMetric. # Prometheus associates the scraped values with this timestamp. If disabled, # i.e. timestamps are not exported, prometheus associates scraped values with # scrape timestamp.

# URL that prometheus scrapes metrics from.

# Prefix to add to all metric names. For example setting this field to # "cloudprober_" will result in metrics with names: # cloudprober_total, cloudprober_success, cloudprober_latency, .. # # As it's typically useful to set this across the deployment, this field can # also be set through the command line flag --prometheus_metrics_prefix.

# GCP project name for pubsub. It's required if not running on GCP, # otherwise it's retrieved from the metadata.

# Pubsub topic name. # Default is cloudprober-{hostname}

# Compress data before writing to pubsub.

# GCP project name for stackdriver. If not specified and running on GCP, # project is used.

# How often to export metrics to stackdriver.

# If allowed_metrics_regex is specified, only metrics matching the given # regular expression will be exported to stackdriver. Since probe type and # probe name are part of the metric name, you can use this field to restrict # stackdriver metrics to a particular probe. # Example: # allowed_metrics_regex: ".*(http|ping).*(success|validation_failure).*" # # Deprecated: Please use the common surfacer options to filter metrics: # https://cloudprober.org/docs/surfacers/overview/#filtering-metrics

# Monitoring URL base. Full metric URL looks like the following: # <monitoring_url>/<ptype>/<probe>/<metric> # Example: # custom.googleapis.com/cloudprober/http/google-homepage/latency

# How many metrics entries to buffer. Incoming metrics # processing is paused while serving data to Stackdriver. This buffer is to # make writes to Stackdriver surfacer non-blocking.

# Metric prefix to use for stackdriver metrics. If not specified, default # is PTYPE_PROBE.

# This name is used for logging. If not defined, it's derived from the type. # Note that this field is required for the USER_DEFINED surfacer type and # should match with the name that you used while registering the user defined # surfacer.

# How many metrics entries (EventMetrics) to buffer. This is the buffer # between incoming metrics and the metrics that are being processed. Default # value should work in most cases. You may need to increase it on a busy # system, but that's usually a sign that you metrics processing pipeline is # slow for some reason, e.g. slow writes to a remote file. # Note: Only file and pubsub surfacer supports this option right now.

# If specified, only allow metrics that match any of these label filters. # Example: # allow_metrics_with_label { # key: "probe", # value: "check_homepage", # }

# Ignore metrics that match any of these label filters. Ignore has precedence # over allow filters. # Example: # ignore_metrics_with_label { # key: "probe", # value: "sysvars", # }

# Allow and ignore metrics based on their names. You can specify regexes # here. Ignore has precendence over allow. # Examples: # ignore_metrics_with_name: "validation_failure" # allow_metrics_with_name: "(total|success|latency)"

# Whether to add failure metric or not. This option is enabled by default.

# If set to true, cloudprober will export all metrics as gauge metrics. Note # that cloudprober inherently generates only cumulative metrics. To create # gauge metrics from cumulative metrics, we keep a copy of the old metrics # and subtract new metrics from the previous metrics. This transformation in # metrics has an increased memory-overhead because extra copies required. # However, it should not be noticeable unless you're producing large number # of metrics (say > 10000 metrics per second).

# Latency metric name pattern, used to identify latency metrics, and add # EventMetric's LatencyUnit to it.

# Environment variable containing additional labels to be added to all # metrics exported by this surfacer. # e.g. "CLOUDPROBER_ADDITIONAL_LABELS=env=prod,app=identity-service" # You can disable this feature by setting this field to an empty string. # Note: These additional labels have no effect if metrics already have the # same label.

# Matching surfacer specific configuration (one for each type in the above # enum)

# It represents the bigquery table columns. # bigquery_columns { # label: "id", # column_name: "id", # column_type: "string", # }

# It represents bigquery client timeout in seconds. So, if bigquery insertion # is not completed within this time period then the request will fail and the # failed rows will be retried later.

# This denotes the time interval after which data will be inserted in # bigquery. Default is 10 seconds. So after every 10 seconds all the em in # current will be inserted in bigquery in a default batch size of 1000

# Column name for metrics name, value and timestamp

# The cloudwatch metric namespace

# The cloudwatch resolution value, lowering this below 60 will incur # additional charges as the metrics will be charged at a high resolution # rate.

# The AWS Region, used to create a CloudWatch session. # The order of fallback for evaluating the AWS Region: # 1. This config value. # 2. EC2 metadata endpoint, via cloudprober sysvars. # 3. AWS_REGION environment value. # 4. AWS_DEFAULT_REGION environment value, if AWS_SDK_LOAD_CONFIG is set. # https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/

# The maximum number of metrics that will be published at one # time. Metrics will be stored locally in a cache until this # limit is reached. 1000 is the maximum number of metrics # supported by the Cloudwatch PutMetricData API. # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# Prefix to add to all metrics.

# Datadog API key. If not set, DD_API_KEY env variable is used.

# Datadog APP key. If not set, DD_APP_KEY env variable is used.

# Datadog server, default: "api.datadoghq.com"

# The maximum number of metrics that will be published at one # time. Metrics will be stored locally in a cache until this # limit is reached. Datadog's SubmitMetric API has a maximum payload # size of 500 kilobytes (512000 bytes). Compressed payloads must have a # decompressed size of less than 5 megabytes (5242880 bytes). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be published when the timer expires, or the buffer is # full, whichever happens first.

# Disable gzip compression of metric payload, when sending metrics to Datadog. # Compression is enabled by default.

# Where to write the results. If left unset, file surfacer writes to the # standard output.

# Compress data before writing to the file.

# If no URL is provided, OpenTelemetry SDK will use the environment variable # OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT in that # preference order.

# HTTP request headers. These can also be set using environment variables.

# Compression algorithm to use for gRPC requests.

# Whether to use insecure gRPC connection.

# If no URL is provided, OpenTelemetry SDK will use the environment variable # OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT in that # preference order.

# HTTP request headers. These can also be set using environment variables.

# Compression algorithm to use for HTTP requests.

# How often metrics will be exported. Note that metrics are accumulated # internally and exported at this interval. Increasing this value will # increase the memory usage.

# Prefix to use for metrics. Defaults to "cloudprober_".

# Label name

# Column to map this label to:

# Postgres connection string. # Example: # "postgresql://root:${PASSWORD}@localhost/cloudprober?sslmode=disable"

# Metrics table name. # To create table (when storing all labels in single column in JSON format): # CREATE TABLE metrics ( # time timestamp, metric_name varchar(80), value float8, labels jsonb # )

# Adding label_to_column fields changes how labels are stored in a Postgres # table. If this field is not specified at all, all the labels are stored as # jsonb values as the 'labels' column (this mode impacts performance # negatively). If label_to_colum entries are specified for some labels, # those labels are stored in their dedicated columns; all the labels that # don't have a mapping will be dropped.

# The maximum number of metric events will be commited in one transaction at one # time. Metrics will be stored locally until this limit is reached. Metrics will # be commited to postgres when the timer expires, or the buffer is full, whichever # happens first.

# The maximum amount of time to hold metrics in the buffer (above). # Metrics will be commited to postgres when the timer expires, or the buffer is full, # whichever happens first.

# default 60s

# Number of points in each timeseries. This field dictates how far back # can you go up to (resolution_sec * timeseries_size). Note that higher # this number, more memory you'll use.

# Max targets per probe.

# ProbeStatus URL # Note that older default URL /probestatus forwards to this URL to avoid # breaking older default setups.

# Page cache time

# Probestatus surfacer is enabled by default. To disable it, set this # option.

# How many metrics entries (EventMetrics) to buffer. Incoming metrics # processing is paused while serving data to prometheus. This buffer is to # make writes to prometheus surfacer non-blocking. # NOTE: This field is confusing for users and will be removed from the config # after v0.10.3.

# Whether to include timestamps in metrics. If enabled (default) each metric # string includes the metric timestamp as recorded in the EventMetric. # Prometheus associates the scraped values with this timestamp. If disabled, # i.e. timestamps are not exported, prometheus associates scraped values with # scrape timestamp.

# URL that prometheus scrapes metrics from.

# Prefix to add to all metric names. For example setting this field to # "cloudprober_" will result in metrics with names: # cloudprober_total, cloudprober_success, cloudprober_latency, .. # # As it's typically useful to set this across the deployment, this field can # also be set through the command line flag --prometheus_metrics_prefix.

# GCP project name for pubsub. It's required if not running on GCP, # otherwise it's retrieved from the metadata.

# Pubsub topic name. # Default is cloudprober-{hostname}

# Compress data before writing to pubsub.

# GCP project name for stackdriver. If not specified and running on GCP, # project is used.

# How often to export metrics to stackdriver.

# If allowed_metrics_regex is specified, only metrics matching the given # regular expression will be exported to stackdriver. Since probe type and # probe name are part of the metric name, you can use this field to restrict # stackdriver metrics to a particular probe. # Example: # allowed_metrics_regex: ".*(http|ping).*(success|validation_failure).*" # # Deprecated: Please use the common surfacer options to filter metrics: # https://cloudprober.org/docs/surfacers/overview/#filtering-metrics

# Monitoring URL base. Full metric URL looks like the following: # <monitoring_url>/<ptype>/<probe>/<metric> # Example: # custom.googleapis.com/cloudprober/http/google-homepage/latency

# How many metrics entries to buffer. Incoming metrics # processing is paused while serving data to Stackdriver. This buffer is to # make writes to Stackdriver surfacer non-blocking.

# Metric prefix to use for stackdriver metrics. If not specified, default # is PTYPE_PROBE.