Resource Discovery Service

Cloudprober internally defines and uses a protocol called resource discovery service (RDS¹) for targets discovery². It helps provide a consistent interface between the targets subsystem, actual resource discovery mechanisms, and probes subsystem. It also provides a way to move targets discovery into an independent process, which can be used to reduce the upstream API traffic.

Protocol (rds_targets) #

To understand the RDS protocol, let’s look at the rds_targets targets type. You can think of rds_targets as a configuration interface to the RDS service. When you configure rds_targets, you’re creating an RDS client that talks to an RDS backend that is either part of the same process (default) or available over gRPC (usefule for centralizing the upstream API calls).

Here are the RDS targets configuration options:

message RDSTargets {
  // RDS server options, for example:
  // rds_server_options {
  //   server_address: "rds-server.xyz:9314"
  //   oauth_config: {
  //     ...
  //   }
  // }
  // Default is to use the local server if any.
  optional rds.ClientConf.ServerOptions rds_server_options = 1;

  // Resource path specifies the resources to return. Resources paths have the
  // following format:
  // <resource_provider>://<resource_type>/<additional_params>
  //
  // Examples:
  // For GCE instances in projectA: "gcp://gce_instances/<projectA>"
  // Kubernetes Pods : "k8s://pods"
  optional string resource_path = 2;

  // Filters to filter resources by. Example:
  // filter {
  //   key: "namespace"
  //   value: "mynamesspace"
  // }
  // filter {
  //   key: "labels.app"
  //   value: "web-service"
  // }
  repeated rds.Filter filter = 3;

  // IP config to specify the IP address to pick for a resource. IPConfig
  // is defined here:
  // https://github.com/cloudprober/cloudprober/blob/master/rds/proto/rds.proto
  optional rds.IPConfig ip_config = 4;
}

Most options are explained in the comments for a quick reference. Here is the further explanation of some of these options:

rds_server_options #

This field specifies how to connect to the RDS server: server address and security options (OAuth and TLS). If left unspecified, it connects to the local server if any (started through rds_server option). Next up it looks for the rds_server_options in global_targets_options.

resource_path #

Resource path specifies the resources we are interested in. It consists of a resource provider, resource type and an optional relative path: <resource_provider>://<resource_type>/<optional_relative_path>

• resource_provider: Resource provider is a generic concept within the RDS protocol but usually maps to the cloud provider. Cloudprober RDS server currently implements the Kubernetes (k8s) and GCP (gcp) resource providers. We plan to add more resource providers in future.
• resource_type: Available resource types depend on the providers, for example, for k8s provider supports the following resource types: pods, endpoints, and services.
• optional_relative_path: For most resource types you can specify resource name in the resource path itself, e.g. k8s://services/cloudprober. Alternatively, you can use filters to filter by name, resource, etc.

filter #

Filters are key-value strings that can be used to filter resources by various fields. Filters depend on the resource types, but most resources support filtering by name and labels.

# Return resources that start with "web" and have label "service:service-a"
...
 filter {
   key: "name"
   value: "^web.*"
 }
 filter {
   key: "labels.service"
   value: "service-a"
 }

• Filters supported by kubernetes resources:
  • k8s filters.
• Filters supported by GCP:
  • GCE Instances
  • Forwarding Rules
  • Pub/Sub Messages

Running RDS Server #

RDS server can either be run as an independent process, or it can be a part of the main prober process. Former mode is useful for large deployments where you may want to reduce the API upcall traffic (for example, to GCP). For example, if you run 1000+ prober processes, it will be much more economical from the API quota usage point of view to have a centralized RDS service with much fewer (2-3) instances instead of having each prober process make its own API calls.

RDS server can be added to a cloudprober process using the rds_server stanza. If you’re running RDS server in a remote process, you’ll have to enable gRPC server in that process (using grpc_port) so that other instances can access it remotely.

Here is an example RDS server configuration:

rds_server {
  # GCP provider to discover GCP resources.
  provider {
    gcp_config {
      # Projects to discover resources in.
      project: "test-project-1"
      project: "test-project-2"

      # Discover GCE instances in us-central1.
      gce_instances {
        zone_filter: "name = us-central1-*"
        re_eval_sec: 60  # How often to refresh, default is 300s.
      }

      # GCE forwarding rules.
      forwarding_rules {}
    }
  }

  # Kubernetes targets are further discussed at:
  # https://cloudprober.org/how-to/run-on-kubernetes/#kubernetes-targets
  provider {
    kubernetes_config {
      endpoints {}
    }
  }
}

For the remote RDS server setup, if accessing over external network, you can secure the underlying gRPC communication using TLS certificates.

Remote RDS Server Example #

Cloudprober config:

probe {
  rds_targets {
    rds_server_options {
      server_address: "rds-service:9314"

      # mTLS configuration
      tls_config {
        ca_cert_file: "/vol/certs/server_ca.crt" # To verify the server
        tls_cert_file: "/vol/certs/client.crt"   # Own cert to present to server
        tls_key_file: "/vol/certs/client.key"    # Own cert's private key
      }
    }

    resource_path: "gcp://gce_instances"
    filter {
      key: "name"
      value: "ins-cf-.*"
    }
  }
}

On a different cloudprober instance:

...
rds_server {
  provider {
    gcp_config {
      instances {
        zone_filter: "us-east1-a"
      }
    }
  }
}

grpc_tls_config {
  ca_cert_file: "/vol/certs/client_ca.crt" # To verify the server
  tls_cert_file: "/vol/certs/server.crt"   # Own cert to present to client
  tls_key_file: "/vol/certs/server.key"    # Own cert's private key
}

# Required for remote access
grpc_port: 9314