Metrics from multiple google projects (external)

[psalaberria002]

I have a GKE cluster in a google project, and I have Google Cloud resources (pubsub, bigtable) in a different project.

I would like to autoscale using the external metrics api based on those resources.

Is it possible to pass the project id to the adapter? My main project already gets stackdriver metrics from all other projects.

Thank you.

[astropuffin]

I am also having this problem. I'm happy to write the code to get it fixed, but will need some guidance. So far I think the main issue is here: https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/pkg/adapter/provider/translator.go, as it references the project (t.config.Project). However, when I modify the code to have a configurable project, I'm getting this:

E0928 21:47:17.044439       1 provider.go:217] Failed request to stackdriver api: Get https://monitoring.googleapis.com/v3/projects/redacted-project/metricDescriptors?alt=json&filter=resource.labels.project_id+%3D+%22redacted-project%22+AND+resource.labels.cluster_name+%3D+%22green%22+AND+resource.labels.container_name+%3D+%22%22+AND+resource.labels.pod_id
+%21%3D+%22%22+AND+resource.labels.pod_id+%21%3D+%22machine%22: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_scope",
  "error_description": "Empty or missing scope not allowed."
}

I'm sure I'm doing something dumb, as I will need to set the scope in the stackdriver service somewhere. I feel like I'm about to fall into a deep, deep rabbithole...

[astropuffin]

Ok I figured out how to do this, which is probably what was originally intended:

1. Install kubernetes cluster, making sure you don't mess with the permissions of the service account (see below).
2. install the stackdriver adapter as is, with no code changes (making sure to follow all the troubleshooting steps as well)
3. Create an app that bridges the gap between different projects. Basically it will read whatever metrics you want in other projects, and create custom metrics in the project that the GKE cluster is in. There are two ways to enable this permissions-wise:
   A. install your kubernetes cluster using a custom service account, and enable that service account to have roles/monitoring.viewer on the other project
   B. (recommended) Create a service account for your bridge app with the appropriate permissions.
4. write an HPA that uses external custom metrics, like so:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: dummy
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: dummy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - external:
      metricName: custom.googleapis.com|foo
      targetAverageValue: "2"
    type: External

While this solution requires essentially two adapters, it should work for us as each HPA will only be scaling on one metric.

[kawych]

@astropuffin the option you proposed is the way to make it work without changes in the adapter itself. To distinguish metrics from different projects, you may consider adding a metric label with project name (or project id).

Another option I see is to allow a special label for the project id: https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/pkg/adapter/provider/translator.go#L46
The labels represent parameters in Stackdriver query, currently they are related only to the filter for metrics, but with some code changes we could use it for project id as well (you would have to do the implementation though, we don't plan to work on this right now). Similarly to your option A, you would also have to enable your SA to read metrics from another project.

[yuvaldrori]

@kawych currently you have to pipe external metrics from other projects to the current monitoring workspace because the gceconfig gets the project id from the metadata server. Is there a way to pass a different project id to the adapter to monitor different projects?
This could be related to #363

[Ekimik]

Hi,
since this issue is still open I want to ask if there is any way how to read metric from stackdriver, which belongs to another Google cloud project. I have service account from owner of other project, which allows me to access monitoring data (tested in Grafana). I 've checked code in MR, which was merged 2 years ago, but I don't know if I should follow those steps since examples introduced there are not in master anymore.

What I want to do exactly is to read num_undelivered_messages metric associated with pubsub service in other project and use it in HPA in my project.

[davidxia]

I think issue is already addressed by #205 which has been merged and released. If so, this issue can be closed.

But I think there's a bug here #396