Consider using Istio readiness check instead of Envoy admin API

[linjmeyer]

Current the ENVOY_ADMIN_API is used to prevent application startup until Istio/Envoy is ready. Istio itself has a readiness probe (used by k8s healthchecks) at http://127.0.0.1:15020/healthz/ready. Looking at the master branch of Istio this endpoint verifies the Istio Admin API is ready (or it will time out), and internally verifies Envoy is operational.

I think using Istio's readiness endpoint makes more sense as we can be more confident that not only Envoy is ready, but Istio itself is too. Most likely this would mean deprecating ENVOY_ADMIN_API as the name of the variable wouldn't make sense.

[JoeJasinski]

Hey @linjmeyer, I tried setting the ENVOY_ADMIN_API to http://127.0.0.1:15020/healthz/ready with an existing release of scuttle, scuttle seems to start up, and envoy reports it is ready, but the main pod never seems to get started and the pod hangs forever. I've separately curled the http://127.0.0.1:15020/healthz/ready url and it returns a 200 when the pod is ready.

Here is where scuttle starts up:

curlpod-2 curlpod-container-jjj 2021-04-10T00:52:33Z scuttle: Scuttle 1.3.1 starting up, pid 1
curlpod-2 curlpod-container-jjj 2021-04-10T00:52:33Z scuttle: Logging is now enabled
curlpod-2 curlpod-container-jjj 2021-04-10T00:52:33Z scuttle: Blocking until Envoy starts

Here the last thing that gets logged.. envoy is ready. But the main workload never runs.

curlpod-2 istio-proxy 2021-04-10T00:52:36.397354Z       info    Envoy proxy is ready

And here is the pod

command:
- scuttle
- bash
- -cx
- --
- curl -v http://myservice:8000/api/
env:
- name: ENVOY_ADMIN_API
  value: http://127.0.0.1:15020/healthz/ready
- name: ISTIO_QUIT_API
  value: http://127.0.0.1:15000

Does scuttle need to be changed to account for the different url?

[JoeJasinski]

Do you think an optional variable could be set that would define the request body content that scuttle would wait for? For example, in the case of the ADMIN_API it would be LIVE, but in the case of the /healthz/ready, it would be empty.

[linjmeyer]

Hey @JoeJasinski, I think I'd prefer to deprecate the ENVOY_ADMIN_API and replace it with something more generic that just looks for a 200 as that matches normal health/liveness/readiness checks. Call it READINESS_ENDPOINT or something and consider it ready as soon as a 200 response it seen. There are a few other config options that really could be more generic too. We forked Scuttle from an Envoy specific project, adapted it for Istio, but I think it could be made to work with other sidecars easily. Thoughts?

[JoeJasinski]

Thanks @linjmeyer, that makes sense to me. I also think checking for a 200 would make it easy to use other places.

[snuggie12]

envoy has a /ready endpoint which returns a 503 if not ready and 200 if LIVE. Seems like a good fit

[snuggie12]

I think the example tried above does not work because scuttle is looking for an IP and port and it tacks on server_info according to the docs.