Azure storage-cli integration

[kathap]

• Add blobstore_cli_config templates (buildpacks/droplets/packages/resource_pool) to api, worker and clock job
• Pass default timeout of 41s if azure storage cli is used
• Each job now renders:
  • storage_cli_config_droplets.json
  • storage_cli_config_buildpacks.json
  • storage_cli_config_packages.json
  • storage_cli_config_resource_pool.json
• Update template specs for per-scope rendering
• Keeps provider optional; non-Azure scopes render {}

New optional property in manifest: cc.[scope].blobstore_provider for each of buildpacks, droplets, packages, and resource_pool.
Default is nil (unset). If you set it to AzureRM, the job renders a storage_cli_config_[scope].json using the values under cc.[scope].connection_config (e.g., azure_storage_account_name, azure_storage_access_key, container_name, etc.).
If you leave it unset or set a non-Azure value, the template renders {} and existing behaviour is unchanged.

Manifest example:

cc:
  packages:
    blobstore_type: storage-cli
    blobstore_provider: AzureRM
    packages_directory_key: some-key
    connection_config:
      azure_storage_account_name: ((azure_account))
      azure_storage_access_key:   ((azure_key))
      container_name:             cc-packages
      ...

• A short explanation of the proposed change:
  bosh-azure-storage-cli based blobstore client integration

• An explanation of the use cases your change solves
  bosh-azure-storage-cli based blobstore client is a potential replacement for deprecated fog libraries like azure fog

• Links to any other associated PRs
  Extend bosh-azure-storage-cli: new blob commands + upload timeout bosh-azure-storage-cli#22
  ADR: Use Bosh Storage CLIs for Blobstore Operations cloud_controller_ng#4443
  POC: bosh-azure-storage-cli based blobstore client cloud_controller_ng#4397

• I have viewed signed and have submitted the Contributor License Agreement

• I have made this pull request to the develop branch

• I have run CF Acceptance Tests on bosh lite

[jochenehret]

Tested on bbl dev env, works.

[sethboyles]

question why do we need to render the .json files at pre-start time, rather than have them rendered via bosh templates?

[kathap]

question why do we need to render the .json files at pre-start time, rather than have them rendered via bosh templates?
When we render them via bosh templates, we need 1 template file for each config json, that are 12 jsons additionally with nearly the same code, if we render them in pre-start and have 1 generic template per job, we just need 3 jsons extra in total.

[sethboyles]

Ah, I see. BOSH templating sadly isn't super flexible for this sort of thing. I went down a rabbit hole today trying to see if there was any good way to share templating logic, but didn't find much.

Still, it's unfortunate that we need to introduce an additional possible failure point in pre-start task for some templating...

Is it possible to do something like the shared_job_templates instead? I think it would be something like 4 different templates in that directory:

storage_cli_config_buildpacks.json.erb
storage_cli_config_resources.json.erb
storage_cli_config_droplets.json.erb
storage_cli_config_packages.json.erb

And then in each of the 3 jobs that needs them, use a symlink. You can see how this is done with setup_local_blobstore or any of the other files in that directory. I know this is less DRY than your method (4 files with duplicate content instead of 3), but has the advantage of being contained to the template rendering step of a BOSH deploy.

[kathap]

Ah, I see. BOSH templating sadly isn't super flexible for this sort of thing. I went down a rabbit hole today trying to see if there was any good way to share templating logic, but didn't find much.

Still, it's unfortunate that we need to introduce an additional possible failure point in pre-start task for some templating...

Is it possible to do something like the shared_job_templates instead? I think it would be something like 4 different templates in that directory:

storage_cli_config_buildpacks.json.erb
storage_cli_config_resources.json.erb
storage_cli_config_droplets.json.erb
storage_cli_config_packages.json.erb

And then in each of the 3 jobs that needs them, use a symlink. You can see how this is done with setup_local_blobstore or any of the other files in that directory. I know this is less DRY than your method (4 files with duplicate content instead of 3), but has the advantage of being contained to the template rendering step of a BOSH deploy.

We thought the shared_templates folder is a bit outdated and not used really, and that symlinks are not the preferred option, https://github.com/cloudfoundry/capi-release/tree/develop/shared_job_templates, commit 2f640cd. That is why we decided against symlinking. We tried with having a template in the shared_templates folder without symlink, that did not work out.

[sethboyles]

My main concern is that if an error is encountered during pre-start, that is much harder for an operator to resolve than if it occurs during template rendering, since it will pause a deployment midway. I would think it'd be preferable to avoid more pre-start logic unless it was necessary.

Maybe I'm missing something, but what about symlinks makes it worth the operational trade-off?

Is it possible to have a step in pre-packaging to render the template variations to each job instead of symlinks? Of course, pre_packaging is discouraged by BOSH as well...

[kathap]

That is true, your thought about not having it in pre-start sounds reasonable.
We will render the templates not in pre-start, have one template for each scope, this we already tested and it is working. With that we could go on and test the stuff on real set up.

[jochenehret]

Shouldn't we have a similar test for "cloud_controller_worker" and "cloud_controller_clock"?