`databricks bundle deploy` does not respect specified `profile`

[gable-grammarly]

Describe the issue

bundle deploy does not pass DATABRICKS_CONFIG_PROFILE to the Terraform subprocess when a profile is specified via workspace.profile in databricks.yml or --profile.

The CLI authenticates successfully for its own operations (validate, upload), but the Terraform apply phase fails because the Terraform provider calls databricks auth token --host <host> instead of databricks auth token --profile <profile>, which is ambiguous when multiple profiles in ~/.databrickscfg share the same host.

While, databricks auth env --profile <name> correctly returns DATABRICKS_CONFIG_PROFILE in its output, it follows a different code path than bundle deploy, meaning databricks auth and databricks bundle return inconsistent results for the same profile.

This appears to be the same class of issue as:

• DATABRICKS_HOST env variable is ignored when using CLI inside DAB directory #4502, where non-bundle and bundle commands follow divergent auth resolution paths
• cli commands unexpectedly inferring host from bundle #1358 where inference is applied when explicit credentials were provided.

Two other callers in the codebase already work around this by manually appending the profile to the subprocess environment:

• cmd/bundle/run.go:284-285
• cmd/apps/run_local.go:62-63

The Terraform code path in bundle/deploy/terraform/init.go:338 does not.

Steps to reproduce the behavior

1. Have multiple profiles in ~/.databrickscfg sharing the same host:

   [DEFAULT]
   host      = https://my-workspace.cloud.databricks.com
   auth_type = databricks-cli
   
   [my-bundle-profile]
   host      = https://my-workspace.cloud.databricks.com
   auth_type = databricks-cli

2. Run databricks bundle deploy --profile my-bundle-profile
3. See error during Terraform apply

Expected Behavior

bundle deploy should pass DATABRICKS_CONFIG_PROFILE to the Terraform subprocess when a profile is specified, so that Terraform's auth resolution can disambiguate profiles sharing the same host, consistent with how databricks auth env --profile behaves.

Actual Behavior

Validation and file uploads succeed. Terraform apply fails:

Error: cannot create job: failed during request visitor: databricks-cli auth:
cannot get access token: Error: DEFAULT and my-bundle-profile match
https://my-workspace.cloud.databricks.com in ~/.databrickscfg.
Use --profile to specify which profile to use.
Config: host=https://my-workspace.cloud.databricks.com. Env: DATABRICKS_HOST

OS and CLI version

• CLI: v0.290.1
• OS: macOS (Darwin 25.3.0)

Is this a regression?

The existing workarounds in cmd/bundle/run.go and cmd/apps/run_local.go as well as the aforementioned related issues suggest this gap has been known.

The fact that two (now three) callers need the same manual patch suggests the fix really belongs in either auth.Env() or b.AuthEnv() rather than at each call site. To that end, it's clear that more robust testing could be beneficial here.

Test Coverage Recommendations

There are no tests that assert the subprocess environment contains DATABRICKS_CONFIG_PROFILE when a profile is specified.

While this looks like a multi-profile host collision error, the underlying defect (profile silently dropped in favor of host-based inferenc) is independently testable:

• Missing test (unit): When b.AuthEnv() is called and workspace.profile is set, assert DATABRICKS_CONFIG_PROFILE is present in the returned env map. (this issue)

• Missing test (integration): When --profile is passed to bundle deploy, assert the Terraform subprocess environment includes DATABRICKS_CONFIG_PROFILE and does NOT fall back to host-based profile resolution.

  • I'd argue there should be a warning issued any time a profile is specified but auth falls back to inference. having a single code path would make this a lot easier to test and maintain.

• Missing assertion (existing tests): Any existing test that passes a profile and spawns a subprocess should assert that the profile propagates, not just that the command succeeds (which it will if only one profile matches the host).

Debug Logs

Debug: Environment variables for Terraform: DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR,  DATABRICKS_USER_AGENT_EXTRA

DATABRICKS_CONFIG_PROFILE is absent from this list.

[andrewnester]

This is already fixed in main and will be released in the upcoming CLI release
#4624

cc @simonfaltum

[gable-grammarly]

@andrewnester glad to hear this particular one-off is patched. I was torn between DAB and CLI, but the reason I picked CLI is due to inconsistencies for the end users across the CLI "UX". For example the auth env {args} running through a different code path than the DAB auth feels like the real root cause here (as well as the missing tests imho).

appreciate the quick follow up and knowing there's a short term solution in place, but would also deeply appreciate stability, consistency, and more visibility into the auth/execution flow.

if you need or want to use different workflows for authentication, disabling auth "inference" for DABs through the databricks.yml would be useful to avoid this confusion.

[gable-grammarly]

@andrewnester just a heads up that I've tested this under 0.294 and it's still failing

[andrewnester]

@gable-grammarly thanks for sharing! This is indeed unexpected. Could you please share the debug logs with us (running bundle command with --debug)?

@simonfaltum, could you please take a look at what's happening here?