Error in `future::evalFuture()` after version `1.67`

[emilianomm]

Describe the bug

Hi. Just wanted to report that my previously working code started to fail since future 1.67. The code runs inside a databricks scheduled job that pulls the newest version of the package every time it runs. This job started failing just after the 1.67 update. I can confirm that it runs successfully with the default version installed on the Databricks Runtime (i.e future 1.38) and everything else constant.

Here is the exact error message:

Error : future::evalFuture() failed on 1229-16200004-99lhyj44gy-10-1-12-13 (pid 44) at 2025-11-24T22:40:27. Using package 'future' v1.68.0. Possible other reasons: cannot change working directory

Reproduce example

A reproducible example using R code.

library(tidyverse)
library(bigrquery)

# Create a generic example dataset
set.seed(123)
example_data <- tibble(
  group1 = rep(letters[1:3], each = 5),
  group2 = rep(LETTERS[1:2], times = 8),
  value = rnorm(15)
)

# Split data by groups
data_chunks <- example_data %>%
  group_by(group1, group2) %>%
  group_split()

tbl_bq <- bq_table(
  "my_project",
  "my_dataset",
  "my_table"
)

bq_table_create(
  tbl_bq,
  as_bq_fields(example_data)
)

# Set up parallel processing
future::plan(future::multisession, workers = future::availableCores() - 1)


# Walk over chunks in parallel
furrr::future_walk(
  data_chunks,
  function(chunk) {
    bq_auth(path = path)
    bq_table_upload(
      table = tbl_bq,
      data = distinct(chunk),
      write_disposition = "WRITE_APPEND"
    )
  }
)

# Reset plan to sequential after
future::plan(future::sequential)

Expected behavior

Run as before future 1.67

Session information
Since the error was not raised in an interactive session, I can not provide session info. Anyway I am able to provide the following details:

1. It runs inside Azure Databricks with 17.3 Runtime
2. It pulls latest version of the future package using databricks's built in package manager (Which is based on posit package manager) at cluster level
3. The MRE should give you a pretty accurate idea of what is being loaded in the session. Note that for future as well furrr I use only qualified names thus not importing the package with library()

[HenrikBengtsson]

There was a big internal redesign in future 1.40.0 (https://future.futureverse.org/news/index.html), which is meant to be backward compatible. The updates since have undergone lots of scrutiny, lots of testing, and user feedback. But, it could very well be that you're affected.

... Possible other reasons: cannot change working directory

That suggests that there is a setwd(some_path) somewhere in the code for a non-existing some_path folder.

There is one setwd() statement in the future ecosystem, but I don't see how that would come in play here (I've been wrong before).

... i.e future 1.38

There has never been a future 1.38 release, cf. https://cran.r-project.org/src/contrib/Archive/future/. If you meant 1.58.0, then that's an important difference, it limits what could have changed, and changes the whole discussion here

PS. future::availableCores(omit = 1) is preferred over future::availableCores() - 1 for reasons explain in https://parallelly.futureverse.org/reference/availableCores.html#avoid-ending-up-with-zero-cores

[HenrikBengtsson]

... i.e future 1.38

@emilianomm, I haven't heard back from you regarding this comment of yours - that version does not exist and it's essential to know which version you last used in order to help you here.

[emilianomm]

@HenrikBengtsson Hi. Thanks for your reply.

The actual version of the future package used in the databricks runtime is 1.34.0 as described in the docs. Sorry for the misspelling.

Anyway, I can confirm that with this regression the code runs successfully, but as I pointed out earlier my original implementation pulled the latest package available on cran every time it ran using install.package("future"). So I think only changes since 1.58 are relevant for this case.

Kind regards

[HenrikBengtsson]

Thanks for this. This turns the focus on the internal rewrites in future (>= 1.40.0). I'd like to try to reproduce your example, but when I run your code example as-is, I get:

Error in `tibble()`:
! Tibble columns must have compatible sizes.
• Size 15: Existing data.
• Size 16: Column `group2`.
ℹ Only values of size one are recycled.

After fixing that with:

example_data <- tibble(
  group1 = rep(letters[1:3], length.out = 15L),
  group2 = rep(LETTERS[1:2], length.out = 15L),
  value = rnorm(15)
)

I get another error from bq_table_create() saying:

Error in `bq_post()`:
! Invalid resource name projects/my_project; Project id: my_project [badRequest]

Could you please try to provide me with an example that works for any R user in the world?

My first goal is to identify the culprit here;

1. Is it specific to bigrquery?
2. Is it specific to Azure Databricks?

Also, there's a bq_auth(path = path) where path is undefined.

[HenrikBengtsson]

@emilianomm , just a friendly reminder that I need a cut'n'pastable small example that I can work with, e.g. reproduce it on my end. If you could provide me with that, that would be great. Thx.

[emilianomm]

Hi

Since both Databricks and BigQuery are subscription dependent services, I am not able to provide a public access to those services ATM. I can, however, provide a few clarifications in case it would eventually help further inquiry.

1. path

The variable path is a single-line string loaded from a Google Cloud Service Account Key

I have generated an example file with an AI bot for illustrative purposes

{
  "type": "service_account",
  "project_id": "my-data-project-123",
  "private_key_id": "a1b2c3d4e5f6...",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC...\n-----END PRIVATE KEY-----\n",
  "client_email": "etl-loader@my-data-project-123.iam.gserviceaccount.com",
  "client_id": "123456789012345678901",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/etl-loader%40my-data-project-123.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

2. Error in bq_post()`

This error raises because the SQL table 3-part my_project.my_dataset.my_table cannot be found, because it is a mock. Once you have a working connection the correct identifier for the table should be replaced in

tbl_bq <- bq_table(
  "my_project", # Put the correct project here
  "my_dataset", # The dataset here
  "my_table" # The table here
)`

Other observations

If the error is related with Databricks/BigQuery it is most likely to be the former, since that is the platform that supports the environment from where the code is called.

I hope this helps in case someone is able to contribute with a working connection to bigquery or a databricks workspace to pursuit this further.

In any case, if someone is facing this same issue may this report be valuable to help them with my suggested workaround.

Cheers!