--shm-size with Google Batch provider

[Rnsras]

Hello,

The current tool requires the --shm-size 12gb flag when starting the container to function properly (run_pangenome_aware_deepvariant at image google/deepvariant:pangenome_aware_deepvariant-sbx-gpu).

I referenced the #191 and it mentioned that container shared memory size would be the total memory size.

but when I created an environment using --machine-type n2-standard-96 , the system still only had 64MB of RAM.

am I missing some parameters?

Thank you for your help!

[mbookman]

Hey @Rnsras!

Let's see what we can figure out here. First on the question of only having 64MB of RAM, what is the evidence that leads you to say that. Also, when you say "the system", do you mean the VM or the container? If so, how are you determining that?

This discussion:

https://discuss.google.dev/t/having-problems-with-runnable-container-options-in-batch-script/161264

indicates that one can pass the shared memory option to the container using the runnable.container.options field. You could test this by adding code in ‎dsub/providers/google_batch_operations.py‎ to the build_container() function. Something like:

def build_container(
    image_uri: str, entrypoint: str, volumes: List[str], commands: List[str]
) -> batch_v1.types.task.Runnable.Container:
  container = batch_v1.types.task.Runnable.Container()
  container.image_uri = image_uri
  container.entrypoint = entrypoint
  container.commands = commands
  container.volumes = volumes
  container.options = "--shm-size 12gb"
  return container

[Rnsras]

Hey @mbookman :

I try to create a job just like:

dsub \
  --unique-job-id \
  --provider google-batch \
  --project test_rosser \
  --regions asia-east1 \
  --location asia-east1 \
  --logging gs://test_rosser/log \
  --image google/deepvariant:pangenome_aware_deepvariant-sbx-gpu \
  --preemptible \
  --boot-disk-size 60 \
  --disk-size 342 \
  --machine-type n2-standard-96 \
  --command "df -h"

and I got shm size:

shm 64M 0 64M 0% /dev/shm

how can I increase shm size inside the job's Docker container?

[mbookman]

Thank-you for the details. We don't have a way to pass this on the command-line, but you can update the source as indicated in #320 (comment) above. If you aren't already running from source, you can find instructions in the README.

[Rnsras]

So there is no parameters on dsub could increase shm size inside the job's Docker container?

the only way is update source code?

#191 means it only works on google-cls-v2 provider?

[mbookman]

There is currently no parameter for this. We can add a parameter for the container options, but we'll need to schedule that work.

The google-cls-v2 provider no longer exists as the Google Cloud Lifesciences API was was turned down.

[Rnsras]

Got it. I modified the source code according to #320 (comment) and it works successfully.

Thanks.

[carbocation]

Minor point but this seems to behave like a breaking API change for pytorch jobs, possibly due to how the dataloader shared memory works. (I don't have a full trace or a strong understanding of the difference here—just an observation that previously-successful jobs now fail with the message ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)..)

[carbocation]

Cribbing from @mbookman 's approach, this lets you choose whether to modify the default based on an environmental variable:

import os

def build_container(
    image_uri: str, entrypoint: str, volumes: List[str], commands: List[str]
) -> batch_v1.types.task.Runnable.Container:
  container = batch_v1.types.task.Runnable.Container()
  container.image_uri = image_uri
  container.entrypoint = entrypoint
  container.commands = commands
  container.volumes = volumes

  # Temporary workaround: only set container.options if the variable is defined
  # in the environment and non-empty:
  shm_size = os.environ.get("DSUB_BATCH_SHM_SIZE")
  if shm_size:
      # Support values like "12g", "8g", "2G", "1024m", etc.
      container.options = f"--shm-size={shm_size}"

  return container

I made a fork with this that you can install:

python3 -m pip uninstall dsub -y
python3 -m pip install git+https://github.com/carbocation/dsub.git@shm 

Then set an environment variable and invoke dsub:

export DSUB_BATCH_SHM_SIZE="12g"

dsub ...