Skip to content

Cel/terraform datacrunch #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chucklever merged 11 commits into from
Dec 16, 2025
Merged

Cel/terraform datacrunch #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d677d36
terraform: Add DataCrunch GPU cloud provider integration
mcgrof Dec 6, 2025
384a6d8
terraform/datacrunch: fix early exit and error suppression in apply_w…
chucklever Dec 14, 2025
1aad135
datacrunch: Use robust config parsing in wrapper scripts
chucklever Dec 14, 2025
4693be2
devconfig: pin uv installer version with checksum verification
chucklever Dec 14, 2025
78200d1
datacrunch: Add timeout to urlopen calls to prevent indefinite hangs
chucklever Dec 14, 2025
17c604f
datacrunch: use git repository root for SSH key name hash
chucklever Dec 14, 2025
b31b017
datacrunch: Fix space handling in volume cache invocation
chucklever Dec 14, 2025
f184648
terraform/datacrunch: use directory checksum in SSH key file path
chucklever Dec 14, 2025
b5125f1
datacrunch: iterate over all locations in on-demand mode
chucklever Dec 14, 2025
e12dad9
terraform/datacrunch: handle save failure in volume cache delete
chucklever Dec 14, 2025
c1ea579
terraform/datacrunch: use home-relative path for ssh_config_name
chucklever Dec 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,4 @@ docs/contrib/kdevops_contributions*
__pycache__/

.ansible
cloud-bill
9 changes: 9 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ ifdef DECLARE_HOSTS
export DECLARED_HOSTS := $(DECLARE_HOSTS)
endif

# Export workflow CLI overrides
ifdef KNLP
export KNLP
endif

ifdef KEEP
export KEEP
endif

include scripts/refs.Makefile

KDEVOPS_NODES_ROLE_TEMPLATE_DIR := $(KDEVOPS_PLAYBOOKS_DIR)/roles/gen_nodes/templates
Expand Down
11 changes: 11 additions & 0 deletions defconfigs/datacrunch-4x-b200
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# DataCrunch 4x B200 (Blackwell) instance - latest GPU architecture
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_4B200_120V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
11 changes: 11 additions & 0 deletions defconfigs/datacrunch-4x-b300
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# DataCrunch 4x B300 (Blackwell) instance - latest GPU architecture
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_4B300_120V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
11 changes: 11 additions & 0 deletions defconfigs/datacrunch-4x-h100-pytorch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# DataCrunch 4x H100 PCIe instance with PyTorch - pay-as-you-go pricing
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_4H100_80S_176V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
11 changes: 11 additions & 0 deletions defconfigs/datacrunch-a100
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# DataCrunch single A100 40GB SXM instance - pay-as-you-go pricing
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_1A100_40S_22V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
13 changes: 13 additions & 0 deletions defconfigs/datacrunch-a100-40-or-less
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# DataCrunch GPU with tier-based fallback (A100-40 maximum tier)
# Uses A100_40_OR_LESS for best available single GPU up to A100-40
# Fallback order: A100-40 → RTX PRO 6000 → RTX 6000 Ada → L40S → RTX A6000 → Tesla V100
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_A100_40_OR_LESS=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
13 changes: 13 additions & 0 deletions defconfigs/datacrunch-a100-80-or-less
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# DataCrunch GPU with tier-based fallback (A100-80 maximum tier)
# Uses A100_80_OR_LESS for best available single GPU up to A100-80
# Fallback order: A100-80 → A100-40 → RTX PRO 6000 → RTX 6000 Ada → L40S → RTX A6000 → Tesla V100
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_A100_80_OR_LESS=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
13 changes: 13 additions & 0 deletions defconfigs/datacrunch-b200-or-less
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# DataCrunch GPU with tier-based fallback (B200 maximum tier)
# Uses B200_OR_LESS for best available single GPU up to B200
# Fallback order: B200 → H100 → A100-80 → A100-40 → RTX PRO 6000 → RTX 6000 Ada → L40S → RTX A6000 → Tesla V100
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_B200_OR_LESS=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
11 changes: 11 additions & 0 deletions defconfigs/datacrunch-b300
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# DataCrunch single NVIDIA Blackwell B300 GPU (latest generation)
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_1B300_30V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
13 changes: 13 additions & 0 deletions defconfigs/datacrunch-b300-or-less
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# DataCrunch GPU with tier-based fallback (B300 maximum tier)
# Uses B300_OR_LESS for best available single GPU (any tier)
# Fallback order: B300 → B200 → H100 → A100-80 → A100-40 → RTX PRO 6000 → RTX 6000 Ada → L40S → RTX A6000 → Tesla V100
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_B300_OR_LESS=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
12 changes: 12 additions & 0 deletions defconfigs/datacrunch-h100-pytorch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# DataCrunch H100 PCIe instance with PyTorch - pay-as-you-go pricing
# Uses ANY_1H100 to automatically select any available single H100 variant
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_ANY_1H100=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
13 changes: 13 additions & 0 deletions defconfigs/datacrunch-h100-pytorch-or-less
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# DataCrunch GPU with tier-based fallback (H100 maximum tier)
# Uses H100_OR_LESS for best available single GPU up to H100
# Fallback order: H100 → A100-80 → A100-40 → RTX PRO 6000 → RTX 6000 Ada → L40S → RTX A6000 → Tesla V100
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_H100_OR_LESS=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
12 changes: 12 additions & 0 deletions defconfigs/datacrunch-v100
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# DataCrunch single Tesla V100 GPU (cheapest option)
# 6 vCPUs, 23 GiB RAM - lowest cost GPU tier
CONFIG_TERRAFORM=y
CONFIG_TERRAFORM_DATACRUNCH=y
CONFIG_TERRAFORM_DATACRUNCH_INSTANCE_TYPE_1V100_6V=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_OVERWRITE=y
CONFIG_TERRAFORM_SSH_CONFIG_GENKEY_EMPTY_PASSPHRASE=y
CONFIG_WORKFLOWS=y
CONFIG_WORKFLOWS_TESTS=y
CONFIG_WORKFLOWS_LINUX_TESTS=y
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
Loading
Loading