Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,14 @@ Every compiled pipeline runs as three sequential jobs:
│ │ └── upload_workitem_attachment.rs
│ ├── runtimes/ # Runtime environment implementations (one dir per runtime)
│ │ ├── mod.rs # Module entry point
│ │ └── lean/ # Lean 4 theorem prover runtime
│ │ ├── mod.rs # Config types, install helpers
│ │ ├── lean/ # Lean 4 theorem prover runtime
│ │ │ ├── mod.rs # Config types, install helpers
│ │ │ └── extension.rs # CompilerExtension impl
│ │ ├── python/ # Python runtime
│ │ │ ├── mod.rs # Config types, install/auth helpers
│ │ │ └── extension.rs # CompilerExtension impl
│ │ └── node/ # Node.js runtime
│ │ ├── mod.rs # Config types, install/auth helpers
│ │ └── extension.rs # CompilerExtension impl
│ ├── data/
│ │ ├── base.yml # Base pipeline template for standalone
Expand Down Expand Up @@ -156,8 +162,8 @@ index to jump to the right page.
in the pipeline UI, including the auto-injected `clearMemory` parameter.
- [`docs/tools.md`](docs/tools.md) — `tools:` configuration (bash allow-list,
`edit`, `cache-memory`, `azure-devops` MCP).
- [`docs/runtimes.md`](docs/runtimes.md) — `runtimes:` configuration (currently
Lean 4).
- [`docs/runtimes.md`](docs/runtimes.md) — `runtimes:` configuration (Lean 4,
Python, Node.js).
- [`docs/targets.md`](docs/targets.md) — target platforms: `standalone` and
`1es`.
- [`docs/safe-outputs.md`](docs/safe-outputs.md) — full reference for every
Expand Down
84 changes: 84 additions & 0 deletions docs/runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,87 @@ When enabled, the compiler:
- Emits a compile-time warning if `tools.bash` is empty (Lean requires bash access)

**Note:** In the 1ES target, the bash command allow-list is updated but elan installation must be done manually via `steps:` front matter. The 1ES target handles network isolation separately.

### Python (`python:`)

Python runtime. Auto-installs Python via `UsePythonVersion@0`, emits `PipAuthenticate@1` for internal feed access, adds Python ecosystem domains to the AWF network allowlist, extends the bash command allow-list, and optionally injects feed URL env vars for pip and uv.

```yaml
# Simple enablement (installs default Python 3.x)
runtimes:
python: true

# With options (pin version, configure feed)
runtimes:
python:
version: "3.12"
feed-url: "https://pkgs.dev.azure.com/myorg/_packaging/myfeed/pypi/simple/"
```

**Fields:**

| Field | Type | Description |
|-------|------|-------------|
| `version` | string | Python version to install (e.g., `"3.12"`, `"3.11"`). Passed to `UsePythonVersion@0` `versionSpec`. Defaults to latest 3.x. |
| `feed-url` | string | Internal PyPI feed URL. Injects `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` env vars into the agent environment. |
| `config` | string | Path to a pip/uv config file. Accepted with a warning — the file will not be available inside the AWF agent environment until proxy-auth support lands. |

When enabled, the compiler:
- Injects `UsePythonVersion@0` into `{{ prepare_steps }}` (runs before AWF)
- If `feed-url` is set, also injects `PipAuthenticate@1` to authenticate the ADO build service identity for internal feeds
- Auto-adds `python`, `python3`, `pip`, `pip3`, `uv` to the bash command allow-list
- Adds Python ecosystem domains to the network allowlist (pypi.org, pythonhosted.org, etc.)
- If `feed-url` is set, injects `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` env vars into the agent environment
- Appends a prompt supplement informing the agent about Python availability
- No AWF mounts or PATH prepends needed — `UsePythonVersion@0` installs to `/opt/hostedtoolcache` (auto-mounted by AWF) and publishes PATH entries that AWF merges via `$GITHUB_PATH`

**Note:** `PipAuthenticate@1` is currently emitted with an empty `artifactFeeds` input, which configures credentials for all feeds accessible to the build service identity. If your internal feed requires scoped authentication to a specific Azure Artifacts feed, this may need future refinement.

### Node.js (`node:`)

Node.js runtime. Auto-installs Node.js via `NodeTool@0`, emits `npmAuthenticate@0` for internal feed access, adds Node ecosystem domains to the AWF network allowlist, extends the bash command allow-list, and optionally injects feed URL env vars for npm.

```yaml
# Simple enablement (installs default Node LTS)
runtimes:
node: true

# With options (pin version, configure feed)
runtimes:
node:
version: "22.x"
feed-url: "https://pkgs.dev.azure.com/ORG/PROJECT/_packaging/FEED/npm/registry/"
```

**Fields:**

| Field | Type | Description |
|-------|------|-------------|
| `version` | string | Node.js version to install (e.g., `"22.x"`, `"20.x"`). Passed to `NodeTool@0` `versionSpec`. Defaults to `"22.x"`. |
| `feed-url` | string | Internal npm registry URL. Injects `NPM_CONFIG_REGISTRY` env var into the agent environment. |
| `config` | string | Path to an .npmrc config file. Accepted with a warning — the file will not be available inside the AWF agent environment until proxy-auth support lands. |

When enabled, the compiler:
- Injects `NodeTool@0` into `{{ prepare_steps }}` (runs before AWF)
- If `feed-url` or `config` is set, also injects `npmAuthenticate@0` (and an ensure-`.npmrc` step) to authenticate the ADO build service identity for internal feeds
- Auto-adds `node`, `npm`, `npx` to the bash command allow-list
- Adds Node ecosystem domains to the network allowlist (npmjs.org, nodejs.org, etc.)
- If `feed-url` is set, injects `NPM_CONFIG_REGISTRY` env var into the agent environment
- Appends a prompt supplement informing the agent about Node.js availability
- No AWF mounts or PATH prepends needed — `NodeTool@0` installs to `/opt/hostedtoolcache` (auto-mounted by AWF) and publishes PATH entries that AWF merges via `$GITHUB_PATH`
- Note: AWF overlays `~/.npmrc` with `/dev/null` for credential security — the `NPM_CONFIG_REGISTRY` env var approach avoids conflicting with this overlay

### Combining Runtimes

Multiple runtimes can be enabled simultaneously:

```yaml
runtimes:
python:
version: "3.12"
node:
version: "22.x"
lean: true
```

All runtime extensions are sorted into `ExtensionPhase::Runtime` and execute before tool extensions (`ExtensionPhase::Tool`), ensuring language toolchains are available before any tools that depend on them.
75 changes: 75 additions & 0 deletions src/compile/common.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1912,6 +1912,71 @@ pub fn collect_awf_path_prepends(extensions: &[super::extensions::Extension]) ->
.collect()
}

/// Collects `agent_env_vars()` from all extensions, validates keys against
/// `BLOCKED_ENV_KEYS`, deduplicates (bails on collision), and formats them
/// as YAML `KEY: "value"` lines for injection into the `{{ engine_env }}` block.
///
/// Returns an empty string if no extensions declare env vars.
pub fn collect_agent_env_vars(extensions: &[super::extensions::Extension]) -> anyhow::Result<String> {
use crate::engine::BLOCKED_ENV_KEYS;
use crate::validate;
use std::collections::HashSet;

let mut lines = Vec::new();
let mut seen_keys = HashSet::new();

for ext in extensions {
for (key, value) in ext.agent_env_vars() {
// Deduplicate: bail on collision
if !seen_keys.insert(key.clone()) {
anyhow::bail!(
"Extension '{}' declares agent env var '{}' which was already declared \
by a previous extension. Each env var key must be unique.",
ext.name(),
key,
);
}

// Validate key is not blocked
if BLOCKED_ENV_KEYS.iter().any(|blocked| key.eq_ignore_ascii_case(blocked)) {
anyhow::bail!(
"Extension '{}' declares agent env var '{}' which conflicts with a \
compiler-controlled environment variable.",
ext.name(),
key,
);
}

// Validate key format
if !validate::is_valid_env_var_name(&key) {
anyhow::bail!(
"Extension '{}' declares agent env var '{}' with invalid key format. \
Keys must contain only ASCII alphanumerics and underscores.",
ext.name(),
key,
);
}

// Validate value for injection (defence in depth — covers ADO expressions,
// pipeline commands, template markers, and newlines)
validate::reject_pipeline_injection(&value, &format!("agent env var '{key}'"))?;

if value.contains('"') || value.contains('\'') {
anyhow::bail!(
"Extension '{}' agent env var '{}' value contains a quote character \
which would produce malformed YAML or bash syntax.",
ext.name(),
key,
);
}

lines.push(format!("{key}: \"{value}\""));
}
}

Ok(lines.join("\n"))
}

// ==================== Shared compile flow ====================

/// Target-specific overrides for the shared compile flow.
Expand Down Expand Up @@ -2104,6 +2169,12 @@ pub async fn compile_shared(
if !awf_path_env.is_empty() {
engine_env = format!("{engine_env}\n{awf_path_env}");
}

// Append extension-declared agent env vars (e.g., PIP_INDEX_URL, NPM_CONFIG_REGISTRY)
let agent_env = collect_agent_env_vars(extensions)?;
if !agent_env.is_empty() {
engine_env = format!("{engine_env}\n{agent_env}");
}
let engine_log_dir = ctx.engine.log_dir();
let acquire_write_token = generate_acquire_ado_token(
front_matter
Expand Down Expand Up @@ -2505,6 +2576,8 @@ mod tests {
});
fm.runtimes = Some(crate::compile::types::RuntimesConfig {
lean: Some(crate::runtimes::lean::LeanRuntimeConfig::Enabled(true)),
python: None,
node: None,
});
let params = CompileContext::for_test(&fm).engine.args(&fm, &crate::compile::extensions::collect_extensions(&fm)).unwrap();
assert!(params.contains("shell(lean)"), "lean command should be allowed");
Expand All @@ -2525,6 +2598,8 @@ mod tests {
});
fm.runtimes = Some(crate::compile::types::RuntimesConfig {
lean: Some(crate::runtimes::lean::LeanRuntimeConfig::Enabled(true)),
python: None,
node: None,
});
let params = CompileContext::for_test(&fm).engine.args(&fm, &crate::compile::extensions::collect_extensions(&fm)).unwrap();
assert!(params.contains("--allow-all-tools"), "wildcard should use --allow-all-tools");
Expand Down
28 changes: 28 additions & 0 deletions src/compile/extensions/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,17 @@ pub trait CompilerExtension {
fn awf_path_prepends(&self) -> Vec<String> {
vec![]
}

/// Environment variables to inject into the agent execution environment.
///
/// Returns `(key, value)` pairs that are emitted as `KEY: "value"` in
/// the `{{ engine_env }}` YAML block. Used by runtimes to configure
/// package managers via env vars (e.g., `PIP_INDEX_URL`, `NPM_CONFIG_REGISTRY`).
///
/// Keys are validated against `BLOCKED_ENV_KEYS` at collection time.
fn agent_env_vars(&self) -> Vec<(String, String)> {
vec![]
}
}

/// Mount access mode for an AWF bind mount.
Expand Down Expand Up @@ -534,6 +545,9 @@ macro_rules! extension_enum {
fn awf_path_prepends(&self) -> Vec<String> {
match self { $( $Enum::$Variant(e) => e.awf_path_prepends(), )+ }
}
fn agent_env_vars(&self) -> Vec<(String, String)> {
match self { $( $Enum::$Variant(e) => e.agent_env_vars(), )+ }
}
}
};
}
Expand All @@ -547,6 +561,8 @@ pub use crate::tools::azure_devops::AzureDevOpsExtension;
pub use crate::tools::cache_memory::CacheMemoryExtension;
pub use github::GitHubExtension;
pub use crate::runtimes::lean::LeanExtension;
pub use crate::runtimes::node::NodeExtension;
pub use crate::runtimes::python::PythonExtension;
pub use safe_outputs::SafeOutputsExtension;
pub use trigger_filters::TriggerFiltersExtension;

Expand All @@ -559,6 +575,8 @@ extension_enum! {
GitHub(GitHubExtension),
SafeOutputs(SafeOutputsExtension),
Lean(LeanExtension),
Python(PythonExtension),
Node(NodeExtension),
AzureDevOps(AzureDevOpsExtension),
CacheMemory(CacheMemoryExtension),
TriggerFilters(TriggerFiltersExtension),
Expand Down Expand Up @@ -593,6 +611,16 @@ pub fn collect_extensions(front_matter: &FrontMatter) -> Vec<Extension> {
extensions.push(Extension::Lean(LeanExtension::new(lean.clone())));
}
}
if let Some(python) = front_matter.runtimes.as_ref().and_then(|r| r.python.as_ref()) {
if python.is_enabled() {
extensions.push(Extension::Python(PythonExtension::new(python.clone())));
}
}
if let Some(node) = front_matter.runtimes.as_ref().and_then(|r| r.node.as_ref()) {
if node.is_enabled() {
extensions.push(Extension::Node(NodeExtension::new(node.clone())));
}
}

// ── First-party tools (ExtensionPhase::Tool) ──
if let Some(tools) = front_matter.tools.as_ref() {
Expand Down
Loading
Loading