feat: use a virtualenv around all python Make targets

CLAUDE.md

@@ -0,0 +1,117 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This repository contains specifications and deployment scripts for managing Open edX REST API routing through an API management layer. **This repository does not contain source code for running an API management service** - it contains Swagger/OpenAPI specifications and AWS API Gateway deployment scripts.
+
+The primary purpose is to:
+- Define a unified API interface for Open edX REST endpoints
+- Provide deployment automation for AWS API Gateway
+- Enable routing to various Open edX services (edx-platform, IDAs) through a single access point
+
+## Development Commands
+
+### Setup
+
+This project requires Python 3.11. The Makefile will automatically:
+1. Check if Python 3.11 is installed
+2. On Ubuntu/Debian systems with apt-get: automatically install Python 3.11 via deadsnakes PPA (requires sudo)
+3. On other systems: provide installation instructions
+
+```bash
+make venv            # Create Python 3.11 virtualenv (auto-created by other targets)
+make requirements    # Install Python dependencies for local development
+```
+
+**Note**: On Ubuntu, the first run will install Python 3.11, python3.11-venv, and python3.11-dev packages via apt-get.
+
+### Testing
+```bash
+make test           # Run all tests (quality, Python tests, and Swagger tests)
+make test_python    # Run Python tests only
+make test_swagger   # Spin up stub server and run integration tests
+```
+
+### Quality Checks
+```bash
+make quality        # Run PEP8 and Pylint on scripts/aws directory
+```
+
+### Building Swagger Documentation
+```bash
+make build          # Flatten Swagger docs into build artifacts
+                    # Requires Java 7+ installed
+                    # Downloads swagger-codegen-cli.jar and generates flattened docs
+```
+
+### Dependency Management
+```bash
+make upgrade        # Update all requirements/*.txt files with latest packages
+```
+
+### Cleanup
+```bash
+make clean          # Remove Python bytecode and build artifacts
+```
+
+## Architecture
+
+### Swagger Specifications
+
+The API definitions use nested Swagger 2.0 specifications with remote references:
+
+- **`swagger/api.yaml`**: Main specification that defines the complete Open edX public API surface
+- **`swagger/index.yaml`**: Index endpoint specification
+- **`swagger/heartbeat.yaml`**: Health check endpoint
+- **`swagger/oauth.yaml`**: OAuth2 token endpoint
+
+The main `api.yaml` uses `$ref` to pull in both local files and remote specifications from upstream services (e.g., course-discovery, edx-enterprise). This allows service teams to maintain their own API specs while the api-manager composes them into a unified interface.
+
+### AWS Deployment Scripts
+
+Located in `scripts/aws/`, these Python scripts manage AWS API Gateway deployments using a ring deployment strategy:
+
+1. **`bootstrap.py`**: Creates a new API Gateway RestApi and deploys a hello-world bootstrap stage
+2. **`deploy.py`**: Uploads flattened Swagger to API Gateway in the next stage of the ring rotation
+3. **`flip.py`**: Updates the custom domain to point to a specific stage (activation/rollback)
+4. **`monitor.py`**: Monitoring utilities for API Gateway instances
+5. **`common/deploy.py`**: Shared deployment logic (stage rotation, API updates, throttling configuration)
+
+#### Deployment Flow
+
+1. Bootstrap: Create initial API Gateway with custom domain
+2. Build: Generate flattened Swagger JSON using swagger-codegen
+3. Deploy: Upload to next stage in ring (e.g., red → black → red)
+4. Flip: Point domain to new stage or rollback to previous
+
+#### Ring Deployment Strategy
+
+The deployment scripts use ordered stages (e.g., "red" and "black") to enable zero-downtime deployments. The live stage serves traffic while the next stage is updated, then traffic is flipped to the new stage.
+
+### Test Structure
+
+- **`scripts/aws/tests/`**: Unit tests for deployment scripts (bootstrap, deploy, flip)
+- **`tests/`**: Integration tests that validate Swagger specifications against a stub server
+
+## AWS Configuration
+
+When working with AWS deployment scripts, ensure these environment variables are set:
+- `AWS_REGION`
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+
+## Key Files
+
+- **`Makefile`**: Build, test, and quality commands
+- **`swagger/api.yaml`**: Main API specification with nested references
+- **`scripts/aws/bootstrap.json`**: Minimal hello-world API for bootstrap stage
+- **`.pep8`** and **`.pylintrc`**: Code quality configuration
+
+## Important Notes
+
+- Java 7+ is required for building Swagger documentation
+- The repository uses pip-tools for dependency management (requirements/*.in → requirements/*.txt)
+- AWS API Gateway is the reference implementation, but the specs are vendor-agnostic
+- Stage variables in API Gateway must be configured for your specific Open edX installation

Makefile

@@ -1,32 +1,72 @@
 SWAGGER_CODEGEN_JAR := https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.4.13/swagger-codegen-cli-2.4.13.jar
 BUILD_OUTPUT_DIR := swagger-build-artifacts
 STUB_SERVER_DIR := edx-api-stub-server
+VENV := venv
+PYTHON_VERSION := 3.11
+PYTHON := $(VENV)/bin/python
+PIP := $(VENV)/bin/pip
+
+# Detect available Python and OS
+PYTHON311 := $(shell command -v python3.11 2> /dev/null)
+HAS_APT := $(shell command -v apt-get 2> /dev/null)
 
 help:
 	@echo '                                                                                        '
 	@echo 'Makefile for api-manager                                                                '
 	@echo '                                                                                        '
 	@echo 'Usage:                                                                                  '
+	@echo '    make venv             Create Python 3.11 virtualenv                                 '
 	@echo '    make clean            Delete generated python byte code and testing remnants        '
 	@echo '    make requirements     Install requirements for local development                    '
 	@echo '    make quality          Run PEP8 and Pylint                                           '
 	@echo '    make build            Flatten the swagger docs                                      '
 	@echo '    make test             Run all tests                                                 '
 	@echo '                                                                                        '
 
+venv:
+	@echo "Checking for Python $(PYTHON_VERSION)..."
+ifndef PYTHON311
+	@echo "Python 3.11 not found, attempting to install..."
+ifeq ($(HAS_APT),)
+	@echo "ERROR: Python $(PYTHON_VERSION) is not installed and apt-get is not available."
+	@echo ""
+	@echo "Please install Python $(PYTHON_VERSION) manually:"
+	@echo "  macOS (using Homebrew): brew install python@$(PYTHON_VERSION)"
+	@echo ""
+	@exit 1
+endif
+	@echo "Installing Python $(PYTHON_VERSION) using apt-get..."
+	sudo apt-get update
+	sudo apt-get install -y software-properties-common
+	sudo add-apt-repository -y ppa:deadsnakes/ppa
+	sudo apt-get update
+	sudo apt-get install -y python$(PYTHON_VERSION) python$(PYTHON_VERSION)-venv python$(PYTHON_VERSION)-dev
+	@echo "Python $(PYTHON_VERSION) installed successfully"
+else
+	@echo "Found python3.11 at $(PYTHON311)"
+endif
+	@if [ ! -d "$(VENV)" ]; then \
+		echo "Creating virtual environment in $(VENV)"; \
+		python3.11 -m venv $(VENV); \
+		$(PIP) install --upgrade pip; \
+	else \
+		echo "Virtual environment $(VENV) already exists, skipping creation"; \
+	fi
+
 clean:
 	find . -name '*.pyc' -delete
 	find . -name '__pycache__' -type d -delete
 	rm -rf $(BUILD_OUTPUT_DIR)
 	rm -rf $(STUB_SERVER_DIR)
+	rm -rf $(VENV)
 
-requirements:
-	pip install -r requirements/pip.txt
-	pip install -qr requirements/test.txt --exists-action w
+requirements: venv
+	$(PIP) install -r requirements/pip.txt
+	$(PIP) install -qr requirements/test.txt --exists-action w
 
-quality:
-	pep8 --config=.pep8 scripts/aws
-	pylint --rcfile=.pylintrc scripts/aws
+quality: venv requirements
+	$(VENV)/bin/pep8 --config=.pep8 scripts/aws
+	$(VENV)/bin/pylint --rcfile=.pylintrc scripts/aws
 
 # Download the swagger codegen jar
 # TODO: verify via checksum that the file is valid.
@@ -43,15 +83,15 @@ build-gocd: codegen.download
 	/gocd-jre/bin/java -jar swagger-codegen-cli.jar generate -l swagger -i swagger/api.yaml -o $(BUILD_OUTPUT_DIR)
 
 test_python: clean requirements
-	cd scripts/aws && python -m pytest
+	cd scripts/aws && ../../$(PYTHON) -m pytest
 
 # Spin up a stub server and hit it with tests
-test_swagger: codegen.download
+test_swagger: codegen.download venv requirements
 	java -jar swagger-codegen-cli.jar generate -l nodejs-server -i swagger/api.yaml -o $(STUB_SERVER_DIR)
 	cd $(STUB_SERVER_DIR) && npm install
 	cd $(STUB_SERVER_DIR) && NODE_ENV=development node index.js 2> /dev/null &
 	sleep 5
-	pyresttest --url=http://localhost:8080 --test=tests/test_all.yaml
+	$(VENV)/bin/pyresttest --url=http://localhost:8080 --test=tests/test_all.yaml
 	killall -9 node
 
 test: clean
@@ -60,14 +100,14 @@ test: clean
 	make test_swagger
 
 # Targets in a Makefile which do not produce an output file with the same name as the target name
-.PHONY: help requirements clean quality test
+.PHONY: help venv requirements clean quality test codegen.download build build-gocd test_python test_swagger upgrade
 
 upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade
-upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
-	pip install -q -r requirements/pip_tools.txt
-	pip-compile --upgrade --allow-unsafe --rebuild -o requirements/pip.txt requirements/pip.in
-	pip-compile --upgrade -o requirements/pip_tools.txt requirements/pip_tools.in
-	pip install -qr requirements/pip.txt
-	pip install -qr requirements/pip_tools.txt
-	pip-compile --upgrade -o requirements/base.txt requirements/base.in
-	pip-compile --upgrade -o requirements/test.txt requirements/test.in
+upgrade: venv ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
+	$(PIP) install -q -r requirements/pip_tools.txt
+	$(VENV)/bin/pip-compile --upgrade --allow-unsafe --rebuild -o requirements/pip.txt requirements/pip.in
+	$(VENV)/bin/pip-compile --upgrade -o requirements/pip_tools.txt requirements/pip_tools.in
+	$(PIP) install -qr requirements/pip.txt
+	$(PIP) install -qr requirements/pip_tools.txt
+	$(VENV)/bin/pip-compile --upgrade -o requirements/base.txt requirements/base.in
+	$(VENV)/bin/pip-compile --upgrade -o requirements/test.txt requirements/test.in

requirements/base.txt

@@ -6,9 +6,9 @@
 #
 boto==2.49.0
     # via google-compute-engine
-boto3==1.42.31
+boto3==1.42.36
     # via -r requirements/base.in
-botocore==1.42.31
+botocore==1.42.36
     # via
     #   -r requirements/base.in
     #   boto3
@@ -19,7 +19,7 @@ google-compute-engine==2.8.13
     # via -r requirements/base.in
 jinja2==3.1.6
     # via -r requirements/base.in
-jmespath==1.0.1
+jmespath==1.1.0
     # via
     #   -r requirements/base.in
     #   boto3

requirements/pip.txt

@@ -4,11 +4,13 @@
 #
 #    make upgrade
 #
-wheel==0.45.1
+packaging==26.0
+    # via wheel
+wheel==0.46.3
     # via -r requirements/pip.in
 
 # The following packages are considered to be unsafe in a requirements file:
 pip==25.3
     # via -r requirements/pip.in
-setuptools==80.9.0
+setuptools==80.10.2
     # via -r requirements/pip.in

requirements/pip_tools.txt

@@ -8,15 +8,17 @@ build==1.4.0
     # via pip-tools
 click==8.3.1
     # via pip-tools
-packaging==25.0
-    # via build
+packaging==26.0
+    # via
+    #   build
+    #   wheel
 pip-tools==7.5.2
     # via -r requirements/pip_tools.in
 pyproject-hooks==1.2.0
     # via
     #   build
     #   pip-tools
-wheel==0.45.1
+wheel==0.46.3
     # via pip-tools
 
 # The following packages are considered to be unsafe in a requirements file:

requirements/test.txt

@@ -20,12 +20,12 @@ boto==2.49.0
     # via
     #   -r requirements/base.txt
     #   google-compute-engine
-boto3==1.42.31
+boto3==1.42.36
     # via
     #   -r requirements/base.txt
     #   aws-sam-translator
     #   moto
-botocore==1.42.31
+botocore==1.42.36
     # via
     #   -r requirements/base.txt
     #   aws-xray-sdk
@@ -40,7 +40,7 @@ cfn-lint==1.41.0
     # via moto
 charset-normalizer==3.4.4
     # via requests
-cryptography==46.0.3
+cryptography==46.0.4
     # via
     #   joserfc
     #   moto
@@ -68,7 +68,7 @@ jinja2==3.1.6
     # via
     #   -r requirements/base.txt
     #   moto
-jmespath==1.0.1
+jmespath==1.1.0
     # via
     #   -r requirements/base.txt
     #   boto3
@@ -111,7 +111,7 @@ openapi-schema-validator==0.6.3
     # via openapi-spec-validator
 openapi-spec-validator==0.7.2
     # via moto
-packaging==25.0
+packaging==26.0
     # via pytest
 pathable==0.4.4
     # via jsonschema-path
@@ -123,7 +123,7 @@ pluggy==1.6.0
     # via pytest
 py-partiql-parser==0.6.3
     # via moto
-pycparser==2.23
+pycparser==3.0
     # via cffi
 pycurl==7.45.7
     # via pyresttest
@@ -135,7 +135,7 @@ pygments==2.19.2
     # via pytest
 pylint==4.0.4
     # via -r requirements/test.in
-pyparsing==3.3.1
+pyparsing==3.3.2
     # via moto
 pyresttest==1.7.1
     # via -r requirements/test.in