Skip to content

Commit f86c995

Browse files
Addressing all feedback
1 parent 7cfb745 commit f86c995

15 files changed

Lines changed: 55 additions & 317 deletions

docs/lambda-features/durable-functions.md

Lines changed: 16 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -14,54 +14,44 @@ description: Using Powertools for AWS Lambda (Python) with Lambda Durable Functi
1414
| **Durable execution** | Complete lifecycle of a durable function, from start to completion |
1515
| **Checkpoint** | Saved state that tracks progress through the workflow |
1616
| **Replay** | Re-execution from the beginning, skipping completed checkpoints |
17-
| **Steps** | Business logic with built-in retries and progress tracking |
18-
| **Waits** | Suspend execution without incurring compute charges |
17+
| **Step** | Business logic with built-in retries and progress tracking |
18+
| **Wait** | Suspend execution without incurring compute charges |
1919

2020
## How it works
2121

2222
Durable functions use a **checkpoint/replay mechanism**:
2323

24-
1. Your code runs from the beginning
24+
1. Your code runs always from the beginning
2525
2. Completed operations are skipped using stored results
26-
3. Execution continues from where it left off
26+
3. Execution of new steps continues from where it left off
2727
4. State is automatically managed by the SDK
2828

2929
## Powertools integration
3030

31-
Powertools for AWS Lambda (Python) works seamlessly with Durable Functions. The [Durable Execution SDK](https://github.com/aws/aws-durable-execution-sdk-python){target="_blank" rel="nofollow"} has native integration with Powertools Logger via `context.set_logger()`.
31+
Powertools for AWS Lambda (Python) works seamlessly with Durable Functions. The [Durable Execution SDK](https://github.com/aws/aws-durable-execution-sdk-python){target="_blank" rel="nofollow"} has native integration with Logger via `context.set_logger()`.
3232

3333
???+ note "Found an issue?"
34-
If you encounter any issues using Powertools for AWS with Durable Functions, please [open an issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new?template=bug_report.yml){target="_blank"}.
34+
If you encounter any issues using Powertools for AWS Lambda (Python) with Durable Functions, please [open an issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new?template=bug_report.yml){target="_blank"}.
3535

3636
### Logger
3737

38-
The Durable Execution SDK provides a `context.logger` that automatically handles **log deduplication during replays**. You can integrate Powertools Logger to get structured JSON logging while keeping the deduplication benefits.
38+
The Durable Execution SDK provides a `context.logger` instance that automatically handles **log deduplication during replays**. You can integrate Logger to get structured JSON logging while keeping the deduplication benefits.
3939

40-
#### Using Powertools Logger with context.set_logger
40+
For the best experience, set the Logger on the durable context. This gives you structured JSON logging with automatic log deduplication during replays:
4141

42-
For the best experience, set the Powertools Logger on the durable context:
43-
44-
```python hl_lines="5 10" title="Integrating Powertools Logger with Durable Functions"
42+
```python hl_lines="5 11 14 22" title="Integrating Logger with Durable Functions"
4543
--8<-- "examples/lambda_features/durable_functions/src/using_logger.py"
4644
```
4745

4846
This gives you:
4947

50-
- **JSON structured logging** from Powertools for AWS
48+
- **JSON structured logging** from Powertools for AWS Lambda (Python)
5149
- **Log deduplication** during replays (logs from completed operations don't repeat)
5250
- **Automatic SDK enrichment** (execution_arn, parent_id, name, attempt)
5351
- **Lambda context injection** (request_id, function_name, etc.)
5452

55-
#### Log deduplication during replay
56-
57-
When you use `context.logger`, the SDK prevents duplicate logs during replays:
58-
59-
```python title="Log deduplication behavior"
60-
--8<-- "examples/lambda_features/durable_functions/src/log_deduplication.py"
61-
```
62-
6353
???+ warning "Direct logger usage"
64-
If you use the Powertools Logger directly (not through `context.logger`), logs will be emitted on every replay:
54+
If you use the Logger directly (not through `context.logger`), logs will be emitted on every replay:
6555

6656
```python
6757
# Logs will duplicate during replays
@@ -76,23 +66,20 @@ When you use `context.logger`, the SDK prevents duplicate logs during replays:
7666
Tracer works with Durable Functions. Each execution creates trace segments.
7767

7868
???+ note "Trace continuity"
79-
Due to the replay mechanism, traces may not show a continuous flow. Each execution (including replays) creates separate trace segments. Use the `execution_arn` to correlate traces.
69+
Due to the replay mechanism, traces may be interleaved. Each execution (including replays) creates separate trace segments. Use the `execution_arn` to correlate traces.
8070

8171
```python hl_lines="5 9" title="Using Tracer with Durable Functions"
8272
--8<-- "examples/lambda_features/durable_functions/src/using_tracer.py"
8373
```
8474

8575
### Metrics
8676

87-
Metrics work with Durable Functions, but be aware that **metrics may be emitted multiple times** during replay if not handled carefully.
77+
Metrics work with Durable Functions, but be aware that **metrics may be emitted multiple times** during replay if not handled carefully. Emit metrics at workflow completion rather than during intermediate steps to avoid counting replays as new executions.
8878

89-
```python hl_lines="6 10 21" title="Using Metrics with Durable Functions"
90-
--8<-- "examples/lambda_features/durable_functions/src/using_metrics.py"
79+
```python hl_lines="6 9 18 19 20 21" title="Using Metrics with Durable Functions"
80+
--8<-- "examples/lambda_features/durable_functions/src/best_practice_metrics.py"
9181
```
9282

93-
???+ tip "Accurate metrics"
94-
Emit metrics at workflow completion rather than during intermediate steps to avoid counting replays as new executions.
95-
9683
### Idempotency
9784

9885
The `@idempotent` decorator integrates with Durable Functions and is **replay-aware**. It's useful for protecting the Lambda handler entry point, especially for Event Source Mapping (ESM) invocations like SQS, Kinesis, or DynamoDB Streams.
@@ -111,14 +98,6 @@ The `@idempotent` decorator integrates with Durable Functions and is **replay-aw
11198

11299
- Steps within a durable function are already idempotent via the checkpoint mechanism
113100

114-
### Parser
115-
116-
Parser works with Durable Functions for validating and parsing event payloads.
117-
118-
```python hl_lines="9 14" title="Using Parser with Durable Functions"
119-
--8<-- "examples/lambda_features/durable_functions/src/using_parser.py"
120-
```
121-
122101
### Parameters
123102

124103
Parameters work normally with Durable Functions.
@@ -128,26 +107,10 @@ Parameters work normally with Durable Functions.
128107
```
129108

130109
???+ note "Parameter freshness"
131-
For long-running workflows (hours/days), parameters fetched at the start may become stale. Consider fetching parameters within steps that need the latest values.
110+
If the replay or execution happens within the cache TTL on the same execution environment, the parameter value may come from cache. For long-running workflows (hours/days), parameters fetched at the start may become stale. Consider fetching parameters within steps that need the latest values, and customize the caching behavior with `max_age` to control freshness.
132111

133112
## Best practices
134113

135-
### Use context.logger for log deduplication
136-
137-
Always use `context.set_logger()` and `context.logger` instead of using the Powertools Logger directly. This ensures logs are deduplicated during replays.
138-
139-
```python title="Recommended logging pattern"
140-
--8<-- "examples/lambda_features/durable_functions/src/best_practice_logging.py"
141-
```
142-
143-
### Emit metrics at workflow completion
144-
145-
To avoid counting replays as new executions, emit metrics only when the workflow completes successfully.
146-
147-
```python title="Metrics at completion"
148-
--8<-- "examples/lambda_features/durable_functions/src/best_practice_metrics.py"
149-
```
150-
151114
### Use Idempotency for ESM triggers
152115

153116
When your durable function is triggered by Event Source Mappings (SQS, Kinesis, DynamoDB Streams), use the `@idempotent` decorator to protect against duplicate invocations.

docs/lambda-features/managed-instances.md

Lines changed: 17 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -7,84 +7,52 @@ description: Using Powertools for AWS Lambda (Python) with Lambda Managed Instan
77

88
[Lambda Managed Instances](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances.html){target="_blank" rel="nofollow"} enables you to run Lambda functions on Amazon EC2 instances without managing infrastructure. It supports multi-concurrent invocations, EC2 pricing models, and specialized compute options like Graviton4.
99

10-
## Key differences from Lambda (default)
10+
## Key differences from Lambda On Demand
1111

12-
| Aspect | Lambda (default) | Lambda Managed Instances |
12+
| Aspect | Lambda On Demand | Lambda Managed Instances |
1313
| ---------------- | ------------------------------------------- | ----------------------------------------------- |
1414
| **Concurrency** | Single invocation per execution environment | Multiple concurrent invocations per environment |
1515
| **Python model** | One process, one request | Multiple processes, one request each |
1616
| **Pricing** | Per-request duration | EC2-based with Savings Plans support |
17-
| **Scaling** | Scale on demand with cold starts | Async scaling based on CPU, no cold starts |
17+
| **Scaling** | Scale on demand with cold starts | Async scaling based on CPU |
1818
| **Isolation** | Firecracker microVMs | Containers on EC2 Nitro |
1919

2020
## How Lambda Python runtime handles concurrency
2121

22-
Unlike Java or Node.js which use threads, the **Lambda Python runtime uses multiple processes** for concurrent requests. Each request runs in a separate process, which provides natural isolation between requests.
22+
The **Lambda Python runtime uses multiple processes** for concurrent requests. Each request runs in a separate process, which provides natural isolation between requests.
2323

2424
This means:
2525

26-
- **Memory is not shared** between concurrent requests
27-
- **Global variables** are isolated per process
26+
- **Each process has its own memory** - global variables are isolated per process
2827
- **`/tmp` directory is shared** across all processes - use caution with file operations
2928

30-
## Isolation model
31-
32-
Lambda Managed Instances use a different isolation model than Lambda (default):
33-
34-
| Layer | Lambda (default) | Lambda Managed Instances |
35-
| ---------------------- | ---------------------------------------- | ------------------------------------------ |
36-
| **Instance level** | Firecracker microVMs on shared AWS fleet | Containers on EC2 Nitro in your account |
37-
| **Security boundary** | Execution environment | Capacity provider |
38-
| **Function isolation** | Strong isolation via microVMs | Container-based isolation within instances |
39-
40-
**Capacity providers** serve as the security boundary. Functions within the same capacity provider share the underlying EC2 instances. For workloads requiring strong isolation between functions, use separate capacity providers.
41-
42-
For Python specifically, the multi-process model adds another layer of isolation - each concurrent request runs in its own process with separate memory space.
29+
For more details on the isolation model, see [Lambda Managed Instances documentation](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances.html){target="_blank" rel="nofollow"}.
4330

4431
## Powertools integration
4532

4633
Powertools for AWS Lambda (Python) works seamlessly with Lambda Managed Instances. All utilities are compatible with the multi-process concurrency model used by Python.
4734

48-
### Logger
49-
50-
Logger works without any changes. Each process has its own logger instance.
51-
52-
```python hl_lines="4 7" title="Using Logger with Managed Instances"
53-
--8<-- "examples/lambda_features/managed_instances/src/using_logger.py"
54-
```
55-
56-
### Tracer
35+
### Logger, Tracer, and Metrics
5736

58-
Tracer works without any changes. X-Ray traces are captured per request.
37+
Core utilities work without any changes. Each process has its own instances, so correlation IDs and traces are naturally isolated per request.
5938

6039
???+ note "VPC connectivity required"
61-
Lambda Managed Instances run in your VPC. Ensure you have [network connectivity](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances-networking.html){target="_blank" rel="nofollow"} to send traces to X-Ray.
40+
Lambda Managed Instances run in your VPC. Ensure you have [network connectivity](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances-networking.html){target="_blank" rel="nofollow"} to send logs to CloudWatch, traces to X-Ray, and metrics to CloudWatch.
6241

63-
```python hl_lines="4 8 12" title="Using Tracer with Managed Instances"
42+
```python hl_lines="5 6 7 10 11 12 20 25" title="Using Logger, Tracer, and Metrics with Managed Instances"
6443
--8<-- "examples/lambda_features/managed_instances/src/using_tracer.py"
6544
```
6645

67-
### Metrics
68-
69-
Metrics work without any changes. Each process flushes metrics independently.
70-
71-
???+ note "VPC connectivity required"
72-
Ensure you have [network connectivity](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances-networking.html){target="_blank" rel="nofollow"} to send metrics to CloudWatch.
73-
74-
```python hl_lines="5 9 12" title="Using Metrics with Managed Instances"
75-
--8<-- "examples/lambda_features/managed_instances/src/using_metrics.py"
76-
```
77-
7846
### Parameters
7947

80-
Parameters utility works correctly, but be aware that **cache is per-process**.
48+
Parameters utility works as expected, but be aware that **caching is per-process**.
8149

8250
```python hl_lines="9" title="Using Parameters with Managed Instances"
8351
--8<-- "examples/lambda_features/managed_instances/src/using_parameters.py"
8452
```
8553

8654
???+ tip "Cache behavior"
87-
Since each process has its own cache, you might see more calls to SSM/Secrets Manager during initial warm-up. Once each process has cached the value, subsequent requests within that process use the cache.
55+
Since each process has its own cache, you might see more calls to SSM/Secrets Manager during initial warm-up. Once each process has cached the value, subsequent requests within that process use the cache. You can customize the caching behavior with `max_age` to control the TTL.
8856

8957
### Idempotency
9058

@@ -94,35 +62,6 @@ Idempotency works without any changes. It uses DynamoDB for state management, wh
9462
--8<-- "examples/lambda_features/managed_instances/src/using_idempotency.py"
9563
```
9664

97-
### Batch Processing
98-
99-
Batch Processing works without any changes. Each batch is processed within a single process.
100-
101-
```python hl_lines="5 8 14" title="Using Batch Processing with Managed Instances"
102-
--8<-- "examples/lambda_features/managed_instances/src/using_batch.py"
103-
```
104-
105-
???+ note "Other utilities"
106-
All other Powertools for AWS utilities (Feature Flags, Validation, Parser, Data Masking, etc.) work without any changes. If you encounter any issues, please [open an issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new?template=bug_report.yml){target="_blank"}.
107-
108-
## Working with shared resources
109-
110-
### The `/tmp` directory
111-
112-
The `/tmp` directory is **shared across all processes** in the execution environment. Use caution when writing files.
113-
114-
```python title="Safe file handling with unique names"
115-
--8<-- "examples/lambda_features/managed_instances/src/tmp_file_handling.py"
116-
```
117-
118-
### Database connections
119-
120-
Since each process is independent, connection pooling behaves differently than in threaded runtimes.
121-
122-
```python title="Database connections per process"
123-
--8<-- "examples/lambda_features/managed_instances/src/database_connections.py"
124-
```
125-
12665
## VPC connectivity
12766

12867
Lambda Managed Instances require VPC configuration for:
@@ -136,18 +75,19 @@ Configure connectivity using one of these options:
13675
1. **VPC Endpoints** - Private connectivity without internet access
13776
2. **NAT Gateway** - Internet access from private subnets
13877
3. **Public subnet with Internet Gateway** - Direct internet access
78+
4. **Egress-only Internet Gateway** - IPv6 outbound connectivity without inbound access ([learn more](https://docs.aws.amazon.com/vpc/latest/userguide/egress-only-internet-gateway.html){target="_blank" rel="nofollow"})
13979

14080
See [Networking for Lambda Managed Instances](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances-networking.html){target="_blank" rel="nofollow"} for detailed setup instructions.
14181

14282
## FAQ
14383

14484
### Does Powertools for AWS Lambda (Python) work with Lambda Managed Instances?
14585

146-
Yes, all Powertools for AWS utilities work seamlessly with Lambda Managed Instances. The multi-process model in Python provides natural isolation between concurrent requests.
86+
Yes, all Powertools for AWS Lambda (Python) utilities work seamlessly with Lambda Managed Instances. The multi-process model in Python provides natural isolation between concurrent requests.
14787

14888
### Is my code thread-safe?
14989

150-
For Python, you don't need to worry about thread safety because Lambda Managed Instances uses **multiple processes**, not threads. Each request runs in its own process with isolated memory.
90+
Lambda Managed Instances uses **multiple processes**, instead of threads. Each request runs in its own process with isolated memory. If you implement multi-threading within your handler, you are responsible for thread safety.
15191

15292
### Why is my cache not shared between requests?
15393

@@ -157,10 +97,6 @@ Each process maintains its own cache (for Parameters, Feature Flags, etc.). This
15797

15898
Yes, but remember they are **per-process**, not shared across concurrent requests. This is actually safer than shared state.
15999

160-
### How should I handle files in `/tmp`?
161-
162-
Use unique file names (include request ID or UUID) to avoid conflicts between concurrent requests. Always clean up files after use to avoid filling the shared `/tmp` directory.
163-
164-
### Do I need to change my existing Powertools for AWS code?
100+
### Do I need to change my existing Powertools for AWS Lambda (Python) code?
165101

166-
No changes are required. Your existing code will work as-is with Lambda Managed Instances.
102+
No changes are required if you are running Powertools for AWS Lambda (Python) version **3.4.0** or later. Your existing code will work as-is with Lambda Managed Instances.

examples/lambda_features/durable_functions/src/best_practice_logging.py

Lines changed: 0 additions & 21 deletions
This file was deleted.

examples/lambda_features/durable_functions/src/best_practice_metrics.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ def handler(event: dict, context: DurableContext) -> str:
1414
name="process",
1515
)
1616

17-
# Emit only at the end
18-
metrics.add_metric(name="WorkflowCompleted", unit=MetricUnit.Count, value=1)
17+
# Emit metrics in a dedicated step to ensure they are only counted once
18+
context.step(
19+
lambda _: metrics.add_metric(name="WorkflowCompleted", unit=MetricUnit.Count, value=1),
20+
name="emit_completion_metric",
21+
)
22+
1923
return result

examples/lambda_features/durable_functions/src/log_deduplication.py

Lines changed: 0 additions & 21 deletions
This file was deleted.

0 commit comments

Comments
 (0)