Fill in the macro implementation #227
Conversation
- Create validation.py module in sentry_streams package - Extract validation logic from runner.py for reusability - Add comprehensive unit tests for validation function - Function validates pipeline configs against config.json schema
- Create PipelineStep class in new pipeline_step.py module - Define PipelineStepContext TypedDict with all required fields - Implement validate_context with pipeline_config schema validation - Add sentry-streams dependency to access validation function - Keep Consumer class for backward compatibility
- Add build_name() to generate RFC 1123 compliant k8s names - Add build_labels() to generate standard k8s labels - Add build_container() to build complete container spec with: - Command/args for running the streaming platform - Resource requests (CPU and memory) - Volume mount for pipeline config - All functions handle deep copying to avoid mutations
- Generate deployment and configmap manifests - Deep copy deployment_template to avoid mutations - Update deployment metadata with name and labels - Add container to deployment spec - Add configmap volume mount to deployment - Create configmap with pipeline_config as YAML data - Preserve namespace from deployment template if present - Return both resources in dictionary with separate keys
- Remove validation.py and test_validation.py from sentry_streams - Keep sentry_streams package unchanged as requested
…implementation - Create validation.py in sentry_streams_k8s with inline config schema - Add validation tests for pipeline config - Update PipelineStep to use local validation module - Remove sentry-streams dependency from pyproject.toml - All 15 tests passing (12 pipeline_step + 3 validation tests) - No changes made to sentry_streams package
- Update validation.py to read config.json from sentry_streams package - Remove duplicated CONFIG_SCHEMA dictionary - Add sentry-streams as regular dependency (not local path) - All 15 tests still passing - No duplication of schema definition - No changes to sentry_streams package
fpacifici
force-pushed
the
fpacifici/fill_k8s
branch
from
ab41754 to
bddfdbf
Compare
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| Merge semantics: | ||
| - Simple types (str, int, bool, None): override replaces base | ||
| - Dictionaries: recursively merge | ||
| - Lists: concatenate (append override elements to base) |
There was a problem hiding this comment.
Why does this append instead of replacing?
There was a problem hiding this comment.
I followed the patch semantics of kubectl patch https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#notes-on-the-strategic-merge-patch.
In this specific case scenarios for this choices are:
- adding a container. Containers are lists not dictionaries. If we replaced the list we would not be able to make the template provide envoy
- adding volumes. They are lists as well.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fpacifici
force-pushed
the
fpacifici/fill_k8s
branch
2 times, most recently
from
eadecdb to
d6db388
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fpacifici
force-pushed
the
fpacifici/fill_k8s
branch
from
d6db388 to
94fdc71
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| labels = { | ||
| "pipeline-app": make_k8s_name(pipeline_module), | ||
| "pipeline": make_k8s_name(pipeline_name), | ||
| } |
There was a problem hiding this comment.
Missing length validation for label values
Medium Severity
Label values created from pipeline_module and pipeline_name via make_k8s_name are not validated or truncated to meet Kubernetes' 63-character limit for label values. Long module or pipeline names like my.very.long.module.name.that.exceeds.the.kubernetes.limit would create labels exceeding 63 characters, causing Kubernetes API to reject the deployment with validation errors.
This is the first pass for the implementation of the
PipelineStepsentry-kubeexternal macro.
This macro takes:
With the above it creates a Deployment that runs the streaming platform
rust runner and a configmap with the streaming pipeline config.
The configmap is also mounted inside the Deployment so the runner can
access the pieline config.
The principle in the macro is that we take a basic deployment template,
we overlay the customer provided template on that, and we overlay the
streaming platform part on top of that.
This means doing a deepmerge of 3 yaml file one over the other.
This could be achieved by patching the three yaml file one over the
other. Doign this with python dictionaries allows us to be a more
flexible and reuse pieces (see the labels).
The next steps will be:
edit a specific field of the result whether or not it is customizable.
This can be achieved by passing in an emergency_patch parameter.