Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: CI

on:
push:
branches-ignore:
- "build-*"
- "v*.*.*"
tags-ignore:
- "build-*"
- "v*.*.*"
pull_request:
branches-ignore:
- "build-*"
- "v*.*.*"

jobs:
fvt:
name: Build + FVT
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.25.9"

- name: Install yamllint
run: |
python3 -m pip install --upgrade pip
python3 -m pip --quiet install yamllint

- name: Set local image variables
run: |
REGISTRY_URL="$(echo "${DOCKER_REGISTRY_LIST}" | tr ',' ' ' | cut -d' ' -f1)"
IMAGE_NAME="${DOCKER_IMAGE_NAME}"

# PR builds may not provide registry/image env vars.
if [ -z "${REGISTRY_URL}" ]; then
REGISTRY_URL="local"
fi
if [ -z "${IMAGE_NAME}" ]; then
IMAGE_NAME="staticroute-operator"
fi

echo "REGISTRY_URL=${REGISTRY_URL}" >> "${GITHUB_ENV}"
echo "DOCKER_IMAGE_NAME=${IMAGE_NAME}" >> "${GITHUB_ENV}"
echo "REGISTRY_REPO=${REGISTRY_URL}/${IMAGE_NAME}" >> "${GITHUB_ENV}"

- name: Show Docker versions
run: |
docker --version

- name: Install deps and run FVT
run: |
make deps INSTALL_LOCATION="${{ github.workspace }}/bin"
make fvt
54 changes: 0 additions & 54 deletions .travis.yml

This file was deleted.

6 changes: 2 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ GIT_COMMIT_SHA:=$(shell git rev-parse HEAD 2>/dev/null)
SHFILES=$(shell find . -type f -name '*fvt*.sh')
SHELLCHECK_EXISTS:=$(shell shellcheck --version 2>/dev/null)
YAMLLINT_EXISTS:=$(shell yamllint --version 2>/dev/null)
INSTALL_LOCATION?=$(GOPATH)/bin
INSTALL_LOCATION?=$(or $(GOPATH),$(HOME)/go)/bin
MAKEFILE_DIR := $(dir $(realpath $(firstword $(MAKEFILE_LIST))))

include Makefile.env
Expand Down Expand Up @@ -50,9 +50,7 @@ endif

lint-yaml:
ifdef YAMLLINT_EXISTS
ifeq ($(TRAVIS),true)
yamllint .travis.yml ./config/
endif
yamllint .github/workflows/ ./config/
else
@echo "yamllint is not installed"
endif
Expand Down
8 changes: 1 addition & 7 deletions Makefile.sdk
Original file line number Diff line number Diff line change
Expand Up @@ -124,14 +124,8 @@ run: manifests generate fmt vet ## Run a controller from your host.
# (i.e. docker build --platform linux/arm64 ). However, you must enable docker buildKit for it.
# More info: https://docs.docker.com/develop/develop-images/build_enhancements/

# Temporary skip s390x build
#docker-build-s390x:
#ifdef TRAVIS
# docker buildx build --platform linux/s390x --progress=plain -t ${IMG}-s390x --build-arg BUILDER_IMAGE=$(GO_BUILDER_IMAGE) --build-arg ARCH=s390x --load .
#endif

.PHONY: docker-build
docker-build: test ## docker-build-s390x
docker-build: test
docker build \
--platform linux/amd64 \
--progress=plain \
Expand Down
110 changes: 61 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/IBM/staticroute-operator)](https://goreportcard.com/report/github.com/IBM/staticroute-operator) [![Active](http://img.shields.io/badge/Status-Active-green.svg)](https://github.com/IBM/staticroute-operator) [![PR's Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat)](https://github.com/IBM/staticroute-operator/pulls) [![Build Status](https://app.travis-ci.com/IBM/staticroute-operator.svg?branch=master)](https://app.travis-ci.com/github/IBM/staticroute-operator) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Code of Conduct](https://img.shields.io/badge/code%20of-conduct-ff69b4.svg?style=flat)](https://www.ibm.com/partnerworld/program/code-of-conduct)

# static-route-operator

[![Go Report Card](https://goreportcard.com/badge/github.com/IBM/staticroute-operator)](https://goreportcard.com/report/github.com/IBM/staticroute-operator) [![Active](http://img.shields.io/badge/Status-Active-green.svg)](https://github.com/IBM/staticroute-operator) [![PR's Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat)](https://github.com/IBM/staticroute-operator/pulls) [![Build Status](https://github.com/IBM/staticroute-operator/actions/workflows/ci.yml/badge.svg)](https://github.com/IBM/staticroute-operator/actions/workflows/ci.yml) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Code of Conduct](https://img.shields.io/badge/code%20of-conduct-ff69b4.svg?style=flat)](https://www.ibm.com/partnerworld/program/code-of-conduct)

Static IP route operator for Kubernetes clusters

This project is under development, use it on your own risk please.

# Usage
## Usage

Public OCI images are not available yet. To give a try to the project you have to build your own image and store it in your image repository. Please follow some easy steps under `Development` section of the page.
After build you have to apply some Kubernetes manifests: `config/crd/bases/static-route.ibm.com_staticroutes.yaml`, `config/rbac/service_account.yaml`, `config/rbac/role.yaml`, `config/rbac/role_binding.yaml` and `config/manager/manager.dev.yaml`.
Expand All @@ -14,7 +15,8 @@ Finaly you have to create `StaticRoute` custom resource on the cluster. The oper
## Sample custom resources

Route a subnet across the default gateway.
```

```yaml
apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
Expand All @@ -24,7 +26,8 @@ spec:
```

Route a subnet to the custom gateway.
```

```yaml
apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
Expand All @@ -35,7 +38,8 @@ spec:
```

Selecting target node(s) of the static route by label(s):
```

```yaml
apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
Expand All @@ -52,78 +56,86 @@ spec:

## Runtime customizations of operator

* Routing table: By default static route controller uses #254 table to configure static routes. The table number is configurable by giving a valid number between 0 and 254 as `TARGET_TABLE` environment variable. Changing the target table on a running operator is not supported. You have to properly terminate all the existing static routes by deleting the custom resources before restarting the operator with the new config.
* Protect subnets: Static route operator allows to set any subnet as routing destination. In some cases users can break the entire network by mistake. To protect some of the subnets you can use a comma separated list in environment variables starting with the string `PROTECTED_SUBNET_` (ie. `PROTECTED_SUBNET_CALICO=172.0.0.1/24,10.0.0.1/24`). The operator will ignore custom route if the subnets (in the custom resource and the protected list) are overlapping each other.
* Fallback IP address for GW selection: if the gateway parameter is not provided in any CR, static route operator will select the gateway based on a predefined IP address (NOT CIDR). The address can be provided via an environment variable: `FALLBACK_IP_FOR_GW_SELECTION`. If the environment variable is not provided for the operator, it will use `10.0.0.1` as a default value.
* Routing table: By default static route controller uses #254 table to configure static routes. The table number is configurable by giving a valid number between 0 and 254 as `TARGET_TABLE` environment variable. Changing the target table on a running operator is not supported. You have to properly terminate all the existing static routes by deleting the custom resources before restarting the operator with the new config.
* Protect subnets: Static route operator allows to set any subnet as routing destination. In some cases users can break the entire network by mistake. To protect some of the subnets you can use a comma separated list in environment variables starting with the string `PROTECTED_SUBNET_` (ie. `PROTECTED_SUBNET_CALICO=172.0.0.1/24,10.0.0.1/24`). The operator will ignore custom route if the subnets (in the custom resource and the protected list) are overlapping each other.
* Fallback IP address for GW selection: if the gateway parameter is not provided in any CR, static route operator will select the gateway based on a predefined IP address (NOT CIDR). The address can be provided via an environment variable: `FALLBACK_IP_FOR_GW_SELECTION`. If the environment variable is not provided for the operator, it will use `10.0.0.1` as a default value.

# Development
## Development

## Prerequisites

The following components are needed to be installed on your environment:
* git
* go 1.18+
* docker
* kubectl v1.22.2 or newer
* KinD v0.11.1 (for testing)
* golangci-lint v1.49.0
* Operator SDK CLI 1.23.0 (more information: https://sdk.operatorframework.io/docs/installation/)
* and access to a Kubernetes cluster on a version v1.21.0 or newer
* before you run any of the make target below, make sure the following are done:
- export `REGISTRY_REPO` environment variable to your docker registry repo url (ie.: quay.io/example/static-route-operator:v0.0.1)
- export `KUBECONFIG` environment variable to the path of kubeconfig file (if not set, default $$HOME/.kube/config will be used)
- login to your docker registry using your credentials (ie.: docker login... , ibmcloud cr login etc.)

* git
* go 1.18+
* docker
* kubectl v1.22.2 or newer
* KinD v0.11.1 (for testing)
* golangci-lint v1.49.0
* Operator SDK CLI 1.23.0 (more information: [https://sdk.operatorframework.io/docs/installation/](https://sdk.operatorframework.io/docs/installation/))
* and access to a Kubernetes cluster on a version v1.21.0 or newer
* before you run any of the make target below, make sure the following are done:
* export `REGISTRY_REPO` environment variable to your docker registry repo url (ie.: quay.io/example/static-route-operator:v0.0.1)
* export `KUBECONFIG` environment variable to the path of kubeconfig file (if not set, default $$HOME/.kube/config will be used)
* login to your docker registry using your credentials (ie.: docker login... , ibmcloud cr login etc.)

## Changing Go build version
You can change the builder Go version for Static Route operator in `Makefile.env`. Please note, that since the docker build is done inside separately in a Go builder image (`GO_BUILDER_IMAGE`), you should also change Travis Go version in `.travis.yaml`, to make sure that the Go tests are running on the same Go version as the build.

You can change the builder Go version for Static Route operator in `Makefile.env`.

## Updating the Custom Resource Definitions (CRDs)

Make sure, that every time you modify anything in `*_types.go` file, run the `make generate` (DeepCopy, DeepCopyInto, and DeepCopyObject...) and `make manifests` (WebhookConfiguration, ClusterRole and CustomResourceDefinition...) to update generated code for `k8s` and `CRDs`.

## Building the static route operator

`make deps` it is strongly recommended to run this make target before trying to build the operator.
`make build-operator` target can be used for updating, building operator. It executes all the static code analyzing.
`make dev-publish-image` publishes a new build of the operator image into your Docker repository.

## Testing the changes

Once you have made changes in the source, you have two option to run and test your operator:
- as a `deployment` inside a Kubernetes cluster
- as a binary program running locally on your development environment

* as a `deployment` inside a Kubernetes cluster
* as a binary program running locally on your development environment
1. Run as a deployment inside your cluster
- run the `make dev-run-operator-remote` target which updates your operator resources, builds the operator, pushes the built operator docker image to the `REGISTRY_REPO`, changes the operator manifest file and creates the Kubernetes resources (CRDs, operator, role, rolebinding and service account) inside the cluster
- you can remove the operator resources using `make dev-cleanup-operator` target
* run the `make dev-run-operator-remote` target which updates your operator resources, builds the operator, pushes the built operator docker image to the `REGISTRY_REPO`, changes the operator manifest file and creates the Kubernetes resources (CRDs, operator, role, rolebinding and service account) inside the cluster
* you can remove the operator resources using `make dev-cleanup-operator` target
2. Run as a Go program on your local development environment
- run `make dev-run-operator-local`
* run `make dev-run-operator-local`

## Functional verification tests

The fvt tests are written is bash and you could find it under the `scripts` directory. By default it uses the [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/) environment to setup a Kubernetes cluster and then it applies all the needed resources and starts the operator.
- run `make fvt` to execute the functional tests

Please note, the fvt test currently does not check network connectivity, it only makes sure that the relevant and necessary routes are setup on the node (container). Travis also runs these tests.
* run `make fvt` to execute the functional tests

Please note, the fvt test currently does not check network connectivity, it only makes sure that the relevant and necessary routes are setup on the node (container). GitHub Actions also runs these tests.

Also there is an option to functionally test the operator on an existing cluster (in a cloud or in on-premise) by customizing test run with environment variables. The only prerequisite is that you shall access your cluster via `kubectl` commands before running the tests.
- set the Prerequisites described above (repo name, kube config, docker login etc.)
- export the following environment variables depending on your needs
- PROVIDER (can be `ibmcloud`, if not set then KinD will be used)
- SKIP_OPERATOR_INSTALL (if you already have an operator, set this to `true`. Default is `false`)
- PROTECTED_SUBNET_TEST1
- PROTECTED_SUBNET_TEST2 (list of protected subnets to test, if either of them are empty then no protected subnet test will run)

* set the Prerequisites described above (repo name, kube config, docker login etc.)
* export the following environment variables depending on your needs
* PROVIDER (can be `ibmcloud`, if not set then KinD will be used)
* SKIP_OPERATOR_INSTALL (if you already have an operator, set this to `true`. Default is `false`)
* PROTECTED_SUBNET_TEST1
* PROTECTED_SUBNET_TEST2 (list of protected subnets to test, if either of them are empty then no protected subnet test will run)

### Handling Restricted Pod Security Admission (PSA)
The fvt test script attempts to install `static-route-operator` and `hostnet` pods into the `default` namespace. These pods require escalated privileges to be able to verify functionality of the operator. Since [Kubernetes 1.25](https://kubernetes.io/docs/concepts/security/pod-security-admission/), PSA has been a stable feature and allows users to restrict or prevent the creation of pods that require higher levels of authority. The fvt test script will temporarily apply labels to the `default` namespace to allow the privileged pods to be created, and upon completion of the fvt test script the label values will be removed (if they did not exist prior to testing) or they will be reset to their previous values (if they were set prior to running the test script).

## Setting Travis-CI
If you want to test, build and publish your changes into your own personal repo after forking this project, you need to following variables set up in Travis instance associated to your github project:
- DOCKER_IMAGE_NAME, this is the name of your docker image ie. myrepo/staticroute-operator
- DOCKER_REGISTRY_LIST, you need at least one docker repository url to publish docker images. This is a comma separated list of repo urls.
- DOCKER_USERNAME, username for your docker repository
- GH_REPO, your github repo with the project name, ie: github.com/myrepo/staticroute-operator
- GH_TOKEN, github token generated to access (tag, and push) to your github repository
- and a set of variables that contains the docker password for each repository url ie. if you set `my.docker.repo.io,quay.io` in DOCKER_REGISTRY_LIST than you need a `my_docker_repo_io` and `quay_io` secrets with the corresponding passwords
(Note: you should take care of GH_TOKEN and docker passwords to be non-visible secrets in Travis!)
The fvt test script attempts to install `static-route-operator` and `hostnet` pods into the `default` namespace. These pods require escalated privileges to be able to verify functionality of the operator. Since [Kubernetes 1.25](https://kubernetes.io/docs/concepts/security/pod-security-admission/), PSA has been a stable feature and allows users to restrict or prevent the creation of pods that require higher levels of authority. The fvt test script will temporarily apply labels to the `default` namespace to allow the privileged pods to be created, and upon completion of the fvt test script the label values will be removed (if they did not exist prior to testing) or they will be reset to their previous values (if they were set prior to running the test script).

## GitHub Actions variables used by this repo

The current `.github/workflows/ci.yml` uses these variables/secrets:

* `DOCKER_IMAGE_NAME` (environment/repository variable)
* `DOCKER_REGISTRY_LIST` (environment/repository variable)

# Contributing
## Contributing

We appreciate your help!

To contribute, please read our contribution guidelines: [CONTRIBUTION.md](CONTRIBUTION.md)

Note that the Static Route Operator project uses the [issue tracker](https://github.com/IBM/staticroute-operator/issues) for bug reports and proposals only. If you have questions, engage our team via Slack by [registering here](https://cloud.ibm.com/kubernetes/slack) and join the discussion in the #general channel on our [public IBM Cloud Kubernetes Service Slack](https://ibm-cloud-success.slack.com/).
Note that the Static Route Operator project uses the [issue tracker](https://github.com/IBM/staticroute-operator/issues) for bug reports and proposals only. If you have questions, engage our team via Slack by [registering here](https://cloud.ibm.com/kubernetes/slack) and join the discussion in the #general channel on our [public IBM Cloud Kubernetes Service Slack](https://ibm-cloud-success.slack.com/).
1 change: 0 additions & 1 deletion config/prometheus/monitor.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

# Prometheus Monitor Service (Metrics)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
Expand Down
Loading
Loading