Skip to content

Commit 8e28eb2

Browse files
delterrdelterr
authored
Concourse pipeline (#49)
* init concourse * correct terraform directory * write policy util to differetiate between data logger and dashboard * test logic in build_image.sh * remove whitespace * add space before ]] * fix typo \o/ * remove (()) notation * export the repo name on the terraform step * remove some env vars in terraform apply * rename an env var... * environment_name -> environment * remove environment var? * re-add previous env vars * fix strig EOL * cd into correct directory terraform_infra.sh * bump hahsicorp/aws version * rm container image * separate out dev and prod yml files * refactor to use api instead of concourse release for tagging * update README and remove redundant code * fix linting issues * fix another linting issue...
1 parent e564aae commit 8e28eb2

8 files changed

Lines changed: 281 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,53 @@ To run mypy (static type checking)
312312
make mypy
313313
```
314314

315+
### Deployments with Concourse
316+
317+
#### Allowlisting your IP
318+
319+
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
320+
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
321+
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
322+
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
323+
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
324+
325+
#### Setting up a pipeline
326+
327+
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
328+
AWS Secrets Manager so you do not need to set up anything yourself. Since this repository has two services that need to be deployed, you can set up a seprate
329+
pipeline for each by either specifying the pipeline name as `github-policy-dashboard` or `github-policy-lambda`.
330+
331+
To set the pipeline, run the following script:
332+
333+
```bash
334+
chmod u+x ./concourse/scripts/set_pipeline.sh
335+
./concourse/scripts/set_pipeline.sh github-policy-dashboard # for policy dashboard
336+
./concourse/scripts/set_pipeline.sh github-policy-lambda # for policy lambda
337+
338+
```
339+
340+
Note that you only have to run chmod the first time running the script in order to give permissions.
341+
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to the current commit hash at the time of setting the pipeline.
342+
343+
The pipeline name itself will usually follow a pattern as follows: `<repo-name>-<branch-name>`
344+
If you wish to set a pipeline for another branch without checking out, you can run the following:
345+
346+
```bash
347+
./concourse/scripts/set_pipeline.sh github-policy-dashboard <branch_name> # For policy dashboard
348+
./concourse/scripts/set_pipeline.sh github-policy-lambda <branch_name> # For policy lambda
349+
```
350+
351+
If the branch you are deploying is "main" or "master", it will trigger a deployment to the sdp-prod environment. To set the ECR image tag, you must draft a Github release pointing to the latest release of the main/master branch that has a tag in the form of vX.Y.Z. Drafting up a release will automatically deploy the latest version of the main/master branch with the associated release tag, but you can also manually trigger a build through the Concourse UI or the terminal prompt.
352+
353+
#### Triggering a pipeline
354+
355+
Once the pipeline has been set, you can manually trigger a build on the Concourse UI, or run the following command:
356+
357+
```bash
358+
fly -t aws-sdp trigger-job -j github-policy-dashboard-<branch-name>/build-and-push # For policy dashboard
359+
fly -t aws-sdp trigger-job -j github-policy-lambda-<branch-name>/build-and-push # For policy lambda
360+
```
361+
315362
### Markdown Linting
316363

317364
To lint the markdown files in this repository, we use `markdownlint-cli`. This repository uses the Dockerised version of the linter, allowing it to be run without needing to install it locally.

‎concourse/ci.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
resources:
2+
- name: resource-repo
3+
type: git
4+
icon: github
5+
source:
6+
uri: https://github.com/ONS-Innovation/github-policy-dashboard
7+
username: ((github_access_token))
8+
password: x-oauth-basic
9+
branch: ((branch))
10+
11+
jobs:
12+
- name: build-and-push
13+
public: true
14+
plan:
15+
- get: resource-repo
16+
timeout: 5m
17+
- task: build-image
18+
privileged: true
19+
config:
20+
platform: linux
21+
image_resource:
22+
type: docker-image
23+
source:
24+
repository: hashicorp/terraform
25+
inputs:
26+
- name: resource-repo
27+
params:
28+
aws_account_id: ((aws_account_sdp_((env))))
29+
aws_role_arn: arn:aws:iam::((aws_account_sdp_((env)))):role/sdp-concourse-((env))
30+
secrets: ((sdp_((env))_github_policy_dashboard_secrets))
31+
run: # binary used to build the image
32+
path: sh
33+
args:
34+
- -cx
35+
- |
36+
apk add --no-cache aws-cli podman jq iptables curl
37+
38+
export repo_name=((repo_name))
39+
chmod u+x ./resource-repo/concourse/scripts/policy_util.sh
40+
source ./resource-repo/concourse/scripts/policy_util.sh
41+
if [[ "((env))" == "prod" ]]; then
42+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/github-policy-dashboard/releases" | jq -r '.[0].tag_name')
43+
export tag
44+
else
45+
export tag=((tag))
46+
fi
47+
git rev-parse --abbrev-ref HEAD
48+
chmod u+x ./resource-repo/concourse/scripts/assume_role.sh
49+
chmod u+x ./resource-repo/concourse/scripts/build_image.sh
50+
source ./resource-repo/concourse/scripts/assume_role.sh
51+
./resource-repo/concourse/scripts/build_image.sh
52+
timeout: 10m
53+
- task: terraform
54+
privileged: true
55+
config:
56+
platform: linux
57+
image_resource:
58+
type: docker-image
59+
source: {repository: hashicorp/terraform}
60+
inputs:
61+
- name: resource-repo
62+
params:
63+
secrets: ((sdp_((env))_github_policy_dashboard_secrets))
64+
github_access_token: ((github_access_token))
65+
run:
66+
path: sh
67+
args:
68+
- -cx
69+
- |
70+
apk add --no-cache jq curl
71+
72+
export repo_name=((repo_name))
73+
chmod u+x ./resource-repo/concourse/scripts/policy_util.sh
74+
source ./resource-repo/concourse/scripts/policy_util.sh
75+
if [[ "((env))" == "prod" ]]; then
76+
tag=$(curl "https://api.github.com/repos/ONS-Innovation/github-policy-dashboard/releases" | jq -r '.[0].tag_name')
77+
export tag
78+
else
79+
export tag=((tag))
80+
fi
81+
chmod u+x ./resource-repo/concourse/scripts/terraform_infra.sh
82+
export env=((env))
83+
./resource-repo/concourse/scripts/terraform_infra.sh
84+
timeout: 30m

‎concourse/scripts/assume_role.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
set -euo pipefail
2+
3+
aws sts assume-role --output text \
4+
--role-arn "${aws_role_arn}" \
5+
--role-session-name concourse-pipeline-run \
6+
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
7+
| awk -F '\t' '{print $1 > ("AccessKeyId")}{print $2 > ("SecretAccessKey")}{print $3 > ("SessionToken")}'
8+
9+
10+
export AWS_ACCESS_KEY_ID="$(cat AccessKeyId)"
11+
export AWS_SECRET_ACCESS_KEY="$(cat SecretAccessKey)"
12+
export AWS_SESSION_TOKEN="$(cat SessionToken)"

‎concourse/scripts/build_image.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
set -euo pipefail
2+
3+
export STORAGE_DRIVER=vfs
4+
export PODMAN_SYSTEMD_UNIT=concourse-task
5+
6+
aws ecr get-login-password --region eu-west-2 | podman --storage-driver=vfs login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com
7+
8+
if [[ ${repo_name} == "github-policy-dashboard" ]]; then
9+
container_image=$(echo "$secrets" | jq -r .container_image)
10+
podman build -t ${container_image}:${tag} resource-repo
11+
else
12+
container_image=$(echo "$secrets" | jq -r .container_image_lambda)
13+
podman build -t ${container_image}:${tag} resource-repo/data_logger/
14+
fi
15+
16+
podman tag ${container_image}:${tag} ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/${container_image}:${tag}
17+
18+
podman push ${aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/${container_image}:${tag}

‎concourse/scripts/policy_util.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
if [[ "${repo_name}" == "github-policy-dashboard" || "${repo_name}" == "github-policy-lambda" ]]; then
2+
echo "${repo_name}"
3+
else
4+
echo "Unknown repository name: ${repo_name}"
5+
exit 1
6+
fi

‎concourse/scripts/set_pipeline.sh‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
repo_name=${1}
2+
3+
if [[ $# -gt 1 ]]; then
4+
branch=${2}
5+
git rev-parse --verify ${branch}
6+
if [[ $? -ne 0 ]]; then
7+
echo "Branch \"${branch}\" does not exist"
8+
exit 1
9+
fi
10+
else
11+
branch=$(git rev-parse --abbrev-ref HEAD)
12+
fi
13+
14+
if [[ ${branch} == "main" || ${branch} == "master" ]]; then
15+
env="prod"
16+
else
17+
env="dev"
18+
fi
19+
20+
if [[ ${env} == "dev" ]]; then
21+
tag=$(git rev-parse HEAD)
22+
else
23+
tag=$(git tag | tail -n 1)
24+
fi
25+
26+
fly -t aws-sdp set-pipeline -c concourse/ci.yml -p ${repo_name}-${branch} -v branch=${branch} -v tag=${tag} -v env=${env} --var repo_name=${repo_name}
27+

‎concourse/scripts/terraform_infra.sh‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
set -euo pipefail
2+
3+
aws_account_id=$(echo "$secrets" | jq -r .aws_account_id)
4+
aws_access_key_id=$(echo "$secrets" | jq -r .aws_access_key_id)
5+
6+
aws_secret_access_key=$(echo "$secrets" | jq -r .aws_secret_access_key)
7+
domain=$(echo "$secrets" | jq -r .domain)
8+
9+
github_app_client_id=$(echo "$secrets" | jq -r .github_app_client_id)
10+
aws_secret_name=$(echo "$secrets" | jq -r .aws_secret_name)
11+
12+
github_org=$(echo "$secrets" | jq -r .github_org)
13+
container_image=$(echo "$secrets" | jq -r .container_image)
14+
15+
service_subdomain=$(echo "$secrets" | jq -r .service_subdomain)
16+
force_deployment=$(echo "$secrets" | jq -r .force_deployment)
17+
18+
container_port=$(echo "$secrets" | jq -r .container_port)
19+
from_port=$(echo "$secrets" | jq -r .from_port)
20+
21+
lambda_name=$(echo "$secrets" | jq -r .lambda_name)
22+
lambda_arch=$(echo "$secrets" | jq -r .lambda_arch)
23+
lambda_timeout=$(echo "$secrets" | jq -r .lambda_timeout)
24+
lambda_memory=$(echo "$secrets" | jq -r .lambda_memory)
25+
schedule=$(echo "$secrets" | jq -r .schedule)
26+
27+
export AWS_ACCESS_KEY_ID=$aws_access_key_id
28+
export AWS_SECRET_ACCESS_KEY=$aws_secret_access_key
29+
30+
git config --global url."https://x-access-token:$github_access_token@github.com/".insteadOf "https://github.com/"
31+
32+
if [[ ${env} != "prod" ]]; then
33+
env="dev"
34+
fi
35+
36+
echo ${env}
37+
38+
39+
40+
if [[ ${repo_name} == "github-policy-dashboard" ]]; then
41+
42+
cd resource-repo/terraform/dashboard
43+
44+
terraform init -backend-config=env/${env}/backend-${env}.tfbackend -reconfigure
45+
46+
terraform apply \
47+
-var "aws_account_id=$aws_account_id" \
48+
-var "aws_access_key_id=$aws_access_key_id" \
49+
-var "aws_secret_access_key=$aws_secret_access_key" \
50+
-var "domain=$domain" \
51+
-var "container_ver=${tag}" \
52+
-var "github_app_client_id=$github_app_client_id" \
53+
-var "aws_secret_name=$aws_secret_name" \
54+
-var "github_org=$github_org" \
55+
-var "container_image=$container_image" \
56+
-var "service_subdomain=$service_subdomain" \
57+
-var "force_deployment=$force_deployment" \
58+
-var "container_port=$container_port" \
59+
-var "from_port=$from_port" \
60+
-auto-approve
61+
62+
63+
else
64+
65+
cd resource-repo/terraform/data_logger
66+
67+
terraform init -backend-config=env/${env}/backend-${env}.tfbackend -reconfigure
68+
69+
terraform apply \
70+
-var "aws_account_id=$aws_account_id" \
71+
-var "aws_access_key_id=$aws_access_key_id" \
72+
-var "aws_secret_access_key=$aws_secret_access_key" \
73+
-var "env_name=$domain" \
74+
-var "aws_secret_name=$aws_secret_name" \
75+
-var "github_org=$github_org" \
76+
-var "github_app_client_id=$github_app_client_id" \
77+
-var "lambda_name=$lambda_name" \
78+
-var "lambda_arch=$lambda_arch" \
79+
-var "lambda_timeout=$lambda_timeout" \
80+
-var "lambda_memory=$lambda_memory" \
81+
-var "lambda_version=${tag}" \
82+
-var "schedule=$schedule" \
83+
-auto-approve
84+
85+
86+
fi

‎terraform/data_logger/providers.tf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ terraform {
22
required_providers {
33
aws = {
44
source = "hashicorp/aws"
5-
version = "~> 5.0"
5+
version = ">= 6.0.0"
66
}
77
}
88
}

0 commit comments

Comments
 (0)