Merge pull request #32 from ONS-Innovation/lambda-rework

KEH-511 | Data Logger Lambda Rework

Author: TotalDwarf03

.gitignore

@@ -49,4 +49,10 @@ override.tf.json
 terraform.rc
 
 # Ignore the terraform/service prority calc
-terraform/service/highest_priority.txt
+terraform/service/highest_priority.txt
+
+.terraform.lock.hcl
+
+# Lambda Logs and Output
+debug.log
+output/

README.md

@@ -1,6 +1,36 @@
-# GitHub Audit Dashboard
+# GitHub Policy Dashboard
 A dashboard which uses organisation data from the GitHub API to monitor how well policy is adhered to in ONS.
 
+## Overview
+
+This repository contains 2 main elements:
+
+- A Streamlit Dashboard to visualise policy data from S3.
+- An AWS Lambda Data Logger to collect information from GitHub to be used by the dashboard.
+
+## Table of Contents
+
+- [GitHub Policy Dashboard](#github-policy-dashboard)
+  - [Overview](#overview)
+  - [Table of Contents](#table-of-contents)
+  - [Prerequisites](#prerequisites)
+  - [Documentation](#documentation)
+  - [Setup - Run outside of Docker](#setup---run-outside-of-docker)
+  - [Setup - Running in a container](#setup---running-in-a-container)
+  - [Storing the container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr)
+  - [Deployment to AWS](#deployment-to-aws)
+    - [Deployment Prerequisites](#deployment-prerequisites)
+      - [Underlying AWS Infrastructure](#underlying-aws-infrastructure)
+      - [Bootstrap IAM User Groups, Users and an ECSTaskExecutionRole](#bootstrap-iam-user-groups-users-and-an-ecstaskexecutionrole)
+      - [Bootstrap for Terraform](#bootstrap-for-terraform)
+      - [Running the Terraform](#running-the-terraform)
+      - [Provision Users](#provision-users)
+    - [Updating the running service using Terraform](#updating-the-running-service-using-terraform)
+    - [Destroy the Main Service Resources](#destroy-the-main-service-resources)
+  - [Linting and Formatting](#linting-and-formatting)
+  - [Future Development](#future-development)
+
+
 ## Prerequisites
 This project uses poetry for package management and colima/docker for containerisation.
 
@@ -344,4 +374,13 @@ make pylint
 To run mypy (static type checking)
 ```bash
 make mypy
-```
+```
+
+## Future Development
+
+This repository still needs the following implemented:
+
+- Linting
+- Testing to a 95% coverage
+- MkDocs documentation refactor / rewrite
+- General repository clean up

data_logger/Dockerfile

@@ -0,0 +1,37 @@
+# Note: 
+# Within docker, this container runs as root as no user is specified.
+# Due to the nature of the lambda function, this is acceptable as Lambda defines the default user
+# to have the least-privilege permissions rather than the root user.
+
+# Therefore we can ignore the linting error for this Dockerfile
+# as it is not a security risk for this specific use case.
+
+# Ignored in .trivyignore also.
+#kics-scan disable=fd54f200-402c-4333-a5a4-36ef6709af2f
+#checkov:skip=CKV_DOCKER_3:Lambda makes default user lowest privilege
+
+FROM public.ecr.aws/lambda/python:3.12
+
+# Install git using dnf (https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-base)
+# For python 3.12, dnf replaces yum for package management
+RUN dnf install -y git-2.40.1 && dnf clean all
+
+# Copy the poetry.lock and pyproject.toml files
+COPY ./pyproject.toml ./poetry.lock ${LAMBDA_TASK_ROOT}/
+
+# Install the dependencies
+WORKDIR ${LAMBDA_TASK_ROOT}
+RUN pip install --no-cache-dir poetry==1.8.3 &&\
+    poetry config virtualenvs.create false &&\ 
+    poetry install
+
+# Copy config folder
+COPY config ${LAMBDA_TASK_ROOT}/config
+
+# Copy function code
+COPY src/main.py src/logger.py src/policy_checks.py src/custom_threading.py ${LAMBDA_TASK_ROOT}/src/
+
+HEALTHCHECK NONE
+
+# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
+CMD [ "src.main.handler" ]