KEH-921 - Post-Migration Cleanup (#57)

* Remove dashboard Terraform

* Remove dashboard related files

* Update Poetry and remove Streamlit

* Remove pandas, matplotlib, plotly

* Update and move CODEOWNERS to .github

* Copy lambda README to root

* Remove dashboard Dockerfile

* Update README prerequisites and add makefile and table of contents sections

* Remove dashboard dockerignore

* Rename lambda folder to src

* Add Makefile commands for applying linting

* Run linters

* Improve README formatting

* Refactor lambda
- Fix var typo
- Capitalise constants
- CoPilot -> Copilot
- Add missing docstring
- Remove unused var

* Move terraform

* Refactor lambda
- Add module docstring
- Use isinstance check
- Use lazy formatting for logging
- Remove unnecessary else and function parameter

* Create LICENSE

* Fail CI if linting fails

* Potential fix for code scanning alert no. 5: Clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Remove old info from docs

* Refactor lambda
- Use Optional typing
- Get Pylint to ignore handler parameters
- Extract repeated logic for updating S3
- Extract logic for converting data to dictionary

* Reduce linter max line length to 100 to fix Pylint conflict

* Replace try except blocks with response type checks

* Split handler into separate functions: reached 10/10 on pylint

* Add tests folder

* Update Makefile
- Clean more temp files
- Run linters on tests folder
- Add new command for running tests

* Update README
- Add table of contents
- Add testing and linting info

* feat: update and setup for testing + coverage.

- Added pytest-cov and pytest-xdist for enhanced testing capabilities.
- Updated Makefile to include new test commands.
- Modified .gitignore to exclude coverage reports.
- Update README.md table of contents.

* chore: update makefile commands

- Remove tests directory from linting and formatting commands in Makefile.
- Add .coverage to clean command in Makefile.

* test: Write some tests for main.py

- Test get_team_history
- Test get_and_update_copilot_teams
- Test update_s3_object
- Add VS Code settings to gitignore

* chore: remove old commented out tests

* Add missing tests to reach 100% coverage

* Add tests to ci.yml

* Fix failing tests

* Install dependencies via Poetry from requirements txt files and update all dependencies

* Group Poetry docs dependencies

* Use Poetry for mkdocs
- Delete requirements txt file
- Update deployment workflow
- Update setup documentation

* Fix Docker run command formatting

* Delete requirements.txt

* refactor: Update Dockerfile to use Poetry

- Moved Dockerfile to root directory for better visibility.
- Update Dockerfile to use Poetry for dependency management instead of pip.
- Update .dockerignore to exclude unnecessary files from build context.
- Update README.md to reflect move of Dockerfile to root directory.

* Add option to run lambda outside of a container
- Update README with setup instructions
- Add commented function call to main.py

* Address comments
- Specify where to find CI workflow
- Update sentence saying linting auto runs on tests
- Fix README env vars
- Use poetry install --only docs for mkdocs deployment
- Add missing docstrings and return types
- Return boolean for update_s3_object
- Add comment above dev code

* Run linting

* Update README and .dockerignore
- Remove disclaimer from README and update overview section with dashboard repository link
- Add .github/ to .dockerignore

* Remove and replace sandbox references
- Remove and replace sandbox from .sh files, README, example_tfvars, variables.tf
- Delete sandbox Terraform and move its example_tfvars to dev folder

* Add backticks to README

* Address comments
- Use lambda full name
- Update documentation bash
- Move handler function to bottom of script

* Address comments
- Move logger to end of handler function
- Remove Digital Landscape links
- Specify Docker endpoint should return message if successful

* Add missing type hints and return types

* Tidy lambda scripts info in README

* Fix function return type

---------

Co-authored-by: Kieran Pritchard <kieran.pritchard@ons.gov.uk>

Author: smstone0

.dockerignore

@@ -1,12 +1,11 @@
-.github/*
-docs/*
-lambda_data_logger/*
-mkdocs_deployment/*
-terraform/*
-.gitignore
-CODEOWNERS
-Makefile
-mkdocs_requirements.txt
-mkdocs.yml
-README.md
-venv/*
+# Virtual Environment
+venv/
+
+# __pycache__
+__pycache__/
+
+.github/
+concourse/
+docs/
+terraform/
+tests/

.github/CODEOWNERS

@@ -0,0 +1 @@
+@ONS-Innovation/keh-dev

.github/workflows/ci.yml

@@ -38,8 +38,11 @@ jobs:
       - name: Install dependencies
         run: make install-dev
 
-      - name: Lint Python
+      - name: Lint Python (check only)
         run: make lint
+
+      - name: Run tests
+        run: make test
 
       - name: Cleanup residue file
         run: make clean

.github/workflows/deploy_mkdocs.yml

@@ -27,6 +27,5 @@ jobs:
           path: .cache 
           restore-keys: |
             mkdocs-material-
-      - run: pip install mkdocs-material
-      - run: pip install -r mkdocs_requirements.txt
+      - run: poetry install --only docs
       - run: mkdocs gh-deploy --force

.gitignore

@@ -59,3 +59,11 @@ terraform/dashboard/highest_alb_listener_priority.txt
 
 .env
 
+# Ignore the .DS_Store file created by macOS
+.DS_Store
+
+# Pytest Coverage Reports
+.coverage
+
+# VS Code settings
+.vscode/

.streamlit/config.toml

@@ -1,21 +1,21 @@
-# Get a base image with a slim version of Python 3.10
-FROM python:3.12-slim
+FROM public.ecr.aws/lambda/python:3.12
 
-# run a pip install for poetry 1.5.0
-RUN pip install poetry==1.5.0
+# Install git using dnf (https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-base)
+# For python 3.12, dnf replaces yum for package management
+RUN dnf install git -y
 
-# Set the working directory
-WORKDIR /app
+# Copy required files into the container
+COPY pyproject.toml poetry.lock ./
 
-# Copy the current directory contents into the container at /app
-COPY . /app
+# Install Poetry
+RUN pip install poetry
 
-# Run poetry install --without dev
-RUN poetry install --only main --no-root 
+# Install dependencies using Poetry
+RUN poetry config virtualenvs.create false && \
+    poetry install --only main --no-root
 
-# Expose the port the app runs on
-EXPOSE 8501
+# Copy the source code into the container
+COPY src/ src/
 
-# Run the dashboard
-# Note: ENTRYPOINT cannot be overriden by docker run command
-ENTRYPOINT ["poetry", "run", "streamlit", "run", "src/app.py", "--server.port=8501", "--server.address=0.0.0.0"]
+# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
+CMD [ "src.main.handler" ]

CODEOWNERS

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 ONS Innovation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Dockerfile

@@ -9,23 +9,40 @@ all: ## Show the available make targets.
 clean: ## Clean the temporary files.
 	rm -rf .mypy_cache
 	rm -rf .ruff_cache
+	rm -rf .pytest_cache
+	rm -rf tests/__pycache__
+	rm -rf .coverage
 
-.PHONY: black
-black: ## Run black.
-	poetry run black src --check || true
+.PHONY: black-check
+black-check: ## Run black for code formatting, without fixing.
+	poetry run black src --check
 
-.PHONY: ruff
-ruff: ## Run ruff without fixing.
-	poetry run ruff check src || true
+.PHONY: black-apply
+black-apply: ## Run black and fix code formatting.
+	poetry run black src
+
+.PHONY: ruff-check
+ruff-check: ## Run ruff for linting and code formatting, without fixing.
+	poetry run ruff check src
+
+.PHONY: ruff-apply
+ruff-apply: ## Run ruff and fix linting and code formatting.
+	poetry run ruff check --fix src
 
 .PHONY: pylint
-pylint: ## Run pylint.
-	poetry run pylint src || true
+pylint: ## Run pylint for code analysis.
+	poetry run pylint src
 
 .PHONY: lint
-lint:  ## Run Python linter
-	make black
-	make ruff
+lint:  ## Run Python linters without fixing.
+	make black-check
+	make ruff-check
+	make pylint
+
+.PHONY: lint-apply
+lint-apply: ## Run black and ruff with auto-fix, and Pylint.
+	make black-apply
+	make ruff-apply
 	make pylint
 
 .PHONY: mypy
@@ -40,8 +57,7 @@ install:  ## Install the dependencies excluding dev.
 install-dev:  ## Install the dependencies including dev.
 	poetry install --no-root
 
-.PHONY: run-local
-run-local:  ## Install the dependencies including dev.
-	poetry run streamlit run src/app.py --server.port 8502
-
+.PHONY: test
+test: ## Run the lambda tests.
+	poetry run pytest -n auto --cov=src --cov-report term-missing --cov-fail-under=95