working deploy and destroy

Author: usrbinkat

docs/user_guide/Troubleshooting_Kubernetes_Resource_State.md

@@ -0,0 +1,128 @@
+# Resolving Kubernetes Cluster State Issues with Pulumi
+
+## Introduction
+
+This document provides a step-by-step guide to resolving issues when Pulumi fails to destroy unreachable Kubernetes resources. It follows the Konductor project’s documentation standards and aligns with Pulumi Python development best practices.
+
+This guide is designed to assist developers in handling errors related to `PULUMI_K8S_DELETE_UNREACHABLE` and Kubernetes provider misconfigurations. By following these steps, you can ensure smooth and complete infrastructure teardown, avoiding unnecessary cloud costs.
+
+## Troubleshooting Steps
+
+### 1. Identify the Problematic Resource
+
+To identify Kubernetes resources causing destroy failures:
+
+1. Export the Pulumi stack state:
+
+   ```bash
+   pulumi stack export > stack.json
+   ```
+
+2. Use `jq` to filter Kubernetes resources:
+
+   ```bash
+   cat stack.json | jq '.deployment.resources[] | select(.type == "kubernetes:core/v1:Pod") | .urn'
+   ```
+
+   **Example Output:**
+
+   ```
+   "urn:pulumi:navteca-aws-credentials-config-smce::konductor::kubernetes:core/v1:Pod::nginx-test-test-eks-cluster"
+   ```
+
+### 2. Use `PULUMI_K8S_DELETE_UNREACHABLE`
+
+Set the environment variable to allow deletion of unreachable resources:
+
+1. Export the variable:
+
+   ```bash
+   export PULUMI_K8S_DELETE_UNREACHABLE=true
+   ```
+
+2. Attempt to destroy the resource:
+
+   ```bash
+   pulumi destroy --target="urn:<resource-URN>" --refresh --skip-preview
+   ```
+
+   If the operation fails, proceed to manually remove the resource from the state.
+
+### 3. Manually Remove the Resource
+
+1. Delete the resource directly from the Pulumi state:
+
+   ```bash
+   pulumi state delete "urn:<resource-URN>"
+   ```
+
+   **Example:**
+
+   ```bash
+   pulumi state delete "urn:pulumi:navteca-aws-credentials-config-smce::konductor::kubernetes:core/v1:Pod::nginx-test-test-eks-cluster"
+   ```
+
+2. Retry the destroy operation for the entire stack:
+
+   ```bash
+   pulumi destroy --skip-preview --refresh --continue-on-error
+   ```
+
+### 4. Validate and Clean Up
+
+1. Ensure all resources have been deleted by reviewing the destroy output.
+2. Remove the Pulumi stack if no longer needed:
+
+   ```bash
+   pulumi stack rm <stack-name>
+   ```
+
+## Best Practices
+
+### Handling Kubernetes Resources
+
+- **Use `PULUMI_K8S_DELETE_UNREACHABLE` Proactively:** Set the environment variable when working with ephemeral or potentially inaccessible clusters.
+- **Monitor Resource Dependencies:** Use `depends_on` in Pulumi to manage resource creation and deletion order.
+- **Backup Pulumi State:** Export the stack state regularly:
+
+  ```bash
+  pulumi stack export > backup-<date>.json
+  ```
+
+### Configurations
+
+- **Namespace Management:** Explicitly set namespaces for Kubernetes resources to avoid conflicts.
+- **Pulumi Provider Configuration:** Ensure the Kubernetes provider is properly configured with an accessible kubeconfig file:
+
+  ```python
+  k8s_provider = pulumi_kubernetes.Provider(
+      "k8s-provider",
+      kubeconfig="~/.kube/config"
+  )
+  ```
+
+### Debugging
+
+- **Use Pulumi’s debug logging to analyze failures:**
+
+  ```bash
+  pulumi destroy --debug --skip-preview
+  ```
+
+## Common Errors and Solutions
+
+| Error Message                                        | Cause                                                | Solution                                                                 |
+|------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------------------|
+| configured Kubernetes cluster is unreachable          | Cluster is deleted or unreachable.                   | Use `PULUMI_K8S_DELETE_UNREACHABLE` or manually remove the resource.     |
+| failed to read resource state due to unreachable API | Pulumi cannot reconcile the resource state.          | Manually delete the resource using `pulumi state delete`.                |
+| preview failed: unable to load schema information    | Invalid or missing Kubernetes provider configuration. | Check and update the kubeconfig path in the Pulumi provider configuration. |
+| error: update failed                                 | Failed to delete a dependent resource.               | Identify and remove dependencies manually from the Pulumi state.         |
+
+## Advanced Techniques
+
+### Recreate a Dummy Cluster
+
+If the original cluster has been deleted but its resources remain in the Pulumi state:
+
+1. Recreate a temporary cluster with the same name and configuration.
+2. Retry the destroy operation to clean up the resources.

modules/aws/eks.py

@@ -297,13 +297,16 @@ def deploy_test_nginx(self, k8s_provider: k8s.Provider, name: str) -> k8s.core.v
                 f"nginx-test-{name}",
                 metadata={"name": f"nginx-test-{name}", "namespace": "default"},
                 spec={"containers": [{"name": "nginx", "image": "nginx:latest", "ports": [{"containerPort": 80}]}]},
-                opts=ResourceOptions(provider=k8s_provider),
+                opts=ResourceOptions(
+                    provider=k8s_provider,
+                    depends_on=[k8s_provider],
+                    custom_timeouts={"create": "5m", "delete": "5m"}),
             )
             return nginx_pod
 
         except Exception as e:
             log.error(f"Failed to deploy test nginx pod: {str(e)}")
-            raise
+            return None
 
     def deploy_cluster(
         self,
@@ -329,6 +332,7 @@ def deploy_cluster(
 
             # Create EKS cluster
             cluster = self.create_cluster(name=name, subnet_ids=subnet_ids, cluster_role=cluster_role, version=version)
+            cluster_name = cluster.name.apply(lambda n: n)
 
             # Create node group
             node_group = self.create_node_group(
@@ -386,7 +390,10 @@ def deploy_cluster(
             # Get cluster auth token using Pulumi's built-in AWS provider
             try:
                 cluster_token = aws.eks.get_cluster_auth(
-                    name=cluster.name, opts=pulumi.InvokeOptions(provider=self.provider.provider)
+                    name=cluster_name, opts=pulumi.InvokeOptions(
+                        provider=self.provider.provider,
+                        depends_on=[cluster]
+                    )
                 )
             except Exception as e:
                 log.error(f"Failed to get cluster auth token: {str(e)}")
@@ -416,11 +423,7 @@ def deploy_cluster(
             )
 
             # Export both kubeconfigs with descriptive names
-            pulumi.export("eks_kubeconfig_external", external_kubeconfig)
-            pulumi.export("eks_kubeconfig_internal", internal_kubeconfig)
-
-            # Export the k8s provider as a pulumi stack output secret for use by other Pulumi stacks
-            pulumi.export("k8s_provider", k8s_provider)
+            pulumi.export("eks_kubeconfig_external", pulumi.Output.secret(external_kubeconfig))
 
             # Deploy test nginx pod
             self.deploy_test_nginx(k8s_provider, name)

modules/kubernetes/__init__.py

@@ -0,0 +1,198 @@
+# Cert Manager Module Guide
+
+# TODO: Convert from Kargo to generalized Konductor Framework Template repo docs content
+
+Welcome to the **Cert Manager Module** for the Konductor IaC Framework! This guide is tailored for both newcomers to DevOps and experienced developers, providing a comprehensive overview of how to deploy and configure the Cert Manager module within the Kargo platform.
+
+---
+
+## Table of Contents
+
+- [Introduction](#introduction)
+- [Why Use Cert Manager?](#why-use-cert-manager)
+- [Getting Started](#getting-started)
+- [Enabling the Module](#enabling-the-module)
+- [Configuration Options](#configuration-options)
+   - [Default Settings](#default-settings)
+   - [Customizing Your Deployment](#customizing-your-deployment)
+
+- [Module Components Explained](#module-components-explained)
+   - [Namespace Creation](#namespace-creation)
+   - [Helm Chart Deployment](#helm-chart-deployment)
+   - [Self-Signed Cluster Issuer Setup](#self-signed-cluster-issuer-setup)
+
+- [Using the Module](#using-the-module)
+   - [Example Usage](#example-usage)
+
+- [Troubleshooting and FAQs](#troubleshooting-and-faqs)
+- [Additional Resources](#additional-resources)
+- [Conclusion](#conclusion)
+
+---
+
+## Introduction
+
+The Cert Manager module automates the management of SSL/TLS certificates in your Kubernetes cluster using [cert-manager](https://cert-manager.io/). It simplifies the process of obtaining, renewing, and managing certificates, enhancing the security of your applications without manual intervention.
+
+---
+
+## Why Use Cert Manager?
+
+- **Automation**: Automatically provisions and renews certificates.
+- **Integration**: Works seamlessly with Kubernetes Ingress resources and other services.
+- **Security**: Enhances security by ensuring certificates are always up-to-date.
+- **Compliance**: Helps meet compliance requirements by managing PKI effectively.
+
+---
+
+## Getting Started
+
+### Prerequisites
+
+- **Kubernetes Cluster**: Ensure you have access to a Kubernetes cluster.
+- **Pulumi CLI**: Install the Pulumi CLI and configure it.
+- **Kubeconfig**: Your kubeconfig file should be properly set up.
+
+### Setup Steps
+
+1. **Navigate to the Kargo Pulumi Directory**:
+
+```bash
+cd Kargo/pulumi
+```
+
+2. **Install Dependencies**:
+
+```bash
+pip install -r requirements.txt
+```
+
+3. **Initialize Pulumi Stack**:
+
+```bash
+pulumi stack init dev
+```
+
+---
+
+## Enabling the Module
+
+The Cert Manager module is enabled by default. To verify or modify its enabled status, adjust your Pulumi configuration.
+
+### Verifying Module Enablement
+
+```yaml
+# Pulumi.<stack-name>.yaml
+
+config:
+  cert_manager:
+    enabled: true  # Set to false to disable
+```
+
+Alternatively, use the Pulumi CLI:
+
+```bash
+pulumi config set --path cert_manager.enabled true
+```
+
+---
+
+## Configuration Options
+
+### Default Settings
+
+The module is designed to work out-of-the-box with default settings:
+
+- **Namespace**: `cert-manager`
+- __Version__: Defined in `default_versions.json`
+- **Cluster Issuer Name**: `cluster-selfsigned-issuer`
+- **Install CRDs**: `true`
+
+### Customizing Your Deployment
+
+You can tailor the module to fit your specific needs by customizing its configuration.
+
+#### Available Configuration Parameters
+
+- **enabled** *(bool)*: Enable or disable the module.
+- **namespace** *(string)*: Kubernetes namespace for cert-manager.
+- **version** *(string)*: Helm chart version to deploy. Use `'latest'` to fetch the most recent stable version.
+- __cluster_issuer__ _(string)_: Name of the ClusterIssuer resource.
+- __install_crds__ _(bool)_: Whether to install Custom Resource Definitions.
+
+#### Example Custom Configuration
+
+```yaml
+config:
+  cert_manager:
+    enabled: true
+    namespace: "my-cert-manager"
+    version: "1.15.3"
+    cluster_issuer: "my-cluster-issuer"
+    install_crds: true
+```
+
+---
+
+## Module Components Explained
+
+### Namespace Creation
+
+A dedicated namespace is created to isolate cert-manager resources.
+
+- **Why?**: Ensures better organization and avoids conflicts.
+- **Customizable**: Change the namespace using the `namespace` parameter.
+
+### Helm Chart Deployment
+
+Deploys cert-manager using Helm.
+
+- **Chart Repository**: `https://charts.jetstack.io`
+- **Version Management**: Specify a version or use `'latest'`.
+- **Custom Values**: Resource requests and limits are set for optimal performance.
+
+### Self-Signed Cluster Issuer Setup
+
+Sets up a self-signed ClusterIssuer for certificate provisioning.
+
+- **Root ClusterIssuer**: Creates a root issuer.
+- **CA Certificate**: Generates a CA certificate stored in a Kubernetes Secret.
+- **Primary ClusterIssuer**: Issues certificates for your applications using the CA certificate.
+- **Exported Values**: CA certificate data is exported for use in other modules.
+
+---
+
+## Using the Module
+
+### Example Usage
+
+After enabling and configuring the module, deploy it using Pulumi:
+
+```bash
+pulumi up
+```
+
+---
+
+## Troubleshooting and FAQs
+
+**Q1: Cert-manager pods are not running.**
+
+- **A**: Check the namespace and ensure that CRDs are installed. Verify the Kubernetes version compatibility.
+
+**Q2: Certificates are not being issued.**
+
+- **A**: Ensure that the ClusterIssuer is correctly configured and that your Ingress resources reference it.
+
+**Q3: How do I update cert-manager to a newer version?**
+
+- **A**: Update the `version` parameter in your configuration and run `pulumi up`.
+
+---
+
+## Additional Resources
+
+- **cert-manager Documentation**: [cert-manager.io/docs](https://cert-manager.io/docs/)
+- **Kargo Project**: [Kargo GitHub Repository](https://github.com/ContainerCraft/Kargo)
+- **Pulumi Kubernetes Provider**: [Pulumi Kubernetes Docs](https://www.pulumi.com/docs/reference/pkg/kubernetes/)
+- **Helm Charts Repository**: [Artifact Hub - cert-manager](https://artifacthub.io/packages/helm/cert-manager/cert-manager)