How can I use Kubernetes core resources (e.g., Pods) with kcp v0.28.0? Is there official documentation?

[tangshibokmpler]

I’ve deployed kcp and the api-syncagent and reviewed the documentation, but I can’t find guidance on using core Kubernetes resources (e.g., Pods). Earlier v0.11 had docs—do those instructions still apply to v0.28, or is there updated guidance for v0.28?

[embik]

Hi @tangshibokmpler. v0.11 had a very different mechanism for syncing to Kubernetes clusters (called TMC - Transparent Multi-Cluster), so no, those don't instructions do not apply anymore. The api-syncagent PublishedResource mechanism allows you to publish any API resource on a Kubernetes cluster that has api-syncagent installed, and that includes core resources like Pods. Have you tried that out so far?

Here is how I published Secrets for some demo I was doing:

apiVersion: syncagent.kcp.io/v1alpha1
kind: PublishedResource
metadata:
  name: publish-secret # name can be freely chosen
spec:
  resource:
    kind: Secret
    apiGroup: ""
    versions: [v1]
  projection:
    group: kubermatic.com
    # all of these options are optional
    kind: KubeSecret

[tangshibokmpler]

Hi, I tried this, but publishing resources to a workspace via PublishedResource seems to require changing the GVK. I’m concerned this would prevent many operators that expect native Kubernetes GVKs (/api/v1, /apis/apps) from working unchanged. Is that correct?

Hi @tangshibokmpler. v0.11 had a very different mechanism for syncing to Kubernetes clusters (called TMC - Transparent Multi-Cluster), so no, those don't instructions do not apply anymore. The api-syncagent PublishedResource mechanism allows you to publish any API resource on a Kubernetes cluster that has api-syncagent installed, and that includes core resources like Pods. Have you tried that out so far?

Here is how I published Secrets for some demo I was doing:

apiVersion: syncagent.kcp.io/v1alpha1
kind: PublishedResource
metadata:
name: publish-secret # name can be freely chosen
spec:
resource:
kind: Secret
apiGroup: ""
versions: [v1]
projection:
group: kubermatic.com
# all of these options are optional
kind: KubeSecret

[embik]

That is correct. If you require the exact functionality that TMC provided, I'm afraid that is no longer available - We would be open to restarting efforts on the TMC component, but we would need community members to join in leading those efforts as our resources are limited.

api-syncagent focuses much more on publishing higher-level APIs to kcp, so not Pods but the CRDs that (stock) operators use. Operators would then run on the Kubernetes cluster running api-syncagent and be unaware of its APIs being published into kcp.

[XZXRJ]

That is correct. If you require the exact functionality that TMC provided, I'm afraid that is no longer available - We would be open to restarting efforts on the TMC component, but we would need community members to join in leading those efforts as our resources are limited.

api-syncagent focuses much more on publishing higher-level APIs to kcp, so not Pods but the CRDs that (stock) operators use. Operators would then run on the Kubernetes cluster running api-syncagent and be unaware of its APIs being published into kcp.

Does this mean that using workload-related APIs in kcp is not a best practice? So, kubeconfig using kcp cannot directly operate workload-related resource such as pod and deployment?

[embik]

Does this mean that using workload-related APIs in kcp is not a best practice? So, kubeconfig using kcp cannot directly operate workload-related resource such as pod and deployment?

No and Yes. So by default, kcp doesn't serve Pods, Deployments, etc. These resources simply do not exist. You can add them via APIExports/APIBindings, but then you need something to fulfill those APIs. TMC (transparent multi-cluster) was such an implementation but is now dormant. If you re-implement this, you could absolutely offer workload-related APIs directly through kcp, it's just a question of what creating a Deployment in kcp should do in your architecture.

[XZXRJ]

Does this mean that using workload-related APIs in kcp is not a best practice? So, kubeconfig using kcp cannot directly operate workload-related resource such as pod and deployment?

No and Yes. So by default, kcp doesn't serve Pods, Deployments, etc. These resources simply do not exist. You can add them via APIExports/APIBindings, but then you need something to fulfill those APIs. TMC (transparent multi-cluster) was such an implementation but is now dormant. If you re-implement this, you could absolutely offer workload-related APIs directly through kcp, it's just a question of what creating a Deployment in kcp should do in your architecture.

Then I hope to operate workload-related APIs in kcp, but in reality, these workload-related resources are run by the physical cluster. Both sides can synchronize their states to kcp for visualization. Can this form be achieved through kcp or api-syncagent?

[xrstf]

Pods are known to have some complicated datastructures in them, so exporting a Pod/Deployment CRD using kcp's crd-puller can be tricky to then apply in kcp. In general, yes, the syncagent should handle Pods just fine, as long as the Pod CRD itself is CRD-compatible.

TL;DR: You can try, but without further massaging the CRDs you create it might be difficult.