OCTRL-1008 Attempt reconnecting to ecs-dcs gateway at core startup

[Copilot]

Problem

The DCS plugin only attempts to reconnect to the Subscribe stream if it has been already established during the plugin lifetime. However, if the ecs-dcs-gateway is unavailable during plugin initialization, the plugin will never attempt to reconnect, leaving it permanently disconnected.

This issue was observed in production on 14/04/2025 at 13:31 when the DCS gateway was unavailable during AliECS core startup.

Root Cause

In core/integration/dcs/plugin.go, the Init() method performs an initial subscription attempt (lines 290-293). If this fails, the method returns an error and the reconnection goroutine (lines 294-348) is never started:

evStream, err := p.dcsClient.Subscribe(context.Background(), in, grpc.EmptyCallOption{})
if err != nil {
    return fmt.Errorf("failed to subscribe to DCS service on %s, possible network issue or DCS gateway malfunction", viper.GetString("dcsServiceEndpoint"))
}
// Reconnection goroutine only starts if initial subscription succeeds
go func() { /* reconnection logic */ }()

Solution

This PR restructures the initialization logic to ensure the reconnection goroutine always starts, regardless of initial connection status:

1. Always start reconnection goroutine: The goroutine now starts even if the initial subscription fails
2. Move subscription into goroutine: Initial subscription attempts are handled within the reconnection loop
3. Resilient initialization: Init() succeeds even when the DCS gateway is unavailable
4. Continuous reconnection: The plugin continuously attempts reconnection with 3-second intervals until successful

Key Changes

• Removed early return on subscription failure that prevented goroutine startup
• Restructured goroutine to handle both initial and subsequent subscription attempts
• Added proper stream state management (evStream = nil on connection failures)
• Enhanced logging for better observability of connection attempts

Testing

• ✅ All existing DCS tests pass (no regression)
• ✅ New test TestPluginInitWithUnavailableGateway validates the fix
• ✅ Manual testing confirms continuous reconnection attempts
• ✅ Core binary builds successfully

Impact

This fix ensures the DCS plugin will reliably reconnect to the ecs-dcs-gateway after it becomes available, preventing the production issue experienced when the gateway is unavailable during startup. The plugin now gracefully handles network outages and service restarts without requiring AliECS core restarts.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• esm.ubuntu.com
  • Triggering command: /usr/lib/apt/methods/https (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[knopers8]

@copilot remove the proto-generated go files from the PR. They have nothing to do with the aim of the PR.

[Copilot]

@copilot remove the proto-generated go files from the PR. They have nothing to do with the aim of the PR.

Removed all proto-generated files from the PR. The changes now only include the DCS plugin fix (core/integration/dcs/plugin.go and core/integration/dcs/plugin_reconnect_test.go). Commit: 22d0313

[knopers8]

Well, that actually looks correct and works well (tested on staging)

[knopers8]

The commits should be squashed when merging.