DLPX-96312 Add InfluxDB/Telegraf infrastructure for Engine Performance Analytics

[dbshah12]

Design Doc

• https://perforce.atlassian.net/wiki/spaces/DLXEN/pages/2740027450/Design+of+Delphix+Engine+Performance+Analytics+in+DCT

Problem

Telegraf is already collecting engine performance metrics and writing them to local JSON files on the appliance. However, there is no local time-series database to store and serve these metrics, making it difficult for tools like DCT Smart Proxy to query historical performance data from the engine directly.

Solution

Add InfluxDB 2.x infrastructure to the appliance, mirroring the existing Telegraf setup pattern. This includes:

• influxdb/influxdb.toml — InfluxDB daemon config: bound to 127.0.0.1:8086, with bolt/engine paths matching the installed package (/var/lib/influxdb/). Named .toml (not .conf) because InfluxDB uses the Viper config library, which determines the file format from the extension — .conf is not recognized and is silently ignored, causing influxd to fall back to defaults (~/.influxdbv2/).
• influxdb/influxdb-init.conf — Tunable init config (org, bucket, retention period, readiness wait parameters) sourced by the init script. Change values here without touching the script.
• influxdb/delphix-influxdb-init — One-time init script that:
  • Exits immediately if /etc/influxdb/influxdb_meta already exists (safe on upgrades and reboots).
  • Waits for InfluxDB to be ready via the /health endpoint.
  • Calls /api/v2/setup to create the org, bucket, and admin credentials (one-shot; uses curl directly, no influx CLI dependency).
  • Is crash-safe: persists a setup state file (including the generated admin password) immediately after /api/v2/setup so a re-run resumes without repeating the one-shot call and without mismatching the stored password.
  • Creates a write-only token for Telegraf and a read-only token for DCT Smart Proxy.
  • Writes the [[outputs.influxdb_v2]] stanza (with the write token, chmod 640) to /etc/telegraf/telegraf.outputs.influxdb and touches /etc/telegraf/INFLUXDB_ENABLED to enable it by default.
  • Atomically writes /etc/influxdb/influxdb_meta (chmod 600) containing: INFLUXDB_ORG, INFLUXDB_BUCKET, INFLUXDB_ADMIN_USER, INFLUXDB_ADMIN_PASSWORD, INFLUXDB_WRITE_TOKEN, INFLUXDB_READ_TOKEN.
• influxdb/delphix-influxdb-service — Wrapper that starts influxd with INFLUXD_CONFIG_PATH=/etc/influxdb/influxdb.toml in the background, runs the init script, then waits on the daemon PID.
• influxdb/delphix-influxdb.service — Systemd unit following the same structure as delphix-telegraf.service (PartOf=delphix.target, Restart=on-failure, runs as root).
• influxdb/perf_influxdb — Toggle script (mirrors perf_playbook) to enable/disable InfluxDB metric output from Telegraf without stopping InfluxDB itself. Manages the /etc/telegraf/INFLUXDB_ENABLED flag and restarts Telegraf.
• telegraf/delphix-telegraf-service — Updated to conditionally append telegraf.outputs.influxdb to the assembled telegraf.conf when the INFLUXDB_ENABLED flag exists (same pattern as PLAYBOOK_ENABLED).
• telegraf/telegraf.base — Replaced the old commented-out v1 InfluxDB stanza with a comment pointing to perf_influxdb enable|disable.
• debian/rules — Installs all influxdb files: scripts to /usr/bin/, systemd unit to /lib/systemd/system/, configs to /etc/influxdb/.
• debian/control — Added influxdb2 and curl to Depends (see notes below).

All influxdb files are installed to /etc/influxdb/, mirroring the /etc/telegraf/ convention used for Telegraf.

Notes to Reviewers

Runtime dependency decisions (debian/control)

When someone runs apt install performance-diagnostics, APT checks each package listed in Depends:

• If already installed → skip (no reinstall, no harm).
• If not installed → automatically download and install it.

So listing a package that is already present on the appliance is always safe — it is simply a no-op.

The init script (delphix-influxdb-init) relies on curl, openssl, and python3 at runtime. Here is why only curl is explicitly added to Depends:

Dependency	Decision	Reason
openssl	Not added	Already a runtime dependency of delphix-platform (debian/rules) — guaranteed on every Delphix appliance.
python3	Not added	Already present via python3-minimal in our existing Depends.
curl	Added	Only in delphix-platform's Build-Depends (build-time only, not a guaranteed runtime dep) — so explicitly declared here to be safe.

Why influxdb.toml instead of influxdb.conf

InfluxDB 2.x uses Viper for config parsing, which determines the file format from the extension. Only .json, .toml, .yaml, and .yml are recognized — .conf is silently ignored and influxd falls back to defaults (~/.influxdbv2/ for root). Verified on InfluxDB v2.8.0: INFLUXD_CONFIG_PATH=influxdb.conf → paths/settings ignored; INFLUXD_CONFIG_PATH=influxdb.toml → config fully respected.

Testing Done

• Ran ab-pre-push: https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/13634/ - 🏃‍♂️
• On the ab-pre-push engine, below files are created:

/etc/influxdb# ls -l
total 4
-rw-r--r-- 1 root root  86 Mar 31 09:56 config.toml
-rw-r--r-- 1 root root 357 Mar 31 09:19 influxdb-init.conf
-rw-r--r-- 1 root root 274 Mar 31 09:19 influxdb.toml
-rw------- 1 root root 347 Mar 31 12:24 influxdb_meta

/etc/influxdb# ls -l /var/lib/influxdb
total 23
drwxr-x--- 5 influxdb influxdb      5 Mar 31 12:46 engine
-rw------- 1 influxdb influxdb  65536 Mar 31 12:46 influxd.bolt
-rw-r----- 1 influxdb influxdb      4 Mar 31 12:22 influxd.pid
-rw-r----- 1 influxdb influxdb 122880 Mar 31 12:23 influxd.sqlite

• InfluxDB setup is also completed, and I can see data there in the UI:

perf_influxdb enable/disable testing

Test	Result
INFLUXDB_ENABLED flag exists on fresh boot	✅
telegraf.outputs.influxdb exists with correct perms (-rw-r-----)	✅
Telegraf loaded influxdb_v2 output on boot	✅
perf_influxdb disable removes flag and telegraf.conf has no influxdb_v2 stanza	✅
After disable, Telegraf loaded file (4x) only (no influxdb output)	✅
perf_influxdb enable recreates flag and telegraf.conf has stanza with real token	✅
After enable, Telegraf loaded influxdb_v2 output	✅
Non-root user blocked with clear error (must be run as root)	✅
No errors in journalctl	✅