Long-term - migrate observability -> datafusion

[thorrester]

For small to medium sized projects the current span postgres backend; however, for large-scale, multi-tenant workflows with terabytes of data, we should consider revamping the datafusion logic

Plan: Create multi-worker datafusion/deltalake DB engine for observability data

Todos:

• Create simple read/write logic with observability schema
• Support for attribute shreddling style logic for quick reads based on attributes (maybe map struct schema)
• Setup local buffer with periodic writes
• Explore partitioning options for datafusion and/or deltalake (z order, etc)
• Setup compaction jobs with deltalake
• Setup vacuum jobs with deltalake
• Build multi-worker system for distributed read/write architecture in order to support high levels of concurrency

[thorrester]

Some of these are completed with the following

• Feat/otel tracing #219
• updates #220