What we'd like to do is accept up to N minutes of loss (i.e. much less than the retention window of our queues) and run XXL single node CH instances with sharding / merging externally managed, since CH replicated mergetree is much slower than large nodes with plain mergetree, and inserts are >10x more ops replicated according to the docs.
However, this requires being able to bring up new nodes pretty quickly without a long rebuild process to handle failover / node replacement. We can then replay inserts very easily from the last row.
Right now backups seem too heavy to run every 60s / every N rows committed. We are self-hosters but I notice this is a feature gap in CH cloud too: backups are only daily.
Any plans to make this more efficient?
What we'd like to do is accept up to N minutes of loss (i.e. much less than the retention window of our queues) and run XXL single node CH instances with sharding / merging externally managed, since CH replicated mergetree is much slower than large nodes with plain mergetree, and inserts are >10x more ops replicated according to the docs.
However, this requires being able to bring up new nodes pretty quickly without a long rebuild process to handle failover / node replacement. We can then replay inserts very easily from the last row.
Right now backups seem too heavy to run every 60s / every N rows committed. We are self-hosters but I notice this is a feature gap in CH cloud too: backups are only daily.
Any plans to make this more efficient?