High FE metadata memory usage with large number of tablets – any plan to further slim FE metadata in Doris 3.0 Cloud?

[Z-SWEI]

High FE metadata memory usage with large number of tablets – any plan to further slim FE metadata in Doris 3.0 Cloud?

We’ve noticed relatively high memory usage on the FE side caused by metadata, especially in clusters with a large number of tablets. As the cluster scales, FE memory pressure becomes quite noticeable.

After a brief analysis, it seems that the main factor is the tablet count:

Each tablet maintains certain metadata structures in FE

With many tables / partitions / replicas, the total number of tablets grows rapidly

Metadata-related objects take an increasing portion of FE JVM memory

We’d like to ask the community a few questions:

In Doris 3.0 Cloud mode, are there any plans to further slim down or refactor FE metadata, especially tablet-related metadata?

Will the Cloud architecture introduce mechanisms such as metadata offloading, layering, or lazy loading to reduce FE resident memory usage?

If there is already an optimization direction, is there a rough timeline or version plan available?

We understand that FE needs to keep global metadata for consistency. Our goal here is mainly to understand the future optimization roadmap for large-scale tablet scenarios, so we can better plan our architecture and capacity.

Thanks in advance for your insights and discussion 🙏

FE 元数据内存占用较高，tablet 数量多场景下压力明显，3.0 Cloud 模式是否有进一步瘦身计划？

目前在使用 Doris 的过程中，发现 FE 端元数据占用内存较高，在集群规模和 tablet 数量较大的情况下，FE 内存压力比较明显。FULL GC 会导致 BDB 集群崩溃。

我们简单分析了一下内存构成，发现主要与 tablet 数量偏多 有关：

每个 tablet 在 FE 中都会维护一定量的元数据信息

当表数量 / 分区数 / 副本数较多时，tablet 总数快速膨胀

FE JVM 内存中元数据对象占比持续上升

想请教下社区：

在 Doris 3.0 Cloud 模式 下，是否有针对 FE 元数据（尤其是 tablet 相关元数据）进一步瘦身或重构 的规划？

是否会通过 Cloud 架构（如元数据分层、下沉、惰性加载等）来降低 FE 常驻内存占用？

如果已经有相关优化方向，是否方便透露一个 大致排期或版本规划？

我们理解 FE 需要维护全局元数据来保证一致性，这里更多是想了解 未来在大规模 tablet 场景下的优化路线，以便提前做架构和容量规划。

感谢社区的解答与讨论 🙏

[dataroaring]

How many tablets and how much memory is consumed?

[Z-SWEI]

How many tablets and how much memory is consumed?

To add more context with concrete numbers:
In our current cluster, the tablet count reaches tens of millions. During FE metadata checkpoint, we observe a significant memory spike, with peak memory usage reaching around 300 GB.

It appears that during the checkpoint process, a large amount of metadata objects are held in memory at the same time, which causes this sharp increase in memory usage. This is why we’re particularly interested in potential optimizations around FE metadata footprint and checkpoint behavior.

当前集群规模下，tablet 数量达到几千万级别。在 FE 执行 metadata checkpoint 过程中，内存占用会出现明显峰值，最高可达到 300GB。

从现象看，checkpoint 阶段会同时持有大量元数据对象，导致瞬时内存压力非常大，这也是我们比较关心 FE 元数据进一步瘦身和 checkpoint 行为优化的原因。

[dataroaring]

Could you give me a histogram of memory footprint? We have dome some optimization on master, maybe you can copy image and load using master to see the effect.