about engram vocab size

[YYYFEI]

Hi! I'm also trying to replicate Qwen+Engram. How did you decide on the Engram vocab size? I noticed your maximum size is much lower than DeepSeek's setting.

[FCY36]

I‘m also confused about this issue. In DeepSeek's paper, the Engram vocab size is about 1 million level.

[Alichua]

Hi, thanks for your careful reading of TinyEngram!

Our initial experiments are not intended to replicate DeepSeek’s large-scale pretraining setup, but rather to explore whether Engram can work as a lightweight PEFT-style component under limited resources. For this reason, we intentionally use much smaller vocab sizes (e.g. 10k for 2-gram, 1k for 3-gram) to keep the memory footprint and training cost low, while still capturing high-frequency local patterns.

In our small-scale exploratory experiments(doc/experiments/engram_scaling_on_small_dataset.md), we observed that Engram performance did not consistently improve with larger vocab sizes. However, these results are based on limited settings and are far from conclusive. We would be very happy to see more systematic studies or replications in different settings.

This project is still in an early stage, and we hope it can serve as a starting point for community exploration on how to adapt external memory mechanisms like Engram to efficient fine-tuning scenarios.

Feel free to adjust the vocab size based on your data distribution, and we’d be happy to see and discuss your findings!

[coruse123]

Hi Alichua,
Thank you so much for the detailed explanation! I truly appreciate your openness and your selfless contribution to the open-source community. Projects like TinyEngram provide a fantastic starting point for those of us exploring efficient memory mechanisms.
I have two follow-up questions regarding the current implementation:

Layer Configuration: I noticed that TinyEngram currently utilizes 4 Engram layers. However, the original paper suggests using 2 layers, and the experimental figures seem to indicate that placing them at layers 2 and 15 yields the best results. Could you share the intuition behind increasing this to 4 layers in this specific implementation?

Vocab Size Ratio: I'm curious about the $10:1$ ratio between the 2-gram and 3-gram vocabulary sizes (10k vs. 1k). Intuitively, one might expect 3-gram patterns to be more diverse, potentially requiring a larger vocabulary to capture the increased complexity. Was this specific ratio chosen based on empirical observations in your small-scale tests, or is there a specific structural reason for keeping the 3-gram vocabulary more compact?

I would love to hear your thoughts on these design choices.

Thanks again for your time and for sharing this inspiring work!

Best regards

[Alichua]

Hi coruse123,
Thanks for the questions, I’ll share our current reasoning and some observations, though these are based on limited experiments and are still evolving.

For Layer Configuration:
The original paper was studied on large-scale model with MoE and full hyper-connections. Our setting is quite different: Qwen-0.6B + dense (hc_mult = 1) + no MoE. Qwen-0.6B uses fewer layers overall (24 layers), of which Layer 2 is very early, and Layer 15 is relatively deep. For such a small model, we expect it to benefit from having more opportunities to query the memory bank.

We tried both 2-layer (e.g., [5,17]) and 4-layer ([5,7,13,17]) setups in a few runs. The 4-layer version shows more stable convergence in eval loss. Based on these early trials, we went with 4 layers.

For Vocab Size: 2-gram (10k) vs. 3-gram (1k)

Indeed, the theoretical space of 3-grams ($V^3$) is much larger than that of 2-grams ($V^2$). However, in our experiments, the choice was mainly guided by data sparsity in small SFT dataset(e.g. biomedical corpora), rather than theoretical capacity.

The example below shows why 3-grams are much sparser and harder to learn:

Consider two sentences with similar structures:

Sentence A: "The quick brown fox jumps over the lazy dog."
Sentence B: "The quick red fox runs over the lazy cat."

• 2-grams (Bigrams): Both sentences share high-frequency structural pairs: The quick, over the, the lazy. These are dense and appear constantly across the dataset, ensuring their embeddings in the 10k table are frequently updated and well-trained.
• 3-grams (Trigrams): Do they share any 3-grams?
  • The quick brown vs The quick red -> Mismatch
  • quick brown fox vs quick red fox -> Mismatch
  • jumps over the vs runs over the -> Mismatch

Even in this simple case, zero 3-grams overlap. Hence, in the SFT scenario that TinyEngram currently focuses, high frequency 3-grams can be quite sparse.

Our intuition is that the 2-gram table mainly captures common structural patterns, while the smaller 3-gram table serves as a lightweight cache for only the most frequent longer patterns.

Overall, these are practical defaults for small-scale experiments—not strict recommendations. The best setup likely depends on model size, data scale, and domain. If you try other setups, we’d be really curious to hear how they work. Forks/experiments/issues are all appreciated.

[Alichua]

For anyone interested in vocab size selection or other TinyEngram hyperparameters: we just released a more complete ablation study.

Full report: engram_parameters_tuning.md

We’ve opened a dedicated discussion thread for hyperparameter tuning: #6