feat(tts-cpp): add 23-language multilingual support with runtime MeCab/Cangjie paths

[freddy311082]

🎯 What problem does this PR solve?

• tts-cpp only supported 18 tier-1 languages; 5 tier-2 languages (ja, he, ru, zh, hi) were missing
• MeCab dictionary path for Japanese was hardcoded to /opt/homebrew/lib/mecab/dic/ipadic, breaking cross-platform builds
• Cangjie TSV path for Chinese was baked in at compile time, same problem

📝 How does it solve it?

• Add text_preprocess.h/.cpp with Japanese (MeCab), Chinese (Cangjie), and Korean (Jamo) preprocessing
• Expand supported_languages() from 18 to 23
• Replace compile-time dictionary paths with runtime resolution: EngineOptions::mecab_dict_path / cangjie_tsv_path, CLI --mecab-dict / --cangjie-tsv, env var fallbacks CHATTERBOX_MECAB_DICT / CHATTERBOX_CANGJIE_TSV
• Add scripts/build_mecab_dict.py to materialize IPAdic from pip (same pattern as tts-onnx)
• Add 23-language CTest integration tests (L0 registry, L1 wav structure, L2 audio sanity)
• Add optional L3 ASR verification tests via whisper-cli

🧪 How was it tested?

• All 23 languages pass L0/L1/L2 synthesis tests
• L3 ASR matrix: 22/23 pass, zh fails (CER ~97% — Cangjie approach needs rework)

🔌 API Changes

struct EngineOptions {
    // ...existing fields...
    std::string mecab_dict_path;   // runtime path to MeCab IPAdic dir (for ja)
    std::string cangjie_tsv_path;  // runtime path to Cangjie5_TC.tsv (for zh)
};

[GustavoA1604]

Need to check the following. Probably can just feed to cursor and it will be able to take care

1. Should zh indeed be in supported_languages()? The PR description states "zh fails (CER ~97% — Cangjie approach needs rework". Also on windows the zh test crashed on windows (see blocker #3)

2. generated.resize(n - 2) in the repetition early-stop trims too aggressively
   In chatterbox_cli.cpp, the 3-token early-stop was changed to 2-token with a kMtlMinTokensBeforeCadence = 60 guard. But generated.resize(n - 2) removes both the repeated token and the legitimate one before it. The old code didn't resize at all — the downstream pop_back() handled the trailing token. This should be resize(n - 1). Add a comment explaining the intent.

3. preprocess_japanese/preprocess_chinese exceptions cause std::terminate instead of a clean error exit. Both ja and zh crash rather than printing an error and returning 1. Root cause: s3gen_preload_thread (line 1031 of chatterbox_cli.cpp) is spawned when --s3gen-gguf is provided. When a preprocessing exception propagates out of the try block during stack unwind, this std::thread's destructor is called while it is still joinable, which unconditionally calls std::terminate(). The codebase has a comment at line 1184 explicitly noting this hazard, and guards against it in the --input-file path — but the normal synthesis path is unguarded. Fix: wrap s3gen_preload_thread in a RAII join guard, or join it in the catch block before rethrowing.

4. --output flag in test_multilingual_synth.cpp::run_synthesis() does not exist in the CLI

if (!args.tokens_dump_path.empty()) {
cli_args.push_back("--output"); // no such CLI arg
cli_args.push_back(args.tokens_dump_path);
}
chatterbox_cli.cpp has --out for the wav file, no --output. If --dump-tokens is passed to the test binary, tts_cpp_cli_main hits the unknown-argument handler and returns failure. Either remove the dead branch or add the matching CLI flag.

5. create_mecab_tagger does not pass -r, so a broken system mecabrc silently kills ja synthesis

Confirmed by local testing: MeCab reads the system mecabrc before honoring -d. On any host where the system rc points to a non-existent dictionary (e.g. MSYS2's default naist-jdic that isn't installed), mecab_new returns nullptr and MeCabTagger::load throws with an empty error message. Fix by passing the materialized mecabrc:

mecab_t * create_mecab_tagger(const std::string & dic_str) {
std::string a0 = "mecab";
std::string a1 = "-d";
std::string a2 = dic_str;
std::string a3 = "-r";
std::string a4 = (std::filesystem::path(dic_str) / "mecabrc").string();
char * argv[] = { a0.data(), a1.data(), a2.data(), a3.data(), a4.data() };
return mecab_new(5, argv);
}
build_mecab_dict.py already copies a neutral mecabrc (; This file can be empty but it needs to exist), so this is self-contained.

6. target_include_directories(tts-cpp PUBLIC ${MECAB_INCLUDE_DIR}) must be PRIVATE

Confirmed by build: PUBLIC propagates the entire MinGW include/ directory to every consumer of tts-cpp. On a Windows Clang-cl build this causes a catastrophic cascade of MinGW vs MSVC header conflicts (hundreds of errors) across every translation unit that includes any tts-cpp header. mecab.h is an implementation detail of text_preprocess.cpp — no public header exposes MeCab types.

was:
target_include_directories(tts-cpp PUBLIC ${MECAB_INCLUDE_DIR})
fix:
target_include_directories(tts-cpp PRIVATE ${MECAB_INCLUDE_DIR})

7. set_mecab_dict_path / set_cangjie_tsv_path — silent no-op if called after first synthesis

The std::call_once blocks in mtl_tokenizer.cpp capture the global path at first call. If set_mecab_dict_path is called after the first ja synthesis (e.g. constructing a second Engine with a different dict), the once_flag is spent and the new path is silently ignored. At minimum document this constraint on the setters; ideally assert or warn.

8. Korean (ko) skips decompose_korean_to_jamo — intentional?

apply_language_preprocessing dispatches on "ja" and "zh" but not "ko". The new decompose_korean_to_jamo function exists but is never called. If ko already works without Jamo decomposition this is fine, but it needs a comment explaining why the function exists without being called here.

9. Duplicate UTF-8 decoder

test_multilingual_asr.cpp contains its own utf8_to_codepoints (lines ~14–40) that duplicates decode_utf8 from text_preprocess.h, which is already a public header in the same library. The ASR test should #include "text_preprocess.h" and use the shared function.

[GustavoA1604]

The changes look good now. The only thing left is that there is no acquisition script for the Cangjie TSV (unlike MeCab, which has build_mecab_dict.py). After some investigation the canonical source is Jackchows/Cangjie5 — the Cangjie5_TC.txt file is already TAB-separated, so it's largely compatible with CangjieTable::load() already. I've written a build_cangjie_tsv.py script following the same pattern as build_mecab_dict.py

Also Cangjie5_TC.txt has ~2,529 lines with a third annotation column, e.g.:

昸 ahem [u]
parse_tsv_line uses line.find('\t') to find only the first tab, then sets row.code = line.substr(tab + 1). For annotated lines this gives code = "ahem\t[u]", which would emit tokens like [cj_a][cj_h][cj_e][cj_m] followed by a literal \t and [u] — breaking those characters. Fix is one line in text_preprocess.cpp:

row.code = line.substr(tab + 1);
// strip optional annotation column
const auto tab2 = row.code.find('\t');
if (tab2 != std::string::npos) row.code = row.code.substr(0, tab2);

build_cangjie_tsv.py strips the annotation at conversion time (so the TSV written to disk is already clean), but parse_tsv_line should still be hardened for any raw files callers might point at.

[freddy311082]

@GustavoA1604 CI here: https://github.com/tetherto/qvac/actions/runs/26643991409

[ogad-tether]

Review — omar.gad@tether.io

I reviewed the full diff (16 files, +2080/−17) against the prior review threads from @GustavoA1604. All 9 items from the first review and both items from the second review appear to be addressed — nice work. Below are new findings.

---

Blockers

1. Data race on global path strings (mtl_tokenizer.cpp)

g_mecab_dict_path and g_cangjie_tsv_path are bare std::string globals written by set_mecab_dict_path() / set_cangjie_tsv_path() and read by resolve_mecab_dict_path() / resolve_cangjie_tsv_path(). The std::call_once guards protect the tagger/table initialization, but the path strings themselves are read/written without synchronization. If two Engine instances are constructed on different threads (plausible in the addon), this is a data race — undefined behavior per the C++ standard.

Fix options:

• Protect the globals with a std::mutex, or
• Move the path into the std::call_once lambda's capture (snapshot at set time), or
• Use std::atomic<std::string*> with a publish pattern.

The simplest fix is probably a std::mutex guarding both the path and the initialized flag together.

2. TTS_CPP_HAS_MECAB compile definition should be PRIVATE, not PUBLIC

target_compile_definitions(tts-cpp PUBLIC TTS_CPP_HAS_MECAB)

MeCabTagger lives in src/text_preprocess.h — an internal header, not under include/. No public header exposes MeCab types. Making this PUBLIC leaks an implementation detail to every consumer of tts-cpp and can cause surprising behavior if a consumer happens to #include "text_preprocess.h" transitively. Same rationale as GustavoA1604's review #6 (include dirs) — this should be PRIVATE.

3. MeCab link library should be PRIVATE, not PUBLIC

target_link_libraries(tts-cpp PUBLIC ${MECAB_LIBRARY})

MeCab is only used in text_preprocess.cpp. No public header exposes MeCab symbols. PUBLIC linkage means install(EXPORT) records the dependency, and every consumer doing find_package(tts-cpp) must also have libmecab installed — even if they never synthesize Japanese. This should be PRIVATE.

---

Medium

4. build_cangjie_tsv.py defaults to master — not reproducible

DEFAULT_TAG = "master"

The MeCab script (build_mecab_dict.py) pins to a specific PyPI package version via ipadic. The Cangjie script downloads from Jackchows/Cangjie5 at whatever master happens to be. If upstream changes column format or renames the file, builds silently break. Pin to a specific commit or tag (e.g., --tag v1.0.0 or a commit SHA) as the default, with master available as an override.

5. text_preprocess.h includes <fstream> unnecessarily

The header declares CangjieTable::read_tsv_entries(std::ifstream &, ...) which requires <fstream>. But this is a private method — it doesn't need to be in the header at all. Consider moving it to the .cpp file (or making it a free function in the anonymous namespace), which lets you drop <fstream> from the header and reduces include bloat for every TU that includes this header.

---

Nits

6. CER metric doesn't match standard definition

In test_multilingual_asr.cpp:

const size_t denom = std::max(exp_n.size(), got_n.size());

Standard CER uses ref_len (expected length) as the denominator, not max(ref, hyp). Using max caps the metric at 1.0, which is convenient for thresholding but makes the logged "CER" values incomparable with published benchmarks. Consider renaming to "normalized edit distance" or switching to exp_n.size() as denominator. Not a blocker since this is test-only, but the log output says "CER" which could confuse someone comparing against paper numbers.

7. sh_quote() in ASR test is POSIX-only

test_multilingual_asr.cpp uses POSIX single-quote escaping + std::system(). On Windows this won't work (cmd.exe quoting is different). Fine if the L3 tests are POSIX-only, but worth a comment or a #ifdef _WIN32 skip.

8. Parakeet patches in this PR

The diff includes 4 new files under parakeet-cpp/patches/ (+659 lines). These appear to have come through the merge commit with master. They're unrelated to the multilingual feature — just flagging in case the intent was a clean feature-only diff for review purposes. If they were already reviewed and merged via another PR, no action needed.

---

Summary

The core multilingual implementation is solid — the MeCab/Cangjie architecture, runtime path resolution with env-var fallbacks, the tiered test matrix, and the zh exclusion are all well-thought-out. The three blockers above are focused on thread safety and build hygiene; they should be straightforward to fix. Happy to re-review once those land.

[ogad-tether]

Re-review — omar.gad@tether.io

Commit 63fc0c93 addresses all 7 items from my first review. Verified each fix:

#	Issue	Status
1	Data race on global path strings	✅ path_mutex() guards all reads/writes
2	TTS_CPP_HAS_MECAB PUBLIC → PRIVATE	✅
3	MeCab link PUBLIC → PRIVATE	✅
4	build_cangjie_tsv.py unpinned default	✅ Pinned to b76ed7fd
5	text_preprocess.h includes <fstream>	✅ Now <iosfwd> + std::istream &
6	CER denominator non-standard	✅ Now exp_n.size()
7	sh_quote POSIX-only undocumented	✅ Comment added

---

One remaining nit (non-blocking)

g_mecab_initialized is set before the tagger is actually initialized — misleading warning in a narrow race window

In mecab_tagger():

std::call_once(once, [] {
    {
        std::lock_guard<std::mutex> lk(path_mutex());
        g_mecab_initialized = true;   // ← set HERE
    }
    const std::string path = resolve_mecab_dict_path();  // ← but init happens AFTER
    ...
    t->load(path);
});

If Thread B calls set_mecab_dict_path("new_path") in the window between the flag being set and resolve_mecab_dict_path() being called:

• Thread B sees g_mecab_initialized == true, prints "new path will be ignored"
• Thread B then sets g_mecab_dict_path = "new_path"
• Thread A calls resolve_mecab_dict_path(), reads the new path, and uses it

So the warning says "will be ignored" but the path IS used. The fix is to move g_mecab_initialized = true (and the Cangjie equivalent) to the end of the call_once lambda, after t->load() completes. That way:

• set_*() before any mecab_tagger() call → sets path, no warning, tagger uses it ✅
• set_*() after mecab_tagger() completes → flag is true, warning fires, path genuinely ignored ✅
• set_*() during call_once (inherently racy by the caller) → no flag yet, no misleading warning ✅

Same pattern for cangjie_table().

This is a very narrow window and the practical impact is a spurious warning message (not UB), so it's non-blocking — but it's a quick one-line move that makes the semantics correct.

---

Verdict

Everything else looks good. The mutex pattern is correct (no deadlocks — the scoped lock releases before resolve_*() re-acquires, and std::call_once prevents re-entrant init). The thread_join_guard RAII is correct (safe on default-constructed thread). The 2-token early-stop with kMtlMinTokensBeforeCadence = 60 and resize(n-1) is correct. The Cangjie strip_trailing_tsv_columns hardening handles annotated lines properly.

Ship it once the flag-ordering nit above is addressed (or documented as intentional).