fix: overhaul conversation extraction and add optional LLM validation

[NotYuSheng]

Closes #1, #2, #3, #4, #5, #6

Summary

• Time gap check references wrong message index #2 Fixed time gap check — now tracks last_unixtime within the chain instead of using the raw previous list index
• Prompt only captures first message when other person sends multiple in a row #1 / Skipped non-message events silently reset time gap reference during response cascade #4 Collects all consecutive messages from the same sender as a single turn; non-message events no longer corrupt the time reference
• No conversation context — each prompt/response pair is extracted in isolation #3 Splits each chat into distinct conversations using a configurable CONVERSATION_GAP_THRESHOLD (default 1 hour)
• No conversation context — each prompt/response pair is extracted in isolation #3 / Output is always single-turn; multi-turn conversations are never grouped #5 Groups full back-and-forth exchanges into multi-turn samples instead of isolated prompt/response pairs
• Output is always single-turn; multi-turn conversations are never grouped #5 Updated convert_to_sharegpt.py for the new multi-turn format (human/gpt role mapping)
• Add optional LLM validation pass for conversation grouping quality #6 New scripts/validator.py: optional LLM pass that scores each sample for coherence and quality — controlled via DIALOGSMITH_LLM_VALIDATE env var, gracefully skipped when disabled or no API key is set
• Added .env.example documenting DIALOGSMITH_LLM_VALIDATE, DIALOGSMITH_LLM_MODEL, ANTHROPIC_API_KEY

Test plan

• Run telegram_extract.py against a real result.json export and verify multi-turn output
• Verify conversations split correctly across long silence gaps
• Verify consecutive messages from same sender are merged into one turn
• Run convert_to_sharegpt.py and verify ShareGPT output structure
• Test validator with DIALOGSMITH_LLM_VALIDATE=false — confirm it skips cleanly
• Test validator with a valid ANTHROPIC_API_KEY — confirm filtering and audit log

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces multi-turn conversation support for Telegram data extraction and ShareGPT conversion. Key changes include a refactored extraction script that groups messages into distinct conversations based on time gaps, and a new LLM-based validator that uses Anthropic's API to score conversation coherence and quality. Feedback focuses on resolving a logic redundancy between message chaining and turn merging, improving the robustness of JSON parsing from LLM responses, and parallelizing API requests to enhance validation performance.

[gemini-code-assist]

The MESSAGE_CHAIN_THRESHOLD (currently 30s) is effectively ignored because consecutive turns with the same role are unconditionally merged here. Since collect_turn already stops when the sender changes, any consecutive calls to it within the same conversation will necessarily have the same role and thus be merged. If the goal is to strictly group all consecutive messages from the same sender into a single turn for ShareGPT compatibility, the threshold check in collect_turn (lines 68-69) is redundant. If you intended to split turns based on time, this merge logic should be removed, though that might produce non-alternating roles which are incompatible with some training formats.

[gemini-code-assist]

Directly calling json.loads() on the LLM response is fragile. LLMs frequently include markdown formatting (e.g., json ... ) or conversational preambles even when instructed to return only JSON. It is more robust to extract the JSON block using a regular expression.

Suggested change

	result = json.loads(raw)
	import re
	match = re.search(r'\{.*\}', raw, re.DOTALL)
	if not match:
	raise ValueError(f"No JSON object found in LLM response: {raw}")
	result = json.loads(match.group())

[gemini-code-assist]

This loop performs LLM validation sequentially, which will be slow for large datasets. Consider using concurrent.futures.ThreadPoolExecutor to parallelize the API requests and significantly improve performance, especially since the validator runs once per sample.