Improve and align tutorials

[yairallouche]

Summary

Improve the quality, consistency, and completeness of the tutorials directory.

Tasks

1. Clean up tutorial 02 (RAG pipeline)

• Fix broken import path (govt_data_loader vs scripts/govt_data_loader)
• Fix threshold inconsistency between show_intermediates (0.4) and run_pipeline (0.5)
• Remove unused ANSWERABILITY_THRESHOLD constant
• Consolidate scattered imports
• Refactor large function cell into smaller, focused cells
• Add explanatory comments for non-obvious design choices (TOP_K, temperature, pipeline ordering)
• Improve govt_data_loader.py error handling and progress feedback
• Add diagram showing the 7-step pipeline flow, including the three exit points (blocked, unanswerable, needs clarification)

2. Align format across all tutorials

• Convert the hello-adapter to a notebook
• Standardize heading hierarchy (H1 title, H2 numbered sections)
• Standardize template (intro -> prerequisite -> content -> next steps )
• Add time estimates and prerequisites links to all tutorials
• Add "Next Steps" sections linking to related tutorials
• Consolidate imports in notebooks (not scattered across cells)
• Replace ANSI color codes with display(Markdown()) in notebook 01
• Use Rich Markdown rendering for the printing method (see RAG tutorial)
• Add "What you will learn" to each notebook
• Improve prose clarity across all tutorials
• Validate clear text in all notebooks
• Add missing context if needed (e.g., aLoRA vs LoRA explanation, attribution instruction context)
• Provide a one-line explanation on each adapter before usage
• Validate all links

3. Consider adding new tutorials

• Evaluate a simpler RAG tutorial (inline docs, no vector DB) as a stepping stone
• Evaluate a complex RAG tutorial with loop between answerability rewriting

[noaakl]

1. Notebooks

• Make sure that LoRA vs LoRA Race matches the Notebook Template.
• Organize and clean up the Next Steps section.

2. RAG

• Move all imports to the top of the file.
• Move the show script into utils.
• Add the following sentence:
  “The benchmark uses the government-services corpus from IBM mt-rag-benchmark as its retrieval dataset.”
• merge the vector DB init with Luis scripts in LoRA vs aLoRA
• address David's comment about the conversation class
• Add the missing actual query rewrite step in 5b; then consider renaming the current 5b to 5c.
• Review whether ModelOption.TEMPERATURE: 0.0 is needed in the mfuncs.act(...) call, or whether temperature should rely on the setting already defined in io.yaml.

3. Colab

• Verify that everything runs properly on Google Colab (similar to LoRA vs LoRA Race).
• Check how many documents are returned in RAG and limit it to 5 so it can run smoothly, while ensuring performance is not affected.

4. README

• Add Google Colab links to the notebooks table in the README.
• Update the following sentence:
  “The HuggingFace examples demonstrate how adapters are activated via control tokens. While this is helpful for understanding the underlying mechanics, for actual inference you should use Mellea (Path 2), which provides constrained decoding, prompt formatting, and proper input/output processing.”
• Add links to the scripts.

5. General

• Remove the Quickstart section.
• Rename 04_alora_vs_lora_race.ipynb to 05_alora_vs_lora_race.ipynb
• Verify that all links are correct and up to date.
• Remove the compose script from the notebooks and add a reference to the compose notebook instead.
• Align duration