fix: Preserve Line Breaks in Code Blocks During Chunking

[eureka928]

Problem

When using chunk_elements() on markdown files containing code blocks, line breaks within the code were being discarded, resulting in unreadable code:

Fixes #4095

# Before fix - code becomes flattened:
"def hello(): print('Hello') return True"

# Expected - preserve formatting:
"def hello():\n    print('Hello')\n    return True"

Root Cause

Two issues were identified:

1. HTML Parser: <pre> elements generated generic Text elements instead of CodeSnippet elements
2. Chunking: The _iter_text_segments() method normalized all whitespace to single spaces, destroying newlines

Solution

1. HTML Parser Change (unstructured/partition/html/parser.py)

Made <pre> elements generate CodeSnippet elements:

class Pre(BlockItem):
    """Custom element-class for `<pre>` element.

    Can only contain phrasing content. Generates CodeSnippet elements to preserve
    code formatting including whitespace and line breaks.
    """

    _ElementCls = CodeSnippet  # Added this line

2. Chunking Change (unstructured/chunking/base.py)

Modified _iter_text_segments() to preserve whitespace for CodeSnippet elements:

def _iter_text_segments(self) -> Iterator[str]:
    """Generate overlap text and each element text segment in order.

    Empty text segments are not included. CodeSnippet elements preserve their
    original whitespace (including newlines) to maintain code formatting.
    """
    if self._overlap_prefix:
        yield self._overlap_prefix
    for e in self._elements:
        if e.text and len(e.text):
            # -- preserve whitespace for code snippets to maintain formatting --
            if isinstance(e, CodeSnippet):
                text = e.text.strip()
            else:
                text = " ".join(e.text.strip().split())
            if text:
                yield text

Files Changed

File	Change
unstructured/partition/html/parser.py	Added CodeSnippet import, set _ElementCls = CodeSnippet in Pre class
unstructured/chunking/base.py	Added CodeSnippet import, special handling in _iter_text_segments()
test_unstructured/partition/html/test_parser.py	Added test for CodeSnippet generation, updated existing test
test_unstructured/chunking/test_base.py	Added 2 tests for whitespace preservation

Contribution by Gittensor, see my contribution statistics at https://gittensor.io/miners/details?githubId=42954461

[eureka928]

This issue is for test_ingest_src fail... I don't know the reason

[eureka928]

@badGarnet finally, this passed all!
Hope you merge this when you have a sec
Have a nice weekend