-
Notifications
You must be signed in to change notification settings - Fork 494
[WIP] fix: preserve whitespace when removing translation tokens (fix #1302) #1324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
serhii73
wants to merge
2
commits into
master
Choose a base branch
from
fix/1302-preserve-whitespace
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+199
−29
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,40 @@ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #!/usr/bin/env python3 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| """Test script to check current whitespace behavior""" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| from dateparser.languages import default_loader | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| from dateparser.conf import settings | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| def test_current_behavior(): | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| locale = default_loader.get_locale("fi") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Test 1: Single spaces | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| input1 = "28 maalis klo 9:37" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| result1 = locale.translate(input1, settings=settings) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Input1: |{input1}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Result1: |{result1}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print("Expected1: |28 march 9:37|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Match: {result1 == '28 march 9:37'}") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Test 2: Double spaces | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| input2 = "28 maalis klo 9:37" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| result2 = locale.translate(input2, settings=settings) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Input2: |{input2}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Result2: |{result2}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print("Expected2: |28 march 9:37|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Match: {result2 == '28 march 9:37'}") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Test 3: Triple spaces | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| input3 = "28 maalis klo 9:37" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| result3 = locale.translate(input3, settings=settings) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Input3: |{input3}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Result3: |{result3}|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print("Expected3: |28 march 9:37|") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Match: {result3 == '28 march 9:37'}") | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+14
to
+36
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| print(f"Input1: |{input1}|") | |
| print(f"Result1: |{result1}|") | |
| print("Expected1: |28 march 9:37|") | |
| print(f"Match: {result1 == '28 march 9:37'}") | |
| print() | |
| # Test 2: Double spaces | |
| input2 = "28 maalis klo 9:37" | |
| result2 = locale.translate(input2, settings=settings) | |
| print(f"Input2: |{input2}|") | |
| print(f"Result2: |{result2}|") | |
| print("Expected2: |28 march 9:37|") | |
| print(f"Match: {result2 == '28 march 9:37'}") | |
| print() | |
| # Test 3: Triple spaces | |
| input3 = "28 maalis klo 9:37" | |
| result3 = locale.translate(input3, settings=settings) | |
| print(f"Input3: |{input3}|") | |
| print(f"Result3: |{result3}|") | |
| print("Expected3: |28 march 9:37|") | |
| print(f"Match: {result3 == '28 march 9:37'}") | |
| print() | |
| assert result1 == "28 march 9:37" | |
| # Test 2: Double spaces | |
| input2 = "28 maalis klo 9:37" | |
| result2 = locale.translate(input2, settings=settings) | |
| assert result2 == "28 march 9:37" | |
| # Test 3: Triple spaces | |
| input3 = "28 maalis klo 9:37" | |
| result3 = locale.translate(input3, settings=settings) | |
| assert result3 == "28 march 9:37" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -80,7 +80,7 @@ def setUp(self): | |||||||||
| # French | ||||||||||
| param("fr", "20 Février 2012", "20 february 2012"), | ||||||||||
| param("fr", "Mercredi 19 Novembre 2013", "wednesday 19 november 2013"), | ||||||||||
| param("fr", "18 octobre 2012 à 19 h 21 min", "18 october 2012 19:21"), | ||||||||||
| param("fr", "18 octobre 2012 à 19 h 21 min", "18 october 2012 19:21"), | ||||||||||
| # German | ||||||||||
| param("de", "29. Juni 2007", "29. june 2007"), | ||||||||||
| param("de", "Montag 5 Januar, 2015", "monday 5 january 2015"), | ||||||||||
|
|
@@ -109,49 +109,57 @@ def setUp(self): | |||||||||
| param("it", "Giovedi Maggio 29 2013", "thursday may 29 2013"), | ||||||||||
| param("it", "19 Luglio 2013", "19 july 2013"), | ||||||||||
| # Portuguese | ||||||||||
| param("pt", "22 de dezembro de 2014 às 02:38", "22 december 2014 02:38"), | ||||||||||
| param("pt", "22 de dezembro de 2014 às 02:38", "22 december 2014 02:38"), | ||||||||||
| # Russian | ||||||||||
| param("ru", "5 августа 2014 г. в 12:00", "5 august 2014 year. 12:00"), | ||||||||||
| param("ru", "5 августа 2014 г. в 12:00", "5 august 2014 year. 12:00"), | ||||||||||
| # Turkish | ||||||||||
| param("tr", "2 Ocak 2015 Cuma, 16:49", "2 january 2015 friday 16:49"), | ||||||||||
| # Czech | ||||||||||
| param("cs", "22. prosinec 2014 v 2:38", "22. december 2014 2:38"), | ||||||||||
| param( | ||||||||||
| "cs", "22. prosinec 2014 v 2:38", "22. december 2014 2:38" | ||||||||||
| ), # Issue #1302: v becomes in, (cleared, leaves 2 spaces) | ||||||||||
|
Comment on lines
+119
to
+120
|
||||||||||
| "cs", "22. prosinec 2014 v 2:38", "22. december 2014 2:38" | |
| ), # Issue #1302: v becomes in, (cleared, leaves 2 spaces) | |
| "cs", "22. prosinec 2014 v 2:38", "22. december 2014 2:38" | |
| ), # Issue #1302: skip token "v" is removed and surrounding spaces are merged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| """ | ||
| Tests for issue #1302: Whitespace preservation during translation | ||
| """ | ||
|
|
||
| from parameterized import param, parameterized | ||
|
|
||
| from dateparser.conf import settings | ||
| from dateparser.languages import default_loader | ||
| from tests import BaseTestCase | ||
|
|
||
|
|
||
| class TestWhitespacePreservation(BaseTestCase): | ||
| """ | ||
| Tests to ensure that whitespace is preserved exactly when translating | ||
| date strings, even when tokens are removed from the skip list (e.g., "klo" in Finnish). | ||
|
|
||
| Issue #1302: Extra whitespace handling during date translation | ||
| """ | ||
|
|
||
| def setUp(self): | ||
| super().setUp() | ||
| self.language = NotImplemented | ||
| self.datetime_string = NotImplemented | ||
| self.translation = NotImplemented | ||
| self.settings = settings | ||
|
|
||
| @parameterized.expand( | ||
| [ | ||
| # Single space preservation - Finnish | ||
| param("fi", "28 maalis klo 9:37", "28 march 9:37"), | ||
| # Double space preservation - Finnish | ||
| param("fi", "28 maalis klo 9:37", "28 march 9:37"), | ||
| # Triple space preservation - Finnish | ||
| param("fi", "28 maalis klo 9:37", "28 march 9:37"), | ||
| # Mixed whitespace - Finnish | ||
| param("fi", "28 maalis klo 9:37", "28 march 9:37"), | ||
| # More complex Finnish date with whitespace | ||
| param( | ||
| "fi", "tiistaina 27. lokakuuta 2015", "tuesday 27. october 2015" | ||
| ), | ||
| ] | ||
| ) | ||
| def test_whitespace_preservation_during_translation( | ||
| self, shortname, datetime_string, expected_translation | ||
| ): | ||
| """Test that exact whitespace is preserved when translating date strings.""" | ||
| self.given_bundled_language(shortname) | ||
| self.given_string(datetime_string) | ||
| self.when_datetime_string_translated() | ||
| self.then_string_translated_to(expected_translation) | ||
|
|
||
| def given_bundled_language(self, shortname): | ||
| self.language = default_loader.get_locale(shortname) | ||
|
|
||
| def given_string(self, datetime_string): | ||
| self.datetime_string = datetime_string | ||
|
|
||
| def when_datetime_string_translated(self): | ||
| self.translation = self.language.translate( | ||
| self.datetime_string, settings=self.settings | ||
| ) | ||
|
|
||
| def then_string_translated_to(self, expected_string): | ||
| self.assertEqual( | ||
| expected_string, | ||
| self.translation, | ||
| f"\nExpected: |{expected_string}|\nGot: |{self.translation}|\n" | ||
| f"Input: |{self.datetime_string}|", | ||
| ) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Locale.translate now performs a custom pass over tokens to merge spaces around removed skip tokens. Given how central translate() is, please add/extend tests that exercise this behavior with keep_formatting=True as well (where separator="" is used) to ensure the change doesn't introduce spacing regressions in formatting-preserving mode.