Feature/i18n react i18next

[zhb-ai]

No description provided.

[zhb-ai]

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

This PR introduces React i18n support (react-i18next) and expands internationalization across the frontend, while also adding a substantial set of backend/frontend tests and improving Unicode handling and model configuration (including server-managed “global models”).

Changes:

• Add i18next + react-i18next setup, locale resources (en/zh), and replace many hard-coded UI strings with t(...) calls.
• Add global model registry + API support (server-managed models) and related backend tests; improve agent prompt language control and diagnostics payloads.
• Add Vitest-based frontend unit tests and broaden Python test coverage around Unicode table-name sanitization, JSON serialization, and upload/parse flows (including legacy .xls parsing).

Reviewed changes

Copilot reviewed 116 out of 121 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
vitest.config.ts	Adds Vitest configuration for frontend unit tests (jsdom + setup file).
tests/frontend/unit/views/safeCellRender.test.tsx	Tests safe rendering of object/boolean cell values in React tables/grids.
tests/frontend/unit/views/checkIsLikelyTextOnlyModel.test.ts	Tests heuristic for text-only model detection.
tests/frontend/unit/data/resolveExcelCellValue.test.ts	Tests Excel cell value normalization for richText/hyperlink/formula/error.
tests/frontend/unit/data/coerceDate.test.ts	Tests date coercion behavior for Date/null/strings/timestamps.
tests/frontend/unit/app/dfSelectors.test.ts	Tests selector logic for active model selection.
tests/frontend/setup.ts	Adds jest-dom matchers for Vitest.
tests/frontend/README.md	Documents frontend test layout and Vitest commands.
tests/conftest.py	Ensures py-src is importable for Python tests.
tests/backend/unit/test_workspace_fresh_names.py	Tests workspace fresh-name generation with Unicode names.
tests/backend/unit/test_unicode_table_name_sanitization.py	Tests Unicode preservation across multiple sanitizers.
tests/backend/unit/test_parquet_utils_table_names.py	Tests parquet sanitizer for safety, casing, and Unicode.
tests/backend/unit/test_model_registry.py	Tests env-var model registry loading + credential isolation.
tests/backend/unit/test_list_global_models_api.py	Tests /api/agent/list-global-models response shape and secret redaction.
tests/backend/unit/test_json_chinese_serialization.py	Regression tests for json.dumps(..., ensure_ascii=False) patterns.
tests/backend/unit/test_global_model_security.py	Tests global model credential resolution + error sanitization properties.
tests/backend/unit/test_file_manager_table_names.py	Tests file-manager table name sanitization with Unicode and edge cases.
tests/backend/unit/test_external_data_loader_table_names.py	Tests external loader table name sanitization rules/limits.
tests/backend/unit/test_duckdb_notes_prompt.py	Ensures DuckDB notes mention non-ASCII identifier quoting rule.
tests/backend/unit/test_client_image_strip.py	Tests client retry logic and stripping image_url blocks for text-only models.
tests/backend/unit/test_agent_utils_sql_table_names.py	Tests SQL sanitizer and DuckDB view creation with Unicode names.
tests/backend/unit/README.md	Documents backend unit test conventions.
tests/backend/integration/test_parse_file_endpoint.py	Integration tests for server-side parse-file endpoint.
tests/backend/integration/test_excel_fixture_parsing.py	Ensures .xls fixture can be parsed by pandas.
tests/backend/integration/test_create_table_xls_upload.py	End-to-end .xls upload flow tests (workspace + list-tables).
tests/backend/integration/README.md	Documents backend integration testing scope.
tests/backend/fixtures/README.md	Documents fixture directory usage.
tests/backend/contract/test_table_name_contracts.py	Contract tests for route-level sanitization guarantees with Unicode.
tests/backend/contract/README.md	Documents contract test intent for boundary stability.
tests/backend/README.md	Documents backend test layering and recommended expansion order.
tests/README.md	Documents overall test tree organization and commands.
src/views/SelectableDataGrid.tsx	Adds i18n strings, fixed table layout/colgroup, safer cell rendering, UI tweaks.
src/views/ReactTable.tsx	Prevents rendering object values directly by stringifying objects.
src/views/MultiTablePreview.tsx	i18n for empty/preview/remove labels and rows×cols display.
src/views/MessageSnackbar.tsx	Adds diagnostics viewer, i18n, and message-button severity indicator.
src/views/EncodingShelfThread.tsx	Adjusts encoding shelf width.
src/views/EncodingBox.tsx	i18n for channel labels/tips and several UI strings; minor refactors.
src/views/DataView.tsx	Makes row sampling deterministic and standardizes row-id column sizing.
src/views/DataThreadCards.tsx	Adds t(...) to table-card tooltips/aria labels for i18n.
src/views/DataLoadingThread.tsx	i18n throughout, adds text-only model heuristic + safer preview formatting.
src/views/DataFormulator.tsx	Uses unified model selector, i18n for landing/footer, changes model auto-select behavior.
src/views/DBTableManager.tsx	i18n for DB manager UI strings and status messages.
src/views/ChatThreadView.tsx	i18n for labels, accessibility improvements for collapsible rows.
src/views/ChatDialog.tsx	i18n for dialog labels and buttons.
src/views/ChartifactDialog.tsx	i18n for report title/footer text via i18n.t(...).
src/views/ChartRecBox.tsx	i18n placeholders/tooltips and minor prompt helper refactor.
src/views/AgentRulesDialog.tsx	i18n for rule dialog labels and buttons.
src/views/About.tsx	i18n for feature descriptions and accessibility labels.
src/scss/DataView.scss	Adjusts styling for row-id header cell sizing.
src/index.tsx	Initializes i18n on app startup.
src/i18n/locales/zh/upload.json	Adds Chinese strings for upload flow.
src/i18n/locales/zh/navigation.json	Adds Chinese strings for navigation.
src/i18n/locales/zh/model.json	Adds Chinese strings for model UI.
src/i18n/locales/zh/messages.json	Adds Chinese strings for messages UI.
src/i18n/locales/zh/index.ts	Aggregates zh locale modules.
src/i18n/locales/zh/encoding.json	Adds Chinese strings for encoding UI.
src/i18n/locales/zh/chart.json	Adds Chinese strings for chart UI.
src/i18n/locales/index.ts	Exports en and zh locale bundles.
src/i18n/locales/en/upload.json	Adds English strings for upload flow.
src/i18n/locales/en/navigation.json	Adds English strings for navigation.
src/i18n/locales/en/model.json	Adds English strings for model UI.
src/i18n/locales/en/messages.json	Adds English strings for messages UI.
src/i18n/locales/en/index.ts	Aggregates en locale modules.
src/i18n/locales/en/encoding.json	Adds English strings for encoding UI.
src/i18n/locales/en/chart.json	Adds English strings for chart UI.
src/i18n/index.ts	Initializes i18next + language detection and registers resources.
src/data/utils.ts	Adds resolveExcelCellValue() and uses it when reading Excel via ExcelJS.
src/data/types.ts	Updates date coercion to convert Date objects to ISO strings.
src/app/utils.tsx	Adds URLs, passes Accept-Language header, and exposes getAgentLanguage().
src/app/useFormulateData.ts	Adds onError callbacks and attaches diagnostics payload to failure messages.
src/app/store.ts	Blacklists globalModels from persistence to avoid stale server-managed models.
src/app/dfSlice.tsx	Adds global models support, new thunks, selector changes, and status handling.
requirements.txt	Adds xlrd dependency for legacy .xls parsing.
pytest.ini	Defines test paths/markers and standardizes pytest discovery.
pyproject.toml	Adds xlrd runtime dep and pytest dev dependency.
py-src/data_formulator/tables_routes.py	Adds parse-file endpoint; improves metadata filling and list-tables fallback behavior.
py-src/data_formulator/sandbox/not_a_sandbox.py	Improves error diagnostics for missing/incorrect output DataFrame variable.
py-src/data_formulator/sandbox/local_sandbox.py	Adds diagnostics (DataFrame variable names) for sandbox execution results/errors.
py-src/data_formulator/sandbox/docker_sandbox.py	Improves error message with DataFrame variable diagnostics.
py-src/data_formulator/model_registry.py	Adds env-based global model registry with safe public listing.
py-src/data_formulator/datalake/workspace.py	Ensures session JSON uses ensure_ascii=False for Unicode readability.
py-src/data_formulator/datalake/parquet_utils.py	Updates table-name sanitization to preserve Unicode while staying safe.
py-src/data_formulator/datalake/file_manager.py	Updates table-name sanitization to preserve Unicode while staying safe.
py-src/data_formulator/datalake/azure_blob_workspace.py	Ensures session JSON uses ensure_ascii=False for Unicode readability.
py-src/data_formulator/data_loader/external_data_loader.py	Improves sanitizer to preserve Unicode and normalize separators safely.
py-src/data_formulator/app.py	Adds AVAILABLE_LANGUAGES to app config; refactors logging config.
py-src/data_formulator/agents/data_agent.py	Adds language instruction injection into the system prompt.
py-src/data_formulator/agents/client_utils.py	Adds image-block stripping + retry logic and a ping() helper.
py-src/data_formulator/agents/agent_utils_sql.py	Updates SQL table-name sanitization for Unicode and safer prefixes.
py-src/data_formulator/agents/agent_utils.py	Adds lenient JSON parsing and output-variable heuristic patch helper.
py-src/data_formulator/agents/agent_sort_data.py	Uses ensure_ascii=False when serializing inputs for LLM prompts.
py-src/data_formulator/agents/agent_report_gen.py	Adds language instruction injection for report generation prompts.
py-src/data_formulator/agents/agent_language.py	Adds shared language-instruction builder for agent prompts.
py-src/data_formulator/agents/agent_interactive_explore.py	Adds language instruction injection into system prompt.
py-src/data_formulator/agents/agent_data_load.py	Adds language instruction injection into system prompt.
py-src/data_formulator/agents/agent_data_clean_stream.py	Adds language instruction injection and uses ensure_ascii=False in stream output.
py-src/data_formulator/agents/agent_code_explanation.py	Adds language instruction injection into system prompt.
py-src/data_formulator/agents/agent_chart_insight.py	Adds language instruction injection into system prompt.
package.json	Adds i18n deps and Vitest/testing-library tooling + test scripts.
.gitignore	Ignores *.egg-info/ and tmp-docs/.
.env.template	Documents LOG_LEVEL, DATA_FORMULATOR_HOME, and AVAILABLE_LANGUAGES.

Comments suppressed due to low confidence (1)

src/views/MessageSnackbar.tsx:133

• This effect uses displayedMessageIdx and dispatch but the dependency array only includes messages. That can lead to stale reads of displayedMessageIdx (and will trip react-hooks/exhaustive-deps). Include displayedMessageIdx and dispatch in the dependency list (or refactor to derive the next index from messages.length if that’s the intended trigger).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

configure_logging() reads LOG_LEVEL, but logging.basicConfig(level=logging.WARNING, ...) sets the root logger to WARNING, so INFO/DEBUG logs from data_formulator will still be filtered out even if LOG_LEVEL=INFO/DEBUG. Set the root/handler level to app_log_level (or configure a dedicated handler for data_formulator) so the env var actually controls emitted log verbosity.

[zhb-ai]

Thank you for the review, but this issue does not apply to the current code. The snippet referenced in your comment (with duplicate level=logging.ERROR / level=logging.WARNING parameters and a separate data_formulator.agents logger) appears to be from an intermediate revision and is not present in the final version.

In the current implementation, logging.basicConfig(level=logging.WARNING, ...) sets the root logger level to WARNING, while logging.getLogger('data_formulator').setLevel(app_log_level) explicitly sets our application logger to the level specified by the LOG_LEVEL env var. This works correctly because Python's log propagation (callHandlers) only checks handler levels when walking up the logger hierarchy — it does not re-check parent logger levels. Since the root StreamHandler defaults to NOTSET, all records that pass the originating logger's effective level check will be emitted. The root WARNING level only serves to suppress noisy INFO/DEBUG output from third-party libraries that don't have an explicit level override, which is the intended behavior.

No changes needed here.

[Copilot]

This README says tests/frontend are pytest-driven contract tests and recommends running them via pytest tests/frontend, but this PR introduces Vitest + React Testing Library tests under tests/frontend/** and adds npm test/vitest scripts. Update this documentation to reflect the new frontend test runner and commands (and clarify what, if anything, remains pytest-driven under tests/frontend).

[zhb-ai]

The root tests/README.md was outdated and contradicted the actual test setup:

It described tests/frontend/ as "pytest-driven contract tests," but all files there are .test.ts / .test.tsx running under Vitest + @testing-library/react.
It stated "not intended for React component-level unit tests yet," but safeCellRender.test.tsx is exactly that.
The suggested command pytest tests/frontend would find zero tests since there are no Python files in that directory.
Meanwhile, tests/frontend/README.md already correctly documented the Vitest setup — so the two READMEs directly contradicted each other.

Fixed by updating the root tests/README.md to accurately reflect the frontend test stack (Vitest), the correct commands (npm test / npm run test:watch), and a cross-reference to tests/frontend/README.md for layout details.
comment ID:
07fe32f

[Copilot]

Client defines _strip_image_blocks, _strip_images_from_messages, and _is_image_deserialize_error twice (the second set starting here duplicates the earlier implementation). This duplication makes future changes error-prone (it’s easy to edit one copy and forget the other). Remove the duplicate method definitions and keep a single authoritative implementation.

Suggested change

	def _strip_image_blocks(self, content):
	"""Remove image_url blocks from multimodal content arrays."""
	if isinstance(content, list):
	sanitized = []
	for item in content:
	if isinstance(item, dict):
	if item.get("type") == "image_url":
	continue
	sanitized.append(item)
	else:
	sanitized.append(item)
	return sanitized
	return content
	def _strip_images_from_messages(self, messages):
	"""Create a copy of messages with image_url blocks removed."""
	sanitized_messages = []
	for msg in messages:
	if isinstance(msg, dict):
	new_msg = dict(msg)
	if "content" in new_msg:
	new_msg["content"] = self._strip_image_blocks(new_msg["content"])
	sanitized_messages.append(new_msg)
	else:
	sanitized_messages.append(msg)
	return sanitized_messages
	def _is_image_deserialize_error(self, error_text: str) -> bool:
	"""Detect provider errors caused by image blocks on text-only models."""
	lowered = error_text.lower()
	return ("image_url" in lowered and "expected `text`" in lowered) or "unknown variant `image_url`" in lowered

[zhb-ai]

Good catch — the three methods (_strip_image_blocks, _strip_images_from_messages, _is_image_deserialize_error) were indeed duplicated with identical implementations. This was a merge artifact. Removed the duplicate definitions, keeping a single authoritative copy. Fixed. fixed in comment ID 6c0da55

[Copilot]

list_tables() falls back to workspace.read_data_as_df(table_name) to infer columns and row_count when metadata/schema is missing. This can load entire tables into memory (and it may happen twice: once for columns and once for row_count), which is expensive for large parquet tables and can degrade API latency. Prefer using parquet schema/row-count metadata only (or a cheap head/sample) and avoid full reads inside the listing endpoint.

[zhb-ai]

Actually, this issue is more complex than described and requires a comprehensive solution. This will be included in the upcoming priority fix plan.

[Copilot]

parse_file() returns str(e) directly to the client on exceptions. This can leak internal details and results in inconsistent error messages vs other routes that sanitize DB/IO errors. Return a stable, user-safe message (and keep the full exception only in logs), ideally reusing the same sanitization pattern used elsewhere in this module.

Suggested change

	logger.error(f"Error parsing file: {str(e)}")
	return jsonify({"status": "error", "message": str(e)}), 400
	# Log full exception details for diagnostics, but return a sanitized message to the client.
	logger.error("Error parsing file", exc_info=True)
	safe_msg, _status_code = sanitize_db_error_message(e)
	return jsonify({"status": "error", "message": safe_msg}), 400

[zhb-ai]

I looked into this carefully and here's my assessment:

The security risk here is minimal. The exceptions in parse_file() come from pandas parsing in-memory streams (pd.ExcelFile(file.stream), pd.read_csv(file.stream)), not from database or filesystem operations. The resulting error messages describe data format issues (e.g. "Excel file format cannot be determined", "Error tokenizing data") and don't leak server internals like file paths or credentials. Returning these messages to the client is intentional — they tell the user what's wrong with their uploaded file.

Using sanitize_db_error_message here would actually be worse:

It's semantically designed for DB errors — none of its patterns match pandas parsing errors.
The fallback (f"An unexpected error occurred: {error_msg}") still exposes the full str(e), so it doesn't actually sanitize anything.
It would return status 500 instead of 400 — incorrect, since a malformed upload is a client error.
What I did adopt: Changed the logger call to use exc_info=True for full stack traces in logs, which is genuinely better for diagnostics than just logging str(e).

fix: improve file parsing error logging in comment ID 9b73ce4

Use exc_info=True to record complete exception stack trace for easier troubleshooting

[Chenglong-MS]

I think the main update to consider is to define a agentCofing / agentOptions to help manage shared configs for different agents; and diagnostic info can be wrapped separately to avoid duplicate.

[Chenglong-MS]

diag component seems to have some duplicates in both agent_data_load and agent_data_rec. It might be good to use a separate class to manage diagnostic messages.

[Chenglong-MS]

I think a better solution is to:

1. in prompt hint it to put output variable in the last line, and
2. ask the agent to also put the output variable in output json object

So the output variable prioritize 2 if it is available, otherwise finds the last line variable output (typically notebook structure)

if neither exists we just consider execution fail and ask agent to repair with an error message "output variable not specified."

[Chenglong-MS]

now that language instruction, repair attempt are becoming a little bit heavy weight to define for every agent. I'm wondering if we should just define an "agentConfig" class to pass these information to the agent, including

1. coding rules
2. exploration rules
3. max repair attempts
4. max iterations
5. language instructions
6. ...
   and we can extend properties later if needed

[Chenglong-MS]

I'm wondering besides manual cleaning here, is there any benefit using libraries like https://github.com/kayak/pypika to deal with sanitization?

[Chenglong-MS]

ChartGallery is tempory for backend visualization library testing and will be updated later... looks good for now.