Skip to content

Conversation

@cka-y
Copy link
Contributor

@cka-y cka-y commented Jan 8, 2026

Summary:
This PR improves the robustness and error handling of the TDG feed import process by adding explicit validation for required fields, introducing a custom exception, and ensuring that database operations for each resource are safely isolated. The changes also refactor some code for clarity and maintainability.

Error handling and validation:

  • Added a custom exception class InvalidTDGFeedError to represent missing required fields in TDG datasets/resources.
  • Updated _update_common_tdg_fields to validate the presence of the publisher name and producer URL, logging a warning and raising InvalidTDGFeedError if they are missing.
  • In _process_tdg_dataset, added try/except blocks to catch InvalidTDGFeedError and IntegrityError per resource, logging and skipping invalid or problematic resources without interrupting the entire import process. [1] [2]

Database transaction safety:

  • Wrapped processing of each resource within a nested transaction (db_session.begin_nested()), ensuring that errors in one resource do not affect others.

Code refactoring and cleanup:

  • Simplified and unified the way static_feeds_by_dataset_id is updated using setdefault. [1] [2]
  • Improved logging for RT feed creation and updates by refactoring string formatting.
  • Minor code formatting and logic improvements for clarity. [1] [2] [3]
    Summarize the changes in the pull request including how it relates to any issues (include the #number, or link them).

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title fix: db transaction + undefined fields fix: tdg import db transaction handling + undefined fields Jan 8, 2026
@cka-y cka-y requested a review from Copilot January 8, 2026 16:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the Transport Data Gouv (TDG) feed import process by adding field validation, custom exception handling, and nested database transactions for resource-level error isolation. The changes aim to make the import more robust by allowing the process to continue even when individual resources fail validation or encounter database integrity errors.

Key changes:

  • Introduced InvalidTDGFeedError exception and validation for required fields (publisher name, producer URL)
  • Wrapped each resource's processing in a nested database transaction using begin_nested() for isolation
  • Added exception handlers to catch and log InvalidTDGFeedError and IntegrityError without stopping the entire import

@cka-y cka-y linked an issue Jan 8, 2026 that may be closed by this pull request
@cka-y cka-y marked this pull request as draft January 9, 2026 15:21
@cka-y cka-y marked this pull request as ready for review January 12, 2026 15:14
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@davidgamez davidgamez merged commit facdd6b into main Jan 15, 2026
2 of 3 checks passed
@davidgamez davidgamez deleted the feat/1531 branch January 15, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TDG feeds with appears with no information

3 participants