Skip to content

fix: CircleCI column naming inconsistency (#8497)#8701

Open
varsis wants to merge 2 commits intoapache:mainfrom
varsis:fix/8497-circle-ci
Open

fix: CircleCI column naming inconsistency (#8497)#8701
varsis wants to merge 2 commits intoapache:mainfrom
varsis:fix/8497-circle-ci

Conversation

@varsis
Copy link

@varsis varsis commented Feb 8, 2026

Fixes #8497

Summary

Fixed CircleCI plugin incremental data collection failure due to column naming inconsistency.

Root Cause

  • CircleCI API v2 returns created_at/updated_at (snake_case)
  • Pipeline model used CreatedDate/UpdatedDate → DB columns created_date/updated_date
  • Workflow/Job models correctly use CreatedAt/UpdatedAt → DB columns created_at/updated_at
  • Collectors queried workflows/jobs with wrong column name created_date (didn't exist in those tables)

Changes

  • Fixed workflow/job collector queries: created_datecreated_at
  • Standardized pipeline model fields to match API: CreatedDate/UpdatedDateCreatedAt/UpdatedAt
  • Created DB migration to rename pipeline table columns
  • Updated archived model for consistency
  • Fixed test CSV headers and removed obsolete ignore fields

Breaking Changes

  • DB schema change (column renames for pipelines table)
  • Requires migration for existing deployments
  • No API contract changes

Test Plan

  • Code builds successfully
  • Migration handles both MySQL and PostgreSQL
  • Safe for clean installs (checks column existence before renaming)
  • Preserves data during migration

CircleCi API returns created_at not created_date.
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. component/plugins This issue or PR relates to plugins pr-type/bug-fix This PR fixes a bug labels Feb 8, 2026
@varsis
Copy link
Author

varsis commented Feb 8, 2026

This pr requires some small tweaks to the tests which I thought ran locally but did not. I'll try and correct them tomorrow

@klesh
Copy link
Contributor

klesh commented Feb 9, 2026

created_at and updated_at are reserved fields that DevLake uses to track when a record is created or updated. Please don’t use them to store data from data sources.

@varsis
Copy link
Author

varsis commented Feb 9, 2026

created_at and updated_at are reserved fields that DevLake uses to track when a record is created or updated. Please don’t use them to store data from data sources.

@klesh The other raw tables also use these values. Would the best approach be to leave those columns as is and pull the created_date etc from the JSON blob?

@klesh
Copy link
Contributor

klesh commented Feb 12, 2026

created_at and updated_at are reserved fields that DevLake uses to track when a record is created or updated. Please don’t use them to store data from data sources.

@klesh The other raw tables also use these values. Would the best approach be to leave those columns as is and pull the created_date etc from the JSON blob?

What other raw tables are you referring to? created_at and updated_at in raw tables are used by DevLake to track the record itself as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/plugins This issue or PR relates to plugins pr-type/bug-fix This PR fixes a bug size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][CircleCI] plugin fails on second (incremental) data collection due to SQL error: unknown column created_date

2 participants