Fix swapped data files, regenerate data, and add frictionless validate to CI#12
Open
olayway wants to merge 1 commit into
Open
Fix swapped data files, regenerate data, and add frictionless validate to CI#12olayway wants to merge 1 commit into
olayway wants to merge 1 commit into
Conversation
Bug: SOURCES and FILE_NAMES in scripts/process.py were ordered such that NY.GDP.DEFL.KD.ZG (GDP deflator) was written to inflation-consumer.csv and FP.CPI.TOTL.ZG (CPI) was written to inflation-gdp.csv — the two output file names were swapped. Fix: swap FILE_NAMES order so index 0 (NY.GDP.DEFL.KD.ZG) writes to inflation-gdp.csv and index 1 (FP.CPI.TOTL.ZG) writes to inflation-consumer.csv. Data regenerated: inflation-gdp.csv now contains GDP deflator values and inflation-consumer.csv now contains CPI values, consistent with their names and the resource descriptions in datapackage.json. CI: added 'frictionless validate datapackage.json' step to actions.yml to catch any future descriptor/data drift before it is merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bug fixed: data files were swapped
scripts/process.pydefines two parallel lists:SOURCES[0]is the GDP deflator indicator butFILE_NAMES[0]isinflation-consumer.csv. As a result:data/inflation-consumer.csvcontained GDP deflator valuesdata/inflation-gdp.csvcontained consumer price (CPI) valuesThis was confirmed by cross-checking the archive CSVs against the data files: Aruba's first value in
archive/NY.GDP.DEFL.KD.ZG.csv(GDP deflator, 1987 = 3.591…) matcheddata/inflation-consumer.csv, notdata/inflation-gdp.csv.Fix
Swapped
FILE_NAMESto['inflation-gdp.csv', 'inflation-consumer.csv']. Regenerated both data files.frictionless validatepasses on the regenerated output.CI addition
Added
frictionless validate datapackage.jsonstep to.github/workflows/actions.yml(runs aftermake data). This will fail the workflow if the descriptor and data files drift out of sync in the future.