adding who indicator import - adolescent birth rate

statvar_imports/who/adolescent_birth_rate/README.md

@@ -0,0 +1,79 @@
+# WHO Adolescent Birth Rate Import
+
+## Overview
+This dataset contains national and regional-level statistics for the Adolescent Birth Rate (per 1,000 women).
+Specifically, it provides data for two distinct age groups:
+- 10-14 years (Y10T14)
+- 15-19 years (Y15T19)
+
+**type of place:** Country, Region (M49 codes), WHO Regions, Overseas Territory, Special Administrative Regions
+**years:** 2000 to 2024
+**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:europe, dcid:LatinAmericaAndCaribbean)
+
+## Data Source
+**Source URL:**
+https://data.who.int/indicators/i/24C65FE/27D371A
+
+**Provenance Description:**
+The data comes from the World Health Organization (WHO). It specifically tracks Sustainable Development Goal (SDG) Indicator 3.7.2, measuring the adolescent birth rate (per 1,000 women aged 10–14 and 15–19) globally across different geographic classifications.
+
+## Refresh Type
+Automatic Refresh
+
+For refresh of the data, the import includes a Python script (`data_download.py`) to automatically download the complete historical dataset directly from the WHO's backend Azure Blob storage endpoint.
+
+## Data Publish Frequency
+Release Frequency = Annual
+
+## How To Download Input Data
+To download the data, run the provided script:
+```bash
+python3 data_download.py
+```
+<<<<<<< HEAD
+This will fetch the latest full dataset and save it locally as `adolescent_birth_rate_data.csv`, making it available for processing.
+=======
+This will fetch the latest full dataset and save it locally as `input_files/adolescent_birth_rate_data.csv`, making it available for processing.
+>>>>>>> 77012e1c (Adding files)
+
+## Processing Instructions
+To process the WHO Adolescent Birth Rate data and generate statistical variables, use the following command from the `adolescent_birth_rate` directory:
+
+**For Data Run**
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+<<<<<<< HEAD
+    --input_data=adolescent_birth_rate_data.csv \
+=======
+    --input_data=input_files/* \
+>>>>>>> 77012e1c (Adding files)
+    --pv_map=adolescent_birth_rate_pvmap.csv \
+    --output_path=adolescent_birth_rate_output \
+    --config_file=adolescent_birth_rate_metadata.csv
+```
+
+This generates the following output files:
+- adolescent_birth_rate_output.csv
+- adolescent_birth_rate_output_stat_vars_schema.mcf
+- adolescent_birth_rate_output_stat_vars.mcf
+- adolescent_birth_rate_output.tmcf
+
+**For Data Quality Checks and validation**
+Validation of the data is done using the lint flag in the DataCommons import tool
+```bash
+java -jar datacommons-import-tool-0.1-jar-with-dependencies.jar lint adolescent_birth_rate_output_stat_vars_schema.mcf adolescent_birth_rate_output.csv adolescent_birth_rate_output.tmcf adolescent_birth_rate_output_stat_vars.mcf
+```
+
+This generates the following output files:
+- report.json
+- summary_report.csv
+- summary_report.html
+
+The report files can be analyzed to check for errors and warnings. Further, linting is performed on the generated output files using the DataCommons import tool.
+
+## Testing
+Testing is performed using the provided `test_data` directory:
+- Input: `test_data/adolescent_birth_rate_input.csv`
+- Output (expected): `test_data/adolescent_birth_rate_output.csv`
+- MCF (expected): `test_data/adolescent_birth_rate_output.tmcf`
+

statvar_imports/who/adolescent_birth_rate/adolescent_birth_rate_metadata.csv

@@ -0,0 +1,2 @@
+parameter,value
+output_columns,"observationDate,value,observationAbout,variableMeasured"

statvar_imports/who/adolescent_birth_rate/adolescent_birth_rate_pvmap.csv

@@ -0,0 +1,230 @@
+key,p1,v1,p2,v2,p3,v3,p4,v4,p5,v5
+RATE_PER_1000_N,value,{Number},populationType,Person,measuredProperty,fertilityRate,scalingFactor,1000,measurementDenominator,-measuredProperty
+DIM_TIME,observationDate,{Number},,,,,,,,
+Y10T14,age,[10 14 Years],,,,,,,,
+Y15T19,age,[15 19 Years],,,,,,,,
+FEMALE,gender,Female,,,,,,,,
+DIM_GEO_CODE_M49: 956,#ignore,ignore,,,,,,,,
+DIM_GEO_CODE_M49: 955,#ignore,ignore,,,,,,,,
+France,observationAbout,dcid:country/FRA,,,,,,,,
+Gabon,observationAbout,dcid:country/GAB,,,,,,,,
+Georgia,observationAbout,dcid:country/GEO,,,,,,,,
+Gambia,observationAbout,dcid:country/GMB,,,,,,,,
+Germany,observationAbout,dcid:country/DEU,,,,,,,,
+Greece,observationAbout,dcid:country/GRC,,,,,,,,
+Guatemala,observationAbout,dcid:country/GTM,,,,,,,,
+Guinea,observationAbout,dcid:country/GIN,,,,,,,,
+Guyana,observationAbout,dcid:country/GUY,,,,,,,,
+Haiti,observationAbout,dcid:country/HTI,,,,,,,,
+Honduras,observationAbout,dcid:country/HND,,,,,,,,
+"China, Hong Kong SAR",observationAbout,dcid:country/HKG,,,,,,,,
+Hungary,observationAbout,dcid:country/HUN,,,,,,,,
+Iceland,observationAbout,dcid:country/ISL,,,,,,,,
+India,observationAbout,dcid:country/IND,,,,,,,,
+Iraq,observationAbout,dcid:country/IRQ,,,,,,,,
+Ireland,observationAbout,dcid:country/IRL,,,,,,,,
+Israel,observationAbout,dcid:country/ISR,,,,,,,,
+Italy,observationAbout,dcid:country/ITA,,,,,,,,
+Côte d'Ivoire,observationAbout,dcid:country/CIV,,,,,,,,
+Jamaica,observationAbout,dcid:country/JAM,,,,,,,,
+Japan,observationAbout,dcid:country/JPN,,,,,,,,
+Kazakhstan,observationAbout,dcid:country/KAZ,,,,,,,,
+Kenya,observationAbout,dcid:country/KEN,,,,,,,,
+Republic of Korea,observationAbout,dcid:country/KOR,,,,,,,,
+Kyrgyzstan,observationAbout,dcid:country/KGZ,,,,,,,,
+Latin America and the Caribbean,observationAbout,dcid:LatinAmericaAndCaribbean,,,,,,,,
+Latvia,observationAbout,dcid:country/LVA,,,,,,,,
+Liberia,observationAbout,dcid:country/LBR,,,,,,,,
+Landlocked developing countries,observationAbout,dcid:undata-geo/G00403000,,,,,,,,
+Lithuania,observationAbout,dcid:country/LTU,,,,,,,,
+Luxembourg,observationAbout,dcid:country/LUX,,,,,,,,
+"China, Macao SAR",observationAbout,dcid:country/MAC,,,,,,,,
+Madagascar,observationAbout,dcid:country/MDG,,,,,,,,
+Malaysia,observationAbout,dcid:country/MYS,,,,,,,,
+Maldives,observationAbout,dcid:country/MDV,,,,,,,,
+Malta,observationAbout,dcid:country/MLT,,,,,,,,
+Mauritius,observationAbout,dcid:country/MUS,,,,,,,,
+Mexico,observationAbout,dcid:country/MEX,,,,,,,,
+Chile,observationAbout,dcid:country/CHL,,,,,,,,
+Colombia,observationAbout,dcid:country/COL,,,,,,,,
+Congo,observationAbout,dcid:country/COG,,,,,,,,
+Democratic Republic of the Congo,observationAbout,dcid:country/COD,,,,,,,,
+Costa Rica,observationAbout,dcid:country/CRI,,,,,,,,
+Croatia,observationAbout,dcid:country/HRV,,,,,,,,
+Cuba,observationAbout,dcid:country/CUB,,,,,,,,
+Cyprus,observationAbout,dcid:country/CYP,,,,,,,,
+Least developed countries,observationAbout,dcid:undata-geo/G00404000,,,,,,,,
+Sub-Saharan Africa,observationAbout,dcid:SubSaharanAfrica,,,,,,,,
+Czechia,observationAbout,dcid:country/CZE,,,,,,,,
+Benin,observationAbout,dcid:country/BEN,,,,,,,,
+Denmark,observationAbout,dcid:country/DNK,,,,,,,,
+Dominica,observationAbout,dcid:country/DMA,,,,,,,,
+Dominican Republic,observationAbout,dcid:country/DOM,,,,,,,,
+Ecuador,observationAbout,dcid:country/ECU,,,,,,,,
+El Salvador,observationAbout,dcid:country/SLV,,,,,,,,
+Ethiopia,observationAbout,dcid:country/ETH,,,,,,,,
+Estonia,observationAbout,dcid:country/EST,,,,,,,,
+Fiji,observationAbout,dcid:country/FJI,,,,,,,,
+Finland,observationAbout,dcid:country/FIN,,,,,,,,
+"occupied Palestinian territory, including east Jerusalem",observationAbout,dcid:country/PSE,,,,,,,,
+Ghana,observationAbout,dcid:country/GHA,,,,,,,,
+Indonesia,observationAbout,dcid:country/IDN,,,,,,,,
+Afghanistan,observationAbout,dcid:country/AFG,,,,,,,,
+Albania,observationAbout,dcid:country/ALB,,,,,,,,
+Oceania,observationAbout,dcid:oceania,,,,,,,,
+Algeria,observationAbout,dcid:country/DZA,,,,,,,,
+Northern Africa,observationAbout,dcid:NorthernAfrica,,,,,,,,
+Andorra,observationAbout,dcid:country/AND,,,,,,,,
+Northern America,observationAbout,dcid:undata-geo/G00136000,,,,,,,,
+Angola,observationAbout,dcid:country/AGO,,,,,,,,
+Eastern Asia,observationAbout,dcid:EasternAsia,,,,,,,,
+Argentina,observationAbout,dcid:country/ARG,,,,,,,,
+Southern Asia,observationAbout,dcid:SouthernAsia,,,,,,,,
+South-eastern Asia,observationAbout,dcid:SouthEasternAsia,,,,,,,,
+Australia,observationAbout,dcid:country/AUS,,,,,,,,
+Austria,observationAbout,dcid:country/AUT,,,,,,,,
+Bahamas,observationAbout,dcid:country/BHS,,,,,,,,
+Bahrain,observationAbout,dcid:country/BHR,,,,,,,,
+Bangladesh,observationAbout,dcid:country/BGD,,,,,,,,
+Barbados,observationAbout,dcid:country/BRB,,,,,,,,
+Australia and New Zealand,observationAbout,dcid:AustraliaAndNewZealand,,,,,,,,
+Belgium,observationAbout,dcid:country/BEL,,,,,,,,
+Central and Southern Asia,observationAbout,dcid:undata-geo/G00116000,,,,,,,,
+Bolivia (Plurinational State of),observationAbout,dcid:country/BOL,,,,,,,,
+Bosnia and Herzegovina,observationAbout,dcid:country/BIH,,,,,,,,
+Brazil,observationAbout,dcid:country/BRA,,,,,,,,
+Belize,observationAbout,dcid:country/BLZ,,,,,,,,
+Bulgaria,observationAbout,dcid:country/BGR,,,,,,,,
+Myanmar,observationAbout,dcid:country/MMR,,,,,,,,
+Belarus,observationAbout,dcid:country/BLR,,,,,,,,
+Cambodia,observationAbout,dcid:country/KHM,,,,,,,,
+Cameroon,observationAbout,dcid:country/CMR,,,,,,,,
+Canada,observationAbout,dcid:country/CAN,,,,,,,,
+Central Asia,observationAbout,dcid:CentralAsia,,,,,,,,
+Sri Lanka,observationAbout,dcid:country/LKA,,,,,,,,
+Europe,observationAbout,dcid:europe,,,,,,,,
+Comoros,observationAbout,dcid:country/COM,,,,,,,,
+Seychelles,observationAbout,dcid:country/SYC,,,,,,,,
+Sierra Leone,observationAbout,dcid:country/SLE,,,,,,,,
+Singapore,observationAbout,dcid:country/SGP,,,,,,,,
+Slovakia,observationAbout,dcid:country/SVK,,,,,,,,
+Viet Nam,observationAbout,dcid:country/VNM,,,,,,,,
+Slovenia,observationAbout,dcid:country/SVN,,,,,,,,
+Somalia,observationAbout,dcid:country/SOM,,,,,,,,
+South Africa,observationAbout,dcid:country/ZAF,,,,,,,,
+Zimbabwe,observationAbout,dcid:country/ZWE,,,,,,,,
+Small Island Developing States,observationAbout,dcid:undata-geo/G00405000,,,,,,,,
+Spain,observationAbout,dcid:country/ESP,,,,,,,,
+Sudan,observationAbout,dcid:country/SDN,,,,,,,,
+Suriname,observationAbout,dcid:country/SUR,,,,,,,,
+Northern Africa and Western Asia,observationAbout,dcid:undata-geo/G00103000,,,,,,,,
+Eswatini,observationAbout,dcid:country/SWZ,,,,,,,,
+Sweden,observationAbout,dcid:country/SWE,,,,,,,,
+Eastern and South-Eastern Asia,observationAbout,dcid:undata-geo/G00122000,,,,,,,,
+Switzerland,observationAbout,dcid:country/CHE,,,,,,,,
+Thailand,observationAbout,dcid:country/THA,,,,,,,,
+Togo,observationAbout,dcid:country/TGO,,,,,,,,
+Trinidad and Tobago,observationAbout,dcid:country/TTO,,,,,,,,
+Tunisia,observationAbout,dcid:country/TUN,,,,,,,,
+Turkmenistan,observationAbout,dcid:country/TKM,,,,,,,,
+Turks and Caicos Islands,observationAbout,dcid:country/TCA,,,,,,,,
+Tuvalu,observationAbout,dcid:country/TUV,,,,,,,,
+Uganda,observationAbout,dcid:country/UGA,,,,,,,,
+Ukraine,observationAbout,dcid:country/UKR,,,,,,,,
+North Macedonia,observationAbout,dcid:country/MKD,,,,,,,,
+Egypt,observationAbout,dcid:country/EGY,,,,,,,,
+United Kingdom of Great Britain and Northern Ireland,observationAbout,dcid:country/GBR,,,,,,,,
+United Republic of Tanzania,observationAbout,dcid:country/TZA,,,,,,,,
+United States of America,observationAbout,dcid:country/USA,,,,,,,,
+Burkina Faso,observationAbout,dcid:country/BFA,,,,,,,,
+Uruguay,observationAbout,dcid:country/URY,,,,,,,,
+Uzbekistan,observationAbout,dcid:country/UZB,,,,,,,,
+Venezuela (Bolivarian Republic of),observationAbout,dcid:country/VEN,,,,,,,,
+Samoa,observationAbout,dcid:country/WSM,,,,,,,,
+Yemen,observationAbout,dcid:country/YEM,,,,,,,,
+Montenegro,observationAbout,dcid:country/MNE,,,,,,,,
+Morocco,observationAbout,dcid:country/MAR,,,,,,,,
+Oman,observationAbout,dcid:country/OMN,,,,,,,,
+Europe and Northern America,observationAbout,dcid:undata-geo/G00126000,,,,,,,,
+Nepal,observationAbout,dcid:country/NPL,,,,,,,,
+Netherlands (Kingdom of the),observationAbout,dcid:country/NLD,,,,,,,,
+Oceania (exc. Australia and New Zealand),observationAbout,dcid:undata-geo/G00145000,,,,,,,,
+New Zealand,observationAbout,dcid:country/NZL,,,,,,,,
+Nicaragua,observationAbout,dcid:country/NIC,,,,,,,,
+Nigeria,observationAbout,dcid:country/NGA,,,,,,,,
+Norway,observationAbout,dcid:country/NOR,,,,,,,,
+Pakistan,observationAbout,dcid:country/PAK,,,,,,,,
+Panama,observationAbout,dcid:country/PAN,,,,,,,,
+Peru,observationAbout,dcid:country/PER,,,,,,,,
+Poland,observationAbout,dcid:country/POL,,,,,,,,
+Portugal,observationAbout,dcid:country/PRT,,,,,,,,
+Guinea-Bissau,observationAbout,dcid:country/GNB,,,,,,,,
+Timor-Leste,observationAbout,dcid:country/TLS,,,,,,,,
+Puerto Rico,observationAbout,dcid:country/PRI,,,,,,,,
+Qatar,observationAbout,dcid:country/QAT,,,,,,,,
+Romania,observationAbout,dcid:country/ROU,,,,,,,,
+Russian Federation,observationAbout,dcid:country/RUS,,,,,,,,
+Rwanda,observationAbout,dcid:country/RWA,,,,,,,,
+Saint Lucia,observationAbout,dcid:country/LCA,,,,,,,,
+Saint Vincent and the Grenadines,observationAbout,dcid:country/VCT,,,,,,,,
+Senegal,observationAbout,dcid:country/SEN,,,,,,,,
+Serbia,observationAbout,dcid:country/SRB,,,,,,,,
+South Sudan,observationAbout,dcid:country/SSD,,,,,,,,
+Jordan,observationAbout,dcid:country/JOR,,,,,,,,
+Kuwait,observationAbout,dcid:country/KWT,,,,,,,,
+Lao People's Democratic Republic,observationAbout,dcid:country/LAO,,,,,,,,
+Lesotho,observationAbout,dcid:country/LSO,,,,,,,,
+Malawi,observationAbout,dcid:country/MWI,,,,,,,,
+Mali,observationAbout,dcid:country/MLI,,,,,,,,
+Mauritania,observationAbout,dcid:country/MRT,,,,,,,,
+Mongolia,observationAbout,dcid:country/MNG,,,,,,,,
+Republic of Moldova,observationAbout,dcid:country/MDA,,,,,,,,
+Mozambique,observationAbout,dcid:country/MOZ,,,,,,,,
+Nauru,observationAbout,dcid:country/NRU,,,,,,,,
+Niger,observationAbout,dcid:country/NER,,,,,,,,
+Niue,observationAbout,dcid:country/NIU,,,,,,,,
+Brunei Darussalam,observationAbout,dcid:country/BRN,,,,,,,,
+Chad,observationAbout,dcid:country/TCD,,,,,,,,
+Zambia,observationAbout,dcid:country/ZMB,,,,,,,,
+Azerbaijan,observationAbout,dcid:country/AZE,,,,,,,,
+Armenia,observationAbout,dcid:country/ARM,,,,,,,,
+Burundi,observationAbout,dcid:country/BDI,,,,,,,,
+Central African Republic,observationAbout,dcid:country/CAF,,,,,,,,
+San Marino,observationAbout,dcid:country/SMR,,,,,,,,
+Sao Tome and Principe,observationAbout,dcid:country/STP,,,,,,,,
+Tajikistan,observationAbout,dcid:country/TJK,,,,,,,,
+Namibia,observationAbout,dcid:country/NAM,,,,,,,,
+Micronesia (Federated States of),observationAbout,dcid:country/FSM,,,,,,,,
+Paraguay,observationAbout,dcid:country/PRY,,,,,,,,
+Philippines,observationAbout,dcid:country/PHL,,,,,,,,
+Kiribati,observationAbout,dcid:country/KIR,,,,,,,,
+Democratic People's Republic of Korea,observationAbout,dcid:country/PRK,,,,,,,,
+Djibouti,observationAbout,dcid:country/DJI,,,,,,,,
+Cabo Verde,observationAbout,dcid:country/CPV,,,,,,,,
+Cook Islands,observationAbout,dcid:country/COK,,,,,,,,
+Saint Kitts and Nevis,observationAbout,dcid:country/KNA,,,,,,,,
+Saudi Arabia,observationAbout,dcid:country/SAU,,,,,,,,
+Tonga,observationAbout,dcid:country/TON,,,,,,,,
+United Arab Emirates,observationAbout,dcid:country/ARE,,,,,,,,
+Marshall Islands,observationAbout,dcid:country/MHL,,,,,,,,
+Palau,observationAbout,dcid:country/PLW,,,,,,,,
+Papua New Guinea,observationAbout,dcid:country/PNG,,,,,,,,
+Eritrea,observationAbout,dcid:country/ERI,,,,,,,,
+Botswana,observationAbout,dcid:country/BWA,,,,,,,,
+Libya,observationAbout,dcid:country/LBY,,,,,,,,
+Türkiye,observationAbout,dcid:country/TUR,,,,,,,,
+Grenada,observationAbout,dcid:country/GRD,,,,,,,,
+Iran (Islamic Republic of),observationAbout,dcid:country/IRN,,,,,,,,
+Antigua and Barbuda,observationAbout,dcid:country/ATG,,,,,,,,
+Lebanon,observationAbout,dcid:country/LBN,,,,,,,,
+Africa,observationAbout,dcid:africa,,,,,,,,
+Americas,observationAbout,dcid:undata-geo/G00134000,,,,,,,,
+Eastern Mediterranean,observationAbout,dcid:undata-geo/G00121000,,,,,,,,
+Western Pacific,observationAbout,dcid:undata-geo/G00160000,,,,,,,,
+World,observationAbout,dcid:Earth,,,,,,,,
+Bhutan,observationAbout,dcid:country/BTN,,,,,,,,
+China,observationAbout,dcid:country/CHN,,,,,,,,
+Solomon Islands,observationAbout,dcid:country/SLB,,,,,,,,
+Equatorial Guinea,observationAbout,dcid:country/GNQ,,,,,,,,
+Syrian Arab Republic,observationAbout,dcid:country/SYR,,,,,,,,
+Vanuatu,observationAbout,dcid:country/VUT,,,,,,,,

statvar_imports/who/adolescent_birth_rate/data_download.py

@@ -0,0 +1,50 @@
+import requests
+import os
+
+output_dir = "input_files"
+
+def download_full_who_data(indicator_id="27D371A"):
+    """
+    Downloads the complete historical dataset for a WHO indicator 
+    directly from the WHDH Azure Blob Storage.
+    """
+    # 1. Create the output directory if it doesn't exist
+    os.makedirs(output_dir, exist_ok=True)
+
+    # 2. Construct the filename inside the output directory
+    # Using indicator_id ensures the file has a unique name
+    filename = os.path.join(output_dir, "adolescent_birth_rate_input.csv")
+
+    # Root URL for the WHDH raw data dumps
+    base_url = f"https://srhdpeuwpubsa.blob.core.windows.net/whdh/DATADOT/INDICATOR/{indicator_id}_ALL_LATEST.csv"
+
+    print(f"Requesting full historical CSV for {indicator_id}...")
+
+    try:
+        # We use stream=True to handle potentially large files efficiently
+        with requests.get(base_url, stream=True) as r:
+            r.raise_for_status()
+            with open(filename, 'wb') as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    f.write(chunk)
+
+        # Verify the file has data beyond the header
+        if os.path.exists(filename):
+            with open(filename, 'r', encoding='utf-8', errors='ignore') as f:
+                line_count = sum(1 for _ in f)
+
+            if line_count <= 1:
+                os.remove(filename)
+                print("No data fetched (empty file or header only)")
+                return "no data fetched"
+
+        print(f"Successfully saved to: {filename} ({line_count} lines found)")
+        return filename
+
+    except requests.exceptions.HTTPError as err:
+        print(f"Error fetching data: {err}")
+        return "error"
+
+if __name__ == "__main__":
+    WHO_INDICATOR_ID = "27D371A"
+    download_full_who_data(WHO_INDICATOR_ID)