Possible variable details structure for date variable

[reikookamoto]

variable	dummyVariable	typeEnd	databaseStart	variableStart	typeStart	recEnd	numValidCat	catLabel	catLabelLong	units	recStart	catStartLabel	variableStartShortLabel	variableStartLabel	notes
death_date	N/A	date	ices	[death_date]	date	copy	N/A	Death date	Death date		[2018-01-01, 2022-12-31]
death_date	N/A	date	ices	[death_date]	date	NA::b	N/A	Missing	Missing		else
birth_date	N/A	date	ices	[birth_date]	date	copy	N/A	Birth date	Birth date		[31OCT1959, 31JAN2000]
birth_date	N/A	date	ices	[birth_date]	date	NA::b	N/A	Birth date	Missing		else

• Could date format (e.g., yyyymmdd, ddmmmyyyy) be specified in units?
• Not sure how to populate last few columns

[DougManuel]

Just a small issue that we use a catch-all else to NA::b to capture anything that slips in as incorrect or missing data.

[DougManuel]

Summary of feedback and suggestions

1. ✅ Use SAS %d%b%Y format with [DDMMMYYYY, DDMMMYYYY] interval notation for recStart
2. ✅ Add "Date" to variables.yaml::variableType enum
3. ✅ Add "date" to variable_details.yaml::typeEnd enum
4. ✅ Add "date" and "date_SAS" to variable_details.yaml::typeStart enum
5. ✅ Use NA::b for all missing date values
6. ✅ Include five date variables from DemPoRT v1: survey_adm_date, dementia_diag_date, death_date, eligbility_date, birth_date. (only suggested names). Note birth_date is unclear in DemPoRTv1.
7. ⚠️ Discuss: Alternative dateFormat field approach vs embedding format in type name
8. ⚠️ Discuss: Whether to include birth_date despite it not being in DemPoRT v1, and if so, privacy approach (full date vs birth year)

1. recStart Notation Format

Rows 3-4 (birth_date) look like the format for date variables. All date variables should use the SAS %d%b%Y format that Rob exports from ICES.

Recommended recStart notation:

• Valid date range: [DDMMMYYYY, DDMMMYYYY] (e.g., [01JAN2001, 31MAR2017])
• Missing values: Empty string "" → recoded to NA::b
• Catch-all: else → recoded to NA::b

Why this format?

Rob's ICES extracts use SAS date format %d%b%Y:

• Dates are exported as character strings: "01JAN2001", "15JUN2008", "31MAR2017"
• Missing events are represented as blank/empty cells in CSV
• When parsed with as.Date(x, format='%d%b%Y') in R, blanks become NA

See:

• DemPoRT-Dev/mock_dementia_outcomes.csv:2-5 shows the format
• DemPoRT-Dev/0- Data Preparation.R:71-74 shows Stacey's conversion code

---

2. Schema updates

We'll need to add date as a variable type to the metadata schemas.

Possible approach to variables.yaml:

Current variableType enum:

constraints:
  enum: ["Categorical", "Continuous"]

Proposed addition:

constraints:
  enum: ["Categorical", "Continuous", "Date"]

Reference: variables.yaml:82-83

Updates to variable_details.yaml:

Current typeEnd enum:

constraints:
  enum: ["cat", "cont"]

Proposed typeEnd addition:

constraints:
  enum: ["cat", "cont", "date"]

Current typeStart enum:

constraints:
  enum: ["cat", "cont", "N/A"]

Proposed typeStart addition:

constraints:
  enum: ["cat", "cont", "date", "date_SAS", "N/A"]

Where:

• "date" = ISO 8601 format (YYYY-MM-DD)
• "date_SAS" = SAS format (DDMMMYYYY, e.g., 01JAN2001)

Reference: variable_details.yaml:83, 123

Discussion point: Alternative dateFormat field

An alternative approach would be to keep typeStart = "date" for all date sources and add a separate optional dateFormat field with enum ["ISO8601", "SAS_DDMMMYYYY", ...]. This separates "what is it" (date) from "how is it formatted" (format string).

To discuss: Which approach is cleaner for the metadata architecture?

---

3. Handling Missing Values

Use NA::b for all missing date values:

• Empty strings "" in source data → recode to NA::b
• Survey administration dates should never be missing (but if they are, use NA::b)
• Outcome dates (diagnosis, death) that didn't occur → NA::b
• Everything else → else recoded to NA::b

Evidence: See mock_dementia_outcomes.csv:3 where patient 1002 has blank diagdate and enddt fields.

---

4. Date variables from ICES data

Reviewig Rob and Stacey's ICES data structure again (see DemPoRT-Dev/data_dictionary_dementia_outcomes.csv), we need the following date variables:

Confirmed variables from DemPoRT v1:

As you suggest, I support updating the names for DemPoRT v2. i.e. survey_adm_date

Variable	Description	Format	Date Range	Missing Rate
surveyadmdate	Survey administration date (baseline)	%d%b%Y	01JAN2001 - 31DEC2013	0%
diagdate	Dementia diagnosis date	%d%b%Y	01JAN2001 - 31MAR2017	~90%
dthdate	Death date	%d%b%Y	01JAN2001 - 31MAR2017	~85%
enddt	End of healthcare eligibility date	%d%b%Y	01JAN2001 - 31MAR2017	~95%

See:

• Variable definitions: data_dictionary_dementia_outcomes.csv:3-6
• Variable extraction: 0- Data Preparation.R:71-74

Variable for discussion:

Variable	Description	Notes
birth_date	Birth date	Not present in DemPoRT v1. Age was captured only as numeric agecchs (age at survey). Normally ICES provides birth date from RPDB, but it's typically deidentified to birth year. Need to discuss privacy implications and whether to include for mock data generation.

Note on birth_date:

• DemPoRT v1 uses only numeric age at survey administration (0- Data Preparation.R:77)
• No birth date or birth year variables exist in the codebase
• No age calculations from dates anywhere in DemPoRT v1 code
• Standard ICES practice is to provide birth date from RPDB but deidentify to birth year
• Flag for discussion: Privacy approach (full date vs birth year vs omit), implications for age calculations

---

[reikookamoto]

• We agree that the end format should be ISO 8601
• How/where do we specify the starting date format? Option 1: typeStart in variable details sheet; Option 2: recStart in variable details sheet

[DougManuel]

Following on laste week's discussion and my note in Teams.

You can keep date variables as variableType = "Continuous" or "Categorical" - either is fine for v2.1.0.

We can focus on completing create_date_var() to:

• Parse date ranges from recStart (e.g., [01JAN2001, 31MAR2017] in SAS format)
• Generate R Date objects
• Return in ISO 8601 format (how R Date displays by default)
• Handle missing values (watch for the same else issue from Issue with generated values #5)

The input format is specified in recStart (SAS format from ICES), and output is always R Date objects. The broader type system changes are deferred to v3.0+.

[DougManuel]

Now addressed in pr #9