Summary of status on various REF reference datasets for REF v2.0

[rswamina]

I combined information from the Diagnostic Project Page , Data Status Project Page and the REF paper and have updated the status on the observational datasets needed for each diagnostic.

Can the REF development team, please read the Notes column for your software and see if you are on the same page as what the note says in the Diagnostic Project Page.

In the table below, the last column is to ensure any code or json tables used to process the data and have standards met is in the GitHub repo for transparency and reproducibility.

Missing from list means I don't see it in the Data Status Project Page page.

For ESMValTool, I don't believe we need the datasets in the Status: Unknown category to be CMOR-ized etc. but can @bouweandela please confirm?

At this time, this is the data status summary by each software provider:

Software	Status	Reference datasets with end dates	Code in GitHub repo	Has Uncertainty Information
ESMValTool	On ESGF	GPCP (30-09-2021), ERA5 (16-12-2022), CERES-EBAF Ed4.2 (15-06-2023), HadISST (16-01-2025)
	Newly CMOR-ized to Obs4MIPS	OSI SAF v3 (2020), ESACCI-Cloud AVHRR-AMPM-fv3.0 (2016), CALIPSO-ICECLOUD-1-00 (2015),
	Status in Progress	HadCRUT 5.0.1.0-analysis, GFED5 (by ILAMB)
	Missing from list	CCI for seaice
	Unknown or not started	MODIS MOD44B, ESA CCI Biomass, ISIMIP3a, GSWP3, W5E5, NOAA-20CR
PMP	On ESGF	ERA5, GPCP, CERES-EBAF, HadISST, TropFlux (16-07-2017)
	Newly CMOR-ized to Obs4MIPS
	Status in Progress
	Missing from list
	Unknown or not started	ESACCI-Ozone, NOAA-20CR
ILAMB	On ESGF
	Newly CMOR-ized to Obs4MIPS	WECANN, LORA, HOFFMAN (check on license status), LAI4g-1-2
	Status in Progress	GFED5 (by ILAMB)
	Missing from list	Wang & Mao, JASMES, AVHRR & VIIRS, GIMMS (Is this LAI4g-1-2?)
	Unknown or not started	DAI , FLUXNET
IOMB	On ESGF
	Newly CMOR-ized to Obs4MIPS	WOA , RAPID array
	Status in Progress
	Missing in list
	Unknown or not started	GLODAP2, IAP

Please update/ correct as needed and tag anyone else as relevant. I am tagging relevant people here: @lewisjared, @Darwyn72 , @nocollier , @bouweandela, @lee1043 , @climate-dude , @hb326 and @CMIP-IPO.

[Darwyn72]

Regarding the table above. I can give an update on the following:

1. JASMES - this was a Jaxa proposed dataset sent to obs4MIPs in February - sent a chase-up email to the proposer as it was accepted and then nothing happened (tanada.kazuhisa@jaxa.jp). No RC has been raised for it, so there are source_ids or source_variable_id's on the obs4MIPs Githu repo. UPDATE: Tanada intends to CMORise and get it on ESGF by Jan 2026.
2. wang&mao - I contacted Jaifu Mao (maoj@ornl.gov) at ORNL to chase up and he responded yesterday to say this was accepted by obs4MIPs then a Github RC issue was raised RC for OLC-ORS soil moisture data PCMDI/obs4MIPs-cmor-tables#388 It looks like the proposers got confused with the Github message trail and then someone from ILAMB told them to to submit it there and they haven't heard anything since. I think this has actually been processed with ILAMB...
3. wang2024-cSoil-v2 - this is not in the list, but get's confused with wang & mao? I contacted Zhuonan Wang (zzw0034@auburn.net) about this dataset, as it has an open RC issue RC for wang2024_cSoil PCMDI/obs4MIPs-cmor-tables#529 and a source_id was created for it on obs4MIPs 'AI-upscaling-cSoil-1-0' with soucr_variable_id 'cSoil'. Zhuonan replied that he hasn't had time to do anything with it, as he moved jobs and might get started on it at the end of October.
4. FLUXNET - Dora contact Dario Papale about this and copied me in. It was passed to Peter Isaac (peter.isaac@jcu.edu.ac) in Australia. I chased him recently about FLUXNET2015-1-0 as it's on obs4MIPs with this source_id and source_variable_id 'gpp' but has not been cmorised as yet. The problem is, as CMOR doesn't support additional variables per file, it will need separate files for each variable (12+ variables) for each ground station, and there are hundreds of stations. Also, this has been superceded by the FLUXNET2025 dataset. He is still keen to do something - would be giving an update after 10 November.
5. HadCRUT-0-2-0 and CRU-TS - Contacted Kate Willet at MetO about these last week. There are no resources at the moment to sort out these data and CMORise, but she will have another look in mid-November.
6. GLODAPv2 has a source_id on the obs4MIPs Github repo 'GLODAP-2-2016b' for source_variable_id's 'so' and 'thetao'. Not sure if this has ever been CMORised.
7. NOAA-20CR is the same as 20CR-V2 and is on ESGF.
8. GFED5 is on the CEDA Archive [25-10-30] after being processed by Morgan at ORNL using ILAMB to be CMOR compliant.
9. CCI for SeaIce? - We had a proposal from FMI which was accepted today [25-11-06] for Snow CCI SWE/GlobSnow SWE, the instructions have been sent to them for CMORising the data etc. Don't expect it to be done in the next month though...

[bouweandela]

The list of files that we need for the ESMValTool recipes in REF and currently cannot obtain from ESGF is available here: https://github.com/Climate-REF/climate-ref/blob/main/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/dataset_registry/data.txt (ignore the obs4MIPs dataset: we can obtain that from ESGF, but the REF cannot ingest it because the REF does not know how to handle CMIP5-style obs4MIPs data Climate-REF/climate-ref#260). As you can see, not all required ERA-5 data is available on ESGF.

I'm not sure how to map the dataset names shown above to the dataset names we use in ESMValTool, so wrote a small script to extract all non CMIP datasets from recipes (click Details to see it)

Details

from itertools import groupby
from pathlib import Path

from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset
from esmvalcore.preprocessor._derive import get_required
from rich.console import Console


def main():
    # Path to the REF source code
    ref_src = Path("/home/bandela/src/Climate-REF/climate-ref")

    def get_variables(short_name, project):
        """Retrieve the input variable names required to derive variable `short_name`."""
        try:
            return [ds["short_name"] for ds in get_required(short_name, project)]
        except NotImplementedError:
            pass
        return [short_name]

    session = CFG.start_session("tmp")
    datasets = [
        ds
        for recipe in ref_src.glob("tests/test-data/regression/esmvaltool/**/recipe.yml")
        for ds in Dataset.from_recipe(recipe, session)
    ]

    obs_facets = {
        (ds["project"], ds["dataset"], short_name, ds.facets.get("version", "None"))
        for ds in datasets
        if ds["project"] != "CMIP6"
        for short_name in get_variables(ds["short_name"], ds["project"])
    }

    def grouper(v):
        return (v[0], v[1], v[3])

    summary = {}
    for group, values in groupby(sorted(obs_facets, key=grouper), key=grouper):
        if group not in summary:
            summary[group] = []
        for v in values:
            summary[group].append(v[2])

    print("Project|Dataset|Version|Variables")
    print("-|-|-|-")
    for group, variables in summary.items():
        print(
            "|".join([*group, ", ".join(variables)]),
        )


if __name__ == "__main__":
    main()

and it shows that these datasets are in use by the REF:

Project	Dataset	Version	Variables
OBS	CALIPSO-ICECLOUD	1-00	cli
OBS	CERES-EBAF	Ed4.2	rlutcs, rlut, rsut, rsutcs
OBS	ESACCI-CLOUD	AVHRR-AMPM-fv3.0	clivi, clt, rlutcs, rsutcs, rlut, rsut, clwvi
OBS	HadCRUT5	5.0.1.0-analysis	tas
OBS	ISCCP-FH	v0	rlutcs, rlut, rsutcs, rsut
OBS	OSI-450-nh	v3	siconc
OBS	OSI-450-sh	v3	siconc
OBS6	TROPFLUX	v1	tos, tauu
native6	ERA5	v1	pr, hus
obs4MIPs	ERA-5	None	tas, ta, psl, ua
obs4MIPs	GPCP-V2.3	None	pr

[bouweandela]

See also https://github.com/Climate-REF/climate-ref/blob/01eabb53ecf47caf5d42f69e3e3601ac002c33ff/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/enso.py#L203-L209, we could potentially use NOAA-ERSSTv5 for the ENSO recipe if it becomes available on ESGF and some further background on current issues in https://github.com/Climate-REF/climate-ref/blob/01eabb53ecf47caf5d42f69e3e3601ac002c33ff/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/cloud_scatterplots.py#L151-L166

[rswamina]

@bouweandela - Can you say which recipes use the ISCCP-FH dataset? We don't seem to have it on any of our lists.

[bouweandela]

The cloud radiative effects diagnostic