Publishing Data to ESGF

Groups that wish to publish datasets as "Obs4MIPs" should conform to the following guidelines, which garantee that datasets are searched and displayed uniformly across agencies and projects.

The recommended esg.ini file to be used for obs4MIPs publication is stored in the ESGF GitHub: esg.ini

General Guidelines

• Each dataset can only contain one physical variable (for example, “hus”), but it can contain other files with errors and number of observations (for example “hus”, “husStderr”, “husNobs”).
• Each dataset must have an accompanying Tech Note.
• The title of the dataset should be in the format:
  • dataset_name_format = %(project)s %(source_id)s %(variable_long)s %(processing_level)s %(time_frequency_long)s Data
  • Example: obs4MIPs AIRS Air Temperature L3 Monthly Data
• The description of the dataset is taken from the NetCDF file global attribute “title".
  • for existing files we have no other option than using what is in the files already.
  • for new files the Obs4MIPs panel should issue a recommendation.
• The dataset version should be in the "date" format yyyymmdd for consistency with CMIP5/6 data
  • This is accomplished by using the option “--use-version-dir” at the time of publishing
• Access Control: obs4MIPs datasets should be protected with the forthcoming "AUTH_ONLY" policy option, which requires users to login before downloading (so we can track their openid), but not to register with any group. You will still need to protect publishing to your local node ("Write") with some other group that you directly control.
  • <200b>Example of statements in file /esg/config/esgf_policies_local.xml:

<policy resource=".*obs4MIPs.*" attribute_type="AUTH_ONLY"
                                attribute_value="" action="Read"/>
<policy resource=".*obs4MIPs.*" attribute_type="NASA OBS"
                                attribute_value="publisher" action="Write"/>

Example commands

# to publish:
esgscan_directory --project obs4MIPs -o obs4MIPs.txt --use-version-dir /esg/data/obs4MIPs/observations
esgpublish --map ./obs4MIPs.txt --project obs4MIPs --service fileservice
esgpublish --map ./obs4MIPs.txt --project obs4MIPs --noscan --thredds --publish --service fileservice

# to unpublish:
esgunpublish --database-delete --map ./obs4MIPs.txt

When publishing a tech note document together with the dataset, first make the document available at some URL, then run the first esgscan command by providing additional arguments as follows:

# to publish with tech note:
esgscan_directory --project obs4MIPs -o obs4MIPs.txt --use-version-dir 
                  --dataset-tech-notes-title <tech_notes_title>
                  --dataset-tech-notes <tech_notes_url>
                  /esg/data/obs4MIPs/observations