Hosted by Department of Energy
Lawrence Livermore National Laboratory
Powered by ESGF logo
Welcome, Guest. | Login | Create Account
obs4MIPs logo
You are at the ESGF@DOE/LLNL node
 

Publishing Data to ESGF

Groups that wish to publish datasets as "Obs4MIPs" should conform to the following guidelines, which garantee that datasets are searched and displayed uniformly across agencies and projects.

The recommended esg.ini file to be used for obs4MIPs publication is stored in the ESGF GitHub: esg.ini

General Guidelines

  • Each dataset can only contain one physical variable (for example, “hus”), but it can contain other files with errors and number of observations (for example “hus”, “husStderr”, “husNobs”).
  • Each dataset must have an accompanying Tech Note.
  • The title of the dataset should be in the format:
    • dataset_name_format = %(project)s %(source_id)s %(variable_long)s %(processing_level)s %(time_frequency_long)s Data
    • Example: obs4MIPs AIRS Air Temperature L3 Monthly Data
  • The description of the dataset is taken from the NetCDF file global attribute “title".
    • for existing files we have no other option than using what is in the files already.
    • for new files the Obs4MIPs panel should issue a recommendation.
  • The dataset version should be in the "date" format yyyymmdd for consistency with CMIP5/6 data
    • This is accomplished by using the option “--use-version-dir” at the time of publishing
  • Access Control: obs4MIPs datasets should be protected with the forthcoming "AUTH_ONLY" policy option, which requires users to login before downloading (so we can track their openid), but not to register with any group. You will still need to protect publishing to your local node ("Write") with some other group that you directly control.
    • <200b>Example of statements in file /esg/config/esgf_policies_local.xml:
<policy resource=".*obs4MIPs.*" attribute_type="AUTH_ONLY"
                                attribute_value="" action="Read"/>
<policy resource=".*obs4MIPs.*" attribute_type="NASA OBS"
                                attribute_value="publisher" action="Write"/>
 

Example commands

# to publish:
esgscan_directory --project obs4MIPs -o obs4MIPs.txt --use-version-dir /esg/data/obs4MIPs/observations
esgpublish --map ./obs4MIPs.txt --project obs4MIPs --service fileservice
esgpublish --map ./obs4MIPs.txt --project obs4MIPs --noscan --thredds --publish --service fileservice

# to unpublish:
esgunpublish --database-delete --map ./obs4MIPs.txt

When publishing a tech note document together with the dataset, first make the document available at some URL, then run the first esgscan command by providing additional arguments as follows:

# to publish with tech note:
esgscan_directory --project obs4MIPs -o obs4MIPs.txt --use-version-dir 
                  --dataset-tech-notes-title <tech_notes_title>
                  --dataset-tech-notes <tech_notes_url>
                  /esg/data/obs4MIPs/observations

 

Last Update: Nov. 6, 2018, 2:35 p.m. by Robert Ferraro
Read News
Obs4MIPs-CMIP6 Planning Meeting Final Report is now available
Go to the left sidebar and click on Planning Meeting Report
Call For Datasets - Nov 2015
The WDAC task team has issued a call for dataset contributions to obs4mIPs. See "Call ...
Title: Dataset Call is now closed!
We have been overwhelmed with the response to the obs4MIPs dataset call! The task team ...
Browse Projects
 
 
 
 
Start typing, or use the 'Delete' key to show all available tags.
obs4MIPs Tags: None