$Header$ -*-text-*-

netCDF Operators NCO version 5.3.1 howl at the Moon

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco/nco (Source Code, Issues, Releases)

What's new?
Version 5.3.1 adds features and fixes to ncremap and ncclimo.
The new autoconversion feature allows these scripts to automagically
convert netCDF4 string variables to netCDF3 character arrays.
The improved treatment of fixed time dimensions allows ncclimo to work
with raw MERRA2 files. Together these improvements make possible
regridding and vertical interpolation of the new formatting of
Copernicus Data Store (CDS) files used for, e.g., ERA5.
Skip this release if these changes do not interest you. 

Work on 5.3.3 is underway and includes improved NC_STRING
autoconversion support for other NCO operators.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. Autoconversion of NC_STRING variables, i.e., automatic conversion
from type NC_STRING to NC_CHAR as dictated by choosing a netCDF3
output from a netCDF4 input file, work for all attributes, but not for
all variables. This is because attributes are at most one-dimensional
and may be of any size whereas variables require gridded dimensions
that usually do not fit the ragged sizes of text strings.
Hence scalar NC_STRING attributes are correctly converted to and
stored as NC_CHAR attributes in the netCDF3 output file, but
not all NC_STRING variables are correctly converted.

As of version 5.3.1, NCO has two distinct levels of support for
autoconversion of NC_STRING variables. ncremap fully supports
autoconversion of scalar NC_STRING variables in both horizontal
regridder and vertical interpolation mode. This means that ncremap
will produce netCDF3 output files from netCDF4 input files (if
requested) and that any NC_STRING variables in the input files will be
translated to NC_CHAR arrays in the netCDF3 output files. There is no
loss of information since the entire contents of the string is
preserved in the character array. Autoconversion stores the character
array with a new dimension named sng_lng_X where X is the smallest
power of ten between 10 and 10000 that is capable of containing the
string contents (including the terminating NUL character).
This type of variable autoconversion fails if the input string length
exceeds 10000 (this is a safety measure, and could be expanded if
people lobby me).

All operators besides ncremap currently retain only the first
character of the input NC_STRING array. This loses information.
We are working to implement the same feature that ncremap enjoys. 
Both types of autoconversion fail if the input string variable is 
multidimensional, i.e., an array of type NC_STRING. A feature to solve
that case is tractable, though we have not yet received any requests
for it. 

Thanks to Waler Hannah (LLNL) for prompting this feature.
http://nco.sf.net/nco.html#autoconversion
http://nco.sf.net/nco.html#ncclimo

B. ncremap's vertical interpolator now automatically extrapolates
variables named "t" (as in ERA5 raw data) as if they were temperature. 
Futhermore the interpolator now automatically extrapolates variables
named "H" (as in MERRA2 raw data) and VerticalLayerMidpoint (SCREAM)
as if they were geopotential height. See the documentation for
specifics about the temperature and geopotential height extrapolation
algorithms.
http://nco.sf.net/nco.html#vrt

C. For many years, users at select supercomputer centers have been
able to access the latest versions of NCO scripts directly from
C. Zender's build directories. However, this feature was not
documented until now. Zender's build directories usually contain the
latest NCO snapshot, and so may be unstable and are not for everyday
use. This feature is instead intended to allow users to test and
provide feedback on new features before they are distributed in a
public release.
Invoking ncclimo and ncremap with the --npo (or long-option equivalent
--nco_path_override) flag as the first option (i.e., as $1) causes
these scripts to utilize the binary NCO executables in Zender's build
directories on the following machines at the national supercomputer
centers in parentheses: 
acme1 (LLNL), andes (ORNL), chrysalis (ANL), compy (PNNL), derecho
(NCAR), frontier (ORNL), and perlmutter (NERSC).

If the latest features that you want are actually implemented in the
NCO binaries or library, then it may suffice to invoke your own
version of the scripts, as in the first examples below. However, if
the latest features that you want are implemented in the scripts
themselves, then you can either invoke Zender's scripts directly (as
in the second set of examples), or copy those scripts into your own
executable path: 
@verbatim
# Access Zender's latest binaries from your default scripts
ncremap --npo -m map.nc in.nc out.nc 
ncclimo --npo -P elm -c ${caseid} ...
# Access Zender's latest binaries from Zender's latest scripts
~zender/bin/ncremap --npo -m map.nc in.nc out.nc 
~zender/bin/ncclimo --npo -P elm -c ${caseid} ...
@end verbatim
@end example
In both cases, the --npo flag must be the first option invoked. 
http://nco.sf.net/nco.html#npo
http://nco.sf.net/nco.html#nco_path_override

BUG FIXES:
   
A. ncremap's vertical interpolator now understands that the time
dimension in input surface pressure fields may be "fixed".
Previously the interpolator assumed that the temporal dimension, if
any, was an unlimited/record dimension. This could cause it to
miscalculate the number of horizontal spatial dimensions, and then
to fail. Now the interpolator considers any fixed dimension named
"time", or most unlimited dimensions (except certain corner cases) to
be temporal dimension. This allows it to interpolate raw MERRA2 or
ERA5 data. Thanks to Naser Mahfouz (PNNL) for prompting this
fix/feature. 
http://nco.sf.net/nco.html#vrt

Full release statement at http://nco.sf.net/ANNOUNCE
    
KNOWN PROBLEMS DUE TO NCO:

This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems. 
These problems occur with NCO 5.2.7 built/tested under
MacOS 15.2 with netCDF 4.9.4-development on HDF5 1.14.3
and with Linux FC40 with netCDF 4.9.2 on HDF5 1.14.3.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

