In straightforward conversion operations where ASCII or unformatted binary data files are already readable by Ferret, the conversion to direct access, self-describing netCDF formatted data can be accomplished by Ferret itself. The following set of examples illustrates these procedures:
Example 1
Consider an ASCII file uv.data, with two variables, u and v, defined on a grid 360 by 180. The following set of commands will properly read in u and v and convert them to a netCDF formatted data set:
yes? DEFINE AXIS/x=1:360:1/units=degrees xaxis yes? DEFINE AXIS/y=1:180:1/units=degrees yaxis yes? DEFINE GRID/x=xaxis/y=yaxis uv_grid yes? FILE/GRID=uv_grid/BAD=-999/VAR="u,v" uv.data yes? SET VARIABLE/TITLE="zonal velocity" u yes? SAVE/FILE=uv.cdf u,v
See command DEFINE AXIS in the Commands Reference. See the chapter "Grids and Regions" for setting up formatted latitude, longitude and time axes.
Example 2
Consider now two separate ASCII files, u.data and v.data, defined on a grid 360 by 180. The following set of commands will properly read in u and v and convert them to a single netCDF formatted data set:
yes? DEF AXIS/x=1:360:1/units=degrees xaxis yes? DEF AXIS/y=1:180:1/units=degrees yaxis yes? DEF GRID/x=xaxis/y=yaxis uv_grid yes? FILE/GRID=uv_grid/BAD=-999/VAR=u u.data yes? FILE/GRID=uv_grid/BAD=-999/VAR=v v.data yes? SAVE/FILE=uv2.cdf u[D=1] yes? SAVE/APPEND/FILE=uv2.cdf v[D=2]
Example 3‚ multiple time steps
Consider 12 ASCII files, uv.data1 to uv.data12, each defined on the same grid as above but each representing a successive time step. The following set of commands illustrates how to save these data into a single netCDF data set (time series):
yes? DEF AXIS/x=1:360:1 xaxis yes? DEF AXIS/y=1:180:1 yaxis yes? DEF AXIS/t=1:1:1 taxis1 yes? DEF GRID/x=xaxis/y=yaxis/t=taxis1 uv_grid1 yes? FILE/GRID=uv_grid1/BAD=-999/VAR="u,v" uv.data1 yes? SAVE/FILE=uv1_12t.cdf u,v yes? CANCEL DATA uv.data1 yes? DEF AXIS/t=2:2:1 taxis1 yes? FILE/GRID=uv_grid1/BAD=-999/VAR="u,v" uv.data2 yes? SAVE/APPEND/FILE=uv1_12t.cdf u,v . . .
and so on, redefining the time axis to be 3:3:1, 4:4:1, .. each time a new file is set.
When the input data is in netCDF files, please see the following FAQ on using data from a set of input files. Note particularly the example showing how to add time information to the variable when the grid of the input data has no time axis.
FAQ: How can I use data as a time series when it exists in multiple files?
Example 4, merging two netCDF files that contain different X-Y regions
The procedure used in example 3, above, is possible because netCDF files can be extended along the time axis. In order to append multiple regions in any dimension other than time, the netCDF file must first be created including the full span of of the non-time dimensions.
Consider two files with different XY ranges. The grids in these files agree in time, but if they differ in their time or other dimensions, add regridding in those directions to the LET commands so that the grids match.
yes? use a1 yes? use b1 yes? show data currently SET data sets: 1> ./a1.nc name title I J K L SST SEA SURFACE TEMPERATURE 1:90 1:55 .. 1:1 2> ./b1.nc (default) name title I J K L TEMP sea surface temperature 1:90 1:46 .. 1:1 yes? show grid sst[d=1] GRID GMH1 name axis # pts start end subset XAXIS LONGITUDE 90mr 1E 179E full YAXIS LATITUDE 55 r 19S 89N full normal Z TIME TIME 1mr 01-JAN-2017 00:00 01-JAN-2017 00:00 full yes? show grid temp[d=2] GRID GLS1 name axis # pts start end subset LONGITUDE LONGITUDE 90mr 179W(-179) 1W(-1) full LATITUDE LATITUDE 46 r 45S 45N full normal Z TIME TIME 1mr 01-JAN-2017 00:00 01-JAN-2017 00:00 full yes? ! Define axes that span both grids. This will be the output grid of the data. yes? define axis/x=1:359:2/units=degrees_east xbig yes? define axis/y=-45:89:2/units=degrees_north ybig yes? set data 1 yes? let/like=sst sst_merged = sst[gx=xbig,gy=ybig] yes? ! Create the file using /XLIMITS and /YLIMITS to write the entire span of these axes yes? ! to the file, and /X= and /Y= to specify the range of the variable we are writing yes? save/clobber/file=bigfile.nc/xlimits=1:359/ylimits=-45:89/x=1:179/y=-19:89 sst_merged LISTing to file bigfile.nc yes? ! Now, from the second file, define a variable with the same name. yes? set data 2 yes? let/like=sst[d=1] sst_merged = temp[gx=xbig,gy=ybig] yes? ! Write this data using the /APPEND qualifier. The data will overwrite yes? ! the data in the file, only over the range of coordinates specified yes? save/append/file=bigfile.nc/x=181:359/y=-45:45 sst_merged LISTing to file bigfile.nc yes? ! Now look at what we have created. The listing spans data from both original datasets. yes? cancel data/all yes? cancel variable/all yes? use bigfile.nc yes? show grid sst_merged GRID GPJ1 name axis # pts start end subset XBIG LONGITUDE 180mr 1E 1W full YBIG LATITUDE 68 r 45S 89N full normal Z TIME TIME 1mr 01-JAN-2017 00:00 01-JAN-2017 00:00 full yes? list/x=170:190/y=0 sst_merged VARIABLE : SEA SURFACE TEMPERATURE (Deg C) FILENAME : bigfile.nc SUBSET : 10 points (LONGITUDE) LATITUDE : 1S TIME : 01-JAN-2017 00:00 1S 23 171E / 86: 29.00 173E / 87: 28.59 175E / 88: 28.31 177E / 89: 28.69 179E / 90: 28.20 179W / 91: 28.86 177W / 92: 27.98 175W / 93: 27.80 173W / 94: 28.29 171W / 95: 27.94
Example 5‚ multiple slabs
The procedure used in example 3 above, is possible because netCDF files can be extended along the time axis. In order to append multiple levels (Z axis), the netCDF file must first be created including all of its vertical levels (the levels initially are filled with a missing data flag).
Consider 5 ASCII files, uv.data1 to uv.data5, each defined on the same grid as above but each representing a successive vertical level. Note that the output grid has an axis containing all the Z levels that the file will contain (and that the other Z axes, zaxis1, zaxis2, .. are defined only for the purpose of reading the data in). The following set of commands illustrates how to save these data into a single netCDF data set.
NOTE: It is best to define a fixed-length axis in the "append" direction, as in the example below. If the axis in that direction is an abstract axis, for instance from a function such as XSEQUENCE or ESEQUENCE, the results will not be appended in Ferret v7.4 and previous versions.
yes? DEF AXIS/x=1:360:1 xaxis yes? DEF AXIS/y=1:180:1 yaxis yes? DEF AXIS/Z=0:100:25/DEPTH zaxis yes? DEF GRID/X=xaxis/Y=yaxis/Z=zaxis uv_grid yes? DEF AXIS/Z=0:0:1 zaxis1 yes? DEF GRID/LIKE=uv_grid/Z=zaxis1 uv_grid1 yes? FILE/GRID=uv_grid1/BAD=-999/VAR="u,v" uv.data1 yes? LET/TITLE="My U data" u1 = u[G=uv_grid@NRST] yes? LET/TITLE="My V data" v1 = v[G=uv_grid@NRST] yes? SAVE/FILE=uv1_5z.cdf/KLIMITS=1:5/K=1 u1, v1 yes? CANCEL DATA uv.data1 yes? DEF AXIS/Z=25:25:1 zaxis1 yes? FILE/GRID=uv_grid1/BAD=-999/VAR="u,v" uv.data2 yes? SAVE/FILE=uv1_5z.cdf/K=2/APPEND u1,v1 . . . yes? CANCEL DATA/ALL ! Cancel definitions before using new file yes? CANCEL VAR/ALL yes? USE uv1_5z.cdf
The @NRST above ensures that the regridding between the uv_grid and uv_grid1 will always work correctly. If the coordinate values are not simple integers, our definition of the axis zaxis and the value used to define the axis zaxis1 may not exactly match. @NRST regridding is needed so that the result will be on our grid uv_grid.
What if the data is in netCDF files to begin with, each file containing a Z level? Often separate files for different levels do not have a grid in the Z direction, so we'll need to handle that. Say the files are ocean_lev0m.nc, ocean_lev5m.nc, ocean_lev20m.nc containing data at those three depths. There might be some attributes in the file that describe the depth, or the information may be just in the dataset name; but there is no z axis.
yes? use ocean_lev0m.nc yes? show data currently SET data sets: 1> ./ocean_lev0m.nc (default) name title I J K L U Zonal Current 1:180 1:90 .. 1:12 V Meridional Current 1:180 1:90 .. 1:12
Define the appropriate output Z axis with all of the depths. Define variables ucur and vcur which are simply the data from the file repeated at each depth. This uses the concept of "conformability", how variables with unlike grids may be combined. Define the new variable that has a Z axis from that newly named file variable, and set the attributes for the new variable to be inherited from the file variable.
yes? define axis/z/depth/units=meters zdepth = {0,5,20}
yes? use ocean_lev0m.nc
yes? sh dat
yes? let ucur = u + 0*z[gz=zdepth]
yes? set attributes/like=u ucur
yes? save/clobber/file=ocean_data.nc/klimits=1:3/k=1 ucur
! Define and save v.
! The first SAVE for each variable needs /KLIMITS
yes? let vcur = v + 0*z[gz=zdepth]
yes? set attributes/like=v vcur
yes? save/append/file=ocean_data.nc/klimits=1:3/k=1 vcur
yes? cancel data ocean_lev0m.nc
! The definitions for ucur and vcur remain and apply to the
! current default dataset as we open each one.
yes? use ocean_lev5m.nc
yes? save/append/file=ocean_data.nc/k=2 ucur
yes? save/append/file=ocean_data.nc/k=2 vcur
yes? cancel data ocean_lev5m.nc
yes? use ocean_lev20m.nc
yes? save/append/file=ocean_data.nc/k=3 ucur
yes? save/append/file=ocean_data.nc/k=3 vcur
yes? cancel data/all
yes? cancel variables/all
yes? use ocean_data.nc
yes? show data
currently SET data sets:
1> ./ocean_data.nc (default)
name title I J K L
UCUR zonal current 1:180 1:90 1:3 1:12
VCUR meridional current 1:180 1:90 1:3 1:12
yes? show grid ucur
GRID GLO1
name axis # pts start end subset
XAX_SUBSET LONGITUDE 180mr 20.5E 18.5E(378.5) full
YAX_SUBSET LATITUDE 90 r 89.5S 88.5N full
ZDEPTH DEPTH (m) 3 i- 0 20 full
TIME TIME 12mr 16-JAN 06:00 16-DEC 01:20 full
Example 6‚ another with multiple slabs
Here we define a variable at each X location of an existing variable, and write them, appending them to a netCDF file. The expression uses the @MAX transformation in the X direction, so the result has no x axis. We'll define a new one-point x axis at each point.
yes? USE levitus_climatology ! Define a variable containing the xaxis locations yes? LET lon = x[gx=temp] ! Define the variable at the first X, put it on a one-point X axis, ! and then onto the full x axis of the variable. yes? DEFINE AXIS/X/UNITS="`temp,return=xunits`" xout = `lon[i=1]` yes? SHOW AXIS xout name axis # pts start end XOUT LONGITUDE 1mr 20.5E 20.5E Axis span (to cell edges) = 1 (modulo length = 360) yes? LET mtmp1 = temp[X=`lon[i=1]`:300@MAX] + 0*x[GX=xout] !-> DEFINE VARIABLE mtmp1 = temp[X=20.5:300@MAX] + 0*x[GX=xout] ! see its grid has a single-point x axis yes? SHOW GRID mtmp1 GRID (G003) name axis # pts start end XOUT LONGITUDE 1mr 20.5E 20.5E YAXLEVITR LATITUDE 180 r 89.5S 89.5N ZAXLEVITR DEPTH (m) 20 i- 0 5000 normal T ! put the variable onto the full X grid. yes? LET mtmp = mtmp1[GX=temp] ! This is the result at the first location saved to the full x axis. ! Write out the first x value, and use /ILIMITS to set up the full ! range in x that we'll be writing to the file yes? SAVE/CLOBBER/FILE=mout.nc/ILIMITS=1:90/I=1 mtmp ! Now do the same for each other X location. The variable ! at each X-step needs its own one-point x axis. yes? PAUSE yes? REPEAT/RANGE=2:30/NAME=inx (LET xstart = lon[I=`inx`];\ DEFINE AXIS/X/UNITS="`temp,RETURN=xunits`" xout = `xstart`;\ LET mtmp1 = temp[X=`xstart`:300@MAX] + 0*x[GX=xout];\ LET mtmp = mtmp1[GX=temp@ave];\ SAVE/APPEND/FILE=mout.nc/i=`inx` mtmp)
The netCDF utilities "ncdump" and "ncgen" can also be combined with a text editor to make final refinements to the netCDF files created by SAVE. (These utilities can be obtained from https://www.unidata.ucar.edu/. Here is a simple example that removes all "history" attributes from a netCDF file using pipes and the Unix "grep" utility:
% ncdump old_file.cdf | grep -v history | ncgen -o new_file.cdf
In addition, the toolset "NetCDF Operator", or NCO, contains a number of operators for manipulating netCDF files. This software can be found at https://nco.sourceforge.net/