FAQ

Using data that spans multiple files.

Question:

How can I plot the time series of a variable which spans multiple data files?

Example:

> I have some netcdf-files with a lot of data. Each file represents the
> status of a certain model time. Now I want to plot one variable (say the
> instantanous mean) as a function of time. How can I do it?
>
> And can I save the calculated time series in a seperate (binary or netcdf)
> file from Ferret?

Explanation:

This classic FAQ describes "descriptor files", a long-standing method of defining a muti-file dataset. However Dataset aggregation is a simpler and more flexible means of handling multi-file datasets, whether the multiple files contain different timesteps of the same variables, or different variables or ensembles of data from model runs. Time aggregation is particularly simple; Ferret/PyFerret will analyze the time information and define the time axis for you.

See the full documentation here: DEFINE DATA/AGGREGATE

The answer to this depends on how your existing NetCDF files are arranged. You say that each file has a single time step. Let us suppose there are 3 spatial dimensions for the variables in your NetCDF files. There are two ways that this could be encoded: either as 3D (time-independent) variables or as 4D variables on a single point time axis. "ncdump -h mynetcdf.cdf" will show if the variables have a 1-point time axis.

Solution:

==> If the files have a 1 point time axis there are three solutions:

If you have a THREDDS server installed, use THREDDS aggregation to make the data into a single virtual dataset. THREDDS aggregation may be used to combine files that contain different timesteps of the same variable(s). It also includes other powerful tools for modifying and combining datasets.
Define a virtual dataset in the Ferret session (Ferret v7 and higher) using DEFINE DATA/AGGREGATE/T. (Virtual dataset aggregation may also be done in the E direction, to define an ensemble, or the F direction, defining a forecast collection.)
```
yes? let file_list = SPAWN("ls -1 tmp/step_*.nc")
yes? TSERIES all_times = file_list
```
Use ferret to concatenate the files:
```
use file1
save/file=concat.cdf var1, var2, var3
use file2
save/append/file=concat.cdf var1,var2,var3
.
.
.
```
note that the names "var1", "var2",... refer to the "current" data set since no explicit data set was specified as in "var1[d=file1]".

Create an "MC" (multi-cdf) data set by writing a descriptor file that ties the files into a single data set. An example of such a descriptor is below.

==> If your files do not have a time word,

then you must use Ferret to impose atime word. Suppose you have 10 files, each separated by one model day, running 1-jan-1980 to 10-jan-1980. Then you might use this technique:

! *** set up
define axis/t=1-jan-1980:10-jan-1980:1/unit=days t10days
define grid/t=t10days gg
 
let var1_with_T = t[g=gg]*0 + var1
set variable/title="xxx"/units="uuu" var1_with_T
 
! *** loop over input files
use file1.cdf
save/file=concat.cdf var1_with_T[l=1]
 
use file2.cdf
save/file=concat.cdf/append var1_with_T[l=2]
 
...

--------------------------------------------------
Here is an example "MC" descriptorfile: my_mc_descriptor.des:

$FORMAT_RECORD
D_TYPE = ' MC',
D_FORMAT = ' 1A',
D_SOURCE_CLASS = 'MODEL OUTPUT',
D_SOURCE = 'PHIL/SIEG',
D_SUBSOURCE = 'TROPICAL PACIFIC',
$END
$BACKGROUND_RECORD
D_EXPNUM = '0068',
D_MODNUM = ' AA',
D_TITLE = 'MOM model output; Hindcast, Double Ramp; NCEP 79-95 twice daily winds'
D_FORCING_SET = 'NCEP 79-95 daily',
D_T0TIME = '14-JAN-1978 14:00:00',
D_TIME_UNIT = 3600.0,
D_TIME_MODULO = .TRUE.,
D_ADD_PARM = 15*' ',
$END
$MESSAGE_RECORD
D_MESSAGE = ' ',
D_ALERT_ON_OPEN = F,
D_ALERT_ON_OUTPUT = F,
$END
*************************************************
*************************************************
$EXTRA_RECORD
$END
 
$STEPFILE_RECORD
s_filename = 'mtaa068-nc.001',
S_AUX_SET_NUM = 0,
S_START = 87623.0,
S_END = 183599.0,
S_DELTA = 24.0,
S_NUM_OF_FILES = 400,
S_REGVARFLAG = ' ',
$END
**************************************************
$STEPFILE_RECORD
s_filename = '**END OF STEPFILES**'
$END
**************************************************