OPeNDAP (formerly called DODS, the Distributed Oceanographic Data System) allows users to access data anywhere from the Internet using a variety of client/server methods, including Ferret. Together, OPeNDAP and Ferret create a powerful tool for the retrieval, sampling, analyzing and displaying of datasets; regardless of size or data format (though there are data format limitations).
For more information, please see the OPeNDAP home page.
2.7.2 Accessing Remote Data Sets
Datasets are accessed through Ferret using their raw Universal Resource Locator (URL) address. Prior to Ferret v7, the address must be enclosed in quotes, as for any dataset name that includes a path. With Ferret/ PyFerret v7, this requirement is lifted. For example, to access datasets hosted at PMEL:
yes? use "http://ferret.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/coads_climatology.nc" ! v7 or higher: yes? use http://ferret.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/etopo5.nc
Once the dataset has been initialized, it is used just like any other local dataset.
yes? list/x=140w/y=2n/t="16-Feb" sst VARIABLE : SEA SURFACE TEMPERATURE (Deg C) FILENAME : coads_climatology.nc FILEPATH : https://data.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/ LONGITUDE: 141W LATITUDE : 1N TIME : 15-FEB 16:29 26.39
To locate OPeNDAP data, you can search the Global Change Master Directory
Beginning with Ferret v6.6, Ferret can read HDF-5 formatted files, and can write files with HDF compression and chunking. Fpr details, see the section on Ferret and NetCDF version 4.
Previous to Ferret v6.6, HDF files could be read by Ferret only via OPeNDAP , that is the HDF file must first be put on an OPeNDAP server and then Ferret can access it by giving its OPeNDAP URL. Even by this means, Ferret will be successful in reading the file only if the HDF file is similar in its structure to a COARDS or CF netCDF file. Often, you will need to apply the USE/ORDER= qualifier to change the ordering of the coordinate axes.
If a file is on an OPeNDAP server, you can look at the OPeNDAP in your browser (the URL that ends in .das). When you look at the attribute data check to see if there are dimension variables with attributes that look like a Latitude or Longitude as in the COARDS conventions.
2.7.3 Debugging Access to Remote OPeNDAP Data Sets
To find out more information about a particular dataset, or to debug problems, there are three elements of the dataset which may be accessed via a web browser. To access this information, merely append a dds, das, or info to the dataset name. For example:
https://data.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/coads_climatology.nc.dds
DDS stands for Data Description Structure and this will return a text description of the data sets structure.
https://data.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/coads_climatology.nc.das
DAS stands for Dataset Attribute Structure and this will return a text description of attributes assigned to the variables in the data set.
https://data.pmel.noaa.gov/pmel/thredds/dodsC/data/PMEL/coads_climatology.nc.info
This will return a text description of the variables in the dataset, if available.
Some OPeNDAP data providers will choose to control access to some or all of their data. When you request data from one of these servers, the OPeNDAP client will prompt you for a username and password. If you want to avoid the prompt, you can embed a username and password in it, like this:
http://user:password@www.dods.org/nph-dods/etc...
2.7.5 Sharing Data Sets via OPeNDAP
One of the most powerful aspect of OPeNDAP is the ease with which it allows for the sharing of data. With just a few simple steps, anyone running a web server can also be an OPeNDAP data server, thereby allowing data set access to anyone with an Internet connection.
Simply copying a few precompiled binaries into the cgi-bin directory of an already configure httpd server is all it takes to become a OPeNDAP server. Once the server is configured, adding or removing data sets is as simple as copying them to the server data directory or deleting them from that directory.
This ability has such immense potential that it bears extra emphasis. Imagine that within seconds of finishing a model run, a remote colleague is able to look at your dataset with whatever DOOPeNDAP DS client he/she desires, be it Ferret, or Matlab, etc. No need for you to package up the data or for your colleague to download and/or reformat it, it is ready to be analyzed right away.
This feature allows caching of frequently accessed OPeNDAP served datasets to produce a quicker response when requesting remote data. The first time you access an OPeNDAP data set, a file in the users home directory will be created called .dodsrc. This file is the OPeNDAP client initialization file. Please see the OPeNDAP documentationfor details of the paramaters that this file contains. Initially, OPeNDAP caching will be turned off. In order to turn caching on, change the line in the newly created ~/.dodsrc file from
USE_CACHE=0
to
USE_CACHE=1
Note that if you edit the .dodsrc file, make sure that the lines within it all start in the first column.
The next time Ferret is run, and a OPeNDAP-served dataset is accessed, a file called .dods_cache will be created, typically in the users home directory. The location of the OPeNDAP cache directory can be controlled by the line
CACHE_ROOT=/home/twaits/.dods_cache/
in the user's .dodsrc file. This directory is where all the cached information is stored. To clear the OPeNDAP cache, simply delete the .dods_cache directory and all of it's contents (for example, rm -rf ~/.dods_cache). This directory will be recreated and repopulated with caching information the next time data is accessed via OPeNDAP , if caching is turned on. All of the paramater values in the .dodsrc file can be modified to better suit individual needs, and will be incorporated the next time Ferret is run and OPeNDAP served data is accessed. Again, see the OPeNDAP User guide, in particular the section "The OPeNDAP Client Initialization File (.dodsrc)" for more detailed information.
It is often a useful diagnostic exercise to turn caching off and/or clear out the cache directory when attempts to access datasets in Ferret appear inconsistent. For example, if Ferret attempted to access an OPeNDAP-served dataset that was unavailable because the OPeNDAP server was down, that information may get cached and adversely effect the next attempt at retrieving the data.
For more detailed information on using OPeNDAP , and on setting up an OPeNDAP server, see the OPeNDAP web pages.>
A OPeNDAP client can negotiate proxy servers, with help from directions in its configuration file. The parameters that control proxy behavior are fully documented in the OPeNDAP Users Guide, see the link above.