NetCDF Output

According to the Unidata website the Network Common Data Form (NetCDF) software was originally intended to create, access, and share array-oriented data in a form that is self-describing and portable. These deal with a variety of data types that encompass single-point observations, time series, regularly-spaced grids, and satellite or radar images.

The NetCDF software functions as an input/output (I/O) library that can be accessed from many scientific data processing and GIS software packages and different programming languages, such as C, FORTRAN, Fortran 90, C++, Java, Perl, Python, MATLAB, and other languages for which a NetCDF library is available. A NetCDF library stores and retrieves data in self-describing, machine-independent datasets, which are described on the Unidata website. The NetCDF user guide provides details on how NetCDF datasets can contain multidimensional, named variables (of various types that include integers, reals, characters, bytes, etc.), and each variable may be accompanied by ancillary data, such as units of measure or descriptive text. The interface includes a method for appending data to existing NetCDF datasets in prescribed ways, functionality that is not unlike a (fixed length) record structure. However, the NetCDF library also allows direct-access storage and retrieval of data by variable name and index and therefore is useful only for disk-resident (or memory-resident) datasets. You can learn about NetCDF compression here.

NetCDF is designed to:

• Facilitate the use of common datasets by distinct applications.

• Permit datasets to be transported between or shared by dissimilar computers transparently, that is, without translation.

• Reduce the programming effort usually spent interpreting formats.

• Reduce errors arising from misinterpreting data and ancillary data.

• Facilitate using the output from one application as input to another.

• Establish an interface standard that simplifies the design of new software for accessing geoscience data.

There is now an unstructured (or flexible mesh) model data format for NetCDF called UGRID.  From the EEMS10.4 version, DSI enhanced the NetCDF capabilities of the EFDC+ Explorer Modeling System (EEMS), so that it is able to handle the UGRID format. Although UGRID is designed for unstructured grids, the advantage for EFDC+ and structured grids is that it writes output for only the active grid cells, rather than the full I,J grid required by web map services for displaying environmental data such as ncWMS. Furthermore, the structure of the output variables is clearly defined, along with the variable name and units. This allows users to easily process EFDC+ output using third-party software.  UGRID is also based on the CF Metadata Conventions, but this approach currently assumes that the horizontal topology should be determined from the cell index of the EFDC+ grid rather than the I,J indices of the structured grid. The UGRID approach adds conventions to specify the topology for unstructured (e.g., triangular) grids.

UGRID also provides powerful new capabilities for display on third-party web tools, some of which are listed here. This will provide an enhanced user experience when accessing real-time models using a WebGIS application.

EFDC+ can currently write the native output in NetCDF-CF format for Windows and Linux. EFDC+ Explorer (the pre-and post-processor for EFDC+) can also convert the binary files output by EFDC+ (*.OUT) to netCDF.nc files, including the derived water quality parameters. EFDC+ currently exports 44 parameters to NetCDF format. This allows EFDC+ output to be read and displayed on web servers. It should be noted that while the display order of some dimensions (i.e. TIME, KC, CELL) in *.nc output files exported from EFDC+ Explorer may differ from that obtained directly from EFDC+, the values of variables are not impacted and retrieved correctly with reference to their descriptive data.

EFDC+ exports the NetCDF files at the same interval that it produces the model linkages files, which is set in the tab for Timing and Linkage | Linkage | NetCDF. 

The main configuration for NetCDF output is in the EFDC_Exlporer Linkage | NetCDF tabs as shown in Figure 1.  Here the user can turn on netCDF output by checking the Output to NetCDF checkbox.  The user may select the level at which the NetCDF data is compressed with the Deflate Level setting. This can range from 1, no compression, to 9 for maximum compression, but is recommended by Unidata to be set to 2 for the best balance between file size and speed. 

The user may also select to Rotate Velocity. If this option is checked, the velocity field is rotated to true east and true north and if not checked then there is no rotation to true east and true north.

Figure 1. NetCDF Setting.

Then, the user can select which parameters they want to output from the water column or the sediment bed by checking the box in Output Parameters frame. Users can select more detailed components when writing to a NetCDF file. Some options for the sediment bed are not yet enabled and will be added to EE in future releases.

In the File Creation frame, the user is given the option of EFDC+ generating a Single File, or Multiple Daily Files.The Multiple Daily Files option provides one netCDF file for each day (24 hours) of model output. NetCDF files (.nc files) are placed in the #output folder along with the EFDC output.  The snapshots for the NetCDF files are determined by the setting for the interval for the EE linkage file. The user can change the snapshot interval by changing the EE linkage options. It is recommended that the linkage frequency should be 10 minutes or more for correct display in ArcGIS tools.  After setting these options, the user should save their model and run EFDC+, which will then export the NetCDF .nc files to the #output folder. With release of EEMS11.3, EFDC+ writes the LPT NetCDF output to a separate .nc file. The output frequency for the drifters in NetCDF is based on the LPT output frequency.

Displaying NetCDF Output

There are a number of options available to the user for displaying netCDF output, such as using ArcGIS, or using web-based tools such as ncWMS. Some of these are outlined here to help the user to become familiar with these tools.

1) Convert to Text

To convert the .nc file to a text file, the user can download the ncdump.exe tool and follow step-by-step instructions in Unidata.

2) Using ArcMap

A detailed guidance video for visualizing NetCDF output using ArcMap has been prepared by DSI and made available here: https://www.youtube.com/watch?v=vgT7dN4UK44

The basic steps involve selecting the ArcToolbox > Multidimensional Tools > Make NetCDF Feature Layer.

Figure 2. ArcGIS reading the NetCDF file.

3) Using NCWMS

ncWMS is a  Web Map Service for geospatial data that are stored in  CF-compliant  NetCDF files. The intention is to create a WMS that requires minimal configuration: the source data files should already contain most of the necessary metadata. ncWMS is developed and maintained by the  Reading e-Science Centre at the  University of Reading, UK. The latest version is 1.3, released in August 2014.

ncWMS is implemented in Java as a web application. To install ncWMS you simply place the WAR (Web ARchive) file in your application server of choice (e.g.  Tomcat).

ncWMS relies heavily on the  Java NetCDF interface from Unidata. This library does a lot of the work of metadata and data extraction. In particular the  GridDatatype class is frequently used to provide a high-level interface to gridded geospatial NetCDF files. The library will also read from NetCDF files on HTTP servers and from  OPeNDAP servers. From http://godiva.reading.ac.uk/ncWMS2/Godiva3.html

ncWMS loads native netCDF files and displays them on a web browser. Making the output the same as EE10 would require further adjustment to the tool. For the purposes of this demonstration, the basic display of the output is shown in the next section.

4) Using Ocean Data View

Ocean Data View (ODV) is a software package for the interactive exploration, analysis, and visualization of oceanographic and other geo-referenced profile, time-series, trajectory, or sequence data.

ODV supports the netCDF format and lets you explore and visualize CF, COARDS, GDT, and CDC-compliant netCDF datasets. This works with netCDF files on your local machine as well as with remote netCDF files served by an OPeNDAP server.

5) Using Python script

Python script can be used to read the netCDF output of the model. For example, a model with 1000 horizontal grids, and 5 layers, and the model simulated with 1-hour output frequency for 10 days. So each day will have 25-time steps → there are 1000*5*(24*10 + 1) = 1,205,000 lines totally. In which of the total lines there are lines with NaN values (dry cells), they should be removed. The User can contact us to get the script.

Examples of Model Output

To demonstrate the netCDF output is consistent with the data viewed in EE, a number of comparisons have been made for different models. Examples of an output from the West Lake WQ model in EE, ArcGIS, and ncWMS are shown in Figure 3, Figure 4 and Figure 5 respectively.  Note that the West Lake model may be downloaded from the EE website.

Figure 3. EE display of EFDC+ output – DO at 137.001

Figure 4. ArcGIS display of EFDC+ netCDF output – DO at 137.001

Figure  5. ncWMS display of EFDC+ netCDF output – DO at 137.001