nccmp : NetCDF Compare

nccmp

Overview
Download
Installation
Examples
Usage
Bugs
Wishlist
References
News

Overview

nccmp compares two NetCDF files bitwise or with a user defined tolerance (absolute or relative percentage). Some logic is used so it isn't blind like regular cmp. Comparisons are done in local memory without requiring temporary files. Also, nccmp (versions prior to 1.0) has been implemented with MPI and built as the command "nccmpi" to allow for parallel comparisons. Highly recommended for regression testing scientific models.

Features

Prints differences and their locations.
Specific variable inclusion or exclusion.
Metadata and/or data comparisons.
Global attributes compared with or without the history attribute.
MPI enabled for supported systems.
Script generation for previewing differences. (pre v1.0)
Exits when first difference is found or optionally continues to process all variables and/or attributes.
Attribute exclusion when comparing metadata.
Bitwise compare or with tolerances (absolute or relative).

Download

Source Code: nccmp-1.1.0.tgz
Binaries

Installation

Download nccmp.
Decompress with
```
tar -zxvf nccmp-*gz
```
./configure
make; make check
make install; make clean;

Examples

###############################################################################
# Compare metadata for non-record variables.
nccmp  -mqf -x recdata,rec test1a.nc test1b.nc

DIFFER : NUMBER OF DIMENSIONS IN FILES : 5 <> 4
DIFFER : LENGTHS : DIMENSION : dim3 : 2 <> 1
DIFFER : NAME : DIMENSION : dim4 : DIMENSION DOESN'T EXIST IN "test1b.nc"
DIFFER : NAME : VARIABLE : data3 : VARIABLE DOESN'T EXIST IN test1b.nc
DIFFER : DIMENSION NAMES FOR VARIABLE data4 : dim4 <> dim2
DIFFER : VARIABLE "data1" IS MISSING ATTRIBUTE WITH NAME "units" IN FILE test1b.nc
DIFFER : LENGTHS : ATTRIBUTE : long_name : VARIABLE : data1 : 9 <> 10
DIFFER : VALUES : ATTRIBUTE : valid_range : VARIABLE : data1
DIFFER : NUMBER OF ATTRIBUTES : VARIABLE : data1 : 3 <> 2

# Compare data using a tolerance threshold.
nccmp -df -t 1e-3 test4a.nc test4b.nc

DIFFER : VARIABLE : data1 : POSITION : 0 1 : VALUES : 2 <> 2.01 : PERCENT : 0.497512
DIFFER : VARIABLE : data2 : POSITION : 0 1 : VALUES : 8 <> 7.99 : PERCENT : 0.125003
DIFFER : VARIABLE : data3 : POSITION : 0 : VALUES : 9 <> 8 : PERCENT : 11.1111
DIFFER : VARIABLE : data3 : POSITION : 1 : VALUES : 10 <> -10 : PERCENT : 200
DIFFER : VARIABLE : data4 : POSITION : 0 : VALUES : 11 <> 110 : PERCENT : 90
DIFFER : VARIABLE : recdata1 : POSITION : 0 1 : VALUES : 4 <> 0 : PERCENT : 100
DIFFER : VARIABLE : recdata2 : POSITION : 0 0 : VALUES : 97 <> 99 : PERCENT : 2.0202
DIFFER : VARIABLE : rec : POSITION : 1 : VALUES : 6 <> 6.1 : PERCENT : 1.63934

###############################################################################
# Compare data written in different formats (classic vs. HDF).
nccmp -dmgfb old.nc new.nc 

INFO: Command-line options parsed.
INFO: Opening input files.
INFO: Creating variable comparison list.
INFO: Using all variables.
INFO: Variables to compare (11):
lon, lat, tile_index, frac, glac, lake, soil, vegn, lwup, e_res_1, e_res_2
INFO: Comparing file formats.
DIFFER : FILE FORMATS : NC_FORMAT_NETCDF4_CLASSIC <> NC_FORMAT_64BIT
INFO: Comparing global attributes.
INFO: Collecting dimension information for first file.
INFO: Collecting dimension information for second file.
INFO: Collecting variable information for first file.
INFO: Collecting variable information for second file.
INFO: Comparing record information.
INFO: Comparing metadata.
INFO: Comparing number of dimensions.
INFO: Getting record dimension names, if they exist.
INFO: Comparing dimension lengths.
INFO: Comparing variable datatypes and rank.
INFO: Comparing variables' dimension names.
INFO: Comparing variables' attributes.
INFO: Comparing data.
INFO: Comparing data for variable "lon".
INFO: Comparing non-record data for variable "lon".
INFO: Comparing data for variable "lat".
INFO: Comparing non-record data for variable "lat".
INFO: Comparing data for variable "tile_index".
INFO: Comparing non-record data for variable "tile_index".
INFO: Comparing data for variable "frac".
INFO: Comparing non-record data for variable "frac".
INFO: Comparing data for variable "glac".
INFO: Comparing non-record data for variable "glac".
INFO: Comparing data for variable "lake".
INFO: Comparing non-record data for variable "lake".
INFO: Comparing data for variable "soil".
INFO: Comparing non-record data for variable "soil".
INFO: Comparing data for variable "vegn".
INFO: Comparing non-record data for variable "vegn".
INFO: Comparing data for variable "lwup".
INFO: Comparing non-record data for variable "lwup".
INFO: Comparing data for variable "e_res_1".
INFO: Comparing non-record data for variable "e_res_1".
INFO: Comparing data for variable "e_res_2".
INFO: Comparing non-record data for variable "e_res_2".
INFO: Finished comparing data.
INFO: Comparisons complete. Freeing memory.

Usage

Compare two NetCDF files.
nccmp is silent when files are equivalent, otherwise it will echo to STDERR whether metadata or a specific variable differs.  By default, comparing stops after the first difference.

Exit code 0 is returned for identical files, 1 for different files, and 2 for a fatal error.

Usage: nccmp [OPTION...] file1 file2

  -A, --Attribute=att1[,...] Ignore attribute(s) att1[,...]
                             for all variables.
  -b, --verbose              Print messages to show progress.
  -d, --data                 Compare data (variable values).
  -D, --debug                Prints verbose debug messages.
  -f, --force                Forcefully compare, do not stop after first difference.
  -F, --fortran              Print position indices using Fortran style (1-based reverse order).
  -g, --global               Compare global attributes.
                             (-m required)
  -G, --globalex att1[,...]  Exclude global attributes att1[,...].
                             (-mg required)
  -h, --history              Compare global history attribute.
                             (-mg required)
  -H, --help                 Give this usage message.
      --usage                
  -m, --metadata             Compare metadata, excluding global attributes.
  -p, --precision='%.17g'    Precision of difference printing (default is '%g').
                             Use '%x' to print bytes.
  -t, --tolerance=TOL        Compare if absolute difference > TOL.
  -T, --Tolerance=TOL        Compare if relative percent difference > TOL.
  -v, --variable=var1[,...]  Compare variable(s) var1[,...] only.
  -V, --version              Print program version.
  -x, --exclude=var1[,...]   Exclude variable(s) var1[,...].
  -w, --warn=tag[,...]       Selectively make certain differences exceptions,
                             so only informational messages are printed for them.
                             Supported tags and their meaning:
                             all: All differences.
                             format: File formats.

Bugs

Exit code isn't consistent for nccmpi.

Wishlist

References

NetCDF Data Format

News

Version 1.1.0 (7/15/2010)

1. Added "-w, --warn" command-line option to make certain differences exceptions and only print messages of their occurance. Thanks to Aleksey Yakovlev.
2. Added a small helper program called 'ncinfo' that echoes a file's format, header size (NetCDF3 files only) and compression characteristics for NetCDF4 files.

Version 1.0.4 (6/27/2010)

1. Bug fix when comparing attributes when a variable was missing in a file. Thanks to Lionel Guez.
2. Pretty print attribute values when they differ.
3. Added simple utility "ncinfo" in /ncinfo to echo file's format and chunk/compression/header characteristics. Thanks to Aleksey Yakovlev.

Version 1.0.3 (4/13/2010)

1. Added hex printing.
2. Added custom precision of float/single value printing.
3. Added fortran-style index printing.

Version 1.0.2 (3/20/2010)

1. Fixed unsafe record queries.
2. Added more information "verbose" progress messages.
3. Made "quiet" mode from pre-1.0 deprecated by making option hidden although still accepted from command-line.

Version 1.0.1 (3/10/2010)

1. Fixed record-based scalar variable comparisons.

Version 1.0.0 (1/20/2010)

1. Overhauled to work generically with Netcdf4 at the cost of lower performance.

Version 0.6.0

1. Added new -G command-line option to ignore specific global attributes. Thanks to V. Balaji.

Version 0.5.4

1. Avoided reading past EOF when number of records differ between files. Record variables won't be compared as a result.
Users should use another tool, like "ncks" to extract record-based hyperslabs if they really need to compare partial netcdf files. Thanks to Lionel Guez.
2. Handles case when both files have record dimensions with length zero. Thanks to Lionel Guez.

Version 0.5.3

1. Fixed memory thrashing in string management code.
2. Made --enable-mpi configure argument and padded header testing more robust.

Version 0.5.2

1. Added command-line option to ignore specific variable attributes.

Version 0.5.1

1. Fixed bug. Single record variables were skipped.

Version 0.5.0

Thanks to Amy Langenhorst for suggestion 1.

1. Compare with absolute or relative percentage tolerance. Obviously slower due to arithmetic. No MPI support for tolerance feature.

Version 0.4.2

Thanks to Amy Langenhorst for these suggestions.

* forces removal of temporary files in generated view script.
* generated view script tests if there exists a user defined NCCMPDIFF,
otherwise sets it to 'diff' as default.
* suppresses global history attribute when extracting data in
generated view script.

Version 0.4.1

* bug fix: prevents premature file closing while still querying for
info such as when generating example view script.

Version 0.4

* builds nccmpi for usage with MPI enabled systems

Version 0.3

* ported completely to C; was a perl script originally

Version 0.2.1

* bug fix: check for comparisons beyond EOF
* bug fix: memory allocation for variable list

Version 0.2

* -q quiet mode, suppresses all but difference message
* -f force mode, process all variables possible; doesn't exit program
after 1st difference
* -x exclude variable list option
* enhanced view script

Version 0.1

* outputs script to do visual comparison of different values