nccmp

1. Overview
2. Download
3. Installation
4. Usage
5. Bugs
6. Wishlist
7. References
8. News

Overview

nccmp compares two NetCDF files bitwise or with a user defined tolerance (absolute or relative percentage). Some logic is used so it isn't blind like regular cmp. Comparisons are done in local memory without requiring temporary files while linearly traversing the data, so it's as fast as possible. Also, nccmp has been implemented with MPI and built as the command nccmpi to allow for parallel comparisons. Highly recommended for regression testing scientific models.

Features

• Specific variable inclusion or exclusion.
• Metadata and/or data comparisons.
• Global attributes compared with or without the history attribute.
• MPI enabled for supported systems.
• Script generation for previewing differences.
• Exits when first difference is found or optionally continues to process all variables and/or attributes.
• Attribute exclusion when comparing metadata.
• Bitwise compare or with tolerances (absolute or relative).

Tested Systems

• SGI Origin 3000 (MIPS 600MHz), IRIX 6.5
• SGI Altix (Itanium-2 1.5 Ghz), Linux 2.4.21 and 2.6.16.27-0.9
• P4 1.8GHz PC, Linux 2.4.18
• Core2 6700 2.66GHz PC, Linux 2.6.9-42.0.10.EL
• i386, Darwin 8.11.1

Wall-Clock Runtimes for Comparing 1Gb Files
Files contained 4 Non-Record Variables, 8 Record Variables, 324 Records

Download

Source Code: nccmp-0.6.0.tar.gz
Binaries

Installation

1. Download the latest NetCDF 3 distribution. You don't need to build it if your system already has NetCDF installed. Regardless, the header files are needed by nccmp for compiling.
2. Download nccmp.
3. Decompress with

   tar -zxvf nccmp-X.tar.gz
   

4. Set the environment variable NETCDF_PKG to the location where you placed the NetCDF distribution of Step 1 or specify it with the --with-netcdf option in Step 5.
5. ./configure [--with-netcdf=/...]
6. make; make check
7. make install; make clean;

Installation (Long Version)

1. Download the latest NetCDF 3 distribution. You don't need to build it if your system already has NetCDF installed. Regardless, the private header files are needed by nccmp for compiling.
2. Extract NetCDF 3 to some location such as

   cd /tmp
   tar xvzf netcdf.tar.Z
   # Creates /tmp/netcdf-3.6.2/.
   # Optionally build and install netcdf if not on system already.
   

3. Download nccmp and extract:

   cd /tmp
   tar zxvf nccmp-0.6.0.tar.gz
   

4. Set the environment variable NETCDF_PKG to the location where you placed the NetCDF distribution of Step 2 or specify it with the --with-netcdf option in Step 5.
5. Configure nccmp:

   cd /tmp/nccmp-0.6.0/src
   # For C-shell.
   setenv CFLAGS '-O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DNDEBUG'
   # For bash.
   #export CFLAGS='-O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DNDEBUG'
   ./configure --disable-mpi --with-netcdf=/tmp/netcdf-3.6.2 --prefix=/tmp/nccmp
   

6. make; make check
   ...
   make[2]: Entering directory `/tmp/nccmp-0.6.0/src/test'
   1.
   2.
   3.
   4.
   5.
   6.
   7.
   8.
   9.
   10.
   11.
   12.
   13.
   14.
   PASS: nccmp_test.sh
   ...
   

7. make install; make clean;

Usage

Compare two NetCDF files.
nccmp is silent when files are equivalent, otherwise it will echo to STDERR whether metadata or a specific variable differs.  By default, comparing stops after the first difference.

Exit code 0 is returned for identical files, 1 for different files, and 2 for a fatal error.

Usage: nccmp [OPTION...] file1 file2

  -A, --Attribute=att1[,...] Ignore attribute(s) att1[,...]
                             for all variables.
  -b, --verbose              Print messages to show progress.
  -d, --data                 Compare data (variable values).
  -f, --force                Forcefully compare, do not stop after first difference.
  -g, --global               Compare global attributes.
                             (-m required)
  -G, --globalex att1[,...]  Exclude global attributes att1[,...].
                             (-mg required)
  -h, --history              Compare global history attribute.
                             (-mg required)
  -H, --help                 Give this usage message.
      --usage
  -m, --metadata             Compare metadata, excluding global attributes.
  -q, --quiet                Quiet mode, prints differences only.
  -t, --tolerance=TOL        Compare if absolute difference > TOL.
  -T, --Tolerance=TOL        Compare if relative percent difference > TOL.
  -v, --variable=var1[,...]  Compare variable(s) var1[,...] only.
  -V, --version              Print program version.
  -x, --exclude=var1[,...]   Exclude variable(s) var1[,...].

Bugs

1. Exit code isn't consistent for nccmpi.

Wishlist

References

1. NetCDF Data Format

News

Version 0.6.0

1. Added new -G command-line option to ignore specific global attributes.  Thanks to V. Balaji.

Version 0.5.4

1. Avoided reading past EOF when number of records differ between files.  Record variables won't be compared as a result.  
Users should use another tool, like "ncks" to extract record-based hyperslabs if they really need to compare partial netcdf files.  Thanks to Lionel Guez.
2. Handles case when both files have record dimensions with length zero. Thanks to Lionel Guez.

Version 0.5.3

1. Fixed memory thrashing in string management code.
2. Made --enable-mpi configure argument and padded header testing more robust.

Version 0.5.2

1. Added command-line option to ignore specific variable attributes.

Version 0.5.1

1. Fixed bug. Single record variables were skipped.

Version 0.5.0

Thanks to Amy Langenhorst for suggestion 1.

1. Compare with absolute or relative percentage tolerance. Obviously slower due to arithmetic. No MPI support for tolerance feature.

Version 0.4.2

Thanks to Amy Langenhorst for these suggestions.

* forces removal of temporary files in generated view script.
* generated view script tests if there exists a user defined NCCMPDIFF,
  otherwise sets it to 'diff' as default.
* suppresses global history attribute when extracting data in 
  generated view script.

Version 0.4.1

* bug fix: prevents premature file closing while still querying for
info such as when generating example view script.

Version 0.4

* builds nccmpi for usage with MPI enabled systems

Version 0.3

* ported completely to C; was a perl script originally

Version 0.2.1

* bug fix: check for comparisons beyond EOF
* bug fix: memory allocation for variable list

Version 0.2

* -q quiet mode, suppresses all but difference message
* -f force mode, process all variables possible; doesn't exit program
  after 1st difference
* -x exclude variable list option
* enhanced view script 

Version 0.1

* outputs script to do visual comparison of different values