SourceForge.net Logo

nccmp

  1. Overview
  2. Download
  3. Installation
  4. Examples
  5. Usage
  6. Bugs
  7. Wishlist
  8. References
  9. News


Overview

nccmp compares two NetCDF files bitwise or with a user defined tolerance (absolute or relative percentage). Some logic is used so it isn't blind like regular cmp. Comparisons are done in local memory without requiring temporary files. Highly recommended for regression testing scientific models or datasets in a test-driven development environment.

Features

Multithreading

Static scheduling is used for threads. Each thread will compare every time-varying variable over strided record indices. This is optional, and configure will determine if POSIX threads are available.

Header Pad

In order to compare the contents of header pads, it is required to download the NetCDF source code and configure nccmp using --with-netcdf=dir in order to access the private netcdf API. This is optional, and nccmp can be built without this feature.


Download

Source Code: nccmp-1.6.1.tgz


Installation

Linux/Unix/MacPorts

# (Optional) Download the NetCDF source code to support header-pad comparisons.
$ cd /tmp && \
  wget -nd -c http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.3.3.1.tar.gz && \
  tar zxvf netcdf-*.tar.gz

# Specify non-standard installation locations for HDF and NetCDF, if necessary.
$ HDF=/usr/local/hdf5-1.8.15-patch1
$ NC=/usr/local/netcdf-4.3.3.1
$ export CFLAGS="-I$HDF/include -I$NC/include"
$ export LDFLAGS="-L$HDF/lib -L$NC/lib"

$ tar zxvf nccmp-*.tgz && cd nccmp-*
$ ./configure --with-netcdf=/tmp/netcdf-4.3.3.1
checking for a BSD-compatible install... /usr/bin/install -c
...
configure:

    CFLAGS       = -I/usr/local/hdf5-1.8.15-patch1/include -I/usr/local/netcdf-4.3.3.1/include -pthread
    CPPFLAGS     = 
    HAVE_NC_H    = yes
    HAVE_PTHREAD = yes
    LDFLAGS      = -L/usr/local/hdf5-1.8.15-patch1/lib -L/usr/local/netcdf-4.3.3.1/lib -pthread
    LIBS         = -lnetcdf -lm 
    with-netcdf  = /tmp/netcdf-4.3.3.1

$ make
$ make check
$ make install
$ make clean

Cygwin

$ C:/cygwin/setup-x86.exe
# Install these prebuilt binary cygwin packages:
# All => Libs  => libnetcdf7      v4.3.3.1-2
#              => libnetcdf-devel v4.3.3.1-2
# All => Devel => libhdf5-devel   v1.8.15-1
#     => Libs  => libhdf5_10      v1.8.15-1

$ ./configure --with-netcdf=/cygdrive/c/Temp/netcdf-4.3.3.1
$ make
$ make check
$ make install

Mac OSX Homebrew

$ brew tap homebrew/science
$ brew install hdf5
$ brew install netcdf

$ NC=/usr/local/Cellar/netcdf/4.3.3.1
$ export CFLAGS="-I$NC/include"
$ export CPPFLAGS="$CFLAGS"
$ export LDFLAGS="-L$NC/lib"
$ export DYLD_LIBRARY_PATH="$DYLD_LIBRARY_PATH":$NC/lib

$ ./configure --prefix=/Users/rsz/bin --with-netcdf=/Users/rsz/source/netcdf-4.3.3.1
$ make
$ make check
$ make install

Testing

$ make check
make  check-TESTS
make[2]: Entering directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
make[3]: Entering directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
PASS: test_strlist.exe
PASS: test_odometer.exe
PASS: test_nccmp_00.sh
PASS: test_nccmp_01.sh
...
PASS: test_nccmp_48.sh
PASS: test_big.sh
make[4]: Entering directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
============================================================================
Testsuite summary for nccmp 1.6.1
============================================================================
# TOTAL: 52
# PASS:  52
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
make[2]: Leaving directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
make[1]: Leaving directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code/test'
make[1]: Entering directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code'
make[1]: Leaving directory '/cygdrive/r/rsz/projects/nccmp/nccmp-code'


Examples

#############################################################
# Compare metadata for non-record variables.
nccmp  -mqf -x recdata,rec test1a.nc test1b.nc

DIFFER : NUMBER OF DIMENSIONS IN FILES : 5 <> 4
DIFFER : LENGTHS : DIMENSION : dim3 : 2 <> 1
DIFFER : NAME : DIMENSION : dim4 : DIMENSION DOESN'T EXIST IN "test1b.nc"
DIFFER : NAME : VARIABLE : data3 : VARIABLE DOESN'T EXIST IN test1b.nc
DIFFER : DIMENSION NAMES FOR VARIABLE data4 : dim4 <> dim2
DIFFER : VARIABLE "data1" IS MISSING ATTRIBUTE WITH NAME "units" IN FILE test1b.nc
DIFFER : LENGTHS : ATTRIBUTE : long_name : VARIABLE : data1 : 9 <> 10
DIFFER : VALUES : ATTRIBUTE : valid_range : VARIABLE : data1
DIFFER : NUMBER OF ATTRIBUTES : VARIABLE : data1 : 3 <> 2

#############################################################
# Compare data using a tolerance threshold.
nccmp -df -t 1e-3 test4a.nc test4b.nc

DIFFER : VARIABLE : data1 : POSITION : 0 1 : VALUES : 2 <> 2.01 : PERCENT : 0.497512
DIFFER : VARIABLE : data2 : POSITION : 0 1 : VALUES : 8 <> 7.99 : PERCENT : 0.125003
DIFFER : VARIABLE : data3 : POSITION : 0 : VALUES : 9 <> 8 : PERCENT : 11.1111
DIFFER : VARIABLE : data3 : POSITION : 1 : VALUES : 10 <> -10 : PERCENT : 200
DIFFER : VARIABLE : data4 : POSITION : 0 : VALUES : 11 <> 110 : PERCENT : 90
DIFFER : VARIABLE : recdata1 : POSITION : 0 1 : VALUES : 4 <> 0 : PERCENT : 100
DIFFER : VARIABLE : recdata2 : POSITION : 0 0 : VALUES : 97 <> 99 : PERCENT : 2.0202
DIFFER : VARIABLE : rec : POSITION : 1 : VALUES : 6 <> 6.1 : PERCENT : 1.63934

##############################################################
# Compare data written in different formats (classic vs. HDF).
nccmp -dmgfb old.nc new.nc 

INFO: Command-line options parsed.
INFO: Opening input files.
INFO: Creating variable comparison list.
INFO: Using all variables.
INFO: Variables to compare (11):
lon, lat, tile_index, frac, glac, lake, soil, vegn, lwup, e_res_1, e_res_2
INFO: Comparing file formats.
DIFFER : FILE FORMATS : NC_FORMAT_NETCDF4_CLASSIC <> NC_FORMAT_64BIT
INFO: Comparing global attributes.
INFO: Collecting dimension information for first file.
INFO: Collecting dimension information for second file.
INFO: Collecting variable information for first file.
INFO: Collecting variable information for second file.
INFO: Comparing record information.
INFO: Comparing metadata.
INFO: Comparing number of dimensions.
INFO: Getting record dimension names, if they exist.
INFO: Comparing dimension lengths.
INFO: Comparing variable datatypes and rank.
INFO: Comparing variables' dimension names.
INFO: Comparing variables' attributes.
INFO: Comparing data.
INFO: Comparing data for variable "lon".
INFO: Comparing non-record data for variable "lon".
INFO: Comparing data for variable "lat".
INFO: Comparing non-record data for variable "lat".
INFO: Comparing data for variable "tile_index".
INFO: Comparing non-record data for variable "tile_index".
INFO: Comparing data for variable "frac".
INFO: Comparing non-record data for variable "frac".
INFO: Comparing data for variable "glac".
INFO: Comparing non-record data for variable "glac".
INFO: Comparing data for variable "lake".
INFO: Comparing non-record data for variable "lake".
INFO: Comparing data for variable "soil".
INFO: Comparing non-record data for variable "soil".
INFO: Comparing data for variable "vegn".
INFO: Comparing non-record data for variable "vegn".
INFO: Comparing data for variable "lwup".
INFO: Comparing non-record data for variable "lwup".
INFO: Comparing data for variable "e_res_1".
INFO: Comparing non-record data for variable "e_res_1".
INFO: Comparing data for variable "e_res_2".
INFO: Comparing non-record data for variable "e_res_2".
INFO: Finished comparing data.
INFO: Comparisons complete. Freeing memory.

##############################################################
# Compare identical files.
nccmp -dmgfs old.nc new.nc 
Files "old.nc" and "new.nc" are identical.

##############################################################
# Get some info on file structure internals.
nccmp -i test_pad1a.nc 
path=test_pad1a.nc
format=NC_FORMAT_CLASSIC
header_size=804 bytes
header_pad_size=180 bytes
var=COADSX3_182 shuffle=0 deflate=0 deflate_level=0 chunked=0
var=COADSY36_55 shuffle=0 deflate=0 deflate_level=0 chunked=0
var=a shuffle=0 deflate=0 deflate_level=0 chunked=0
var=b shuffle=0 deflate=0 deflate_level=0 chunked=0


Usage

Compare two NetCDF files.
nccmp is silent when files are equivalent, otherwise it will echo to STDERR whether metadata or a specific variable differs.  By default, comparing stops after the first difference.

Exit code 0 is returned for identical files, 1 for different files, and 2 for a fatal error.

Usage: nccmp [OPTION...] file1 [file2]

  -A, --Attribute=att1[,...] Ignore attribute(s) att1[,...]
                             for all variables.
  -b, --verbose              Print messages to show progress.
  -c, --var-diff-count=CNT   Print messages up until CNT data differences
                             per variable.
  -C, --diff-count=CNT       Print messages up until CNT data differences total.
  -d, --data                 Compare data (variable values).
  -D, --debug                Prints verbose debug messages.
  -f, --force                Forcefully compare, do not stop after first
                             difference.
  -F, --fortran              Print position indices using Fortran style
                             (1-based reverse order).
  -g, --global               Compare global attributes.
                             (auto enables -m)
  -G, --globalex att1[,...]  Exclude global attributes att1[,...].
                             (auto enables -g)
  -h, --history              Compare global history attribute.
                             (auto enables -g)
  -H, --help                 Give this usage message.
      --usage                
  -i, --info                 Print file structure info for one or both files.
  -m, --metadata             Compare metadata.
                             Use -g to include global attributes.
  -M, --missing              Ignore difference between values that have
                             different missing_value and/or _FillValue.
                             Attribute differences are still reported.
  -n, --threads=N            Use N POSIX threads. Default is 1.
  -N, --nans-are-equal       Allow NaNs to be equal.
  -p, --precision='%.17g'    Precision of difference printing
                             (default is '%g').
                             Use '%x' to print bytes.
  -P, --header-pad           Compare header padding in classic or 64bit
                             file formats.
  -r, --group=group1[,...]   Compare group(s) group1[,...] only.
                             Group names can be full or short.
  -s, --report-identical-files
                             Report when files are the same.
  -t, --tolerance=TOL        Compare if absolute difference > TOL.
  -T, --Tolerance=TOL        Compare if relative percent difference > TOL.
  -v, --variable=var1[,...]  Compare variable(s) var1[,...] only.
  -V, --version              Print program version.
  -x, --exclude=var1[,...]   Exclude variable(s) var1[,...].
  -w, --warn=tag[,...]       Selectively make certain differences
                             exceptions, but still print messages.
                             Supported tags and their meanings:
                             all: All differences.
                             format: File formats.
                             eos: Extra attribute end-of-string nulls.

Report bugs to Remik . Ziemlinski @ noaa . gov.
nccmp 1.6.1
Built with NetCDF 4.3.3.1 of Jul  8 2015 09:51:52 $

Features Enabled
pthreads   = yes
header-pad = yes


Bugs

  1. Exit code isn't consistent for nccmpi.


Wishlist


References

  1. NetCDF Data Format


News

Version 1.6.1 (7/29/2015)

1. Added multithreading with posix threads 
   (-n, --threads=N).
2. Compare groups (and recursively) for all or by short or full name 
   (-r, --group=group1[,...]).
   Thanks to Sean W. Bailey (NASA) and Rick Healy (NASA) for the idea and contributing code.
3. Dump file traits like shuffle, chunk, deflate, header-pad for one or 
   both input files, with no need to do compare
   (-i, --info).
4. Limit comparisons and diff messages printed with global and/or per-variable count 
   (-c, --var-diff-count=CNT, -C, --diff-count=CNT).
   Thanks to Sean W. Bailey (NASA) and Rick Healy (NASA) for the idea and contributing code.
5. Optionally build with header-pad and thread support, and print features enabled
   with --version or --help.
   
Version 1.5.0 (11/30/2014)

1. Added new -N, --nans-are-equal option.
   By default, NaN does not equal anything including itself.
   Now you can override this definition when comparing floats or doubles
   that might have NaNs.
   Thanks to Bob Fischer for this request.
2. Added new -P, --header-pad option.
   Some tools may alter the header pad after filtering classic/64bit NetCDF3 files.
   These changes can now be detected.
   Thanks to Andrew Wittenberg.

Version 1.4.0 (5/30/2013)

1. Added new -M,--missing option.
   If both files have different missing_value and/or _FillValue,
   consider them the same.  Thanks to Dave Allured for the idea.

Version 1.3.0 (4/4/2013)

1. Added new warning flag (-w eos) to ignore extra end-of-string nulls in (global and variable) attribute strings (NC_CHAR type).
2. Added missing support for -D debug flag.

Version 1.2.0 (2/23/2013)

1. Added configure option --enable-ncinfo (default is yes). This will build and install the small helper tool. Use --enable-ncinfo=no to skip it.
2. Added new option (-s, --report-identical-files) to report when two files are the same, just like "diff". Will print a string like:
Files "foo.nc" and "bar.nc" are identical.
3. Created git repo.

Version 1.1.2 (2/10/2013)

1. Fixed building for macports (MacOSX 10.8, Xcode 4.6, gcc4.7.2). Thanks to Dave Allured.

Version 1.1.1 (9/24/2012)

1. Bugfix to handle HDF group dimensions with dimids that don't start at zero. Thanks to Lynton Appel.
2. Removed default build and install of 'ncinfo' because not core to nccmp mission. If you still need it, cd into the ncinfo subdir and do make there.

Version 1.1.0 (7/15/2010)

1. Added "-w, --warn" command-line option to make certain differences exceptions and only print messages of their occurance.  Thanks to Aleksey Yakovlev.
2. Added a small helper program called 'ncinfo' that echoes a file's format, header size (NetCDF3 files only) and compression characteristics for NetCDF4 files.  

Version 1.0.4 (6/27/2010)

1. Bug fix when comparing attributes when a variable was missing in a file. Thanks to Lionel Guez.
2. Pretty print attribute values when they differ.
3. Added simple utility "ncinfo" in /ncinfo to echo file's format and chunk/compression/header characteristics. Thanks to Aleksey Yakovlev.

Version 1.0.3 (4/13/2010)

1. Added hex printing.
2. Added custom precision of float/single value printing.
3. Added fortran-style index printing.

Version 1.0.2 (3/20/2010)

1. Fixed unsafe record queries.
2. Added more information "verbose" progress messages.
3. Made "quiet" mode from pre-1.0 deprecated by making option hidden although still accepted from command-line.

Version 1.0.1 (3/10/2010)

1. Fixed record-based scalar variable comparisons.

Version 1.0.0 (1/20/2010)

1. Overhauled to work generically with Netcdf4 at the cost of lower performance.

Version 0.6.0

1. Added new -G command-line option to ignore specific global attributes.  Thanks to V. Balaji.

Version 0.5.4

1. Avoided reading past EOF when number of records differ between files.  Record variables won't be compared as a result.  
Users should use another tool, like "ncks" to extract record-based hyperslabs if they really need to compare partial netcdf files.  Thanks to Lionel Guez.
2. Handles case when both files have record dimensions with length zero. Thanks to Lionel Guez.

Version 0.5.3

1. Fixed memory thrashing in string management code.
2. Made --enable-mpi configure argument and padded header testing more robust.

Version 0.5.2

1. Added command-line option to ignore specific variable attributes.

Version 0.5.1

1. Fixed bug. Single record variables were skipped.

Version 0.5.0

Thanks to Amy Langenhorst for suggestion 1.

1. Compare with absolute or relative percentage tolerance. Obviously slower due to arithmetic. No MPI support for tolerance feature.

Version 0.4.2

Thanks to Amy Langenhorst for these suggestions.

* forces removal of temporary files in generated view script.
* generated view script tests if there exists a user defined NCCMPDIFF,
  otherwise sets it to 'diff' as default.
* suppresses global history attribute when extracting data in 
  generated view script.

Version 0.4.1

* bug fix: prevents premature file closing while still querying for
info such as when generating example view script.

Version 0.4

* builds nccmpi for usage with MPI enabled systems

Version 0.3

* ported completely to C; was a perl script originally

Version 0.2.1

* bug fix: check for comparisons beyond EOF
* bug fix: memory allocation for variable list

Version 0.2

* -q quiet mode, suppresses all but difference message
* -f force mode, process all variables possible; doesn't exit program
  after 1st difference
* -x exclude variable list option
* enhanced view script 

Version 0.1

* outputs script to do visual comparison of different values


Copyright (C) 2004-2014 Remik Ziemlinski
Licensed under the terms of GPL.