SourceForge.net Logo

nccmp

  1. Overview
  2. Download
  3. Installation
  4. Examples
  5. Usage
  6. Bugs
  7. Wishlist
  8. References
  9. News


Overview

nccmp compares two NetCDF files bitwise or with a user defined tolerance (absolute or relative percentage). Some logic is used so it isn't blind like regular cmp. Comparisons are done in local memory without requiring temporary files. Highly recommended for regression testing scientific models.

Prior to versions 1.0+, nccmp was implemented with MPI to allow for parallel comparisons. If you'd like parallel comparisons to return with MPI or OpenMP, please let the author know.

Features


Download

Source Code: nccmp-1.3.0.tgz


Installation

  1. Download nccmp.
  2. Decompress with
    tar -zxvf nccmp-*gz
    
  3. ./configure
  4. make
  5. make check
  6. make install
  7. make clean


Examples

#############################################################
# Compare metadata for non-record variables.
nccmp  -mqf -x recdata,rec test1a.nc test1b.nc

DIFFER : NUMBER OF DIMENSIONS IN FILES : 5 <> 4
DIFFER : LENGTHS : DIMENSION : dim3 : 2 <> 1
DIFFER : NAME : DIMENSION : dim4 : DIMENSION DOESN'T EXIST IN "test1b.nc"
DIFFER : NAME : VARIABLE : data3 : VARIABLE DOESN'T EXIST IN test1b.nc
DIFFER : DIMENSION NAMES FOR VARIABLE data4 : dim4 <> dim2
DIFFER : VARIABLE "data1" IS MISSING ATTRIBUTE WITH NAME "units" IN FILE test1b.nc
DIFFER : LENGTHS : ATTRIBUTE : long_name : VARIABLE : data1 : 9 <> 10
DIFFER : VALUES : ATTRIBUTE : valid_range : VARIABLE : data1
DIFFER : NUMBER OF ATTRIBUTES : VARIABLE : data1 : 3 <> 2

#############################################################
# Compare data using a tolerance threshold.
nccmp -df -t 1e-3 test4a.nc test4b.nc

DIFFER : VARIABLE : data1 : POSITION : 0 1 : VALUES : 2 <> 2.01 : PERCENT : 0.497512
DIFFER : VARIABLE : data2 : POSITION : 0 1 : VALUES : 8 <> 7.99 : PERCENT : 0.125003
DIFFER : VARIABLE : data3 : POSITION : 0 : VALUES : 9 <> 8 : PERCENT : 11.1111
DIFFER : VARIABLE : data3 : POSITION : 1 : VALUES : 10 <> -10 : PERCENT : 200
DIFFER : VARIABLE : data4 : POSITION : 0 : VALUES : 11 <> 110 : PERCENT : 90
DIFFER : VARIABLE : recdata1 : POSITION : 0 1 : VALUES : 4 <> 0 : PERCENT : 100
DIFFER : VARIABLE : recdata2 : POSITION : 0 0 : VALUES : 97 <> 99 : PERCENT : 2.0202
DIFFER : VARIABLE : rec : POSITION : 1 : VALUES : 6 <> 6.1 : PERCENT : 1.63934

##############################################################
# Compare data written in different formats (classic vs. HDF).
nccmp -dmgfb old.nc new.nc 

INFO: Command-line options parsed.
INFO: Opening input files.
INFO: Creating variable comparison list.
INFO: Using all variables.
INFO: Variables to compare (11):
lon, lat, tile_index, frac, glac, lake, soil, vegn, lwup, e_res_1, e_res_2
INFO: Comparing file formats.
DIFFER : FILE FORMATS : NC_FORMAT_NETCDF4_CLASSIC <> NC_FORMAT_64BIT
INFO: Comparing global attributes.
INFO: Collecting dimension information for first file.
INFO: Collecting dimension information for second file.
INFO: Collecting variable information for first file.
INFO: Collecting variable information for second file.
INFO: Comparing record information.
INFO: Comparing metadata.
INFO: Comparing number of dimensions.
INFO: Getting record dimension names, if they exist.
INFO: Comparing dimension lengths.
INFO: Comparing variable datatypes and rank.
INFO: Comparing variables' dimension names.
INFO: Comparing variables' attributes.
INFO: Comparing data.
INFO: Comparing data for variable "lon".
INFO: Comparing non-record data for variable "lon".
INFO: Comparing data for variable "lat".
INFO: Comparing non-record data for variable "lat".
INFO: Comparing data for variable "tile_index".
INFO: Comparing non-record data for variable "tile_index".
INFO: Comparing data for variable "frac".
INFO: Comparing non-record data for variable "frac".
INFO: Comparing data for variable "glac".
INFO: Comparing non-record data for variable "glac".
INFO: Comparing data for variable "lake".
INFO: Comparing non-record data for variable "lake".
INFO: Comparing data for variable "soil".
INFO: Comparing non-record data for variable "soil".
INFO: Comparing data for variable "vegn".
INFO: Comparing non-record data for variable "vegn".
INFO: Comparing data for variable "lwup".
INFO: Comparing non-record data for variable "lwup".
INFO: Comparing data for variable "e_res_1".
INFO: Comparing non-record data for variable "e_res_1".
INFO: Comparing data for variable "e_res_2".
INFO: Comparing non-record data for variable "e_res_2".
INFO: Finished comparing data.
INFO: Comparisons complete. Freeing memory.

##############################################################
# Compare identical files.
nccmp -dmgfs old.nc new.nc 
Files "old.nc" and "new.nc" are identical.


Usage

Compare two NetCDF files.
nccmp is silent when files are equivalent, otherwise it will echo 
to STDERR whether metadata or a specific variable differs.  By
default, comparing stops after the first difference.

Exit code 0 is returned for identical files, 1 for different files,
and 2 for a fatal error.

Usage: nccmp [OPTION...] file1 file2

  -A, --Attribute=att1[,...] Ignore attribute(s) att1[,...]
                             for all variables.
  -b, --verbose              Print messages to show progress.
  -d, --data                 Compare data (variable values).
  -D, --debug                Prints verbose debug messages.
  -f, --force                Forcefully compare, do not stop after first difference.
  -F, --fortran              Print position indices using Fortran style (1-based reverse order).
  -g, --global               Compare global attributes.
                             (-m required)
  -G, --globalex att1[,...]  Exclude global attributes att1[,...].
                             (-mg required)
  -h, --history              Compare global history attribute.
                             (-mg required)
  -H, --help                 Give this usage message.
      --usage                
  -m, --metadata             Compare metadata, excluding global attributes.
  -M, --missing              Ignore difference between values that have
                             different missing_value and/or _FillValue.
                             Attribute differences are still reported.
  -p, --precision='%.17g'    Precision of difference printing (default is '%g').
                             Use '%x' to print bytes.
  -s, --report-identical-files
                             Report when files are the same.
  -t, --tolerance=TOL        Compare if absolute difference > TOL.
  -T, --Tolerance=TOL        Compare if relative percent difference > TOL.
  -v, --variable=var1[,...]  Compare variable(s) var1[,...] only.
  -V, --version              Print program version.
  -x, --exclude=var1[,...]   Exclude variable(s) var1[,...].
  -w, --warn=tag[,...]       Selectively make certain differences
                             exceptions, but still print messages.
                             Supported tags and their meanings:
                             all: All differences.
                             format: File formats.
                             eos: Extra attribute end-of-string nulls.


Bugs

  1. Exit code isn't consistent for nccmpi.


Wishlist


References

  1. NetCDF Data Format


News

Version 1.4.0 (5/30/2013)

1. Added new -M,--missing option.
   If both files have different missing_value and/or _FillValue,
   consider them the same.  Thanks to Dave Allured for the idea.

Version 1.3.0 (4/4/2013)

1. Added new warning flag (-w eos) to ignore extra end-of-string nulls in (global and variable) attribute strings (NC_CHAR type).
2. Added missing support for -D debug flag.

Version 1.2.0 (2/23/2013)

1. Added configure option --enable-ncinfo (default is yes). This will build and install the small helper tool. Use --enable-ncinfo=no to skip it.
2. Added new option (-s, --report-identical-files) to report when two files are the same, just like "diff". Will print a string like:
Files "foo.nc" and "bar.nc" are identical.
3. Created git repo.

Version 1.1.2 (2/10/2013)

1. Fixed building for macports (MacOSX 10.8, Xcode 4.6, gcc4.7.2). Thanks to Dave Allured.

Version 1.1.1 (9/24/2012)

1. Bugfix to handle HDF group dimensions with dimids that don't start at zero. Thanks to Lynton Appel.
2. Removed default build and install of 'ncinfo' because not core to nccmp mission. If you still need it, cd into the ncinfo subdir and do make there.

Version 1.1.0 (7/15/2010)

1. Added "-w, --warn" command-line option to make certain differences exceptions and only print messages of their occurance.  Thanks to Aleksey Yakovlev.
2. Added a small helper program called 'ncinfo' that echoes a file's format, header size (NetCDF3 files only) and compression characteristics for NetCDF4 files.  

Version 1.0.4 (6/27/2010)

1. Bug fix when comparing attributes when a variable was missing in a file. Thanks to Lionel Guez.
2. Pretty print attribute values when they differ.
3. Added simple utility "ncinfo" in /ncinfo to echo file's format and chunk/compression/header characteristics. Thanks to Aleksey Yakovlev.

Version 1.0.3 (4/13/2010)

1. Added hex printing.
2. Added custom precision of float/single value printing.
3. Added fortran-style index printing.

Version 1.0.2 (3/20/2010)

1. Fixed unsafe record queries.
2. Added more information "verbose" progress messages.
3. Made "quiet" mode from pre-1.0 deprecated by making option hidden although still accepted from command-line.

Version 1.0.1 (3/10/2010)

1. Fixed record-based scalar variable comparisons.

Version 1.0.0 (1/20/2010)

1. Overhauled to work generically with Netcdf4 at the cost of lower performance.

Version 0.6.0

1. Added new -G command-line option to ignore specific global attributes.  Thanks to V. Balaji.

Version 0.5.4

1. Avoided reading past EOF when number of records differ between files.  Record variables won't be compared as a result.  
Users should use another tool, like "ncks" to extract record-based hyperslabs if they really need to compare partial netcdf files.  Thanks to Lionel Guez.
2. Handles case when both files have record dimensions with length zero. Thanks to Lionel Guez.

Version 0.5.3

1. Fixed memory thrashing in string management code.
2. Made --enable-mpi configure argument and padded header testing more robust.

Version 0.5.2

1. Added command-line option to ignore specific variable attributes.

Version 0.5.1

1. Fixed bug. Single record variables were skipped.

Version 0.5.0

Thanks to Amy Langenhorst for suggestion 1.

1. Compare with absolute or relative percentage tolerance. Obviously slower due to arithmetic. No MPI support for tolerance feature.

Version 0.4.2

Thanks to Amy Langenhorst for these suggestions.

* forces removal of temporary files in generated view script.
* generated view script tests if there exists a user defined NCCMPDIFF,
  otherwise sets it to 'diff' as default.
* suppresses global history attribute when extracting data in 
  generated view script.

Version 0.4.1

* bug fix: prevents premature file closing while still querying for
info such as when generating example view script.

Version 0.4

* builds nccmpi for usage with MPI enabled systems

Version 0.3

* ported completely to C; was a perl script originally

Version 0.2.1

* bug fix: check for comparisons beyond EOF
* bug fix: memory allocation for variable list

Version 0.2

* -q quiet mode, suppresses all but difference message
* -f force mode, process all variables possible; doesn't exit program
  after 1st difference
* -x exclude variable list option
* enhanced view script 

Version 0.1

* outputs script to do visual comparison of different values


Copyright (C) 2004-2013 Remik Ziemlinski
Licensed under the terms of GPL.