README-tools

============================
README file for GASNet tools
============================
GASNet tools specification version: 1.21 // work-in-progress, not "closed"
Authors: Dan Bonachea <dobonachea@lbl.gov>
         Paul H. Hargrove <PHHargrove@lbl.gov>

The GASNet tools are a set of communication-independent utilities that are used
to implement GASNet, and constitute a useful portability tool for GASNet
clients and even for other portable software that might not require the GASNet
communication services. The GASNet tools are available to all regular GASNet
clients, and are also available in a stripped-down "tools-only" software
distribution which is intended for bundling with third-party software that does
not require communication services. This file documents both.

============================================
Tools-Only Distribution Install Instructions
============================================

The GASNet_Tools distribution contains just the sources required to build and
use the GASNet tools, without any GASNet conduit support. It can only be used
in executables which do not link a GASNet conduit (which already provide the
tools).

* Step 1: Configure

  Unpack the distribution and run:

    configure (options)
    
  A few of the more important options available:

  --help :  display all available configure options
  --prefix=/install/path : set the directory where GASNet Tools will be installed
  --enable-debug : Enables hundreds of system-wide sanity checks, at a cost in performance.
                   Highly recommended during software development.
  --disable-pthreads: Can be used to disable the thread-safe version of the tools.

* Step 2: Build
  
  Use make to build the tools library:
    
    make
    make check   (optional, but recommended - builds and runs a correctness test)

* Step 3: Install

  Use make to install the tools library:

   make install

* Step 4: Use the library

  Once installed, client code should #include <gasnet_tools.h> from
  $prefix/include and link the appropriate library in $prefix/lib.
  Clients which use multiple pthreads should link the thread-safe library, 
  and define GASNETT_THREAD_SAFE before including gasnet_tools.h, eg:
  
   cc -o myprogram myprogram.c -I$prefix/include -DGASNETT_THREAD_SAFE=1 \
        -L$prefix/lib -lgasnet_tools-par -lpthreads

  Where $prefix is the prefix passed during Step 1. 
  
  Clients which never use pthreads may link the single-threaded version of the
  tools using -lgasnet_tools-seq and the GASNETT_THREAD_SINGLE define.

  Additionally, client code used to build shared libraries (compiled with
  -fPIC, -KPIC or a similar compiler-dependent flag) should pass
  -DGASNETI_FORCE_PIC=1 to ensure use of PIC-safe code in gasnet_tools.h.

  On a few platforms, additional system libraries or compiler flags may be required
  for gasnet_tools to work correctly. Clients seeking maximum portability 
  are recommended to get their compiler and linker flags by including the 
  generated Makefile fragments $prefix/include/gasnet_tools-{seq,par}.mak
  in their Makefile and using the provided make variables they provide.
  See the comments at the top of those files for exact usage documentation.
  Alternatively, pkg-config files for the gasnet_tools-{seq,par} packages
  are installed in $prefix/lib/pkgconfig and provide the same build information.
  See README for pkg-config usage instructions.

===============================
GASNet Tools User Documentation
===============================

The remainder of this file documents the usage of GASNet tools, regardless of
which distribution is in use.

-------------
General Usage
-------------

* All clients of GASNet tools should #include <gasnet_tools.h> before any other header.
  The only exception is source files that use both a GASNet conduit and GASNet
  tools, which must include gasnetex.h before gasnet_tools.h.

* All of the supported public interfaces in GASNet tools are named using the
  'gasnett_' or 'GASNETT_' prefix. Clients of GASNet tools should *ONLY* invoke
  names with this prefix. Use of names with any other prefix (notably including
  'gasneti_') is totally unsupported and subject to change and breakage without
  notice.

* Many of the 'functions' provided by GASNet tools are actually implemented as 
  macros or inline functions for performance reasons. This distinction is explicitly
  undocumented and open to change without notice, and may even differ across 
  platforms in a given release. To ensure correctness, clients should never
  attempt to take the address of a GASNet tool 'function' or #undef its name.

* The GASNet tools have been ported to all the platforms listed in the main
  GASNet README. They may work on others as well. Please contact us if you have
  a new platform you'd like to see supported.

* For questions on using the GASNet tools, contact gasnet-users@lbl.gov. 
  It's especially recommended to contact us before bundling the tools in your
  software package.

------
Timers
------

GASNet tools provides high-granularity, low-overhead wall-clock timers using 
system-specific support, where available.

  gasnett_tick_t - timer datatype representing an integer number of "ticks"
    where a "tick" has a system-specific interpretation
    safe to be handled using integer operations (+,-,<,>,==)

  gasnett_tick_t gasnett_ticks_now() - returns the current tick count 
    note that tick values are THREAD-specific, and do NOT represent a globally-synchronized timer.
    In specific, tick values are very likely to have a different base value across nodes, and 
    might even advance at substantially different rates on different nodes.
    Therefore tick values and tick intervals from different threads should never be directly compared or 
    arithmetically combined, without first converting the relevant tick intervals to wall time intervals.

  uint64_t gasnett_ticks_to_ns(gasnett_tick_t ticks) - convert ticks to nanoseconds as a uint64_t

  GASNETT_TICK_MIN - a value representing the minimum value storable in a gasnett_tick_t

  GASNETT_TICK_MAX - a value representing the maximum value storable in a gasnett_tick_t

  Environment:
    For Linux on x86, x86-64 or MIC processors, the default timer is the TSC
    which requires choosing a mechanism for calibration.  This can be controlled
    via environment variables:

    * GASNET_TSC_RATE
      GASNET_TSC_RATE=walltime
        Measures the TSC tick rate against the OS-provided wallclock time.
        This is the default.
      GASNET_TSC_RATE=cpuinfo
        Obtains the TSC tick rate from information in /proc/cpuinfo
        This is known to be incorrect for certain recent CPU models.
      GASNET_TSC_RATE=[Hz]
        If given an integer in the range 1 Million to 100 Billion, this will be
        taken as the TSC tick rate in Hz (cycles per second).  To avoid the
        ambiguity between binary (M=2^20) and decimal (M=10^6), no suffixes are
        accepted.

    * GASNET_TSC_RATE_TOLERANCE
      This is a floating-point value (read by gasnett_getenv_dbl_withdefault())
      which indicates the relative error permitted in the calibration of the
      TSC.  For instance the value 0.001 would permit a relative error as
      large as 1 part in 1000, or 0.1%.  Exceeding this level of permitted
      relative error will result in a warning message.

    * GASNET_TSC_RATE_HARD_TOLERANCE
      This environment variable functions like GASNET_TSC_RATE_TOLERANCE, except
      that exceeding this value results in a fatal error.

    * GASNET_TSC_VERBOSE
      This boolean setting requests console output regarding calibration activity

-----
Sleep
-----

  int gasnett_nsleep(uint64_t ns_delay) - nanosecond resolution sleep
    Sleep for at least the indicated number of nanoseconds.  If interrupted by a
    signal, may terminate the sleep early returning non-zero with errno = EINTR.
    If ns_delay is zero, this function returns without sleeping.

---------------
Memory barriers
---------------

Memory barriers are used to implement lock-free synchronization and data sharing across 
the threads of a process.

 gasnett_local_wmb:
   A local memory write barrier - ensure all stores to local mem from this thread are
   globally completed across this SMP before issuing any subsequent loads or stores.
   (i.e. all loads issued from any CPU subsequent to this call
      returning will see the new value for any previously issued
      stores from this proc, and any subsequent stores from this CPU
      are guaranteed to become globally visible after all previously issued
      stores from this CPU)
   This must also include whatever is needed to prevent the compiler from reordering
   loads and stores across this point.

 gasnett_local_rmb:
   A local memory read barrier - ensure all subsequent loads from local mem from this thread
   will observe previously issued stores from any CPU which have globally completed.  
   For instance, on the Alpha this ensures
   that queued cache invalidations are processed and on the PPC this discards any loads
   that were executed speculatively.
   This must also include whatever is needed to prevent the compiler from reordering
   loads and stores across this point.
 
 gasnett_local_mb:
   A "full" local memory barrer.  This is equivalent to both a wmb() and rmb().
   All outstanding loads and stores must be completed before any subsequent ones
   may begin.

 gasnett_weak_wmb:
 gasnett_weak_rmb:
 gasnett_weak_mb:
   These are equivalent to the corresponding gasnett_local_* except that in a build
   without threads these compile away to nothing.
  
 gasnett_compiler_fence:
   A barrier to compiler optimizations that would reorder any memory references across
   this point in the code.

  Note that for all of the memory barriers, we require only that a given architecture's
  "normal" loads and stores are ordered as required.  "Extended" instructions such as
  MMX, SSE, SSE2, Altivec and vector ISAs on various other machines often bypass some
  or all of the machine's memory hierarchy and therefore may not be ordered by the same
  instructions.  Authors of MMX-based memcpy and similar code must therefore take care
  to add appropriate flushes to their code.

  For more info on memory barriers: http://gee.cs.oswego.edu/dl/jmm/cookbook.html

-----------------
Atomic operations
-----------------

GASNet tools provides portable atomic memory operations for efficient inter-thread coordination.

Note the default atomic operations exposed by GASNet tools only expand to architecturally 
atomic instructions in GASNETT_THREAD_SAFE mode. In single-threaded mode, they all expand to
appropriate regular (non-atomic) operations, which are often more efficient
than their atomic equivalents and should be indistinguishable in behavior for
programs with no concurrency. 

The default atomics exposed by GASNet tools are *not* guaranteed to be atomic with respect 
to signal handlers, and therefore should not be used for synchronizing with signal handlers.
If you need signal-safe atomics or atomic memory access in single-threaded codes, see
the section on strong atomics below.

 * gasnett_atomic_t

 This interface provides a special datatype (gasnett_atomic_t) representing an atomically
  updated unsigned integer value and a set of atomic ops
 Atomicity is guaranteed only if ALL accesses to the gasnett_atomic_t data happen
  through the provided operations (i.e. it is an error to directly access the
  contents of a gasnett_atomic_t), and if the gasnett_atomic_t data is only
  addressable by the current process (e.g. not in a System V shared memory segment)
 It is also an error to access an uninitialized gasnett_atomic_t with any operation
  other than gasnett_atomic_set().
 We define an unsigned type (gasnett_atomic_val_t) and a signed type
 (gasnett_atomic_sval_t) and provide the following operations on all platforms:

  gasnett_atomic_init(gasnett_atomic_val_t v)
      Static initializer (macro) for an gasnett_atomic_t to value v.

  void gasnett_atomic_set(gasnett_atomic_t *p,
                          gasnett_atomic_val_t v,
                          int flags);
      Atomically sets *p to value v.

  gasnett_atomic_val_t gasnett_atomic_read(gasnett_atomic_t *p, int flags);
      Atomically read and return the value of *p.

  void gasnett_atomic_increment(gasnett_atomic_t *p, int flags);
      Atomically increment *p (no return value).

  void gasnett_atomic_decrement(gasnett_atomic_t *p, int flags);
      Atomically decrement *p (no return value).

  int gasnett_atomic_decrement_and_test(gasnett_atomic_t *p, int flags);
      Atomically decrement *p, return non-zero iff the new value is 0.

 * Semi-portable atomic operations

 The following two groups of useful atomic operations are available on most
 platforms, but not all.  Preprocessor definitions indicate what is available.

 + Group 1: add and subtract

  gasnett_atomic_val_t gasnett_atomic_add(gasnett_atomic_t *p,
                                          gasnett_atomic_val_t op,
                                          int flags);
  gasnett_atomic_val_t gasnett_atomic_subtract(gasnett_atomic_t *p,
                                               gasnett_atomic_val_t op,
                                               int flags);

 These implement atomic (unsigned) addition and subtraction.
 If the result would lie outside the range of gasnett_atomic_val_t,
 then the excess high-order bits of the exact result are truncated.
 Both return the value after the addition or subtraction.

 GASNETT_HAVE_ATOMIC_ADD_SUB will be defined to 1 when these operations are available.
 They are always either both available, or neither is available.

 + Group 2: conditional and unconditional swap

  int gasnett_atomic_compare_and_swap(gasnett_atomic_t *p,
                                      gasnett_atomic_val_t oldval,
                                      gasnett_atomic_val_t newval,
                                      int flags);

   This operation is the atomic equivalent of:
    if (*p == oldval) {
      *p = newval;
      return NONZERO;
    } else {
      return 0;
    }

  gasnett_atomic_val_t gasnett_atomic_swap(gasnett_atomic_t *p,
                                           gasnett_atomic_val_t newval,
                                           int flags);

   This operation is the atomic equivalent of:
    gasnett_atomic_val_t oldval = *p;
    *p = newval;
    return oldval;

 GASNETT_HAVE_ATOMIC_CAS will be defined to 1 when these operations are available.
 They are always either both available, or neither is available.

 * Range of atomic type

 Internally a gasnett_atomic_t is an unsigned type of at least 24-bits.  No special
 action is needed to store signed values via gasnett_atomic_set(), however because
 the type may use less than a full word, gasnett_atomic_signed() is provided to
 perform any required sign extension if a value read from a gasnett_atomic_t is
 to be used as a signed type.

  gasnett_atomic_signed(v)      Converts a gasnett_atomic_val_t returned by
                                gasnett_atomic_{read,add,subtract} to a signed
                                gasnett_atomic_sval_t.
  GASNETT_ATOMIC_MAX            The largest representable unsigned value
                                (the smallest representable unsigned value is always 0).
  GASNETT_ATOMIC_SIGNED_MIN     The smallest (most negative) representable signed value.
  GASNETT_ATOMIC_SIGNED_MAX     The largest (most positive) representable signed value.

 The atomic type is guaranteed to wrap around at its minimum and maximum values in
 the normal manner expected of two's-complement integers.  This includes the 'oldval'
 and 'newval' arguments to gasnett_atomic_compare_and_swap(), and the 'v' arguments
 to gasnett_atomic_init() and gasnett_atomic_set() which are wrapped (not clipped)
 to the proper range prior to assignment (for 'newval' and 'v') or comparison (for
 'oldval').

 * Memory fence properties of atomic operations

 NOTE: Atomic operations have no default memory fence properties, as this
 varies by platform.  Every atomic operation except _init() includes a 'flags'
 argument to indicate the caller's minimum fence requirements.
 Depending on the platform, the implementation may use fences stronger than
 those requested, but never weaker. 

 Most cases where atomics are used to implement thread synchronization (eg where 
 the atomic operation indicates the availability or consumption of other data)
 will need to include some fences to ensure consistency of other data (this includes
 both non-atomic data, and other atomic variables).
 Specifying the necessary fence properties
 as arguments to the atomic operation helps to reduce duplication of fences on some 
 platforms (relative to issuing explicit fences before/after the atomic op), because it
 allows the data fence to be combined with whatever fences are used to implement the 
 atomic operation.

 The following fence flags are recognized and may be OR'd together for the flags argument of any
 atomic operation:

  GASNETT_ATOMIC_NONE  - no fence (equivalent to passing 0)
		  
  GASNETT_ATOMIC_RMB_PRE - enforce a read/write/full fence before the atomic operation	
  GASNETT_ATOMIC_WMB_PRE
  GASNETT_ATOMIC_MB_PRE
	
  GASNETT_ATOMIC_RMB_POST - enforce a read/write/full fence after the atomic operation	
  GASNETT_ATOMIC_WMB_POST
  GASNETT_ATOMIC_MB_POST
	
  GASNETT_ATOMIC_RMB_POST_IF_TRUE 
  GASNETT_ATOMIC_RMB_POST_IF_FALSE
    - These enforce a read fence after a boolean atomic operation that succeeds (true) or
      fails (false). 
    - The boolean atomic operations are compare-and-swap and decrement-and-test.

 Convenience names for specifying acquire/release semantics in critical sections built from atomics:
  
  GASNETT_ATOMIC_REL		equivalent to: GASNETT_ATOMIC_WMB_PRE
  GASNETT_ATOMIC_ACQ		equivalent to: GASNETT_ATOMIC_RMB_POST
  GASNETT_ATOMIC_ACQ_IF_TRUE	equivalent to: GASNETT_ATOMIC_RMB_POST_IF_TRUE
  GASNETT_ATOMIC_ACQ_IF_FALSE	equivalent to: GASNETT_ATOMIC_RMB_POST_IF_FALSE

 * Storage of atomic type

 Internally an atomic type may use storage significantly larger than the number
 of significant bits.  This additional space may be needed, for instance, to
 meet platform-specific alignment constraints, or to hold a mutex on platforms
 lacking any other means of ensuring atomicity.

 * Fixed-width atomic types

 The following fixed-width (32- and 64-bit) types/operations are available
 on all platforms.  These are guaranteed to consume exactly the "natural"
 storage, without padding or any extra alignment.  However, one or both may
 use mutexes or lack signal-safety, even where gasnett_atomic_t does not.
 Additionally, unlike gasnett_atomic_t, the same set of operations is present
 on all platforms, even if that requires a mutex-based approach to support
 the full range of operations.  Therefore, there are no GASNETT_HAVE_ defines
 for the fixed-width atomic operations.

  gasnett_atomic32_t
  gasnett_atomic64_t
    Typedef

  gasnett_atomic32_init(uint32_t v)
  gasnett_atomic64_init(uint64_t v)
    Static initializer (macro).

  void gasnett_atomic32_set(gasnett_atomic32_t *p, uint32_t v, int flags);
  void gasnett_atomic64_set(gasnett_atomic64_t *p, uint64_t v, int flags);
    Atomically set *p to value v.

  uint32_t gasnett_atomic32_read(gasnett_atomic32_t *p, int flags);
  uint64_t gasnett_atomic64_read(gasnett_atomic64_t *p, int flags);
    Atomically read and return the value of *p.

  int gasnett_atomic32_compare_and_swap(gasnett_atomic32_t *p, uint32_t oldval,
                                        uint32_t newval, int flags);
  int gasnett_atomic64_compare_and_swap(gasnett_atomic64_t *p, uint64_t oldval,
                                        uint64_t newval, int flags);
    Atomic compare-and-swap of *p from oldval to newval.

  uint32_t gasnett_atomic32_swap(gasnett_atomic32_t *p, uint32_t v, int flags);
  uint64_t gasnett_atomic64_swap(gasnett_atomic64_t *p, uint64_t v, int flags);
    Atomically set *p to value v, returning the previous value.

  uint32_t gasnett_atomic32_add(gasnett_atomic32_t *p, uint32_t v, int flags);
  uint64_t gasnett_atomic64_add(gasnett_atomic64_t *p, uint64_t v, int flags);
    Atomically add value v to *p, returning the new value.

  uint32_t gasnett_atomic32_subtract(gasnett_atomic32_t *p, uint32_t v,
                                     int flags);
  uint64_t gasnett_atomic64_subtract(gasnett_atomic64_t *p, uint64_t v,
                                     int flags);
    Atomically subtract value v from *p, returning the new value.

  void gasnett_atomic32_increment(gasnett_atomic32_t *p, int flags);
  void gasnett_atomic64_increment(gasnett_atomic64_t *p, int flags);
    Atomically add 1 to *p.

  void gasnett_atomic32_decrement(gasnett_atomic32_t *p, int flags);
  void gasnett_atomic64_decrement(gasnett_atomic64_t *p, int flags);
    Atomically subtract 1 from *p.

  int gasnett_atomic32_decrement_and_test(gasnett_atomic32_t *p, int flags);
  int gasnett_atomic64_decrement_and_test(gasnett_atomic64_t *p, int flags);
    Atomically subtract 1 from *p, returning non-zero if *p becomes zero.

  While some platforms do not enforce the same alignment constraints for all
  types of a given width, the implementation of the fixed-width atomics
  guarantee correct atomic operations on storage declared as any of the 4-byte
  and 8-byte integer or floating point scalar types on a given platform.  So,
  assuming 4-byte float and 8-byte double, fixed-width atomic operations via
  pointers generated by the following casts are correct:
    (int32_t *) or (float *) cast to (gasnett_atomic32_t *)
    (int64_t *) or (double *) cast to (gasnett_atomic64_t *)
  where any signed or unsigned integral type of the same width may be used in
  place of int32_t and int64_t.  However, the fixed-width atomic operations do
  NOT guarantee correct operation on arbitrarily aligned blocks of data.  For
  instance the following two examples are NOT permitted
  EX1:
    struct { int16_t a, b; } X;
    gasnett_atomic32_set((gasnett_atomic32_t *)&X, 0, 0);
  EX2: 
    struct { float Real, Img; } Y;
    gasnett_atomic64_set((gasnett_atomic64_t *)&Y, 0, 0);
  because some platforms might align these structures less strictly than the
  integral and floating point types of equal size.  However, since in C
  unions are always aligned by their most-restrictive constituent type,
  the following two examples ARE legal:
  EX3:
    union { float f;
            struct { int16_t a, b; } u16s;
          } X2;
    gasnett_atomic32_set((gasnett_atomic32_t *)&X2, 0, 0);
  EX4:
    union { uint64_t u64;
            struct { float Real, Img; } cplx32;
          } Y2;
    gasnett_atomic64_set((gasnett_atomic64_t *)&Y2, 0, 0);

  Additionally, casts from (gasnett_atomic32_t *) or (gasnett_atomic64_t *) to
  pointers to other types are NOT safe in general, because the alignment of
  the atomic type might be less than required for the other type.  When this
  under-alignment occurs such casts could result in a fatal SIGBUS when the
  pointer is dereferenced.  To avoid this problem apply this rule-of-thumb:
    Storage to be accessed via both a pointer to a fixed-width atomic
    type and another pointer type must be declared as the non-atomic type,
    or as a union of both types.
  This will ensure the storage is suitably aligned for accesses via pointers
  to both the atomic and non-atomic types (assuming, of course, that the
  non-atomic type is one allowed by the previous paragraph.) Note that as a
  general rule, use of union types may be preferable because they avoid
  running afoul of pointer-aliasing rules in C/C++ that might otherwise
  lead to incorrect behavior at high optimization levels.

  It is not safe to concurrently access the same memory location as both
  an atomic type and a non-atomic type.  For the purpose of this distinction
  only references using the gasnett_atomic32_ or gasnett_atomic64_ prefixes
  are atomic.  All non-GASNet references and any other GASNet references are
  non-atomic (including all gets and puts, Active Message calls, etc.).

  The client code is responsible for providing sufficient synchronization
  (such as barriers or mutexes) to prevent the concurrent use of any given
  memory location as both atomic and non-atomic.  Use of non-atomic "flag"
  variables is not sufficient synchronization (even when volatile) in the
  presence of certain compiler optimizations.  Additionally, use of an
  atomic variable as a "flag" is only sufficient when memory fences are
  used correctly.  When practical, one possible mechanism to have the
  client code separate the atomic and non-atomic treatment of memory into
  distinct phases of the computation, separated by a barrier.
   
 * Strong atomics

  GASNet tools offers a "strong" atomics interface, which expands to the strongest available
  atomic operations on a given platform, even in single threaded-codes. The syntax and semantics
  for these operations is identical to those described above, with all name prefixes changed as follows: 

    gasnett_atomic_X    to   gasnett_strongatomic_X
    gasnett_atomic32_X  to   gasnett_strongatomic32_X
    gasnett_atomic64_X  to   gasnett_strongatomic64_X

  On most, but not all, platforms, operations on gasnett_strongatomic_t are signal safe.  
  On the few platforms where this is not the case GASNETT_STRONGATOMIC_NOT_SIGNALSAFE
  will be defined to 1.

  Similarly, GASNETT_STRONGATOMIC32_NOT_SIGNALSAFE and GASNETT_STRONGATOMIC64_NOT_SIGNALSAFE
  are defined to 1 IFF the implementation of the fixed-width atomics is not signal-safe.
  Note that these two are set independently.

  GASNETT_HAVE_STRONGATOMIC_ADD_SUB will be defined to 1 when gasnett_strongatomic_add()
  and gasnett_strongatomic_subtract() are available.  GASNETT_HAVE_STRONGATOMIC_CAS will
  be defined to 1 when gasnett_strongatomic_compare_and_swap() and gasnett_strongatomic_swap()
  are available.  As with the non-strong case, these operations are always available for
  the fixed-width types and thus there are no GASNETT_HAVE_ defines for the fixed-width
  strong atomic operations.

-------------------------
Portable platform defines
-------------------------

Most systems have predefined preprocessor tokens for identifying the compiler, OS and architecture
in use. However, there is no uniform naming convention for such platform features, and often 
a given feature (such as CPU family) will be indicated using a different name under 
different combinations of OS and compiler. 

GASNet tools provides a uniform naming scheme for detecting these preprocessor-provided 
platform features, so that #if tests can be written concisely with expressions like:

#if PLATFORM_COMPILER_GNU && PLATFORM_OS_SOLARIS && PLATFORM_ARCH_X86

See the comments in gasnet_portable_platform.h for the details of the provided defines.

----------------------------------
Portable fixed-width integer types
----------------------------------

inttypes.h is part of the POSIX and C99 specs, but in practice support for it 
varies wildly across systems. GASNet tools portably provides the fixed-bit-width 
integral types via the following typedefs:

    int8_t, uint8_t     signed/unsigned 8-bit integral types
   int16_t, uint16_t    signed/unsigned 16-bit integral types
   int32_t, uint32_t    signed/unsigned 32-bit integral types
   int64_t, uint64_t    signed/unsigned 64-bit integral types
   intptr_t, uintptr_t  signed/unsigned types big enough to hold any pointer offset

--------------------
Compiler annotations
--------------------

Many compilers have pragmas, attributes or other compiler-specific mechanisms for annotating
declarations and code in useful ways which are not standardized by the C specification.
The following macros expand to appropriate annotations when available, or to safe, unannotated 
versions when the given annotation is unavailable.  See also "Feature control", below.

GASNETT_INLINE(fnname) 
definition

  Most forceful inlining demand available.
  Might generate errors in cases where inlining is semantically impossible 
  (eg recursive functions, varargs fns)
  fnname should be the name of the function, and definition should be the actual
  definition of the function (declaration and body)
          
GASNETT_NEVER_INLINE(fnname,definition)

  Most forceful demand available to disable inlining for function.
     
GASNETT_RESTRICT     
  
  The C99 'restrict' keyword, if supported by the compiler, or empty otherwise.
        
GASNETT_FORMAT_PRINTF(fnname,fmtarg,firstvararg,declarator)
GASNETT_FORMAT_PRINTF_FUNCPTR(fnname,fmtarg,firstvararg,declarator)

  Annotate function fnname (defined by definition) as a printf-like function, 
  whose arguments should be checked for type compatibility with a format string whenever possible.
  fmtarg is the 1-based index of the argument providing the format character string, 
  firstvararg is the 1-based index of the first ... argument which corresponds to 
  arguments to the format string.

declaration GASNETT_NORETURN;
GASNETT_NORETURNP(fnname)

  Declare the given function as one that will never return (ie program will exit before return)

GASNETT_MALLOC    
declarator           
GASNETT_MALLOCP(fnname)     
        
  Declare the given function as one that returns new, unaliased memory (as with malloc)

GASNETT_PURE                 
declarator           
GASNETT_PUREP(fnname)        

  Declare as pure function: one with no effects except the return value, and 
  return value depends only on the parameters and/or global variables.
  prohibited from performing volatile accesses, compiler fences, I/O,
  changing any global variables (including statically scoped ones), or
  calling any functions that do so       

GASNETT_CONST                
declarator           
GASNETT_CONSTP(fnname)         

  Declare as const function: a more restricted form of pure function, with all the
  same restrictions, except additionally the return value must NOT
  depend on global variables or anything pointed to by the arguments

GASNETT_HOT
declarator

  Declare a function as frequently called.
  Compilers may do many different things with this information.

GASNETT_COLD
declarator

  Declare a function as infrequently called.
  Compilers may do many different things with this information.

GASNETT_DEPRECATED
declarator

  Declare a function as deprecated (subject to future removal).
  Attempts to generate a warning if the function is called.

GASNETT_WARN_UNUSED_RESULT   
declarator

  Attempt to generate a warning if the return value of the declared function is ignored by caller.

GASNETT_USED                
declarator           

  Declare the given function as one that must not be omitted, even if the compiler
  believes the function cannot ever be called.

GASNETT_PREDICT_TRUE(expr)
GASNETT_PREDICT_FALSE(expr)

  These macros yield a non-zero value if and only if expr has non-zero value.

  Additionally, they pass a hint to the compiler that one expects the value to
  be non-zero or zero, respectively.  Use them to wrap a branch-controlling
  expression when you have strong reason to believe the branch will frequently
  go in one direction and that the branch is a bottleneck.

  The macros if_pf() and if_pf() are implemented in terms of these macros.
  Examples:
    do { S; } while(GASNETT_PREDICT_FALSE(expr)); // single-trip is common case
    V = GASNETT_PREDICT_TRUE(expr) ? (val1) : (val2); // val1 is common case

if_pf(cond) S;
if_pt(cond) S;

  Drop-in replacements for the standard C 'if' keyword with branch-prediction hints.
  if_pf and if_pt behave just like 'if' except they give the C compiler a hint that 
  the condition is predicted to be false (if_pf) and the branch not taken, 
  or predicted to be true (pt) and the branch taken.
  These are equivalent to
    if(GASNETT_PREDICT_TRUE(expr)) S;
  and
    if(GASNETT_PREDICT_FALSE(expr)) S;
  respectively.

gasnett_constant_p(expr)

  This expands to use of __builtin_constant_p() on compilers with the necessary
  support, or to the constant 0 otherwise.

gasnett_unreachable()

  This annotation marks the current code location as unreachable (using compiler-specific
  mechanisms), to assist optimization of surrounding code.

gasnett_assume(cond)

  States simple expression cond is always true, as an annotation directive to guide compiler analysis.
  Becomes an assertion in DEBUG mode and an analysis directive (when available) in NDEBUG mode.
  This notably differs from typical assertions in that the expression must remain valid in NDEBUG mode
  (because it is not preprocessed away), and furthermore may or may not be evaluated at runtime.
  To ensure portability and performance, cond should NOT contain any function calls or side-effects.
  WARNING: passing a cond which is ever false in NDEBUG mode could lead the compiler 
  to mis-optimize surrounding code in unintuitive/unexpected ways.

GASNETT_FALLTHROUGH

 This annotation is used to annotate a deliberate/explicit fallthrough of control from
 one case statement to another, suppressing warnings about implicit fallthrough (the absence
 of a break statement), such as those generated by GCC's -Wimplicit-fallthrough (also -Wextra).

 Example:
   switch (whatever) {
     case 1: do_something(); GASNETT_FALLTHROUGH
     case 2: something_else();
   }
 Note it should be placed after the last statement in the case that
 falls-through and should NOT be followed by a semi-colon.

-----------------------------------------------------
Error-checking System Mutexes and Condition Variables
-----------------------------------------------------

GASNet tools provides convenience wrappers around the system's pthread mutexes
and condition variables.  In debug mode, these wrappers add error checking
capabilities to detect common usage violations (such as attempts to recursively
acquire a mutex, or release a mutex that has not been acquired).  The wrappers
also implement workarounds for known bugs in the pthread implementations of
several systems.  

In non-threaded builds, these wrappers still compile and expand to
appropriate no-ops, unless compiled with -DGASNETT_USE_TRUE_MUTEXES=1
which will force gasnett_mutex_t to always use true locking (even 
without -DGASNETT_THREAD_SAFE=1).

Unlike pthread_mutex_t, these locks may NEVER be obtained recursively, and
in debug builds this is detected as a usage violation. Similarly, they are
not safe to use for inter-process synchronization in shared memory segments.

* Otherwise, the following function similarly to the pthread_mutex symbols of the same name:

gasnett_mutex_t             
GASNETT_MUTEX_INITIALIZER
void gasnett_mutex_init(gasnett_mutex_t *)
void gasnett_mutex_destroy(gasnett_mutex_t *)
int gasnett_mutex_destroy_ignoreerr(gasnett_mutex_t *)

     mutex creation and destruction, as with pthread_mutex_t
     gasnett_mutex_destroy_ignoreerr performs no error checking and silently returns any errors 
     (eg as may occur when attempting to destroy a locked mutex)

void gasnett_mutex_lock(gasnett_mutex_t *)
void gasnett_mutex_unlock(gasnett_mutex_t *)

     lock and unlock (checks for recursive locking errors)

int gasnett_mutex_trylock(gasnett_mutex_t *)

     non-blocking trylock - returns EBUSY on failure, 0 on success

* Additional mutex utilities:

void gasnett_mutex_assertlocked(gasnett_mutex_t *)
void gasnett_mutex_assertunlocked(gasnett_mutex_t *) 

  In debug builds, these functions respectively assert that the given mutex is 
  currently locked or not locked by the calling thread, generating a fatal error
  if the assertion is violated. Has no effect in non-debug builds.

* The following function identically to the pthread_cond symbols of the same name:

gasnett_cond_t             
GASNETT_COND_INITIALIZER
void gasnett_cond_init(gasnett_cond_t *pc)
void gasnett_cond_destroy(gasnett_cond_t *pc)

     condition variable creation and destruction, as with pthread_cond_t

void gasnett_cond_signal(gasnett_cond_t *pc)
void gasnett_cond_broadcast(gasnett_cond_t *pc)

    signal at least one / all current waiters on a gasnet_cond_t, while holding the associated mutex

void gasnett_cond_wait(gasnett_cond_t *pc, gasnett_mutex_t *pl)

    release gasnett_mutex_t pl (which must be held) and block WITHOUT POLLING 
    until gasnett_cond_t pc is signalled by another thread, or until the system
    decides to wake this thread for no good reason (which it may or may not do).
    Upon wakeup for any reason, the mutex will be reacquired before returning.

    It's an error to wait if there is only one thread, and can easily lead to 
    deadlock if the last thread goes to sleep. No thread may call wait unless it
    can guarantee that (A) some other thread will eventually signal it to wake
    up and (B) some other thread is still polling (except in tools-only mode,
    where there is no polling).  The system may or may not also randomly signal
    threads to wake up for no good reason, so upon awaking the thread MUST
    verify using its own means that the condition it was waiting for has
    actually been signalled (ie that the client-level "outer" condition has
    been set).

    In order to prevent races leading to missed signals and deadlock, signaling
    threads must always hold the associated mutex while signaling, and ensure the
    outer condition is set *before* releasing the mutex. Additionally, all waiters
    must check the outer condition *after* acquiring the same mutex and *before*
    calling wait (which atomically releases the lock and puts the thread to sleep).

-------------------
Reader/Writer locks
-------------------

As with the gasnett_mutex_t wrappers in the previous section, we also provide
wrappers around POSIX reader/writer locks (pthread_rwlock_t). In a nutshell,
these allow multiple threads to concurrently acquire a "read" lock (for
concurrent read-only access to the protected data structures), but provide
mutual exclusion when a thread obtains a "write" lock to update the shared data.

CAUTION: The additional opportunities for concurrency provided by reader/writer
locks come at a SIGNIFICANT cost in additional serial overhead, relative to
simple mutexes.  The overhead for obtaining and releasing a read lock on an
uncontended pthread_rwlock_t is commonly 50%-300% more expensive than the
corresponding operation on simple mutex. Also, write locks still need to enforce
mutual exclusion, thus frequent write locks can sharply degrade achieved concurrency.
Consequently, rwlock's are only expected to provide a net performance win
relative to mutexes when there is a high-degree of concurrency for long-running
reader critical sections, and writers are VERY infrequent. In all other cases,
one should probably be using a mutex instead.

On systems lacking reader/writer locks (or when configured with --disable-rwlock), 
these compile down to regular gasnett_mutex_t operations - with full
serialization and no read concurrency. Some implementations also have a limit
on the number of threads that can concurrently obtain a reader lock.  For these
reasons, client code should be designed to remain deadlock-free when some or
all read locks are serialized, even lacking writers.

Unlike pthread_rwlock_t, these locks may NOT be obtained recursively, and
in debug builds this is detected as a usage violation. Similarly, they are
not safe to use for inter-process synchronization in shared memory segments.

* Otherwise, the following function similarly to the pthread_rwlock symbols of the same name:

gasnett_rwlock_t
GASNETT_RWLOCK_INITIALIZER
void gasnett_rwlock_init(gasnett_rwlock_t *)
void gasnett_rwlock_destroy(gasnett_rwlock_t *)

     rwlock creation and destruction, as with pthread_rwlock_t

void gasnett_rwlock_rdlock(gasnett_rwlock_t *)
void gasnett_rwlock_wrlock(gasnett_rwlock_t *)
void gasnett_rwlock_unlock(gasnett_rwlock_t *)

     blocking read lock, blocking write lock and unlock
     POSIX errors due to reader concurrency limits are masked as blocking

int gasnett_rwlock_tryrdlock(gasnett_rwlock_t *)
int gasnett_rwlock_trywrlock(gasnett_rwlock_t *)

     non-blocking trylock - returns EBUSY or EAGAIN on failure, 0 on success

* Additional rwlock utilities:

void gasnett_rwlock_assertrdlocked(gasnett_rwlock_t *)
void gasnett_rwlock_assertwrlocked(gasnett_rwlock_t *)
void gasnett_rwlock_assertlocked(gasnett_rwlock_t *)
void gasnett_rwlock_assertunlocked(gasnett_rwlock_t *)

  In debug builds, these functions respectively assert that the given rwlock is
  currently locked (for read, write or either) or not locked by the calling
  thread, generating a fatal error if the assertion is violated. Has no effect
  in non-debug builds.

--------------------
Thread-specific data
--------------------

GASNet tools provides wrappers to define and access pointers to thread-specific data, 
using an interface that expands to the fastest available mechanism provided by the
current platform for thread-specific data on threaded configurations
(eg __thread or pthread_getspecific()), and expands to simple dereference of 
process-global storage for non-threaded configurations.
Automatically handles the hassle of pthread key creation if required.

A thread-specific data pointer (mykey) must be declared as:

  GASNETT_THREADKEY_DEFINE(mykey); - must be defined in exactly one C file at global scope
  GASNETT_THREADKEY_DECLARE(mykey); - optional, use in headers to reference externally-defined key

and then can be used as:

  void *val = gasnett_threadkey_get(mykey);
  gasnett_threadkey_set(mykey,val);

no initialization is required (happens automatically on first access).

Initialization can optionally be performed using:

  gasnett_threadkey_init(mykey);

which then allows subsequent calls to:

  void *val = gasnett_threadkey_get_noinit(mykey);
  gasnett_threadkey_set_noinit(mykey,val);

these save a branch by avoiding the initialization check.
gasnett_threadkey_init is permitted to be called multiple times and
from multiple threads - calls after the first one will be ignored.

---------------------
Environment utilities
---------------------

Following utilities support querying the environment and manipulating the result.
Most of the query functions will report their actions to the console when the user
selects verbose reporting mode, to support self-documenting environment settings.

char *gasnett_format_number(int64_t val, char *buf, size_t bufsz, int is_mem_size);

  format a integer value as a human-friendly string, with appropriate mem suffix 

int64_t gasnett_parse_int(const char *str, uint64_t mem_size_multiplier);

  parse an integer value back out again
  if mem_size_multiplier==0, it's a unitless quantity
  otherwise, it's a memory size quantity, and mem_size_multiplier provides the 
    default memory unit (ie 1024=1KB) if the string provides none  

void gasnett_setenv(const char *key, const char *value);
void gasnett_unsetenv(const char *key);

  set/unset an environment variable, for the local process ONLY 

char *gasnett_getenv(const char *keyname);

  raw environment query function, bypasses reporting
  uses the gasnet conduit-provided global environment if available or regular getenv otherwise
  legal to call before gasnet_init, but may malfunction if
  the conduit has not yet established the contents of the environment

char *gasnett_getenv_withdefault(const char *keyname, const char *defaultval);

  environment query for a string parameter
  if user has set value the return value indicates their selection
  if value is not set, the provided default value is returned
  call is reported to the console in verbose-environment mode,
   (only the first call with a given key is reported)
  legal to call before gasnet_init, but may malfunction if
  the conduit has not yet established the contents of the environment

int gasnett_getenv_yesno_withdefault(const char *keyname, int defaultval);

   environment query for a yes/no parameter
   if user has set value to 'Y|YES|y|yes|1' or 'N|n|NO|no|0', 
   the return value indicates their selection
   if value is not set, the provided default value is returned

int64_t gasnett_getenv_int_withdefault(const char *keyname, int64_t defaultval, uint64_t mem_size_multiplier);

   environment query for an integral parameter
   if mem_size_multiplier non-zero, expect a (possibly fractional) memory size with suffix (B|KB|MB|GB|TB)
     and the default multiplier is mem_size_multiplier (eg 1024 for KB)
   otherwise, expect a positive or negative integer in decimal or hex ("0x" prefix)
   the return value indicates their selection
   if value is not set, the provided default value is returned

double gasnett_getenv_dbl_withdefault(const char *keyname, double defaultval);

  environment query for a floating-point parameter
  if user has set value the return value indicates their selection
   which must be a valid floating-point value or a fraction (e.g "1.5", "-1e4", or "3/8")
  if value is not set, the provided default value is returned
  call is reported to the console in verbose-environment mode,
   (only the first call with a given key is reported)
  legal to call before gasnet_init, but may malfunction if
  the conduit has not yet established the contents of the environment

int gasnett_verboseenv();

   returns true iff GASNET_VERBOSEENV reporting is enabled on this node 
   note the answer may change during initialization

void gasnett_envint_display(const char *key, int64_t val, int is_dflt, int is_mem_size);
void gasnett_envstr_display(const char *key, const char *val, int is_dflt);
void gasnett_envdbl_display(const char *key, double val, int is_dflt);

   display an integral/string/double environment setting iff gasnett_verboseenv()

-------------------------------
Backtracing and debugger attach
-------------------------------

GASNet tools provides some utilities to automatically freeze your process and wait for
a debugger attach when errors occur, and generate automatic backtraces during a crash.

void gasnett_freezeForDebuggerNow(volatile int *flag, const char *flagsymname);

   freeze immediately for debugger attach, and prompt the user to unfreeze by changing flag

void gasnett_freezeForDebuggerErr(); 

   freeze for debugger attach iff user enabled error freezing (GASNET_FREEZE_ON_ERROR=1)

void gasnett_backtrace_init(const char *exename);

   should be called early at startup with argv[0] in programs that intend to use the 
   automatic backtrace functionality. Alternatively one can pass gasnett_exe_name().

int gasnett_print_backtrace(int fd);

   print a human-readable backtrace immediately to the provided file descriptor.
   The mechanism used for generating the backtrace is system specific - on some systems
   several mechanisms are available and can be prioritized using GASNET_BACKTRACE_TYPE
   (see GASNet README for details).

int (*gasnett_print_backtrace_ifenabled)(int fd);

   This version is called by all internal GASNet errors, and is a pointer to a function
   that invokes gasnett_print_backtrace iff GASNET_BACKTRACE is enabled. 
   The pointer can be changed to modify the default backtracing mechanism 
   used for errors (eg to wrap gasnett_print_backtrace with a language-specific 
   symbol demangler).


GASNETT_CURRENT_FUNCTION

  Expands to const char * indicating the current function name, if available.

gasnett_current_loc
  
  Macro that evaluates to a dynamically-allocated char * describing the current location
  (file, line number and function) for use in error messages.

Backtrace extensibility

  The GASNet tools auto-backtrace mechanisms can be extended by the client, by defining 
  a variable called gasnett_backtrace_user in the client code, as follows:

   extern int myapp_do_backtrace(int fd) {
      /* write backtrace for calling thread to file descriptor fd */
      ...
      return 0; /* indicate success */
   }
   #if GASNETT_SPEC_VERSION_MAJOR > 1 || \
       (GASNETT_SPEC_VERSION_MAJOR == 1 && GASNETT_SPEC_VERSION_MINOR >= 1)
     gasnett_backtrace_type_t gasnett_backtrace_user = 
       { "MYAPP", /* name of backtrace mechanism to be added */
       &myapp_do_backtrace, /* pointer to user-provided function that writes backtrace */
       1 /* supports backtracing of multi-threaded executables? */ 
     };
   #endif

  This code will cause MYAPP to be added to the default GASNET_BACKTRACE_TYPE list, 
  and when GASNET_BACKTRACE_TYPE=MYAPP your function will be called to produce backtraces.
  The function should return non-zero if backtrace generation fails for whatever reason 
  (eg if the call occurs too early), so that other backtrace mechanisms can be attempted.

-----------------
System properties
-----------------

GASNETT_SYSTEM_TUPLE

  Configure-detected human-readable target tuple of this system. 
  Intended for informational display purposes.

GASNETT_CACHE_LINE_BYTES

  Compile-time constant positive int which estimates byte width of cache lines 
  shared between CPUs in an SMP. Set to a conservative value if unknown.

GASNETT_PAGESIZE

  Compile-time constant positive int which provides the size in bytes of the system's 
  virtual memory pages. Set to a conservative value for systems lacking VM or a fixed page size.

GASNETT_PAGESHIFT
  
  Compile-time constant positive int which is log_2(GASNETT_PAGESIZE)

const char *gasnett_gethostname();

  Returns the current system hostname, as reported by gethostname(), with possible
  normalization (such as of case).
  
const char *gasnett_exe_name();

  Returns a fully-qualified pathname to the currently running executable, or an
  empty string if the information is unknown.

int gasnett_cpu_count();    

  Returns the count of physical CPU's on this node (ie sharing a virtual memory), 
  or zero if that cannot be determined. Multiple cores may or may not be counted 
  as separate CPUs, depending on the system.

uint64_t gasnett_getPhysMemSz(int failureIsFatal);
 
  Return the size of the physical memory (in bytes) which is directly addressable 
  on this node. If that cannot be determined, issue a fatal error if failureIsFatal,
  or return zero otherwise.

int gasnett_isLittleEndian();

  Return true iff this architecture stores multi-word integral types in memory with
  the least-significant-byte in the lowest-numbered byte address.

const char *gasnett_performance_warning_str(void);

  Returns a string containing a human-readable warning summarizing attributes of the
  current configuration that may negatively affect performance (e.g. as a result
  of settings determined at configure time). If there are no such warnings, the
  resulting string is empty.

const char *gasnett_release_version_str(void);

  Returns a human-readable string containing the package release version number.
  Also see GASNETT_RELEASE_VERSION_* below.

uint64_t gasnett_release_version(void);

  Return an monotonically advancing integral representation of the compiled library's package version.
  Also see GASNETT_RELEASE_VERSION_* below.

-------------------
Trace/Stats support
-------------------

This section documents interfaces used to control the implementation-defined (non-normative)
tracing and statistical collection features of the GASNet library implementation.
These features are conditionally available based on configure-time settings, and are currently
only active for a "appropriately configured" conduit-mode GASNet library, meaning one configured 
with the --enable-trace/--enable-stats arguments (or the --enable-debug argument that implies these).

The interfaces specified in this section are ONLY functional in translation units meeting ALL
of the following conditions:

1. Must `#include <gasnetex.h>` before `#include <gasnet_tools.h>`
2. The included headers must reference an appropriately configured GASNet library.
3. Must eventually link an appropriately configured conduit-mode GASNet library.

In all other cases, the macros described in this section may compile away to
nothing, or appropriate constant values indicating lack-of-support.

See "GASNet tracing & statistical collection" in the GASNet README for more details on
the command-line user interface to generate and control trace and stats output.
In a nutshell, each process can generate an output stream of trace and stats
output (which is usually sent to a file). The trace stream outputs event
records as they occur, and the stats stream is used to output statistics that
are aggregated in memory before output (usually at process exit). Both streams
are optionally filtered by category via user-controllable masks.  Note that
depending on library build settings, all stats output may additionally be
echoed into the trace log (if any).

const char *GASNETT_TRACE_GETMASK()
const char *GASNETT_STATS_GETMASK()
GASNETT_TRACE_SETMASK(const char *newmask)
GASNETT_STATS_SETMASK(const char *newmask)

  Get and set the trace and stats masks for the current process at runtime. 
  The mask is represented as a character string of categories, as defined in the GASNet README.
  The environment variables described there are used to initialize the mask of
  active categories at job start, and these calls modify the mask programmatically.
  Note that envvars GASNET_{TRACE,STATS}FILE (and optionally GASNET_{TRACE,STATS}NODES)
  must still be set appropriately to enable output, otherwise the mask has no effect.

bool GASNETT_TRACE_ENABLED

  Expands to a non-zero value iff tracing for the `H` category is currently enabled and
  active (i.e. not filtered by the trace-mask) on the calling process.
  In general this is NOT a compile-time constant value.

GASNETT_TRACE_PRINTF(const char *format, ...)
GASNETT_TRACE_PRINTF_FORCE(const char *format, ...)
  
  Print a message into the trace log using the 'H' or 'U' categories, respectively, 
  when such tracing is enabled and active. The 'U' category is not subject to trace-mask 
  filtering, thus the _FORCE variant should be used sparingly for low-frequency messages.
  Argument syntax follows C99 printf conventions (format string followed by arguments).

GASNETT_STATS_PRINTF(const char *format, ...)
GASNETT_STATS_PRINTF_FORCE(const char *format, ...)
  
  Print a message into the stats log using the 'H' or 'U' categories, respectively, 
  when the stats log is enabled and active. The 'U' category is not subject to stats-mask 
  filtering, thus the _FORCE variant should be used sparingly for low-frequency messages.
  Argument syntax follows C99 printf conventions (format string followed by arguments).

GASNETT_STATS_DUMP(bool reset)

  This call directs the statistical collection system on this process to
  immediately dump a report of all active statistical counters to the stats
  output stream (appending to any earlier stats output).
  If the `reset` argument is non-zero, then statistical counters
  are additionally reset to their initial values after they are output.
  This output-then-reset is performed with "best effort" atomicity on a 
  counter-by-counter basis, but atomicity is not strongly guaranteed, especially 
  across separate counters or in the presence of concurrent activity
  by other threads.
  If the stats output stream is inactive on this process, this call has no effect.

-------------------
Miscellaneous tools
-------------------

GASNETT_SPEC_VERSION_MAJOR
GASNETT_SPEC_VERSION_MINOR

  Integral values corresponding to the major and minor version numbers of the GASNet tools 
  specification version adhered to by a particular implementation. The minor version is 
  incremented whenever new functionality is added without breaking backward compatibility.
  The major version is incremented whenever changes require breaking backward compatibility.
  The specification version is provided at the top of this document.

GASNETT_RELEASE_VERSION_MAJOR
GASNETT_RELEASE_VERSION_MINOR
GASNETT_RELEASE_VERSION_PATCH

  Integral values corresponding to the major, minor and patch version numbers of the public release
  identifiers corresponding the packaging on this implementation of GASNet/GASNet tools.

GASNETT_IDENT(identName, identText);

  Macro that should appear at global scope which takes a globally-unique identifier 
  and a textual string and embeds the textual string in the executable file. 
  The text to be embedded is arbitrary, but if you intend to extract it using the RCS 'ident' 
  utility, it will need to match the pattern:  
      "$[A-Za-z]+: [A-Za-z0-9_()<>.,|-]+ $"
  Note ident is particularly picky about the part before the initial colon and the final " $".

int gasnett_set_affinity(int rank);

  THIS CALL IS DEPRECATED.  Users are encouraged to use hwloc instead.

  On systems where GASNet Tools lacks cpu-binding support, the preprocessor
  identifier GASNETT_SET_AFFINITY_SUPPORT will be undefined and this function
  always returns non-zero.

  On systems where GASNet Tools supports cpu-binding, GASNETT_SET_AFFINITY_SUPPORT
  will be defined to 1 and the following description applies.

  Attempt to "pin" the calling thread to the processor indicated by rank, so that
  this thread will run only on the named processor (does not guarantee exclusive use of the
  processor, only tries to ensure the thread will not migrate to other processors).
  The definition and numbering of "processors" follow those of the system-specific
  underlying API, but typically treat each thread of an HT/SMT/HMT CPU as a distinct
  processor.
  Rank is interpreted mod the number of processors in the system.
  This call returns 0 on success and non-zero otherwise.

void gasnett_sched_yield();

  Cause the calling thread to yield (as in sched_yield()), if supported by the system.

void gasnett_flush_streams();

  Make the best effort possible to flush the stdout/stderr streams to their destinations.
  Errors are ignored, for instance if one or both streams have been closed.

void gasnett_close_streams();

  Close the stdin/stdout/stderr streams, usually in preparation for shutdown.
  Errors are ignored, for instance if any of the streams have been previously closed.

void gasnett_fatalerror(const char *msg, ...);
void gasnett_fatalerror_nopos(const char *msg, ...);

  Issue an fatal error message to the console, as specified by the arguments 
  (which follow a printf format convention). Then freeze for debugger and/or
  print a backtrace (depending on current settings) and issue an abort().
  The first form includes information about call site (i.e. function name,
  source file and line number), whereas the second omits this information.

void gasnett_killmyprocess(int exitcode); 

  Terminate the calling process as quickly as possible with the given exitcode, 
  including killing any sibling threads. Bypass any atexit handlers.

typedef void (*gasnett_sighandlerfn_t)(int);
extern gasnett_sighandlerfn_t gasnett_reghandler(int sigtocatch, gasnett_sighandlerfn_t fp);

  Register the provided signal handler function to service the specified signal.
  Return the previous handler function for that signal. 
  Valid fp values include SIG_DFL (system default handler for selected signal) and
  SIG_IGN (ignore the selected signal).

uint64_t gasnett_checksum(const void *p, int numbytes);

  Compute a very simplistic (insecure) but relatively efficient 64-bit checksum 
  from an untyped block of data [*p...*(p+numbytes-1)].

int gasnett_count0s_uint32_t(uint32_t x);
int gasnett_count0s_uint64_t(uint64_t x);
int gasnett_count0s_uintptr_t(uintptr_t x);

  Efficiently count the number of bytes with value 0 in the machine representation
  of a value of type uint32_t, uint64_t or uintptr_t, respectively.

size_t gasnett_count0s(const void *p, size_t numbytes);

  Efficiently count the number of bytes with value 0 in an untyped block of data
  [*p...*(p+numbytes-1)].

size_t gasnett_count0s_copy(void * restrict dst, const void * restrict src, size_t numbytes);

  Efficiently count the number of bytes with value 0 in an untyped block of data
  [*src...*(src+numbytes-1)], while also copying from src to dst.
  This function is equivalent (excluding any side-effects of evaluating numbytes) to
  the following expression:
      gasnett_count0s(memcpy(dst, src, numbytes), numbytes)
  but is generally more efficient.

gasnett_spinloop_hint()

   Some processors get measurably better performance when a special instruction
   is inserted in spin-loops (eg to avoid a memory hazard stall on spin loop exit 
   and reduce power consumption).  If such an instruction exists for this
   architecture, this macro issues the instruction and a compiler fence.
   Otherwise, this macro expands to the same as gasnett_compiler_fence().

int gasnett_maximize_rlimits();
int gasnett_maximize_rlimit(int res, const char *lim_desc);
 
   Maximize an rlimit indicated by res (an RLIMIT_* constant from sys/resource.h), with
   associated limit description. gasnett_maximize_rlimits() maximizes cpu time and all
   the in-memory execution rlimits associated with the current process (does not affect
   file system related limits). These functions return non-zero on success.

   If the environment variable GASNET_MAXIMIZE_[desc] is set to a false value (as
   defined by gasnett_getenv_yesno_withdefault()) then the corresponding limit will
   NOT be maximized, though the result will still indicate success.  For instance,
   setting GASNET_MAXIMIZE_RLIMIT_CPU=0 will suppress maximizing the limit on cpu
   time, but will not (by itself) cause a zero (failure) return value.
  
-------------------------
Macro argument evaluation
-------------------------

When implementing function-like macros that (perhaps conditionally) ignore one
or more of their arguments, it is usually desirable to ensure every argument is
evaluated exactly once (for side-effects).  Unfortunately, the simple idiom
`((void)(arg1),whatever(arg2))` will yield warnings from some compilers.

The following combine exactly-once evaluation and compile-specific warning
suppression (when needed):

  GASNETT_UNUSED_ARGS1(a1)
  GASNETT_UNUSED_ARGS2(a1,a2)
  GASNETT_UNUSED_ARGS3(a1,a2,a3)
  GASNETT_UNUSED_ARGS4(a1,a2,a3,a4)
  GASNETT_UNUSED_ARGS5(a1,a2,a3,a4,a5)
  GASNETT_UNUSED_ARGS6(a1,a2,a3,a4,a5,a6)
  GASNETT_UNUSED_ARGS7(a1,a2,a3,a4,a5,a6,a7)
  GASNETT_UNUSED_ARGS8(a1,a2,a3,a4,a5,a6,a7,a8)

Contrived example:
  #define OPTION_1_OF_3(x,y,z)  (GASNETT_UNUSED_ARGS2(y,z),(x))
  #define OPTION_2_OF_3(x,y,z)  (GASNETT_UNUSED_ARGS2(x,z),(y))
  #define OPTION_3_OF_3(x,y,z)  (GASNETT_UNUSED_ARGS2(x,y),(z))

---------------
Feature control
---------------

There are many features of the compilation and execution environment which are probed
by GASNet at configure-time.  The results of these configure probes are used to determine
if/how to implement certain of the macros or functions listed above.  In some cases there
may be a reasonable need to override the results of the configure probes.  This section
describes a family of pre-processor symbols that the GASNet client may define in order to
control the use of certain features, overriding the default behavior based on the
configure-time probes.

Use of these preprocessor symbols takes precedence over any information GASNet may have
probed from the compiler(s) and libraries at configure-time.  THESE SHOULD BE USED WITH
CARE, SINCE SYNTAX ERRORS CAN RESULT IF ONE ENABLES A FEATURE NOT SUPPORTED BY THE CURRENT
COMPILER.

* Feature control for compiler annotations

  The "Compiler annotations" section above describes a family of macros provided to the
  client for portably applying certain useful annotations.  However, these annotations
  are implemented based on configure-time tests of the compilers ($CC for GASNet-tools,
  and optionally $CXX and $MPI_CC for full GASNet).  In the event that a GASNet header
  is processed by a compiler different from the one(s) probed at configure-time, these
  annotation macros are reduced to their "safe" (usually empty) implementations, since
  GASNet cannot know that the "new" compiler will accept the same __attribute__(()) or
  #pragma syntax as the compiler(s) it probed.

  The client may define any of the following preprocessor symbols, prior to inclusion of
  gasnetex.h or gasnet_tools.h, to inform GASNet's headers that the current compiler does
  (#define to 1) or does NOT (#define to 0) support the corresponding syntax.
  When using a configure-recognized compiler, these default to their configure-detected
  values.

    GASNETT_USE_GCC_ATTRIBUTE_ALWAYSINLINE
      Compiler supports __attribute__((__always_inline__))
    GASNETT_USE_GCC_ATTRIBUTE_NOINLINE
      Compiler supports __attribute__((__noinline__)) 
    GASNETT_USE_GCC_ATTRIBUTE_MALLOC
      Compiler supports __attribute__((__malloc__)) 
    GASNETT_USE_GCC_ATTRIBUTE_WARNUNUSEDRESULT
      Compiler supports __attribute__((__warn_unused_result__)) 
    GASNETT_USE_GCC_ATTRIBUTE_USED
      Compiler supports __attribute__((__used__)) 
    GASNETT_USE_GCC_ATTRIBUTE_MAYALIAS
      Compiler supports __attribute__((__may_alias__)) 
    GASNETT_USE_GCC_ATTRIBUTE_NORETURN
      Compiler supports __attribute__((__noreturn__)) 
    GASNETT_USE_GCC_ATTRIBUTE_PURE
      Compiler supports __attribute__((__pure__)) 
    GASNETT_USE_GCC_ATTRIBUTE_CONST
      Compiler supports __attribute__((__const__)) 
    GASNETT_USE_GCC_ATTRIBUTE_DEPRECATED
      Compiler supports __attribute__((__deprecated__))
    GASNETT_USE_GCC_ATTRIBUTE_FORMAT
      Compiler supports __attribute__((__format__ (...))) 
    GASNETT_USE_GCC_ATTRIBUTE_FORMAT_FUNCPTR
      Compiler supports __attribute__((__format__ (...))) applied to a function pointer
    GASNETT_USE_GCC_ATTRIBUTE_FORMAT_FUNCPTR_ARG
      Compiler supports __attribute__((__format__ (...))) applied to a function pointer
      as an argument to a function (in its declaration and/or definition).
    GASNETT_USE_BUILTIN_CONSTANT_P
      Compiler supports __builtin_constant_p()
    GASNETT_USE_BUILTIN_PREFETCH
      Compiler supports __builtin_prefetch()
    GASNETT_USE_BUILTIN_EXPECT
      Compiler supports __builtin_expect()
    GASNETT_USE_BUILTIN_UNREACHABLE
      Compiler supports __builtin_unreachable()
    GASNETT_USE_BUILTIN_ASSUME
      Compiler supports __builtin_assume()
    GASNETT_USE_ASSUME
      Compiler supports __assume()

  The following can be defined to control use of C++ attributes:
    GASNETT_USE_CXX11_ATTRIBUTE_FALLTHROUGH
      C++ compiler supports [[fallthrough]]
    GASNETT_USE_CXX11_ATTRIBUTE_CLANG__FALLTHROUGH
      C++ compiler supports [[clang::fallthrough]]

  The following can be defined to control use of "restrict"
    GASNETT_USE_RESTRICT
      Set to the (possibly empty) keyword to use for GASNETT_RESTRICT
      Might be, for instance, "restrict", "__restrict" or "__restrict__".
    GASNETT_USE_RESTRICT_ON_TYPEDEFS
      Set to "1" to allow use in GASNet's headers of GASNETT_RESTRICT to qualify arguments
      declared via typedefs (and thus don't look like pointers until the typedef has been
      expanded).  Setting to "0" will disable such use.
      One may not set GASNETT_USE_RESTRICT_ON_TYPEDEFS without setting GASNETT_USE_RESTRICT.

  The following can be defined to control use of compiler-specific pragmas
    GASNETT_USE_PRAGMA_GCC_DIAGNOSTIC
      Use "#pragma GCC diagnostic" to suppress undesirable diagnostics

* Feature control for ctype.h wrappers

  There are systems on which ctype.h is implemented in such a way that passing char-typed
  arguments (for instance to isalpha() or tolower()) results in compiler warnings.  The
  GASNet configure script attempts to detect such systems and if found will replace the
  ctype.h interfaces with wrappers that promote the argument to an int prior to calling
  the system-provided implementation.  One can force (#define to 1) or prohibit (#define
  to 0) use of these wrappers by defining GASNETT_USE_CTYPE_WRAPPERS.

* Feature control for mutexes and condition variables

  In non-threaded builds, the debugging wrappers around pthreads functions expand to
  appropriate no-ops, unless compiled with GASNETT_USE_TRUE_MUTEXES=1 which will force
  unconditional use of true locking (even without GASNETT_THREAD_SAFE=1).
  NOTE: For proper operation this must be defined BOTH when the GASNet(-tools) library
  is built, and when compiling client code that includes gasnetex.h or gasnet_tools.h.

--------------------------------------------------------------------------
  The canonical version of this document is located here:
    https://bitbucket.org/berkeleylab/gasnet/src/master/README-tools

  For more information, please email: gasnet-users@lbl.gov
  or visit the GASNet home page at:   https://gasnet.lbl.gov
--------------------------------------------------------------------------