Inconsistent behaviour of zero-cost fencing with 0 arguments

[anyzelman]

In the zero_engine_MR, setting a synchronization attribute s1 with zero expected and sent messages combined with some valid tag, calling lpf_sync( ctx, s1 ) results in a barrier over s1.

However, setting a synchronization attribute s2 with zero expected and sent messages combined with some invalid tag, calling lpf_sync( ctx, s2 ) does not result in a barrier.

The samentics here need to be revisited for consistency (after which the implementation should be brought up to date).

[anyzelman]

While needing a closer look, it seems like in the s2 case, the lpf_sync turns into a non-blocking variant, i.e., a test for a fence -- while in the s1 case the lpf_sync is blocking. It seems cleaner to have an lpf_sync variant that provides this non-blocking behaviour. However, this increases the number of primitives by one, while it seems like then the blocking lpf_sync could be implemented as a higher-level function around the non-blocking variant. Simply updating the semantics of lpf_sync to become non-blocking always seems unwanted because it breaks backwards compatibility, but I also quickly do not see a way how to remain model compliant either.

So as a very early conclusion, simply adding a non-blocking lpf_sync variant seems the way to go. Note that requesting nonblocking behaviour via the sync attribute is not the way to go because

1. the LPF core spec demands that replacing any valid sync attribute with the default sync attribute should still yield correct behaviour; and
2. the non-blocking lpf_sync should give feedback to the callee on whether the sync has completed.