Forks done right

[larskuhtz]

This PR replaces the current service dates by forks that are voted for by miners, usually by installing the next version of the software. But it would also possible to vote via, say, a command line flag or some other configuration option.

It guarantees continuous operation of the network without the need of regular service releases. It also allows to react in a more agile way to issues and rollout changes on an on-demand basis.

Service date releases are well suited for incremental software improvements that are non-controversial among the the users of the system. Updates that affect economic incentives of the system should be affirmed explicitly by the block producers. This PR introduces such a mechanism.

• Define fork number and related block validation logic
• Replace feature flag validation by fork state validation

Experimental results related to the algorithm for vote counting can be found in this repository in this notebook

[larskuhtz]

@edmundnoble here is some code to simulate propagation of fork votes in the chain graph at the end of each fork epoch: https://github.com/larskuhtz/quant-avg-consensus-for-chainweb

For d4k4 a precision of 1000 limits the average error to [-0.5,0.5]. There is a sharp phase transition for full convergence between 60 and 62 steps. For d5k4 that transition is at 43 steps. For the twenty chain graph at 51 steps.

[larskuhtz]

Here are some results for 500000 samples with precision 1000 and 59-62 steps for d4k4:

$ for ((i=59; i<63; i++)) ; do ./quant-avg-consensus 'd4k4' 500000 1000 $i ; done

{"convergenceCount":499177,"delta":{"avg":1.646e-3,"max":1,"median":0,"min":0},"err":{"avg":2.537244897959155e-2,"max":0.5204081632655289,"median":2.0408163265528856e-2,"min":-0.47959183673447114},"steps":59}
{"convergenceCount":499910,"delta":{"avg":1.8e-4,"max":1,"median":0,"min":0},"err":{"avg":2.58265510204081e-2,"max":0.5204081632655289,"median":3.061224489738379e-2,"min":-0.47959183673447114},"steps":60}
{"convergenceCount":500000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.5171673469387243e-2,"max":0.5204081632655289,"median":2.0408163265528856e-2,"min":-0.47959183673447114},"steps":61}
{"convergenceCount":500000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.5971000000000573e-2,"max":0.5204081632655289,"median":3.061224489738379e-2,"min":-0.47959183673447114},"steps":62}

[larskuhtz]

Results for d5k4:

$ for ((i=48; i<52; i++)) ; do ./quant-avg-consensus 'd5k4' 100000 1000 $i ; done

{"convergenceCount":99877,"delta":{"avg":1.23e-3,"max":1,"median":0,"min":0},"err":{"avg":1.8993301886793032e-2,"max":0.518867924528422,"median":1.886792452842201e-2,"min":-0.481132075471578},"steps":48}
{"convergenceCount":99992,"delta":{"avg":8.0e-5,"max":1,"median":0,"min":0},"err":{"avg":2.0584103773584193e-2,"max":0.518867924528422,"median":1.886792452842201e-2,"min":-0.481132075471578},"steps":49}
{"convergenceCount":100000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.0781462264151077e-2,"max":0.518867924528422,"median":2.3584905660754885e-2,"min":-0.481132075471578},"steps":50}
{"convergenceCount":100000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":1.9994905660378235e-2,"max":0.518867924528422,"median":1.886792452842201e-2,"min":-0.481132075471578},"steps":51}

[larskuhtz]

Results for twenty chain graph:

$ for ((i=43; i<47; i++)) ; do ./quant-avg-consensus 'twenty' 1000000 1000 $i ; done

{"convergenceCount":999547,"delta":{"avg":4.53e-4,"max":1,"median":0,"min":0},"err":{"avg":2.5327500000000343e-2,"max":0.5,"median":4.9999999999272404e-2,"min":-0.4500000000007276},"steps":43}
{"convergenceCount":999980,"delta":{"avg":2.0e-5,"max":1,"median":0,"min":0},"err":{"avg":2.5574049999999855e-2,"max":0.5,"median":4.9999999999272404e-2,"min":-0.4500000000007276},"steps":44}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.4516199999999887e-2,"max":0.5,"median":0,"min":-0.4500000000007276},"steps":45}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.485050000000043e-2,"max":0.5,"median":0,"min":-0.4500000000007276},"steps":46}

[larskuhtz]

Some more

$ for ((i=19; i<23; i++)) ; do ./quant-avg-consensus 'petersen' 1000000 1000 $i ; done
{"convergenceCount":925866,"delta":{"avg":7.4134e-2,"max":1,"median":0,"min":0},"err":{"avg":3.699799999999824e-2,"max":0.5,"median":0,"min":-0.5},"steps":19}
{"convergenceCount":994597,"delta":{"avg":5.403e-3,"max":1,"median":0,"min":0},"err":{"avg":5.036630000000296e-2,"max":0.5,"median":9.999999999990905e-2,"min":-0.5},"steps":20}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":5.006199999999943e-2,"max":0.5,"median":9.999999999990905e-2,"min":-0.40000000000009095},"steps":21}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":5.0148000000001094e-2,"max":0.5,"median":9.999999999990905e-2,"min":-0.40000000000009095},"steps":22}

$ for ((i=77; i<81; i++)) ; do ./quant-avg-consensus 'd3k4' 1000000 1000 $i ; done
{"convergenceCount":999884,"delta":{"avg":1.16e-4,"max":1,"median":0,"min":0},"err":{"avg":2.705292105263337e-2,"max":0.5263157894733013,"median":2.6315789473301265e-2,"min":-0.47368421052669873},"steps":77}
{"convergenceCount":999998,"delta":{"avg":2.0e-6,"max":1,"median":0,"min":0},"err":{"avg":2.6407052631578436e-2,"max":0.5263157894733013,"median":2.6315789473301265e-2,"min":-0.47368421052669873},"steps":78}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.6724710526318012e-2,"max":0.5263157894733013,"median":2.6315789473301265e-2,"min":-0.47368421052669873},"steps":79}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.6195131578946717e-2,"max":0.5263157894733013,"median":2.6315789473301265e-2,"min":-0.47368421052669873},"steps":80}

$ for ((i=43; i<47; i=i+1)) ; do ./quant-avg-consensus 'd4k3' 500000 1000 $i ; done
{"convergenceCount":499786,"delta":{"avg":4.28e-4,"max":1,"median":0,"min":0},"err":{"avg":2.374951219512216e-2,"max":0.5121951219516632,"median":2.439024390241684e-2,"min":-0.48780487804833683},"steps":43}
{"convergenceCount":499993,"delta":{"avg":1.4e-5,"max":1,"median":0,"min":0},"err":{"avg":2.4486536585366585e-2,"max":0.5121951219516632,"median":2.439024390241684e-2,"min":-0.48780487804833683},"steps":44}
{"convergenceCount":500000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.418434146341461e-2,"max":0.5121951219516632,"median":2.439024390241684e-2,"min":-0.4634146341468295},"steps":45}
{"convergenceCount":500000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.3976292682927256e-2,"max":0.5121951219516632,"median":2.439024390241684e-2,"min":-0.4634146341468295},"steps":46}

$ for ((i=39; i<43; i=i+1)) ; do ./quant-avg-consensus 'd5k3' 500000 1000 $i ; done
{"convergenceCount":498858,"delta":{"avg":2.284e-3,"max":1,"median":0,"min":0},"err":{"avg":1.9913305555555424e-2,"max":0.5138888888886868,"median":1.3888888888686779e-2,"min":-0.4861111111113132},"steps":39}
{"convergenceCount":499919,"delta":{"avg":1.62e-4,"max":1,"median":0,"min":0},"err":{"avg":2.1310805555555496e-2,"max":0.5138888888886868,"median":2.7777777778283053e-2,"min":-0.4861111111113132},"steps":40}
{"convergenceCount":499999,"delta":{"avg":2.0e-6,"max":1,"median":0,"min":0},"err":{"avg":2.04324444444443e-2,"max":0.5138888888886868,"median":1.3888888888686779e-2,"min":-0.47222222222262644},"steps":41}
{"convergenceCount":500000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":2.0466888888889142e-2,"max":0.5138888888886868,"median":1.3888888888686779e-2,"min":-0.47222222222262644},"steps":42}

$ for ((i=13; i<17; i=i+1)) ; do ./quant-avg-consensus 'hoffman' 1000000 1000 $i ; done
{"convergenceCount":960855,"delta":{"avg":3.9145e-2,"max":1,"median":0,"min":0},"err":{"avg":5.305559999999742e-3,"max":0.5200000000004366,"median":0,"min":-0.5200000000004366},"steps":13}
{"convergenceCount":982527,"delta":{"avg":1.7473e-2,"max":1,"median":0,"min":0},"err":{"avg":9.252180000000334e-3,"max":0.5,"median":0,"min":-0.5},"steps":14}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":9.924380000000142e-3,"max":0.5,"median":2.0000000000436557e-2,"min":-0.47999999999956344},"steps":15}
{"convergenceCount":1000000,"delta":{"avg":0,"max":0,"median":0,"min":0},"err":{"avg":9.863420000000234e-3,"max":0.5,"median":0,"min":-0.47999999999956344},"steps":16}

[larskuhtz]

Convergence step count for different degree diameter graphs:

[edmundnoble]

resetVotes sounds like a function, how about noVotes?

[edmundnoble]

I think we're still missing a test for the fork vote counting.