Aungkog

.gitignore

@@ -0,0 +1 @@
+*.class

README.md

@@ -1,44 +1,88 @@
-# P2PSP Simulation Project
+# Bitcoin transaction relay simulator
 
-[![Join the chat at https://gitter.im/P2PSP/PeerSim-simulator](https://badges.gitter.im/P2PSP/Qt.svg)](https://gitter.im/P2PSP/PeerSim-simulator?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+This project helps to measure different trade-offs in transaction relay protocols for Bitcoin.
+It was primarily designed to compare configurations of Erlay-like protocols, but it also helps just
+explore how flooding works.
 
-This is the PeerSim simulation branch for P2PSP. Its purpose is to simulate practical conditions with large sets of clients in order to obtain more knowledge about its behaviour.
+The simulator assumes knowledge of the existing Bitcoin p2p stack.
 
-To run this simulations you need to download the PeerSim simulator from [PeerSim download page](http://sourceforge.net/projects/peersim/).
+It currently omits to send GETDATA/TX messages, because it is not necessary for the current
+case, but can be easily expanded for that logic (as well as more advanced peer selection, block
+propagation research, etc.).
 
-You can set up your IDE (i.e. Eclipse) to work with PeerSim project as it described [here](http://miromannino.com/blog/integrating-peersim-with-eclipse/).
+Beware, research code.
 
-Configuration file is located at `config/config.txt`.
+## Organization
 
-## Running simulation
+This project consists of several main classes:
+1. Peer represents a normal Bitcoin node.
+2. Peer initializer spawns and configures Bitcoin nodes.
+3. Source represents a special node from which transactions initially propagate to random nodes.
+4. Source Initializer spawns and configures source nodes.
+5. Helpers contain custom message types to send between nodes.
+6. InvObserver is a class to collect results at the end of the experiment.
+
+## HOWTO
+
+The configuration file is located at `config/config.txt`. In this file, you can specify network size,
+connectivity, and other protocol-specific constants.
+
+Also, you will need JDK for this.
 
 1. Create a directory:
 
     ```shell
-    $ mkdir p2psp-peersim && cd p2psp-peersim
+    mkdir p2p-simulations && cd p2p-simulations
     ```
-    
+
 2. Clone a repository:
 
     ```shell
-    $ git clone git@github.com:P2PSP/sim.git
+    git clone git@github.com:naumenkogs/txrelaysim.git
     ```
-    
+
 3. Download PeerSim simulator engine (and unzip it):
 
     ```shell
-    $ wget downloads.sourceforge.net/project/peersim/peersim-1.0.5.zip && unzip peersim-1.0.5.zip
+    wget downloads.sourceforge.net/project/peersim/peersim-1.0.5.zip && unzip peersim-1.0.5.zip
     ```
-    
-4. Compile source files of P2PSP protocol:
+
+4. Compile source files:
 
     ```shell
-    $ javac -cp ./peersim-1.0.5/peersim-1.0.5.jar:./peersim-1.0.5/jep-2.3.0.jar:./peersim-1.0.5/djep-1.0.0.jar ./sim/src/*.java
-    ``` 
-   
+    javac -cp ./peersim-1.0.5/peersim-1.0.5.jar:./peersim-1.0.5/jep-2.3.0.jar:./peersim-1.0.5/djep-1.0.0.jar ./txrelaysim/src/*.java ./txrelaysim/src/helpers/*.java
+    ```
+
 5. Run simulation:
 
     ```shell
-    $ java -cp ./:./peersim-1.0.5/peersim-1.0.5.jar:./peersim-1.0.5/jep-2.3.0.jar:./peersim-1.0.5/djep-1.0.0.jar peersim.Simulator ./sim/config/config.txt 
+    java -cp ./:./peersim-1.0.5/peersim-1.0.5.jar:./peersim-1.0.5/jep-2.3.0.jar:./peersim-1.0.5/djep-1.0.0.jar peersim.Simulator ./txrelaysim/config/config.txt
     ```
-
+
+## Result interpretation
+
+We usually receive something like this at the end of the run:
+```
+1.7822426377729141 extra inv per tx on average.
+2.155010635147142 shortInvs per tx on average.
+23.500275013750688 success recons on average.
+0.08350417520876044 failed recons on average.
+Avg max latency: 7348.884615384615
+```
+
+For every transaction, no matter which protocol is used, the cost is always at least `INV + GETDATA + TX`.
+This data demonstrates extra cost: `+ 1.78 INV + 2.15 * SHORT_INV`, where `SHORT_INV = INV / 4`.
+
+`Avg max latency` represents the time it takes for a transaction to reach 95% of nodes.
+
+## Results
+
+Some results generated from the output of this simulator you can find in the Results folder.
+
+## Scalability
+
+On my 16" MacBook Pro 2019 it takes no more than a couple of minutes to simulate transaction relay across 30,000 nodes.
+If you increase connectivity (more than 8) or the number of nodes, you might run out of memory.
+
+For a more large-scale experiment, I may suggest using a machine with more RAM.
+To make it faster, you probably want a faster CPU.

config/config.txt

@@ -1,35 +1,29 @@
-# network size
-SIZE 10
+#### All the Erlay-specific configurations are in the "ERLAY" section above.
+#### Everything else is irrelevant framework-related stuff (apart from the network.size parameter,
+#### which might be interesting to adjust).
 
-# parameters of periodic execution
-CYCLES 200
-CYCLE SIZE*10
-
-# parameters of message transfer
-# delay values here are relative to cycle length, in percentage,
-# eg 50 means half the cycle length, 200 twice the cycle length, etc.
+CYCLES 600
+CYCLE 100 # milliseconds
 MINDELAY 5
-MAXDELAY 50
-# drop is a probability, 0<=DROP<=1
+MAXDELAY 100
 DROP 0
 
 random.seed 9098797865656766578567
-network.size SIZE
+network.size 15000
 simulation.endtime CYCLE*CYCLES
-simulation.logtime CYCLE
-simulation.experiments 1000
+simulation.logtime CYCLE * 50
+simulation.experiments 1
 
 ################### protocols ===========================
 protocol.0 peersim.core.IdleProtocol
 protocol.0.step CYCLE
 
-protocol.1 sim.src.Source
+protocol.1 txrelaysim.src.Source
 protocol.1.linkable 0
-protocol.1.step CYCLE
+protocol.1.step 1000 # trigger every seconds to handle tps easier
 protocol.1.transport tr
 
-protocol.2 sim.src.Peer
-protocol.2.buffer_size 32
+protocol.2 txrelaysim.src.Peer
 protocol.2.linkable 0
 protocol.2.step CYCLE
 protocol.2.transport tr
@@ -43,13 +37,39 @@ protocol.tr.transport urt
 protocol.tr.drop DROP
 
 ################### initialization ======================
-init.1 sim.src.SourceInitializer
+init.1 txrelaysim.src.SourceInitializer
 init.1.protocol 1
+init.1.tps 7
 
-init.2 sim.src.PeerInitializer
+############### ERLAY #####################
+init.2 txrelaysim.src.PeerInitializer
 init.2.protocol 2
-init.2.malicious_count 1
-init.2.trusted_count 1
+# How many outbound connections legacy (flooding) nodes make
+init.2.out_peers_legacy 8
+# How many outbound connections erlay (reconciling) nodes make
+init.2.out_peers_recon 8
+# How many reachable nodes we have in the network (total nodes is above called `network.size`)
+init.2.reachable_count 1500
+# Poisson delays applied by legacy nodes when relaying transactions (to inbounds and outbounds)
+init.2.in_relay_delay_legacy_peer 5000
+init.2.out_relay_delay_legacy_peer 2000
+# Poisson delays applied by erlay nodes when relaying transactions (to inbounds and outbounds)
+init.2.in_relay_delay_recon_peer 0
+init.2.out_relay_delay_recon_peer 0
+# Fraction of all nodes that support reconciliation
+init.2.reconcile_percent 100
+# Intervals between reconciliations with a given peer (Alice reconciles with Bob every 8 seconds)
+init.2.reconciliation_interval 16000
+# For erlay nodes, to how many in/out peers they have they will flood.
+# First, they flood to all legacy peers. Then, if the specified % is not reached, they pick random
+# peers among the remaining (erlay) peers, and flood to them. To the rest, they reconcile. This
+# choice is made per-transaction.
+init.2.in_flood_peers_percent 0
+init.2.out_flood_peers_percent 0
+# A coefficient for set difference estimation (used in Erlay)
+init.2.default_q 0.25
+# How many of the nodes are black holes
+init.2.private_black_holes_percent = 0
 
 init.sch1 CDScheduler
 init.sch1.protocol 1
@@ -59,13 +79,9 @@ init.sch2 CDScheduler
 init.sch2.protocol 2
 init.sch2.randstart
 
-
 ################ control ==============================
 
-#control.0 sim.src.PeerObserver
-#control.0.protocol 2
-#control.0.step CYCLE*1
+control.0 txrelaysim.src.InvObserver
+control.0.protocol 2
+control.0.step CYCLE * 100
 
-control.1 sim.src.PoisonedChunksObserver
-control.1.protocol 2
-control.1.step CYCLE*1

results/flood-erlay/README.md

@@ -0,0 +1,63 @@
+## Pick best Erlay configuration
+
+These experiments were done to explore all reasonable Erlay configuations, and to pick the best for
+the current network and for extended connectivities.
+
+For these experiments, the network had 2,000 reachable nodes and 18,000 non-reachable nodes.
+Increasing it by 2x doesn't change the results much.
+
+The following Erlay parameters could be tweaked:
+1. q choice
+2. reconciliation frequency
+3. in/out flooding delay
+4. in/out number of peers a node floods to
+
+## Latency
+We measure the time it takes for a transaction to reach 95% of the nodes, assuming every node has 8
+tx-relay peers. For this experiment, only 2, 3, 4 matter.
+
+We label the configurations as following:
+
+| Name        | In/Out flood delay | Reconciliation interval | In/out flood destinations |
+| ------------- |:-------------:| -----:| -----:|
+| Erlay-a      | 2s/5s | 2s | 2/2 |
+| Erlay-b      | 2s/5s | 2s | 3/3 |
+| Erlay-c      | 1s/2s | 2s | 2/2 |
+| Erlay-d      | 1s/2s | 2s | 3/3 |
+| Erlay-e      | 1s/2s | 1s | 2/2 |
+| Erlay-f      | 1s/2s | 1s | 3/3 |
+
+# Optimal bandwidth
+
+The second experiment considers the first parameter, Q. We try values between 0.01 and 0.2,
+and find out that q=0.01 make most sense for all, because it's best at conserving bandwidth.
+
+It's also possible to see bandwidth breakdown (how many reconciliations failed, etc.),
+and try to optimize it from there, but we leave it for further research.
+
+# Conclusions
+
+<img src="plots/erlay_configurations.png" width="520" height="400" />
+
+For now, we conclude that Erlay-e-0.01 is the most optimal configuration, because it provides
+the lowest bandwidth overhead (except for Erlay-a-0.01, which is way slower)
+and one of the lowest latencies on par with legacy flooding.
+
+If broken down, Erlay-e-0.01 requires sending 1.77 txid and 1.47 short-txid per transaction, in addition
+to necessary 2 * txid (inv + getdata, as per the latest erlay protocol).
+
+Since the reconciliation failure rate is very low, the first aspect is not caused by it, but is
+rather caused by the natural fanout which happens faster than flooding, and thus it can't be reduced,
+unless we use yet another configuration.
+
+This is, however, problematic: either we increase flood delays (and increase the latency overall),
+or make reconciliations more frequent (and make them less efficient).
+
+The latter aspect can be optimized by better estimating set diffenence (by choosing q better),
+but it's just (1.47 * 4) bytes, which is minor compared to the overall INV traffic,
+so we don't do that for now.
+
+# Scaling with connections
+
+Erlay-e-0.01 also scales well with the number of connections. If we increase the connectivity from
+8 to 12, we get just 77 bytes of extra overhead.

results/flood-erlay/plots/compare_protocols.py

@@ -0,0 +1,23 @@
+import matplotlib.pyplot as plt
+
+latencies = [4.3, 8.1, 7, 6.1, 5.1, 4.3, 3.7]
+bandwidth_overhead = [227, 74, 101, 77, 101, 73, 98]
+
+protocols = ['Legacy flooding', 'Erlay-a-0.01', 'Erlay-b-0.01', 'Erlay-c-0.01', 'Erlay-d-0.01',
+             'Erlay-e-0.01', 'Erlay-f-0.01']
+
+
+fig, ax = plt.subplots()
+ax.set_xlabel('Latency (s)')
+ax.set_ylabel('Bandwidth overhead (bytes per tx)')
+ax.scatter(latencies, bandwidth_overhead)
+
+for i, txt in enumerate(protocols):
+    if i == 0:
+        ax.annotate(txt, (latencies[i], bandwidth_overhead[i] - 10))
+    elif i == 1:
+        ax.annotate(txt, (latencies[i] - 0.3, bandwidth_overhead[i] + 5))
+    else:
+        ax.annotate(txt, (latencies[i], bandwidth_overhead[i] + 5))
+
+plt.show()