Running BriskStream benchmarks

[mencagli]

Dear Tony,

I am trying to run BriskStream benchmarks in my multi-core machine (stable branch). The compilation works fine but I am not able to run any program. Since your scripts are very complex, I am trying to run by hand the WordCount benchmark using my local dataset (I have changed the path in the configuration file). Unfortunately, the application hangs foreever without processing anything. I run it using a command like:

java -cp target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar applications.BriskRunner

which should use the default configuration and the default application (WordCount). By inspecting the code (by adding proper prints), I discover that the code hangs in the file ExecutionGraph.java in the Loading function (the two double loops inside). I understand that there are some problems in loading the configuration and nothing happens: the topology is not working and not processing any data.

Can you please help me in understanding how to run BriskStream?

Thanks!

Gabriele

[ShuhaoZhangTony]

Hi Gabriele,

I guess the system hangs at the "load()" function (see the following code). The reason is that it seeks to load statistics file that is required by the cost model, but you haven't prepare it.
while (!tmpfile.exists()) { target = executionNode.getOP() + Math.max(1, (numTasks--)) + srcNode.getOP(); tmpfile = new File(dir + OsUtils.OS_wrapper(target + ".txt")); }

If you want to test the raw performance of briskstream without invoke the cost model and optimization process, you can append a "--native" argument. It's like "java -cp target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar applications.BriskRunner --native".

If you want to test the optimization, you need to first profile the application by specifying a "--profile" argument. An example argument is as follows (you can find it in ``nus" script).
-st 1 -sit 1 -tt 40 --num_socket 8 --num_cpu 18 --THz 500000 --runtime 5000 --loop 100000 --size_tuple 10 --repeat 1 -bt 100--percentile 50"

After profiliation is done, you should be able to see some files like follows.
../briskstream/STAT/wc/50/10/wordCount1splitSentence.txt
Then, you can test briskstream with optimization enabled.
An example argument is as follows.
--THz 500000 --runtime 30 --loop 1000 --num_socket 8 --num_cpu 10 --size_tuple 10

You can find full list of arguments in ``abstractRunner.Java" class.
Hope it helps. :)

I'll pin this issue in case someone else encounters the same problem.
Thanks!
Tony.

[mencagli]

Thanks for the quick reply!!

I tried with "java -cp target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar applications.BriskRunner --native" and the problem is now solved. Unfortunately, there is still a NullPointerExpection raised at

2019-06-11 14:10:20 INFO Optimizer:54 - number of CPUs:8
2019-06-11 14:10:20 INFO Optimizer:55 - number of num_socket:1
2019-06-11 14:10:20 INFO Optimizer:56 - GC factor:1.0
2019-06-11 14:10:20 INFO TopologySubmitter:64 - DB initialize starts @2019-06-11T14:10:20.173+02:00
Exception in thread "main" java.lang.NullPointerException
at brisk.topology.TopologySubmitter.submitTopology(TopologySubmitter.java:69)
at applications.BriskRunner.runTopologyLocally(BriskRunner.java:100)
at applications.BriskRunner.run(BriskRunner.java:415)
at applications.BriskRunner.main(BriskRunner.java:91)

Probably, there is still some configuration issues.

Thanks a lot for your help and guide in this.

[ShuhaoZhangTony]

No problem. :) In fact, I should apologize for not having time to make it more user-friendly.

For the NullPointerException, please ensure
enable_shared_state @ Line 62 is false.
You can find this flag in common/src/main/java/applications/CONTROL.Java.
This flag is an experimental flag, and should be disabled for testing BriskStream.

Another suggestion is that, when you test the performance of BriskStream, remember to use "-bt " argument to configure different jumbo tuple size. A good starting point is "-bt 10", that is 10 tuples are merged into a single joint tuple. This is to reduce instruction cache miss and improve cross-operator communication efficiency.

More details can be found in the paper. A personal version of the paper is here https://www.comp.nus.edu.sg/~shuhao-z/docs/briskstream.pdf

Thanks!
Tony

[mencagli]

Thanks a lot!
Everything seems to start now. Unfortunately with still some problems:

Exception in thread "Operator:splitSentence Executor ID:2" java.lang.NullPointerException
at java.lang.String.(String.java:166)
at applications.bolts.wc.SplitSentenceBolt.execute(SplitSentenceBolt.java:102)
at brisk.components.operators.executor.BasicBoltBatchExecutor.execute(BasicBoltBatchExecutor.java:38)
at brisk.execution.runtime.boltThread._execute_noControl(boltThread.java:183)
at brisk.execution.runtime.boltThread._execute(boltThread.java:193)
at brisk.execution.runtime.executorThread.routing(executorThread.java:359)
at brisk.execution.runtime.boltThread.run(boltThread.java:267)

It seems that the SplitSentenceBolt picked an Exception. But it is good that the topology is running now. Today I will try to understand what happened and in case I will post tomorrow some updates hoping to find some help from you!

Thanks!

Gabriele

[ShuhaoZhangTony]

Ha, I guess that's related to the input data sets.
I have assumed a delimiter of ",".
The input data is like "eating, an, apple, is, good".
You can change the delimiter, for example, use space, at Line 100 of SplitSentenceBolt.
String[] split = new String(value).split(" ");
Thanks!
Tony

[mencagli]

Dear Tony,

WordCount is running well. I am collecting performance results now. Thanks.

I would like to run also FraudDectection. I have the two dataset files (credit-card.dat and model.txt). I am trying the run the application using the following command:

java -cp target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar applications.BriskRunner --native -bt 10 -a FraudDetection

and I receive immediately the following exception

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at brisk.controller.output.PartitionController.(PartitionController.java:97)
at brisk.controller.output.partition.FieldsPartitionController.(FieldsPartitionController.java:31)
at brisk.execution.ExecutionGraph.partitionController_create(ExecutionGraph.java:370)
at brisk.execution.ExecutionGraph.init_pc(ExecutionGraph.java:338)
at brisk.execution.ExecutionGraph.build_streamController(ExecutionGraph.java:436)
at brisk.execution.ExecutionGraph.setup(ExecutionGraph.java:165)
at brisk.execution.ExecutionGraph.Configuration(ExecutionGraph.java:129)
at brisk.execution.ExecutionGraph.(ExecutionGraph.java:55)
at brisk.topology.TopologyComiler.generateEG(TopologyComiler.java:16)
at brisk.topology.TopologySubmitter.submitTopology(TopologySubmitter.java:45)
at applications.BriskRunner.runTopologyLocally(BriskRunner.java:100)
at applications.BriskRunner.run(BriskRunner.java:415)
at applications.BriskRunner.main(BriskRunner.java:91)

Maybe another configuration issue? Thanks for your great help.

Best,

Gabriele

[ShuhaoZhangTony]

Hi Gabriele,

You need to at least configure "-tt 2" (default is 1) to the system in order to run FraudDetection application. Otherwise, the parallelism is zero for some operators.
I have updated the program to handle such corner case. Thanks!

Tony

[mencagli]

Dear Tony,

thank you for your help in the past. In this weeks, I have executed some applications in BriskStream in my multicore machine. All the experiments have been performed using the --native option. Now, I would like to test your interesting optimizations.

I have tried with the following command to generate the profile statistics as explained in your previous post answering my issue:

java -jar target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar --profile -st 1 -sit 1 -tt 40 --num_socket 2 --num_cpu 16 --THz 500000 --runtime 5000 --loop 100000 --size_tuple 10 --repeat 1 -bt 100 --percentile 50

Please, note that my machine is quite small, two sockets only, but I am going to test BriskStream in a larger shared-memory system soon.

The problem that I have with the previous command is that the system hangs forever without doing anything. By running "htop", I see that there is only one thread continuously consuming 100% of a given CPU core. I am wondering if there is a similar problem as the one in my previous post (hanging on load() or whatever). By looking into the code, it seems that the program hangs on this call:

ExecutionGraph g = new TopologyComiler().generateEG(topology, conf);

in TopologySubmitter.java

I hope that you can give me some insights into how to proceed. Many thanks for your attention,

Gabriele

[ShuhaoZhangTony]

Hi Gabriele,
I just checked the code and confirm it's a logic bug, it happens when the user doesn't have any statistic files.
Thanks for pointing it out, I will commit an updated version of the repo. Alternatively, you can simply update the following code in STAT.java in your local copy to work around:

           while (!tmpfile.exists()) {
                target = executionNode.getOP() + Math.max(1, (numTasks--)) + srcNode.getOP();
                tmpfile = new File(dir + OsUtils.OS_wrapper(target + ".txt"));
                if (numTasks == 0) // please add these two lines.
                    break;
            }

Furthermore, please remember that under profiling mode, "-tt" only specifies the number of threads of ``stateful" operator (e.g., Counter in WordCount application), the rest operators (as they are stateless) will be automatically configured with a parallelism of one.

One more thing, since you are playing with the optimizations, please remember to configure your own machines' specifications as a new class file such as briskstream\common\src\main\java\applications\HP_Machine.java.

I have used two machines, HP and HUAWEI. Hence, there're two class files correspondingly.

Besides, you need to configure
public static ArrayList[] getNodes(int machine)
method in Platform.java to indicates the NUMA mapping of your machine.
After that, you need to specify --machine to indicate your machine of usage.

As of now, I haven't prepare the code to generate all those static information automatically but are prepared offline manually and hard-coded into the system. Sorry for such inconvenience! To generate all necessary information of your machine (e.g., latency, memory bandwidth etc.), you can use Intel mlc tool (https://software.intel.com/en-us/articles/intelr-memory-latency-checker). To generate NUMA mapping, you can use NUMACTL.
Feel free to ask me if you are unsure about how to get those information. Thanks!

Tony.

[mencagli]

Dear Tony,

with your help, I am very close to be able to do the profiling. I did all the passages in your answer above. However, I have still a problem. I prepared a file like your HP_Machine.java with the latency profiling information obtained by the Intell MLC tool. Great! I have also modified the getNodes() method as follows:

    else if (machine == 4) { // pianosa
        Integer[] no_0 = {0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23};
        node_0 = new ArrayList<>(Arrays.asList(no_0));
        Integer[] no_1 = {8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31};
        ArrayList<Integer> node_1 = new ArrayList<>(Arrays.asList(no_1));
        return new ArrayList[]{
                node_0,
                node_1
        };
    }

My machine is named "Pianosa", two sockets with 8 cores each (16 with hyperthhreading). Totally 16 cores (32 contexts). Everything seems ready. When I run:

java -jar target/BriskBenchmarks-1.2.0-jar-with-dependencies.jar --profile -a FraudDetection -st 1 -sit 1 -tt 29 --THz 500000 --runtime 5000 --loop 100000 --size_tuple 10 --repeat 1 -bt 100 --percentile 50 --machine 4

everything starts and the program fails in assigning a core to one thread. In particular this is a fragment of the output:

...
2019-07-16 12:19:53 INFO boltThread:234 - Successfully create boltExecutors 13 for bolts:predictorBolt on node: 0binding:000000000000000000000000000000000000000000000000000000000000
2019-07-16 12:19:53 INFO executorThread:127 - predictorBolt(14)
2019-07-16 12:19:53 INFO executorThread:132 - binding to node:0 cpu:23
2019-07-16 12:19:53 INFO boltThread:234 - Successfully create boltExecutors 14 for bolts:predictorBolt on node: 0binding:000000000000000000000000000000000000000000000000000000000000
2019-07-16 12:19:53 INFO AffinityController:141 - No available CPUs

The problem should be in AffinityController.java in the requirePerCore() method. It raises an exception maybe here:

        try {
            cpus[0] = (this.mapping_node[node].get(cnt));
        } catch (Exception e) {
            LOG.info("No available CPUs");
            System.out.println("Node: " + node + " cnt " + cnt);
            System.exit(-1);
        }

After debugging, I see that node is 0 and cnt il 16. In fact, each numa node in my machine has 16 SMT cores, so the last valid cnt should be 15 and not 16. Maybe, there is still some problems (ArrayIndexOfOutBoundException). Do you have any idea about how to fix this?

Many thanks!

gabriele

[mencagli]

I don't know if this might help. I also receive a message from the AffinityLock saying that

2019-07-16 12:33:03 INFO AffinityLock:126 - No isolated CPUs found, so assuming CPUs 1 to 31 available.

[ShuhaoZhangTony]

Hi Gabriele,

The problem is that cpu0 must be reserved for operating system.
Hence, I have a line cpu_pnt.put(0, 1); at AffinityController.java line 42 to explicitly leave out cpu0.
In your case, to make your profiling works, you need to make sure -tt vary from 1 to 28 (i.e., maximally you use 1+1+28+1=31 cores of your machine).

By the way, I have explicitly avoided the usage of hyper threading (HT) cores in my experiments so I'm not sure how will it goes if HT is enabled. At least, it will most likely make the cost model less accurate. It should be an interesting study to explore through.

Thanks!

Tony.

[ShuhaoZhangTony]

AffinityLock's message is saying you didn't explicitly make CPU cores isolated (isolcpus). That means, the OS will still try to schedule its works (e.g., other processes running on the same machine) on those CPU cores that you have explicitly pined for the streaming operators. Based on my experience, isolated cpu cores will make the system performance even more stable but that's not a must requirement. You may simply ignore that warning message.

[mencagli]

Dear Tony,

I understand that core0 is reserved to the OS. Unfortunately, the exception still happens as soon as the -tt value is greater than 12. It seems that it is looking for the core with index cnt = 16 that does not exist (on node 0 I have 16 cores so index from 0 to 15)

cpus[0] = (this.mapping_node[node].get(cnt));