Prevent nested parallelism in HNSW bench

[julianmi]

Setting the gbench number of threads and the HNSWlib config number of threads can lead to nested parallelism. This patch proposes to either use throughput mode using multiple gbench threads or latency mode using batch parallelism. Additionally, there is a significant overhead in going through the thread pool. It is skipped in the search method to handle single query batch size efficiently.

[aamijar]

Hi @julianmi, what is the UX for using multiple threads in HSNW bench? Does the user set the gbench threads parameter, or the num_threads_ parameter?

[achirkin]

To answer @aamijar

In the latency mode, gbench measures how long does it take to execute a single search call for the given algorithm and batch size. In this mode, gbench is always single-threaded. To make the use of the whole CPU, HNSW has its own threading logic. This makes the HNSW measures more realistic and fair against GPU algorithms.

In the throughput mode, gbench measures how many requests can the given algorithm serve per second. Thus, gbench provides independent threads to do the search calls. This clashes with the internal HNSW threading. Because gbench creates its threads and manages batching outside the measured benchmark loop, the performance of HNSW generally looks better with gbench threads than with the internal threads. Hence we just disable internal batching completely in the throughput mode.