Benchmark Output Verification Missing

[marshp7516]

Hello all,

We have been working with the Codeplay engineers ( @rafbiels ) using the Velocity Benchmarking suite to investigate the overhead of OneAPI. When looking at the performance, we want to make sure we are getting valid output before making our comparisons. A solid selection of benchmarks contain output verification, but during our investigation, we noticed the following benchmarks are unclear or do not contain output verification:

• SeisAcoMod2D
• DL-CIFAR
• DL-MNIST
• ETHMiner
• RTM
• Sobel Filter
• Voxelizer
• QuickSilver
• Bitcracker

We want to work with the maintainers of these benchmarks to learn about any current output verification, or if not, would we be able to have it implemented so we can be more confident in the benchmark results before making our comparisons.

Thanks!

[rmukhopa]

Hello @marshp7516, yes, output verification is not present in some workloads. The reasons vary. In some cases, workloads were originally taken from other open source repos and optimized. So they basically follow the original structure. In other cases, it might not be straight-forward to implement verification due to various reason and for other cases still, there might be some forms of verification/validation that can be activated through specific commands/options. We currently do not have the bandwidth to start working on this. But you are most welcome to submit PRs related to verification, and we will be happy to review them.