freebsd: make check bench_bitcoin illegal instruction

[grubles]

OS: FreeBSD 14.2
Tag: Elements-23.2.4
Arch: x86_64

The bitcoind v23.2 bench_bitcoin runs fine.

% ./src/bench/bench_bitcoin 

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|       33,918,912.00 |               29.48 |    0.4% |      0.32 | `AddrManAdd`
|       41,379,012.00 |               24.17 |    0.2% |      0.46 | `AddrManAddThenGood`
|          143,318.71 |            6,977.46 |    0.2% |      0.01 | `AddrManGetAddr`
|              148.36 |        6,740,300.72 |    0.5% |      0.01 | `AddrManSelect`
|          349,355.33 |            2,862.42 |    0.2% |      0.01 | `AssembleBlock`

|             ns/byte |              byte/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|               68.69 |       14,557,263.45 |    0.7% |      0.01 | `Base58CheckEncode`
|               15.00 |       66,656,073.03 |    0.3% |      0.01 | `Base58Decode`
|               50.84 |       19,670,034.41 |    0.3% |      0.01 | `Base58Encode`
|                7.09 |      141,086,434.00 |    0.1% |      0.01 | `Bech32Decode`
|               15.34 |       65,201,841.52 |    0.2% |      0.01 | `Bech32Encode`

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|              134.03 |        7,461,269.48 |    0.8% |      0.01 | `BenchLockedPool`
|               28.87 |       34,637,621.14 |    0.0% |      0.01 | `BenchTimeDeprecated`
|               32.33 |       30,929,578.06 |    0.1% |      0.01 | `BenchTimeMillis`
|               32.31 |       30,953,487.11 |    0.2% |      0.01 | `BenchTimeMillisSys`
|                2.30 |      433,852,393.89 |    0.0% |      0.01 | `BenchTimeMock`
|       53,562,096.00 |               18.67 |    0.2% |      0.59 | `BlockToJsonVerbose`
|       33,862,387.00 |               29.53 |    0.3% |      0.38 | `BlockToJsonVerboseWrite`
|          677,365.50 |            1,476.31 |    0.1% |      0.01 | `BnBExhaustion`
Illegal instruction (core dumped)

[delta1]

does this also occur in 22.1.1 ?

[grubles]

22.1.1 doesn't seem to want to build:

gmake[3]: Leaving directory '/home/user/builds/elements'
  CXX      wallet/libbitcoin_wallet_a-db.o
wallet/db.cpp:22:20: error: no member named 'no_push' in 'boost::filesystem::recursive_directory_iterator'
   22 |                 it.no_push();
      |                 ~~ ^
wallet/db.cpp:39:27: error: no member named 'level' in 'boost::filesystem::recursive_directory_iterator'
   39 |             } else if (it.level() == 0 && it->symlink_status().type() == fs::regular_file && IsBDBFile(it->path())) {
      |                        ~~ ^
wallet/db.cpp:54:16: error: no member named 'no_push' in 'boost::filesystem::recursive_directory_iterator'
   54 |             it.no_push();
      |             ~~ ^
3 errors generated.
gmake[2]: *** [Makefile:10853: wallet/libbitcoin_wallet_a-db.o] Error 1
gmake[2]: Leaving directory '/home/user/builds/elements/src'
gmake[1]: *** [Makefile:18615: all-recursive] Error 1
gmake[1]: Leaving directory '/home/user/builds/elements/src'
gmake: *** [Makefile:838: all-recursive] Error 1

[apoelstra]

You need to use boost 1.75 for 22.x. In Nix you can do this with boost = nixpkgs.boost175;. On Arch IIRC I would spin up an old Debian VM :P.

[grubles]

Got it to build with --disable-wallet. Looks like it may be crashing at the same spot?

% ./src/bench/bench_bitcoin 

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|       16,591,847.00 |               60.27 |    0.3% |      0.26 | `AddrManAdd`
|          174,358.00 |            5,735.33 |    0.3% |      0.00 | `AddrManGetAddr`
|      121,458,287.00 |                8.23 |    3.7% |      0.61 | `AddrManGood`
|              163.28 |        6,124,431.80 |    0.7% |      0.00 | `AddrManSelect`
|          344,166.00 |            2,905.57 |    0.8% |      0.00 | `AssembleBlock`

|             ns/byte |              byte/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|               67.57 |       14,799,154.33 |    0.3% |      0.00 | `Base58CheckEncode`
|               14.94 |       66,946,376.65 |    0.7% |      0.00 | `Base58Decode`
|               50.72 |       19,715,838.02 |    0.5% |      0.00 | `Base58Encode`
|                7.44 |      134,468,374.26 |    1.7% |      0.00 | `Bech32Decode`
|               15.71 |       63,665,606.81 |    0.9% |      0.00 | `Bech32Encode`

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|              136.73 |        7,313,505.61 |    6.7% |      0.00 | :wavy_dash: `BenchLockedPool` (Unstable with ~434.0 iters. Increase `minEpochIterations` to e.g. 4340)
|               28.87 |       34,633,484.32 |    0.0% |      0.00 | `BenchTimeDeprecated`
|               32.31 |       30,950,946.93 |    0.0% |      0.00 | `BenchTimeMillis`
|               32.31 |       30,951,971.08 |    0.0% |      0.00 | `BenchTimeMillisSys`
|                2.06 |      484,928,442.87 |    0.0% |      0.00 | `BenchTimeMock`
|       43,577,664.00 |               22.95 |    0.2% |      0.48 | `BlockToJsonVerbose`
|       32,694,395.00 |               30.59 |    0.4% |      0.36 | `BlockToJsonVerboseWrite`
Illegal instruction (core dumped)

@apoelstra well my intention is to see how elementsd runs on FreeBSD. But yeah 14.2 seems to have boost 1.85 which I guess is too new for 22.x.

[apoelstra]

If "how it runs on X" means "how it runs with the default version of all dependencies provided by X" then it pretty-much doesn't run anywhere :).

[grubles]

I hear you. FWIW that's the only boost-libs package available for FBSD 14.x, unless it's possible to finesse ports to build an older one. I'll try to see if I can get 1.75 on this machine.

[apoelstra]

Actually let me double-check on this. It looks like elements 0.21 worked for me with boost 1.81 (and does not work with 1.75).

[grubles]

Ok got a backtrace with 23.2.4 but no idea if it's helpful:

|          680,709.00 |            1,469.06 |    0.8% |      0.01 | `BnBExhaustion`
Process 52928 stopped
* thread #1, name = 'b-test', stop reason = signal SIGILL: privileged opcode
    frame #0: 0x0000019c3b1b3f6d bench_bitcoin`CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) [inlined] std::__1::vector<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*, std::__1::allocator<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*>>::operator[][abi:se180100](this=<unavailable>, __n=<unavailable>) at vector:1393:3
   1390 template <class _Tp, class _Allocator>
   1391 _LIBCPP_CONSTEXPR_SINCE_CXX20 inline _LIBCPP_HIDE_FROM_ABI typename vector<_Tp, _Allocator>::reference
   1392 vector<_Tp, _Allocator>::operator[](size_type __n) _NOEXCEPT {
-> 1393   _LIBCPP_ASSERT_VALID_ELEMENT_ACCESS(__n < size(), "vector[] index out of bounds");
   1394   return this->__begin_[__n];
   1395 }
   1396
(lldb) bt
* thread #1, name = 'b-test', stop reason = signal SIGILL: privileged opcode
  * frame #0: 0x0000019c3b1b3f6d bench_bitcoin`CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) [inlined] std::__1::vector<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*, std::__1::allocator<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*>>::operator[][abi:se180100](this=<unavailable>, __n=<unavailable>) at vector:1393:3
    frame #1: 0x0000019c3b1b3f6d bench_bitcoin`CCheckQueueSpeedPrevectorJob(bench=0x000001a45bae3668) at checkqueue.cpp:0
    frame #2: 0x0000019c3b19a0e8 bench_bitcoin`benchmark::BenchRunner::RunAll(benchmark::Args const&) [inlined] std::__1::__function::__value_func<void (ankerl::nanobench::Bench&)>::operator()[abi:se180100](this=0x000001f6bf81b350, __args=0x000001a45bae3668) const at function.h:428:12
    frame #3: 0x0000019c3b19a0cb bench_bitcoin`benchmark::BenchRunner::RunAll(benchmark::Args const&) [inlined] std::__1::function<void (ankerl::nanobench::Bench&)>::operator()(this= Function = CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) , __arg=0x000001a45bae3668) const at function.h:981:10
    frame #4: 0x0000019c3b19a0cb bench_bitcoin`benchmark::BenchRunner::RunAll(args=0x000001a45bae3850) at bench.cpp:80:13
    frame #5: 0x0000019c3b196835 bench_bitcoin`main(argc=<unavailable>, argv=<unavailable>) at bench_bitcoin.cpp:116:5
    frame #6: 0x000001a460bdcc3a libc.so.7`__libc_start1 + 298
    frame #7: 0x0000019c3b19302d bench_bitcoin`_start at crt1_s.S:83

[delta1]

what about building with the vendored deps in depends ?

[grubles]

Doesn't look like the 22.1.1 depends supports FreeBSD, though it appears support was added more recently.

Invalid configuration `freebsd': machine `freebsd-unknown' not recognized

[apoelstra]

re dependencies, I was able to build elements 22.1.1 with clang and

db 4.8.30
boost 1.75.0
zlib 1.3.1
zeromq 4.3.5
miniupnpc 2.2.7
protobuf 25.4
libevent 2.1.12

But as I said above, I believe boost 1.81 would've worked fine (and maybe even later versions). miniupnpc 2.2.8 would not have worked. And your compilation error looked like it was related to db.

Not sure what to do about the arch being explicitly unsupported tho.

[grubles]

The db related build error seems to be a separate issue? With both elements-23.2.4 and 22.1.1 bench_bitcoin seems to crash at the same point.

Here is an lldb backtrace for 22.1.1.

|       32,956,597.00 |               30.34 |    0.2% |      0.36 | `BlockToJsonVerboseWrite`
Process 97505 stopped
* thread #1, name = 'b-test', stop reason = signal SIGILL: privileged opcode
    frame #0: 0x000000bf2e40a87d bench_bitcoin`CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) [inlined] std::__1::vector<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*, std::__1::allocator<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*>>::operator[][abi:se180100](this=<unavailable>, __n=<unavailable>) at vector:1393:3
   1390 template <class _Tp, class _Allocator>
   1391 _LIBCPP_CONSTEXPR_SINCE_CXX20 inline _LIBCPP_HIDE_FROM_ABI typename vector<_Tp, _Allocator>::reference
   1392 vector<_Tp, _Allocator>::operator[](size_type __n) _NOEXCEPT {
-> 1393   _LIBCPP_ASSERT_VALID_ELEMENT_ACCESS(__n < size(), "vector[] index out of bounds");
   1394   return this->__begin_[__n];
   1395 }
   1396
(lldb) bt
* thread #1, name = 'b-test', stop reason = signal SIGILL: privileged opcode
  * frame #0: 0x000000bf2e40a87d bench_bitcoin`CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) [inlined] std::__1::vector<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*, std::__1::allocator<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*>>::operator[][abi:se180100](this=<unavailable>, __n=<unavailable>) at vector:1393:3
    frame #1: 0x000000bf2e40a87d bench_bitcoin`CCheckQueueSpeedPrevectorJob(bench=0x000000c74f0d6ee8) at checkqueue.cpp:0
    frame #2: 0x000000bf2e3f0ba1 bench_bitcoin`benchmark::BenchRunner::RunAll(benchmark::Args const&) [inlined] std::__1::__function::__value_func<void (ankerl::nanobench::Bench&)>::operator()[abi:se180100](this=0x000019966401b350, __args=0x000000c74f0d6ee8) const at function.h:428:12
    frame #3: 0x000000bf2e3f0b8b bench_bitcoin`benchmark::BenchRunner::RunAll(benchmark::Args const&) [inlined] std::__1::function<void (ankerl::nanobench::Bench&)>::operator()(this= Function = CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&) , __arg=0x000000c74f0d6ee8) const at function.h:981:10
    frame #4: 0x000000bf2e3f0b8b bench_bitcoin`benchmark::BenchRunner::RunAll(args=0x000000c74f0d70f0) at bench.cpp:65:13
    frame #5: 0x000000bf2e3ef9f4 bench_bitcoin`main(argc=<unavailable>, argv=<unavailable>) at bench_bitcoin.cpp:63:5
    frame #6: 0x000000c756d56c3a libc.so.7`__libc_start1 + 298
    frame #7: 0x000000bf2e3e902d bench_bitcoin`_start at crt1_s.S:83

[grubles]

Maybe related but I also hit this running on ppc64le linux and elements @ 811d835.

$ ./src/bench/bench_bitcoin 
Warning, results might be unstable:
* CPU frequency scaling enabled: CPU 0 between 2,166.0 and 3,800.0 MHz
* CPU governor is 'schedutil' but should be 'performance'
* Turbo is enabled, CPU frequency will fluctuate

Recommendations
* Use 'pyperf system tune' before benchmarking. See https://github.com/psf/pyperf

|               ns/op |                op/s |    err% |          ins/op |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|---------------:|--------:|----------:|:----------
|       67,675,319.00 |               14.78 |    0.1% |            0.00 |           0.00 |    0.0% |      0.75 | `AddrManAdd`
|      172,938,634.00 |                5.78 |    0.1% |            0.00 |           0.00 |    0.0% |      1.90 | `AddrManAddThenGood`
|          265,687.00 |            3,763.83 |    0.3% |            0.00 |           0.00 |    0.0% |      0.01 | `AddrManGetAddr`
|              171.89 |        5,817,540.77 |    0.2% |            0.00 |           0.00 |    0.0% |      0.01 | `AddrManSelect`
|        1,139,993.00 |              877.20 |    0.3% |            0.00 |           0.00 |    0.0% |      0.01 | `AssembleBlock`

|             ns/byte |              byte/s |    err% |        ins/byte |       bra/byte |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|---------------:|--------:|----------:|:----------
|              143.68 |        6,959,997.46 |    0.2% |            0.00 |           0.00 |    0.0% |      0.01 | `Base58CheckEncode`
|               32.82 |       30,471,320.18 |    0.5% |            0.00 |           0.00 |    0.0% |      0.01 | `Base58Decode`
|               96.66 |       10,345,367.57 |    0.2% |            0.00 |           0.00 |    0.0% |      0.01 | `Base58Encode`
|               11.68 |       85,623,387.79 |    0.2% |            0.00 |           0.00 |    0.0% |      0.01 | `Bech32Decode`
|               31.11 |       32,145,785.66 |    0.1% |            0.00 |           0.00 |    0.0% |      0.01 | `Bech32Encode`

|               ns/op |                op/s |    err% |          ins/op |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|---------------:|--------:|----------:|:----------
|              217.31 |        4,601,776.70 |    0.8% |            0.00 |           0.00 |    0.0% |      0.01 | `BenchLockedPool`
|               14.59 |       68,528,355.06 |    0.1% |            0.00 |           0.00 |    0.0% |      0.01 | `BenchTimeDeprecated`
|               49.09 |       20,369,186.14 |    0.1% |            0.00 |           0.00 |    0.0% |      0.01 | `BenchTimeMillis`
|               48.54 |       20,603,644.43 |    0.0% |            0.00 |           0.00 |    0.0% |      0.01 | `BenchTimeMillisSys`
|                2.31 |      431,978,466.60 |    0.2% |            0.00 |           0.00 |    0.0% |      0.01 | `BenchTimeMock`
|      159,718,570.00 |                6.26 |    0.1% |            0.00 |           0.00 |    0.0% |      1.77 | `BlockToJsonVerbose`
|       52,024,438.00 |               19.22 |    0.1% |            0.00 |           0.00 |    0.0% |      0.59 | `BlockToJsonVerboseWrite`
/usr/lib/gcc/powerpc64le-unknown-linux-gnu/14/include/g++-v14/bits/stl_vector.h:1130: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*; _Alloc = std::allocator<CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*>; reference = CCheckQueueSpeedPrevectorJob(ankerl::nanobench::Bench&)::PrevectorJob*&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
Aborted