Openwrt support

[wen-long]

Thanks for your work! it's wonderful
Do you hava any plan to make this work on openwrt?

[hack3ric]

Hi, I'm definitely interested in creating OpenWrt packages for Mimic!

However, Mimic currently requires at least Linux 6.1 (due to usage of ringbuf dynptr), while OpenWrt 23.05 uses 5.15. It may be feasible to retrofit bpf_perf_event_output or add an option to complete disable its usage, since it is only used to cache packets before handshake and resend them afterwards.

Even this is resolved, availability on MIPS and 32-bit Arm still needs to be tested. I don't have any spare router to test now but it's definitely worth trying.

[wen-long]

how about drop packets before handshake?
a small and fast working demo can tell the performance on normal openwrt hareware
after then maybe more developer will come to help

I have a router(GL-AXT1800) to try this, but dont have any idea to start a kernel module dev
do you like to give some suggest?

~# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 1
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 2
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 3
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

Hardware	: Qualcomm Technologies, Inc IPQ6000
Revision	: 0000
Serial		: 0000000000000000
~# cat /proc/version
Linux version 4.4.60 (glinet@glinet) (gcc version 5.5.0 (OpenWrt GCC 5.5.0 r16399+165-c67509efd7) ) #0 SMP PREEMPT Tue Jan 23 12:33:08 2024

[hack3ric]

how about drop packets before handshake?
a small and fast working demo can tell the performance on normal openwrt hareware
after then maybe more developer will come to help

Sure, I'll add build flags to disable that part first.

I have a router(GL-AXT1800) to try this, but dont have any idea to start a kernel module dev
do you like to give some suggest?

I'm trying OpenWrt SDK today and I plan to post build scripts shortly after. You can then add the package feed to OpenWrt SDK and install the produced .ipk file directly to test it. Thanks in advance!

[hack3ric]

Just noticed CONFIG_DEBUG_INFO_BTF isn't enabled by default on OpenWrt, so one may need to rebuild the base image in order to get Mimic running on wireless routers...

[hack3ric]

Seems BTF kfuncs support is only added in around 5.18~6.0, so we're out of luck with OpenWrt 23.05's 5.15 kernel. Building on main should be fine, but we need to wait for their new major release with 6.1/6.6 kernel to be stable.

[wen-long]

GL-AXT1800 is not officaly supported by openwrt, and no IPQ6000 under make menuconfig -> Target System
GLinet stop open their repo publicly, I've send mail to ask for permission as the thread mentioned https://forum.gl-inet.com/t/32901/2
or maybe start from openwrt x86 is also a chooice?

[hack3ric]

Built Mimic against snapshot OpenWrt armsr/armv7 with BTF on, sadly 32-bit Arm's BPF isn't capable of calling kernel function even on Linux 6.6.

root@OpenWrt:/# modprobe mimic
[ 2865.893256] mimic: loading out-of-tree module taints kernel.
root@OpenWrt:/# mimic run lo
 Warn libbpf: prog 'egress_handler': BPF program load failed: No error information
 Warn libbpf: prog 'egress_handler': -- BEGIN PROG LOAD LOG --
JIT does not support calling kernel function
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
 Warn libbpf: prog 'egress_handler': failed to load: -524
 Warn libbpf: failed to load object 'mimic_bpf'
 Warn libbpf: failed to load BPF skeleton 'mimic_bpf': -524
Error failed to load BPF program: No error information
 Info cleaning up
root@OpenWrt:/# uname -a
Linux OpenWrt 6.6.41 #0 SMP Fri Jul 19 14:53:24 2024 armv7l GNU/Linux
root@OpenWrt:/# 

or maybe start from openwrt x86 is also a chooice?

Mimic should run fine on x86_64 and aarch64. Will push build script and try OpenWrt on more platforms using QEMU later.

[hack3ric]

Encountered some issues in building kmod-sched-bpf in OpenWrt, which is required by Mimic for its TC egress path. Reported to openwrt/openwrt#16071.

Other than this, Mimic should be able to run on OpenWrt main branch.

[hack3ric]

I've create a OpenWrt package feed for Mimic: https://github.com/hack3ric/mimic/tree/openwrt

After some hacking, kernel BTF and dynptr dependency is disabled in OpenWrt package script. However, on 23.05 it still fails with "Invalid argument" and no reason. I guess it is from the verifier.

It works on snapshot build and should run fine on all (little-endian) platforms, including 32-bit ARM, though.

[wen-long]

make[4]: Entering directory '/home/xx/openwrt/build_dir/target-x86_64_musl/linux-x86_64/bpf-headers/linux-5.15.161'
/bin/sh: 0: Illegal option -W
: unknown compiler
: unknown assembler invoked
scripts/Kconfig.include:50: Sorry, this assembler is not supported.
make[5]: *** [scripts/kconfig/Makefile:77: oldconfig] Error 1
make[4]: *** [Makefile:628: oldconfig] Error 2

not lucky for me, waste too much time on these build script shit, give up now.
hope you keep work on this, maybe one can enjoy mimic outofbox someday

[hack3ric]

Now that OpenWrt 24.10 has released, and I've successfully built Mimic for 24.10 and SNAPSHOT with GitHub Actions: https://github.com/hack3ric/mimic/actions/runs/13183413756. Testing would be appreciated.

Some notes about testing Mimic on OpenWrt: You don't have to install kmod-mimic for Mimic to run properly, as most NIC on commodity routers (MediaTek, Realtek, Qualcomm) don't need checksum hacks (as I've described in https://github.com/hack3ric/mimic/blob/master/docs/openwrt.md). Just install the package in each zip file's packages/ directory and it will be fine.

Init scripts and more documentation still need to be done before releasing Mimic with OpenWrt support.

[wen-long]

没订阅所以最近才了解到进展.
我按上面的指南编译了固件,用的openwrt源码是link,编译过了,内核模块也能加载, mimic 运行时遇到了错误
报错很长, 见 log

 Warn libbpf: prog 'egress_handler': BPF program load failed: Permission denied
 Warn libbpf: prog 'egress_handler': -- BEGIN PROG LOAD LOG --
...
R4 invalid zero-sized read: u64=[0,15]
processed 721 insns (limit 1000000) max_states_per_insn 1 total_states 44 peak_states 44 mark_read 18
-- END PROG LOAD LOG --
 Warn libbpf: prog 'egress_handler': failed to load: -13
 Warn libbpf: failed to load object 'mimic_bpf'
 Warn libbpf: failed to load BPF skeleton 'mimic_bpf': -13
Error failed to load BPF program: Permission denied

搜了下, BPF 程序本身有问题会导致类似的报错 相关资料

[hack3ric]

这种报错和编译器/内核版本关系挺大的，比如今天刚有个 PR 修了 clang 20 + linux 6.15 上的类似 verifier 报错。LLVM 侧可能因为后端改了一些优化使得生成的 eBPF 指令不同，Linux 内核的每个版本 verifier 逻辑也会有不同（不过大体上都是在保证正确性的情况下 relax 一些不必要的限制，问题没有 clang 的大）。看了一下 log，那个 PR 的环境下也没有出现 OpenWrt 环境下你遇到的这种报错。

我有时间修一下这个问题（希望我能复现），可能还得加一下不同内核和编译器版本的 regression test

[wen-long]

openwrt 成功用上了
之前错误的在 make 时使用 CLANG=/usr/bin/clang, 调用了 host 的低版本 clang
看了文档, 现在 debian13上 安装 clang-19, 编译出来的就没问题了,成功运行
一开始会报几个sendto Operation not permitted, 原因是 openwrt 内置的防火墙规则拒绝了 conntrack invalid 的包
修改 /etc/config/firewall
在 wan zone 下添加 option masq_allow_invalid '1' 就可以了

有时候 iperf3 tcp 测速几乎没有速度, udp 正常. ethtool -K eth0 gro off 可以解决

有时候 openwrt 连接时,服务端每个回包都会接一个 1 字节的 tcp 段, 会影响性能, 用 amd64 客户端连接没发现这个问题,待查