For arm 32bit android build, the "__sync_synchronize" symbol is UNDEFINED in liblkl-hijack.so

[mxi1]

Have you ever tried the ARM 32bit build with arm-linux-androideabi-gcc (4.9.x, 5.x, 6.x, whatever)? I am trying to make hijacking work on Android shell.

After the CROSS_PATH, CROSS_COMPILE and SYS_ROOT environmental variables are set, and some endian-related macro redefinitions are removed, the build process looks nice and the liblkl-hijack.so file is generated. Then, I pushed the file to one root-ed Android device, and setup the LD_PRELOAD=/blah/blah/liblkl-hijack.so to run the ip addr show-like commands.

Here, the issues show up with symbols cannot located. Actually, there're more than one UNDEFINED symbols like dpdk and vde-related symbols. I removed all the dpdk and vde-related code blocks because they don't exist on my root-ed android device. Then, the confusing __sync_synchronize comes up.

I searched via Google, and knew that this symbol is gcc-internal, and should not appear in symbol table of the dynamic library. I tried to remove the --nodefaultlibs option, and this symbol is not UDNEFINED, but the segment fault error will show up when hijack is working.

Has anyone met the similar issue on any ARM 32bit platforms? Please give me your hands for this. Thank you!

[thehajime]

I also faced atomic ops symbols issue which I tried to solve by #273. not upstreamed yet but it would be great if you can try the patch if it works or not.

[mxi1]

@thehajime Thanks for your help.
I added "-march=armv8-a" option to the CFLAGS and LDFLAGS options, and the undefined "__sync_synchronize" is gone.
However, there're still 4 undefined symbols for the build, all of which are related with atomic ops.

They are:

00000000         *UND*	00000000              __sync_fetch_and_or_4
00000000         *UND*	00000000              __sync_fetch_and_and_4
00000000         *UND*	00000000              __sync_fetch_and_add_4
00000000         *UND*	00000000              __sync_fetch_and_sub_4

From the pull request #273 , I can see some similar ops are defined for ARMEL arch. However, all of them are not with "_4" ending.

What should we can do about these symbols? Could you give some advices on it?
Thank you!

[mxi1]

@thehajime Here is one solution mentioned in one old (about 5 years ago) blog: How to solve __sync_add_and_fetch_4, but we need to rebuild the compiler, which is more complicated for me.

[thehajime]

@mxi1 the basic idea of #273 is to give us a chance to provide a single interface to different atomic ops implementations. currently only two defines (ARMEL and other: I assumed x86_64) are included but you can add any ifdefs if it is appropriated.

[mxi1]

@thehajime By adding "-march=armv8-a" to both arch/lkl/Makeifle and tools/lkl/Makefile, these __sync_xx -related issues has gone.
However, when I am trying to hijack with LD_PRELOAD=, the segmentation fault shows up. The only message comes from dmesg, like:

115760.588470s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]iperf3[938]: unhandled level 2 translation fault (11) at 0xf7b4ce28, esr 0x92000006
[115760.588500s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]pgd = ffffffc08b0f6000
[115760.588531s][2017:05:16 20:36:26][pid:938,cpu0,iperf3][f7b4ce28] *pgd=000000008c95f003, *pmd=0000000000000000
[115760.588623s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]
[115760.588653s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]CPU: 0 PID: 938 Comm: iperf3 Tainted: G        W    3.10.90-gda247c4 #1
[115760.588653s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]task: ffffffc0944f1700 ti: ffffffc0d57f8000 task.ti: ffffffc0d57f8000

I can track to lkl_start_kernel(...), but cannot gather more information further. For the Unhandled level 2 translation fault (11), I can only find this mail thread and this.

BTW, for comparison, the lkl hijacking for aarch64 works fine for me.

[thehajime]

@thehajime By adding "-march=armv8-a" to both arch/lkl/Makeifle and tools/lkl/Makefile, these __sync_xx -related issues has gone.
However, when I am trying to hijack with LD_PRELOAD=, the segmentation fault shows up. The only message comes from dmesg, like:

115760.588470s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]iperf3[938]: unhandled level 2 translation fault (11) at 0xf7b4ce28, esr 0x92000006
[115760.588500s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]pgd = ffffffc08b0f6000
[115760.588531s][2017:05:16 20:36:26][pid:938,cpu0,iperf3][f7b4ce28] *pgd=000000008c95f003, *pmd=0000000000000000
[115760.588623s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]
[115760.588653s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]CPU: 0 PID: 938 Comm: iperf3 Tainted: G W 3.10.90-gda247c4 #1
[115760.588653s][2017:05:16 20:36:26][pid:938,cpu0,iperf3]task: ffffffc0944f1700 ti: ffffffc0d57f8000 task.ti: ffffffc0d57f8000
I can track to lkl_start_kernel(...), but cannot gather more information further. For the Unhandled level 2 translation fault (11), I can only find this mail thread.

a stack trace with gdb (from core dump if you have) will be helpful to see what's wrong with.
another way to debug is setting LKL_HIJACK_DEBUG=1 environment variable to generate more messages.

does make test under tools/lkl directory successfully work ?
this will be clarify if hijack library behaves wrong, or lkl itself.

BTW, for comparison, the lkl hijacking for aarch64 works fine for me.

That's a great news.

[mxi1]

@thehajime

a stack trace with gdb (from core dump if you have) will be helpful to see what's wrong with.
I got one strace but it doesn't work well. I can try others if I can get another one.
another way to debug is setting LKL_HIJACK_DEBUG=1 environment variable to generate more messages.

Yes, I enabled LKL_HIJACK_DEBUG, but couldn't get helpful information.
Because it crashes in very-early stage in lkl_start_kernel, might be when memory operations, thread operations or atomic operations are called.

does make test under tools/lkl directory successfully work ?

No. boot can be generated but could not run under Android shell, which says only Position independent executables (PIE) are supported.
Compiling net-test.c fails because of incomplete type sturct icmphdr.

this will be clarify if hijack library behaves wrong, or lkl itself.

[mxi1]

@thehajime Here is the final parts of strace output.

Anything useful for debug? Thanks.

pipe2([4, 5], 0)                        = 0
fcntl64(4, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
futex(0xf6a2af24, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap2(NULL, 1044480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xf6741000
madvise(0xf6741000, 1044480, MADV_MERGEABLE) = 0
mprotect(0xf6741000, 4096, PROT_NONE)   = 0
clone(child_stack=0xf683f928, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xf683f938, tls=0xf683f978,
child_tidptr=0xf683f938) = 18931
mmap2(NULL, 1044480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xf6642000
madvise(0xf6642000, 1044480, MADV_MERGEABLE) = 0
mprotect(0xf6642000, 4096, PROT_NONE)   = 0
clone(child_stack=0xf6740928, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xf6740938, tls=0xf6740978,
child_tidptr=0xf6740938) = 18932
futex(0xf6853008, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGSTOP {si_signo=SIGSTOP, si_code=SI_USER, si_pid=521, si_uid=0} ---
--- stopped by SIGSTOP ---
futex(0xf6853008, FUTEX_WAIT_PRIVATE, 0, NULL <ptrace(SYSCALL):No such process>
+++ killed by SIGSEGV +++
Segmentation fault

[thehajime]

@mxi1 thanks for the trace, but I wasn't able to imagine how it works bad..

maybe I think we can fix the build issue you faced only Position independent executables (PIE) are supported, and then see how the make test works to see.

If PIE build is a constraint on Android binary, we can add the linker flag (-pie) on the lkl build.

[mxi1]

@thehajime Thanks for your advice.
Here is the Makefile changes. The executables and the dynamic libraries should be separated here, because -pie is for executables, -shared is for dynamic libraries.

diff --git a/tools/lkl/Makefile b/tools/lkl/Makefile
index fd6c698b3524..8bb7eaeb83c3 100644
--- a/tools/lkl/Makefile
+++ b/tools/lkl/Makefile
@@ -133,9 +133,12 @@ $(OUTPUT)lib/lkl.o:
 $(OUTPUT)liblkl.a: $(OUTPUT)lib/lkl-in.o $(OUTPUT)lib/lkl.o
        $(QUIET_AR)$(AR) -rc $@ $^
 
-$(OUTPUT)liblkl$(SOSUF) $(OUTPUT)liblkl-hijack$(SOSUF) $(OUTPUT)lklfuse$(EXESUF) $(OUTPUT)fs2tar$(EXESUF) $(OUTPUT)cptofs$(EXESUF) $(OUTPUT)tests/boot $(OUTPUT)tests/net-test:
+$(OUTPUT)liblkl$(SOSUF) $(OUTPUT)liblkl-hijack$(SOSUF) $(OUTPUT)lklfuse$(EXESUF) $(OUTPUT)fs2tar$(EXESUF) $(OUTPUT)cptofs$(EXESUF):
        $(QUIET_LINK)$(CC) $(LDFLAGS) -o $@ $^ $(LDLIBS)
 
+$(OUTPUT)tests/boot $(OUTPUT)tests/net-test:
+       $(QUIET_LINK)$(CC) $(LDFLAGS) -o $@ $^ $(LDLIBS) -pie
+
 $(OUTPUT)cpfromfs$(EXESUF): cptofs$(EXESUF)
        $(Q)if ! [ -e $@ ]; then ln -s $< $@; fi

Now, I can run the 32bit boot command (without any parameters) under Android shell, the output looks like the following:

mutex                passed [1]
semaphore            passed [1]
join                 passed [joined -151783120]
Segmentation fault 

From the source code, the segmentation fault comes from lkl_start_kernel.
Any suggestions for this? Thank you!

[mxi1]

@thehajime In order to find where the code crashes, I added several checkpoints, and found the location:
This line in arch/lkl/kernel/setup.c

init_sem = lkl_ops->sem_alloc(0);

it will call the sem_alloc function defined in tools/lkl/lib/posix-host.c.

In sem_alloc(), all the code works fine, until the last line return sem;. It seems the return value triggers the unhandled level 2 translation fault.

[thehajime]

BTW, for comparison, the lkl hijacking for aarch64 works fine for me.

@mxi1 now I'm trying to test this on aarch64 but still struggling to build liblkl-hijack.so. which toolchain did you use when aarch64 hijack works fine ?

I'm right now using the tarball generated by make_standalone_toolchain.py --arch arm64, included in ndk version 15.0.4075724.

btw, please close this issue if you solved the original issue.

[mxi1]

@thehajime Okay, I think I can close this issue, and I am still preparing one patch for both arm32 and arm64 supports.

There's one bug in aarch64 android gcc toolchain 4.9, you need to use one higher-version toolchain, I remember version 5.4 should work, which I got from linaro git repo. Let me try to find the url for you from my home laptop

[mxi1]

@thehajime Use this one: https://android-git.linaro.org/platform/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-5.4-linaro.git/