Critical - crash in multithreaded environment, when using nedrealloc (yes, again)

[Gerilgfx]

Critical - crash in multithreaded environment, when using nedrealloc (yes, again)

crash appears when nedrealloc being called on multiple threads, reallocating small (or null) memory area to larger buffers again and again. The crash occurs mostly before reaching the first percent in the test. If the algo able to reach that point, software mostly survives. To reproduce the crash, its good to have other processes working too, for example, watching hd yourube video in the front.

Crash type: memory corruption

Version affected: newest (older versions not yet tested)

compiler flag:

g++ nedmalloctester3.c -o nedmalloctester -O3 -s -lpthread -m64

compiler version:

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.7/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.7 --enable-ssp --disable-libssp --disable-libitm --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --program-suffix=-4.7 --enable-linux-futex --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux
Thread model: posix
gcc version 4.7.1 20120723 [gcc-4_7-branch revision 189773](SUSE Linux)

system version:

uname -r -a
Linux a1 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux

output:

g++ nedmalloctester3.c -o nedmalloctester -O3 -s -lpthread

./nedmalloctester

test 3 begins...
nedmalloc: nedprealloc() called with a block not created by nedmalloc!
Aborted

./nedmalloctester

test 3 begins...
0 percent finished
^C

./nedmalloctester

test 3 begins...
0 percent finished
^C

./nedmalloctester

test 3 begins...
0 percent finished
^C

./nedmalloctester

test 3 begins...
nedmalloc: nedprealloc() called with a block not created by nedmalloc!
Aborted

testcase:

// g++ nedmalloctester3.c -o nedmalloctester3 -O3 -s -pthread

include <stdio.h>

include <stdlib.h>

include <string.h>

include <pthread.h>

define USE_LOCKS 1

define USE_DL_PREFIX 1

define NDEBUG

define NO_NED_NAMESPACE

include "nedmalloc/nedmalloc_2013_apr/ori/nedmalloc.h"

include "nedmalloc/nedmalloc_2013_apr/ori/nedmalloc.c"

define malloc_vpool nedmalloc

define free_vpool nedfree

define realloc_vpool nedrealloc

/*#define malloc_vpool malloc

define free_vpool free

define realloc_vpool realloc*/

define TESTMEMMAX 1024_1024_2

void ** test=NULL;

int div_w=8; // block size to be sure that we touching pointers allocated from different thread ID-s

void malt(int thread){
for(int iteracio=1;iteracio<80;iteracio+=4){
for(int i=0;i<TESTMEMMAX;i++){
if(((i/div_w)%10)!=thread) continue; // 10 thread
// printf("%d\n", i);
test[i]=realloc_vpool(test[i], iteracio);
memset(test[i], 1, iteracio);
}
}
}

void *malt2(void * threadid){malt(1);}
void *malt3(void * threadid){malt(2);}
void *malt4(void * threadid){malt(3);}
void *malt5(void * threadid){malt(4);}
void *malt6(void * threadid){malt(5);}
void *malt7(void * threadid){malt(6);}
void *malt8(void * threadid){malt(7);}
void *malt9(void * threadid){malt(8);}
void *malt10(void * threadid){malt(9);}

void MallocStabTest3(){
printf("test 3 begins...\n");

test=(void**)malloc_vpool(128+(TESTMEMMAX*sizeof(void*)));
for(int i=0;i<(TESTMEMMAX);i++) test[i]=NULL;

for(int Z=0;Z<100;Z++){
    div_w=2+(rand()%40);  // random block size to be sure that we touching pointers allocated from different thread ID-s

    pthread_t TMP2=0;
    pthread_t TMP3=0;
    pthread_t TMP4=0;
    pthread_t TMP5=0;
    pthread_t TMP6=0;
    pthread_t TMP7=0;
    pthread_t TMP8=0;
    pthread_t TMP9=0;
    pthread_t TMP10=0;

    pthread_create(&TMP2, NULL, malt2, NULL);
    pthread_create(&TMP3, NULL, malt3, NULL);
    pthread_create(&TMP4, NULL, malt4, NULL);
    pthread_create(&TMP5, NULL, malt5, NULL);
    pthread_create(&TMP6, NULL, malt6, NULL);
    pthread_create(&TMP7, NULL, malt7, NULL);
    pthread_create(&TMP8, NULL, malt8, NULL);
    pthread_create(&TMP9, NULL, malt9, NULL);
    pthread_create(&TMP10, NULL, malt10, NULL);

    malt(0);

    pthread_join(TMP2, NULL);
    pthread_join(TMP3, NULL);
    pthread_join(TMP4, NULL);
    pthread_join(TMP5, NULL);
    pthread_join(TMP6, NULL);
    pthread_join(TMP7, NULL);
    pthread_join(TMP8, NULL);
    pthread_join(TMP9, NULL);
    pthread_join(TMP10, NULL);

    printf("%d percent finished\n", Z);
}

for(int i=0;i<(TESTMEMMAX);i++) if(test[i]) free_vpool(test[i]);
free_vpool(test);
printf("success.\n");

}

int main(){
MallocStabTest3();
}

[Gerilgfx]

previous versions from 2013 crashing too

[ned14]

The last time I put together some form of release was for v1.10 beta 3 in 2012. I agree that nedmalloc definitely needs a regularly executed stress test suite, and in fact I have recently purchased a server for a Jenkins CI which you can see at https://ci.nedprod.com/.

As it happens, I was fired from BlackBerry on Monday, so I suddenly have some free time. I'll look into figuring out some form of automated solution to the many breakages which have slipped into nedmalloc over the years by accident.

Thanks for reporting the bug Geri. You're a trooper.

Niall

[Gerilgfx]

thankyou for creating and supporting this wonderfull software. i suggest to create a stresstest based on my multithreaded testcases, like the current one, and those i posted before. they are simply enough, and its easy to debug them.

[Gerilgfx]

hi. was you able to repeat the crash?

[ned14]

It'll be a few days yet. I'm currently mentoring gsoc and I need to finish two work items to enable the student to proceed as he is waiting on me.

[Gerilgfx]

i did some test:
-both -m64 and -m32 crashing
-both with -o3 and without o3, crashing, both -s or/without s crashing
so i guess its an algorithmic bug

[ned14]

It is on my radar. Integrating the new items from http://boostafio.uservoice.com/forums/218980-boost-afio-feature-request before GSoC ends in ten days has proved harder than expected.

[Gerilgfx]

i wrapped nedrealloc to nedfree and nedmalloc functions in my code until the fix done, no need to hurry

[ned14]

I should be able to look into this now. Can you put the test case above, which is too mangled to make much sense, into a gist so I can get it demangled? Thanks.

[Gerilgfx]

https://gist.github.com/Gerilgfx/6953861 i hope it was this one.

[ned14]

Bad news: I can't replicate this on my Ubuntu 12.04 x64 machine with a i7-3770K CPU. I tried:

GCC v4.6.4
GCC v4.7.3
GCC v4.8.1

It could be a timing issue where your CPU finds a race mine doesn't. Or it could be a bug in GCC 4.7.1 which has since been fixed. There is some order sensitive code in the threadcache, a slight reordering from what is specified in the code would introduce exactly this kind of race. In theory a compiler shouldn't do such a reorder, but maybe there was a bug in GCC v4.7.1.

Niall

[ned14]

Also, try setting THREADCACHEMAX to 0. That will help me determine if it's dlmalloc or the thread cache which is at fault.

[Gerilgfx]

with:
#define THREADCACHEMAX 0

test 3 begins...
nedmalloc: nedprealloc() called with a block not created by nedmalloc!
Aborted

[Gerilgfx]

changing:

test[i]=realloc_vpool(test[i], iteracio);

to:

if(test[i]) free_vpool(test[i]);
test[i]=malloc_vpool(iteracio);

works.

i think the bug is precisely in your realloc implementation.