[Bug] Memory error in libsixel

[ShangzhiXu]

Hi team, thanks for your time reviewing this. I found several bugs in the system through fuzzing
Let me explain

---

1. GIF Transparent Color Index Heap Buffer Overflow

PoC: https://github.com/ShangzhiXu/PoC/blob/main/libsixel/poc1_gif_oob.gif

cmd: ./converters/img2sixel -B '#000000' poc1_gif_oob.gif

Root Cause

The allocation at src/fromgif.c:195-198 sizes the palette based on the GIF color table size, but the write at
src/fromgif.c:241-243 indexes the palette with pg->transparent read from the file at src/fromgif.c:528 without bounds
checking. When the GIF has a 2-color table (ncolors=2, 6-byte palette) but transparent=200, the write hits offset
600 in a 6-byte buffer.

  src/fromgif.c:195-198 (allocation):
  ncolors = 2 << (((pg->lflags & 0x80) ? pg->lflags : pg->flags) & 7);
  palette_size = (size_t)ncolors * 3;
  if (frame->palette == NULL) {
      frame->palette = (unsigned char *)sixel_allocator_malloc(frame->allocator, palette_size);

  src/fromgif.c:526-528 (transparent index from file, unchecked):
  g->eflags = gif_get8(s);
  g->delay = gif_get16le(s); /* delay */
  g->transparent = gif_get8(s);

  src/fromgif.c:241-243 (OOB write, pg->transparent=200 vs 6-byte palette):
  frame->palette[pg->transparent * 3 + 0] = bgcolor[0];
  frame->palette[pg->transparent * 3 + 1] = bgcolor[1];
  frame->palette[pg->transparent * 3 + 2] = bgcolor[2];

Crash Stack

  ==2770952==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000308 at pc 0x5620d308cc70 bp
  0x7ffc6e7a7eb0 sp 0x7ffc6e7a7ea8
  WRITE of size 1 at 0x602000000308 thread T0
      #0 0x5620d308cc6f in gif_init_frame /srv/scratch/z5500277/target/libsixel/src/fromgif.c:241:61
      #1 0x5620d308cc6f in load_gif /srv/scratch/z5500277/target/libsixel/src/fromgif.c:675:22
      #2 0x5620d302863e in load_with_builtin /srv/scratch/z5500277/target/libsixel/src/loader.c:948:18
      #3 0x5620d302863e in sixel_helper_load_image_file /srv/scratch/z5500277/target/libsixel/src/loader.c:1462:18
      #4 0x5620d2fe24b2 in sixel_encoder_encode /srv/scratch/z5500277/target/libsixel/src/encoder.c:1816:14
      #5 0x5620d2fd7996 in main /srv/scratch/z5500277/target/libsixel/converters/img2sixel.c:455:22
      #6 0x7f276b076864 in __libc_start_main (/lib64/libc.so.6+0x3a864)
      #7 0x5620d2f02d0d in _start (/srv/scratch/z5500277/target/libsixel/converters/img2sixel+0xb4d0d)

  0x602000000308 is located 567 bytes after 1-byte region [0x6020000000d0,0x6020000000d1)
  allocated by thread T0 here:
      #0 0x5620d2f9b85e in malloc
      #1 0x5620d3088a72 in gif_init_frame /srv/scratch/z5500277/target/libsixel/src/fromgif.c:214:42
      #2 0x5620d3088a72 in load_gif /srv/scratch/z5500277/target/libsixel/src/fromgif.c:675:22
      #3 0x5620d302863e in load_with_builtin /srv/scratch/z5500277/target/libsixel/src/loader.c:948:18
      #4 0x5620d302863e in sixel_helper_load_image_file /srv/scratch/z5500277/target/libsixel/src/loader.c:1462:18
      #5 0x5620d2fe24b2 in sixel_encoder_encode /srv/scratch/z5500277/target/libsixel/src/encoder.c:1816:14
      #6 0x5620d2fd7996 in main /srv/scratch/z5500277/target/libsixel/converters/img2sixel.c:455:22

---

2. SIXEL DCS Parameter Signed Integer Overflow

PoC: https://github.com/ShangzhiXu/PoC/blob/main/libsixel/crash_w0_23_runtime_error_1772605651.sixel

cmd: ./converters/sixel2png -i crash_w0_23_runtime_error_1772605651.sixel -o /tmp/out.png

Root Cause

The DCS parameter Pn3 (context->params[2]) is parsed via safe_addition_for_params at src/fromsixel.c:482 which allows
values up to INT_MAX. When the DCS 'q' command is processed, lines 538-539 multiply attributed_pan / attributed_pad
by this unchecked Pn3 without overflow protection. With default attributed_pan=2 (initialized at src/fromsixel.c:367)
and params[2]=2147483647, the multiplication 2 * 2147483647 overflows signed int.

  src/fromsixel.c:367 (init, attributed_pan defaults to 2):
  context->attributed_pan = 2;
  context->attributed_pad = 1;

  src/fromsixel.c:482 (Pn3 parsed from file, can reach INT_MAX):
  status = safe_addition_for_params(context, p);

  src/fromsixel.c:535-539 (unchecked multiplication overflows):
  if (context->params[2] == 0) {
      context->params[2] = 10;
  }
  context->attributed_pan = context->attributed_pan * context->params[2] / 10;
  context->attributed_pad = context->attributed_pad * context->params[2] / 10;

Crash Stack

  fromsixel.c:538:71: runtime error: signed integer overflow: 2 * 2147483647 cannot be represented in type 'int'
      #0 0x55e90f561b63 in sixel_decode_raw_impl /home/shangzhi/fuz/libsixel/src/fromsixel.c:538
      #1 0x55e90f566cff in sixel_decode_raw /home/shangzhi/fuz/libsixel/src/fromsixel.c:943
      #2 0x55e90f55c656 in sixel_decoder_decode /home/shangzhi/fuz/libsixel/src/decoder.c:280
      #3 0x55e90f55ae7a in main /home/shangzhi/fuz/libsixel/converters/sixel2png.c:226
      #4 0x7f25de029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
      #5 0x7f25de029e3f in __libc_start_main_impl ../csu/libc-start.c:392
      #6 0x55e90f55a6b4 in _start (/home/shangzhi/fuz/libsixel/converters/sixel2png+0x1a6b4)

  fromsixel.c:539:71: runtime error: signed integer overflow: 2147483647 * 2 cannot be represented in type 'int'
      #0 0x55e90f561b63 in sixel_decode_raw_impl /home/shangzhi/fuz/libsixel/src/fromsixel.c:539
      #1 0x55e90f566cff in sixel_decode_raw /home/shangzhi/fuz/libsixel/src/fromsixel.c:943
      #2 0x55e90f55c656 in sixel_decoder_decode /home/shangzhi/fuz/libsixel/src/decoder.c:280
      #3 0x55e90f55ae7a in main /home/shangzhi/fuz/libsixel/converters/sixel2png.c:226
      #4 0x7f25de029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
      #5 0x7f25de029e3f in __libc_start_main_impl ../csu/libc-start.c:392
      #6 0x55e90f55a6b4 in _start (/home/shangzhi/fuz/libsixel/converters/sixel2png+0x1a6b4)

3. Integer overflow in encoder.c

PoC https://github.com/ShangzhiXu/PoC/blob/main/libsixel/crash_w13_10334_runtime_error_1772618330.gif
cmd: ./img2sixel -h 1890177820 -e -I -C 637555838 -S -o /dev/null crash_w13_10334_runtime_error_1772618330.gif
crash stack

root@server:/home/shangzhi/fuz/libsixel/converters# UBSAN_OPTIONS=print_stacktrace=1 ./img2sixel -h 1890177820 -e -I -C 637555838 -S -o /dev/null crashes_gif_v1/crash_w13_10334_runtime_error_1772618330.gif
encoder.c:707:31: runtime error: signed integer overflow: 1890177820 * 8 cannot be represented in type 'int'
    #0 0x5652bd51be42 in sixel_encoder_do_resize /home/shangzhi/fuz/libsixel/src/encoder.c:707
    #1 0x5652bd51da89 in sixel_encoder_encode_frame /home/shangzhi/fuz/libsixel/src/encoder.c:1039
    #2 0x5652bd51fba5 in load_image_callback /home/shangzhi/fuz/libsixel/src/encoder.c:1752
    #3 0x5652bd5a6f42 in load_gif /home/shangzhi/fuz/libsixel/src/fromgif.c:680
    #4 0x5652bd598c03 in load_with_builtin /home/shangzhi/fuz/libsixel/src/loader.c:948
    #5 0x5652bd59eec7 in sixel_helper_load_image_file /home/shangzhi/fuz/libsixel/src/loader.c:1462
    #6 0x5652bd529c10 in sixel_encoder_encode /home/shangzhi/fuz/libsixel/src/encoder.c:1816
    #7 0x5652bd51b009 in main /home/shangzhi/fuz/libsixel/converters/img2sixel.c:455
    #8 0x7f77b7429d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #9 0x7f77b7429e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #10 0x5652bd519004 in _start (/home/shangzhi/fuz/libsixel/converters/img2sixel+0x12d004)

4. SEVG when using libpng

PoC https://github.com/ShangzhiXu/PoC/blob/main/libsixel/crash_w3_1161_AddressSanitizer_1772622371.bmp
cmd ./img2sixel -r lanczos2 -o /dev/null crash_w3_1161_AddressSanitizer_1772622371.bmp

crash stack

root@server:/home/shangzhi/fuz/libsixel/converters# ./img2sixel -r lanczos2 -o /dev/null crashes_bmp_v1/crash_w3_1161_AddressSanitizer_1772622371.bmp
AddressSanitizer:DEADLYSIGNAL
=================================================================
==3016244==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000a1 (pc 0x7fed1f1723ea bp 0x51a000000080 sp 0x7ffe37d3e410 T0)
==3016244==The signal is caused by a READ memory access.
==3016244==Hint: address points to the zero page.
    #0 0x7fed1f1723ea in png_free_data (/lib/x86_64-linux-gnu/libpng16.so.16+0xc3ea)
    #1 0x7fed1f172a0c in png_destroy_info_struct (/lib/x86_64-linux-gnu/libpng16.so.16+0xca0c)
    #2 0x7fed1f175a28 in png_destroy_read_struct (/lib/x86_64-linux-gnu/libpng16.so.16+0xfa28)
    #3 0x55b965ea2d50 in load_png /home/shangzhi/fuz/libsixel/src/loader.c:657
    #4 0x55b965ee0867 in load_with_builtin /home/shangzhi/fuz/libsixel/src/loader.c:925
    #5 0x55b965ee6ec7 in sixel_helper_load_image_file /home/shangzhi/fuz/libsixel/src/loader.c:1462
    #6 0x55b965e71c10 in sixel_encoder_encode /home/shangzhi/fuz/libsixel/src/encoder.c:1816
    #7 0x55b965e63009 in main /home/shangzhi/fuz/libsixel/converters/img2sixel.c:455
    #8 0x7fed1de29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #9 0x7fed1de29e3f in __libc_start_main_impl ../csu/libc-start.c:392
    #10 0x55b965e61004 in _start (/home/shangzhi/fuz/libsixel/converters/img2sixel+0x12d004)

5. Integer overflow in encode.c

PoC: https://github.com/ShangzhiXu/PoC/blob/main/libsixel/crash_w18_17470_runtime_error_1772617376.gif
cmd: ./img2sixel -S -o /dev/null crash_w18_17470_runtime_error_1772617376.gif

Crash Stack

root@server:/home/shangzhi/fuz/libsixel/converters# UBSAN_OPTIONS=print_stacktrace=1 ./img2sixel -S -o /dev/null crash_w18_17470_runtime_error_1772617376.gif
encoder.c:881:60: runtime error: signed integer overflow: 330239999 * 1000 cannot be represented in type 'int'

6. Integer overflow in quant.c

PoC: https://github.com/ShangzhiXu/PoC/blob/main/libsixel/crash_w31_8521_runtime_error_1772617557.gif
cmd: ./img2sixel -C 2147483647 -o /dev/null crash_w31_8521_runtime_error_1772617557.gif

Crash Stack

root@server:/home/shangzhi/fuz/libsixel/converters# UBSAN_OPTIONS=print_stacktrace=1 ./img2sixel -C 2147483647 -o /dev/null crash_w31_8521_runtime_error_1772617557.gif
quant.c:1121:85: runtime error: signed integer overflow: 36 * 2147483647 cannot be represented in type 'int'

[saitoha]

@ShangzhiXu Thank you for all your reports. May I add the PoC file you provided to the repository for regression testing?

[ShangzhiXu]

Thanks for your rely. Yes, please feel free to use the PoC files~