Skip to content

Unsigned integer underflow in ippevepcl margin calculation causes heap overflow #1544

@Tomer-PL

Description

@Tomer-PL

Summary

In tools/ippevepcl.c:130-142, the PCL margin calculation uses unsigned subtraction without checking for underflow. When cupsWidth < 8 * HWResolution[0] (A4 path) or cupsWidth < HWResolution[0] / 4 + 1 (other sizes), the result wraps to a huge positive value. The subsequent dithering loop reads past the input buffer and writes past pcl_line.

Details

// tools/ippevepcl.c:135-136 (A4 path)
pcl_left  = (cupsWidth - 8 * HWResolution[0]) / 2;  // Unsigned underflow!
pcl_right = pcl_left + 8 * HWResolution[0] - 1;

// Line 243: small allocation based on cupsWidth
pcl_line = malloc(cupsWidth / 8 + 1);  // 13 bytes for cupsWidth=100

// Line 348: dithering loop uses wrapped range
for (x = pcl_left, ..., outptr = pcl_line; x <= pcl_right; x++, line++)
{
    if (*line <= ditherline[x & 63])  // Reads past line buffer
        byte |= bit;
    *outptr++ = byte;                  // Writes past pcl_line buffer
}

With cupsWidth=100, HWResolution=300: pcl_left = (100-2400)/2 wraps to ~2 billion. The loop iterates 2400 times, reading 2300 bytes past the 100-byte line buffer and writing 287 bytes past the 13-byte pcl_line buffer.

Reproducer

unsigned cupsWidth = 100, HWResolution0 = 300;
unsigned pcl_left = (cupsWidth - 8 * HWResolution0) / 2;  // Wraps
unsigned pcl_right = pcl_left + 8 * HWResolution0 - 1;
// pcl_right - pcl_left + 1 = 2400 iterations
// line buffer = 100 bytes → 2300 byte over-read
// pcl_line = 13 bytes → 287 byte overflow

ASan output:

ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1 — 0 bytes after 100-byte region

Suggested Fix

Add underflow checks before the subtraction:

+ if (cupsWidth < 8 * HWResolution[0])
+ {
+     pcl_left = 0;
+     pcl_right = cupsWidth - 1;
+ }
+ else
+ {
      pcl_left  = (cupsWidth - 8 * HWResolution[0]) / 2;
      pcl_right = pcl_left + 8 * HWResolution[0] - 1;
+ }

Same pattern needed for the non-A4 path at lines 141-142.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions