Unbounded recursion in cupsEncodeOptions2 / _cupsEncodeOption for nested collections

[Tomer-PL]

Summary

cupsEncodeOptions2() and _cupsEncodeOption() have mutual recursion with no depth limit when processing nested collection values. A text option like {a={a={a=...}}} with ~31,000 nesting levels causes a stack overflow crash.

Details

The recursion cycle (cups/encode.c):

1. cupsEncodeOptions2() (line 870) calls _cupsEncodeOption() for each option
2. _cupsEncodeOption() (line 662-678) detects { in values, calls cupsParseOptions(), then recursively calls cupsEncodeOptions2() (line 675)
3. No depth counter or limit exists

Note: All server-side code paths that reach cupsEncodeOptions2 have natural depth limits:

• read_job_ticket() uses a 256-byte line buffer, capping nesting at ~57 levels
• IPP wire protocol uses binary collection tags, bypassing text parsing entirely
• The crash is reproducible in client-side tools (lp -o "media-col={a={a=...}}")

Reproducer

#include <cups/cups.h>
int main(void) {
    char value[200000];
    char *p = value;
    for (int i = 0; i < 50000; i++) *p++ = '{'; *p++ = 'a'; *p++ = '='; *p++ = 'x';
    for (int i = 0; i < 50000; i++) *p++ = '}'; *p = '\0';
    
    cups_option_t *options = NULL;
    int num = cupsAddOption("media-col", value, 0, &options);
    
    ipp_t *ipp = ippNew();
    cupsEncodeOptions2(ipp, num, options, IPP_TAG_JOB);  // Stack overflow
    ippDelete(ipp);
    return 0;
}

ASan output:

ERROR: AddressSanitizer: stack-overflow
    #N   in cupsEncodeOptions2 encode.c:870
    #N+1 in _cupsEncodeOption encode.c:675
    (hundreds of alternating frames)

Suggested Fix

Add a depth parameter to _cupsEncodeOption (or use a static/thread-local counter), reject nesting beyond a reasonable limit (e.g., 32):

+ #define CUPS_MAX_COLLECTION_DEPTH 32

  static ipp_tag_t
  _cupsEncodeOption(ipp_t *ipp, ipp_tag_t group_tag,
-                   _ipp_option_t *map, const char *name, const char *value)
+                   _ipp_option_t *map, const char *name, const char *value,
+                   int depth)
  {
+   if (depth > CUPS_MAX_COLLECTION_DEPTH)
+     return (IPP_TAG_ZERO);
    ...
-   cupsEncodeOptions2(collection, num_cols, cols, IPP_TAG_JOB);
+   cupsEncodeOptions2_depth(collection, num_cols, cols, IPP_TAG_JOB, depth + 1);

[michaelrsweet]

Sigh... API misuse but since options are sometimes passed as user input we should protect it (but not as proposed).

[michaelrsweet]

Changes, max depth is 3 for any known IPP collection attributes, set the limit to 4:

[master c3f57e3] Protect against deep collection values (Issue #1539)

[2.4.x e9ddf7b] Protect against deep collection values (Issue #1539)