Missing filemods from processes with over 10,000 filemods in Cb Response

[nphull]

If I pull up a process in the Cb Response web GUI and it has over 10,000 filemods, and then I pull up the same process using cbapi, the cbapi Process claims it only has 10,000 filemods, and if I try enumerating them, I only get about 9,920 of them. What happened to all of the other filemods? How do I get them? Is this a limitation of the Response REST API?

Example: the web GUI shows process 0000cbf9-0000-1728-01d4-e63e96539a3a had 29969 filemods. So I pull it up with cbapi:

c = CbEnterpriseResponseAPI()
process = c.select(Process, '0000cbf9-0000-1728-01d4-e63e96539a3a')
print(process.filemod_count)
# 10000
fms = []
for fm in process.filemods:
    fms.append(fm)

print(len(fms))
# 9917

Why does it claim the process only had 10000 filemods? Why am I only able to get 9917 of those 10000? How do I get the other filemods?

The REST endpoint used to retrieve the filemods is /api/v4/process/0000cbf9-0000-1728-01d4-e63e96539a3a/0/event. What endpoint would I use to get the rest of the filemods? Changing the 0 at the end to a 1 gets a 404 Not Found.

Cb Response version: 6.2.4.190118.1044

[jgarman]

What happens if you try using the all_filemods() method on the process? This may be an edge case that needs addressing

[nphull]

I get the exact same result using all_filemods() as iterating over process.filemods. For a process with 29,968 filemods, both approaches give me the same 9,465 filemods in the same order.

[mtscher]

I am curious to know if there was ever an update on this issue. I can consistently reproduce it on processes that contain more than 10k events. It seems to be avoidable by not selecting the process "by object" but rather using a direct query ("process_id: ") and then looping the returned processes and accumulating the events.

It looks to me like the issue is in the code for Process.require_all_events() method which (for recent versions) uses a single call to get the events with the default (0) process segment. I suspect this call is subject to the documented 10k limit indicated here where it indicates "There is a hard limit of 5k rows on the /devices API per call, and a hard limit of 10k rows on the /events API at maximum"