VECTOR_ELT issues with empty data.tables with R 4.6.

[rickhelmus]

Hello,

Since R 4.6 I am experiencing some errors in a project that heavily uses data.table. The following MWE throws an error:

> data.table::data.table()[, 1, by = numeric()]
Error in `[.data.table`(data.table::data.table(), , 1, by = numeric()) : 
  attempt access index 0/0 in VECTOR_ELT

I see this error on my MacBook with R 4.6 and the latest data.table CRAN release and from GitHub. This error is also visible on CI with Windows. For either my MacBook and CI R 4.5 doesn't throw this error.

Thanks,
Rick

[MichaelChirico]

I can't speak to why this doesn't error on R 4.5.0 -- best guess is somewhere higher in the code we get an early return that doesn't happen on 4.6.0. The fix in #7750 should be applied regardless.

[aitap]

The out-of-bounds access happens on R < 4.6 as well, but only R >= 4.6 checks for it. This was mostly harmless because the `dt` arrives here over-allocated and filled with NULLs. (But if someone creates this table with datatable.overalloc = 0L, that's nasal demon time.)