setDT() fails in case of nested calls

[MichaelChirico]

The {marginaleffects} case in #6575 is pretty hard to tease apart, but I found this example that could be related where 1.16.4 and current master differ in behavior:

baz = function(x) setDT(x)
foo = function(x) {
  bar = function() baz(x)
  x = data.frame(a=1)
  bar()
  class(x)
}
foo()

## 1.16.4
# [1] "data.table" "data.frame"

## current master
# [1] "data.frame"

From debugging of {marginaleffects} thus far, I only found that, in contrast to #6701 and #6725, {marginaleffects} (in example(comparisons) where R CMD check is failing, at least) only calls simply setDT(<name>).

[rikivillalba]

I do not even need foo nor bar to reproduce in devel:

library(data.table)
baz = function(x) setDT(x)
x = data.frame(a=1)
baz(x)
class(x)
# [1] "data.frame"

[MichaelChirico]

Nice, thanks! I added it to #6755 as a regression test