data.table: another non-exportable object.

[JamesThompsonC]

Thank you Henrik for your generosity in developing and sharing this package.

I discovered another non-exportable object.
Here's the repex:

library(data.table)
library(tibble)
library(foreach)
library(fulure.callr)
library(doFuture)

doFuture::registerDoFuture()
plan(callr, workers = 3)

This gives me what I'm expecting. a three row, two column data.table, each cell a data.table:

o <-
foreach(ii = 1:3) %do% {
data.table(df_a = list(data.table(e = 1:10)),
df_b = list(data.table(d = 1:10)))
}

rbindlist(o)

This returns an error:

o <-
foreach(ii = 1:3) %dopar% {
data.table(df_a = list(data.table(e = 1:10)),
df_b = list(data.table(d = 1:10)))
}

Error: Detected a non-exportable reference (‘externalptr’) in the value (of class ‘list’) of the resolved future

I got %dopar% to work by using tibble's instead of data.tables.

My problem is that most of my functions return data.table's.

o <-
foreach(ii = 1:3) %dopar% {
tibble(df_a = list(tibble(e = 1:10)),
df_b = list(tibble(d = 1:10)))
}

rbindlist(o)

I'm not sure you can do anything about it.
If not, maybe include it in your
"A Future for R: Non-Exportable Objects"
vignette.

~Jim

[HenrikBengtsson]

Hi. I think you have:

options(future.globals.onReference = "error")

set when doing this, cf. ?future::future.options. Because without that set, the future framework will not look for "non-exportable reference" and therefore not throw an error in this case.

Section 'Packages that rely on external pointers' of the 'Non-Exportable Objects' vignette mentions data.table as a special type of objects holding "external pointers" but still can be exported (back and forth).

I hope this helps

PS. Please have a look at https://docs.github.com/en/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax on how to format your code examples here in GitHub to make them easier to read.

[JamesThompsonC]

Yep. Thanks. And thank you to pointing me to the how-to on formating.