R/stremr: Streamlined Causal Inference for Static, Dynamic and Stochastic Regimes in Longitudinal Data

Analysis of longitudinal data, with continuous or time-to-event (binary) outcome and time-varying confounding. Allows adjustment for all measured time-varying confounding and informative right-censoring. Estimate the expected counterfactual outcome under static, dynamic or stochastic interventions. Includes doubly robust and semi-parametrically efficient Targeted Minimum Loss-Based Estimator (TMLE), along with several other estimators. Perform data-adaptive estimation of the outcome and treatment models with Super Learner sl3.

Authors: Oleg Sofrygin, Mark van der Laan, Romain Neugebauer

Available estimators

Currently available estimators can be roughly categorized into 4 groups:

• Propensity-score / Inverse Probability Weighted (IPW):
  • direct (bounded) IPW (directIPW)
  • IPW-adjusted Kaplan-Meier (survNPMSM)
  • MSM-IPW for the survival hazard (survMSM)
• Outcome regression:
  • longitudinal G-formula (GCOMP) (Bang and Robins, 2005)
• Doubly-robust (DR) approaches:
  • TMLE for longitudinal data (TMLE) (van der Laan and Gruber, 2012)
  • iterative TMLE (iterTMLE)
  • cross-validated TMLE (CVTMLE)
• Sequentially doubly-robust (SDR) approaches:
  • infinite-dimensional TMLE (iTMLE) (Luedtke, Sofrygin, van der Laan, and Carone, 2017)
  • doubly robust unbiased transformations (DR_transform) (Rubin and van der Laan, 2006, Luedtke, Sofrygin, van der Laan, et al. (2017)`)

Input data format

The exposure, monitoring and censoring variables can be coded as either binary, categorical or continuous. Each can be multivariate (e.g., can use more than one column of dummy indicators for different censoring events). The input data needs to be in long format.

• Possibly right-censored data has to be in long format.
• Each row must contain a subject identifier (ID) and the integer indicator of the current time (t), e.g., day, week, month, year.
• The package assumes that the temporal ordering of covariates in each row is fixed according to (ID, t, L,C,A,N,Y), where
  • L -- Time-varying and baseline covariates.
  • C -- Indicators of right censoring events at time t; this can be either a single categorical or several binary columns.
  • A -- Exposure (treatment) at time t; this can be multivariate (more than one column) and each column can be binary, categorical or continuous.
  • N -- Indicator of being monitored at time point t+1 (binary).
  • Y -- Outcome (binary 0/1 or continuous between 0 and 1).
• Categorical censoring can be useful for representing all of the censoring events with a single column (variable).

Estimation of the outcome and treatment models

• Separate models are fit for the observed censoring, exposure and monitoring mechanisms. Jointly, these make up what is known as the propensity score.
• Separate outcome regression models can be specified for each time-point.
• Each propensity score model can be stratified (separate model is fit) by time or any other user-specified stratification criteria. Each strata is defined with by a single logical expression that selects specific observations/rows in the observed data (strata).
• By default, all models are fit with the logistic regression.
• Alternatively, model fitting can be performed via any machine learning (ML) algorithm available in sl3 and gridisl R packages. See xgboost and h2o for a sample description of possible ML algorithms.
• One can select the best model from an ensemble of many learners via model stacking or Super Learning (Breiman, 1996; van der Laan, Polley, and Hubbard, 2007), which finds the optimal convex combination of all models in the ensemble via cross-validation.

Brief overview of stremr

• Installing stremr and Documentation
• Issues
• Documentation
• Example with Simulated Data
• Sequential G-Computation (GCOMP) and Targeted Maximum Likelihood Estimation (TMLE) for longitudinal survival data
• Machine Learning
• Ensemble Learning with Discrete SuperLearner (based on gridisl R package)
• Details on some implemented estimators

Installation

To install the development version (requires the devtools package):

devtools::install_github('osofr/stremr')

For ensemble-learning with Super Learner algorithm we recommend installing the latest development version of the sl3 R package. It can be installed as follows:

devtools::install_github('jeremyrcoyle/sl3')

For optimal performance, we also recommend installing the latest version of data.table package:

remove.packages("data.table")                         # First remove the current version
install.packages("data.table", type = "source",
    repos = "http://Rdatatable.github.io/data.table") # Then install devel version

Issues

If you encounter any bugs or have any specific feature requests, please file an issue.

Documentation

To obtain documentation for specific relevant functions in stremr package:

?stremr
?importData
?fitPropensity
?getIPWeights
?directIPW
?survNPMSM
?survMSM
?fit_GCOMP
?fit_iTMLE

Simulated data example

Load the data:

require("magrittr")
#> Loading required package: magrittr
require("data.table")
#> Loading required package: data.table
require("stremr")
#> Loading required package: stremr

data(OdataNoCENS)
datDT <- as.data.table(OdataNoCENS, key=c("ID", "t"))

Define some summaries (lags):

datDT[, ("N.tminus1") := shift(get("N"), n = 1L, type = "lag", fill = 1L), by = ID]
datDT[, ("TI.tminus1") := shift(get("TI"), n = 1L, type = "lag", fill = 1L), by = ID]

Define counterfactual exposures. In this example we define one intervention as always treated and another as never treated. Such intervention can be defined conditionally on other variables (dynamic intervention). Similarly, one can define the intervention as a probability that the counterfactual exposure is 1 at each time-point t (for stochastic interventions).

datDT[, ("TI.set1") := 1L]
datDT[, ("TI.set0") := 0L]

Import input data into stremr object DataStorageClass and define relevant covariates:

OData <- importData(datDT, ID = "ID", t = "t", covars = c("highA1c", "lastNat1", "N.tminus1"), CENS = "C", TRT = "TI", OUTCOME = "Y.tplus1")

Once the data has been imported, it is still possible to inspect it and modify it, as shown in this example:

get_data(OData)[, ("TI.set0") := 1L]
get_data(OData)[, ("TI.set0") := 0L]

Regressions for modeling the propensity scores for censoring (CENS) and exposure (TRT). By default, each of these propensity scores is fit with a common model that pools across all available time points (smoothing over all time-points).

gform_CENS <- "C ~ highA1c + lastNat1"
gform_TRT <- "TI ~ CVD + highA1c + N.tminus1"

Stratification, that is, fitting separate models for different time-points, is enabled with logical expressions in arguments stratify_... (see ?fitPropensity). For example, the logical expression below states that we want to fit the censoring mechanism with a separate model for time point 16, while pooling with a common model fit over time-points 0 to 15. Any logical expression can be used to define such stratified modeling. This can be similarly applied to modeling the exposure mechanism (stratify_TRT) and the monitoring mechanism (stratify_MONITOR).

stratify_CENS <- list(C=c("t < 16", "t == 16"))

Fit the propensity scores for censoring, exposure and monitoring:

OData <- fitPropensity(OData,
                       gform_CENS = gform_CENS,
                       gform_TRT = gform_TRT,
                       stratify_CENS = stratify_CENS)

Estimate survival based on non-parametric/saturated IPW-MSM (IPTW-ADJUSTED KM):

AKME.St.1 <- getIPWeights(OData, intervened_TRT = "TI.set1") %>%
             survNPMSM(OData) %$%
             estimates

The result is a data.table that contains the estimates of the counterfactual survival for each time-point, for the treatment regimen TI.set1. In this particular case, the column St.NPMSM contains the survival estimates for IPW-NPMSM and the first row represents the estimated proportion alive at the end of the first cycle / time-point. Note that the column St.KM contains the unadjusted/crude estimates of survival (should be equivalent to standard Kaplan-Meier estimates for most cases).

head(AKME.St.1[],2)
#>    est_name time sum_Y_IPW sum_all_IPAW   ht.NPMSM  St.NPMSM      ht.KM
#> 1:    NPMSM    0 1.6610718     38.13840 0.04355379 0.9564462 0.04733728
#> 2:    NPMSM    1 0.8070748     48.10323 0.01677797 0.9403990 0.01863354
#>        St.KM rule.name
#> 1: 0.9526627   TI.set1
#> 2: 0.9349112   TI.set1

Estimate survival with bounded IPW:

IPW.St.1 <- getIPWeights(OData, intervened_TRT = "TI.set1") %>%
            directIPW(OData) %$%
            estimates

As before, the result is a data.table with estimates of the counterfactual survival for each time-point, for the treatment regimen TI.set1, located in column St.directIPW.

head(IPW.St.1[],2)
#>     est_name time sum_Y_IPW  sum_IPW St.directIPW rule.name
#> 1: directIPW    0  9.828827 225.6710    0.9564462   TI.set1
#> 2: directIPW    1 14.841714 308.6067    0.9519073   TI.set1

Estimate hazard with IPW-MSM then map into survival estimate. Using two regimens and smoothing over two intervals of time-points:

wts.DT.1 <- getIPWeights(OData = OData, intervened_TRT = "TI.set1", rule_name = "TI1")
wts.DT.0 <- getIPWeights(OData = OData, intervened_TRT = "TI.set0", rule_name = "TI0")
survMSM_res <- survMSM(list(wts.DT.1, wts.DT.0), OData, tbreaks = c(1:8,12,16)-1,)

In this particular case the output is a little different, with separate survival tables for each regimen. The output of survMSM is hence a list, with one item for each counterfactual treatment regimen considered during the estimation. The actual estimates of survival are located in the column(s) St.MSM. Note that survMSM output also contains the standard error estimates of survival at each time-point in column(s) SE.MSM. Finally, the output table also contains the subject-specific estimates of the influence-curve (influence-function) in column(s) IC.St. These influence function estimates can be used for constructing the confidence intervals of the counterfactual risk-differences for two contrasting treatments (see help for get_RDs function for more information).

head(survMSM_res[["TI0"]][["estimates"]],2)
#>    est_name time      ht.MSM    St.MSM      SE.MSM rule.name
#> 1:      MSM    0 0.004214338 0.9957857 0.002105970       TI0
#> 2:      MSM    1 0.013068730 0.9827720 0.004100295       TI0
#>                                                                       IC.St
#> 1: 0.004543242,0.004543242,0.004543242,0.004543242,0.004543242,0.004543242,
#> 2: 0.004483868,0.016683119,0.016797415,0.017900770,0.017900770,0.017900770,
head(survMSM_res[["TI1"]][["estimates"]],2)
#>    est_name time     ht.MSM    St.MSM     SE.MSM rule.name        IC.St
#> 1:      MSM    0 0.04355379 0.9564462 0.01521910       TI1 0,0,0,0,0,0,
#> 2:      MSM    1 0.01677797 0.9403990 0.01786105       TI1 0,0,0,0,0,0,

Longitudinal GCOMP (G-formula) and TMLE

Define time-points of interest, regression formulas and software to be used for fitting the sequential outcome models:

tvals <- c(0:10)
Qforms <- rep.int("Qkplus1 ~ CVD + highA1c + N + lastNat1 + TI + TI.tminus1", (max(tvals)+1))

To run iterative means substitution estimator (G-Computation), where all at risk observations are pooled for fitting each outcome regression (Q-regression):

gcomp_est <- fit_GCOMP(OData, tvals = tvals, intervened_TRT = "TI.set1", Qforms = Qforms)

The output table of fit_GCOMP contains the following information, with the column St.GCOMP containing the survival estimates for each time period:

head(gcomp_est$estimates[],2)
#>    est_name time  St.GCOMP St.TMLE   type    cum.inc              IC.St
#> 1:    GCOMP    0 0.9837583      NA pooled 0.01624168 NA,NA,NA,NA,NA,NA,
#> 2:    GCOMP    1 0.9699022      NA pooled 0.03009778 NA,NA,NA,NA,NA,NA,
#>    fW_fit rule.name
#> 1:   NULL   TI.set1
#> 2:   NULL   TI.set1

To run the longitudinal long format Targeted Minimum-Loss Estimation (TMLE), stratified by rule-followers for fitting each outcome regression (Q-regression):

tmle_est <- fit_TMLE(OData, tvals = tvals, intervened_TRT = "TI.set1", Qforms = Qforms)
#> GLM TMLE update cannot be performed since the outcomes (Y) are either all 0 or all 1, setting epsilon to 0
#> GLM TMLE update cannot be performed since the outcomes (Y) are either all 0 or all 1, setting epsilon to 0

The output table of fit_TMLE contains the following information, with the column St.TMLE containing the survival estimates for each time period. In addition, the column SE.TMLE contains the standard error estimates and the column and the column IC.St contains the subject-specific estimates of the efficient influence curve. The letter estimates are useful for constructing the confidence intervals of risk differences for two contrasting treatments (see help for get_RDs function for more information).

head(tmle_est$estimates[],2)
#>    est_name time St.GCOMP   St.TMLE   type    cum.inc     SE.TMLE
#> 1:     TMLE    0       NA 0.9839271 pooled 0.01607286 0.003449949
#> 2:     TMLE    1       NA 0.9707676 pooled 0.02923243 0.004492235
#>                                                                             IC.St
#> 1: -0.007292922,-0.007292922,-0.010190141,-0.007292922,-0.007292922,-0.007292922,
#> 2: -0.009469707,-0.009469707,-0.010503891,-0.009469707,-0.009469707,-0.009469707,
#>    fW_fit rule.name
#> 1:   NULL   TI.set1
#> 2:   NULL   TI.set1

To parallelize estimation over several time-points (tvals) for either GCOMP or TMLE use argument parallel = TRUE:

require("doParallel")
registerDoParallel(cores = parallel::detectCores())
tmle_est <- fit_TMLE(OData, tvals = tvals, intervened_TRT = "TI.set1", Qforms = Qforms, parallel = TRUE)

Data-adaptive estimation, cross-validation and Super Learning

Nuisance parameters can be modeled with any machine learning algorithm supported by sl3 R package. For example, for GLMs use learner Lrnr_glm_fast, for xgboost use learner Lrnr_xgboost, for h2o GLM learner use Lrnr_h2o_glm, for any other ML algorithm implemented in h2o use Lrnr_h2o_grid$new(algorithm = "algo_name"), for glmnet use learner Lrnr_glmnet. All together, these learners provide access to a wide variety of ML algorithms. To name a few: GLM, Regularized GLM, Distributed Random Forest (RF), Extreme Gradient Boosting (GBM) and Deep Neural Nets.

Model selection can be performed via V-fold cross-validation or random validation splits and model stacking and Super Learner combination can be accomplished by using the learner Lrnr_sl and specifying the meta-learner (e.g., Lrnr_solnp). In the example below we define a Super Learner ensemble consisting of several learning algorithms.

First, we define sl3 learners for for xgboost, two types of GLMs and glmnet. Then we will stack these learners into a single learner called Stack:

library("sl3")
lrn_xgb <- Lrnr_xgboost$new(nrounds = 5)
lrn_glm <- Lrnr_glm_fast$new()
lrn_glm2 <- Lrnr_glm_fast$new(covariates = c("CVD"))
lrn_glmnet <- Lrnr_glmnet$new(nlambda = 5, family = "binomial")
## Stack the above candidates:
lrn_stack <- Stack$new(lrn_xgb, lrn_glm, lrn_glm2, lrn_glmnet)

Next, we will define a Super Learner on the above defined stack, by feeding the stack into the Lrnr_sl object and then specifying the meta-learner that will find the optimal convex combination of the learners in a stack (Lrnr_solnp):

lrn_sl <- Lrnr_sl$new(learners = lrn_stack, metalearner = Lrnr_solnp$new())

We will now use stremr to estimate the exposure / treatment propensity model with the above defined Super Learner (lrn_sl):

OData <- fitPropensity(OData,
                       gform_CENS = gform_CENS,
                       gform_TRT = gform_TRT,
                       models_TRT = lrn_sl,
                       stratify_CENS = stratify_CENS)

Details on some implemented estimators

Currently implemented estimators include:

• Kaplan-Meier Estimator. No adjustment for time-varying confounding or informative right-censoring.
• Inverse Probability Weighted (IPW) Kaplan-Meier (survNPMSM). Also known as the Adjusted Kaplan Meier (AKME). Also known as the saturated (non-parametric) IPW-MSM estimator of the survival hazard. This estimator inverse weights each observation based on the exposure/censoring model fits (propensity scores).
• Bounded Inverse Probability Weighted (B-IPW) Estimator of Survival('directIPW'). Estimates the survival directly (without hazard), also based on the exposure/censoring model fit (propensity scores).
• Inverse Probability Weighted Marginal Structural Model (survMSM) for the hazard function, mapped into survival. Currently only logistic regression is allowed where covariates are time-points and regime/rule indicators. This estimator is also based on the exposure/censoring model fit (propensity scores), but allows additional smoothing over multiple time-points and includes optional weight stabilization.
• Longitudinal G-formula (GCOMP). Also known as the iterative G-Computation formula or Q-learning. Directly estimates the outcome model while adjusting for time-varying confounding. Estimation can be stratified by rule/regime followed or pooled across all rules/regimes.
• Longitudinal Targeted Minimum-Loss-based Estimator (TMLE). Also known as L-TMLE. Doubly robust and semi-parametrically efficient estimator that de-biases each outcome regression fit with a targeting step, using IPW.
• Iterative TMLE (iterTMLE) for longitudinal data. Fits sequential G-Computation and then iteratively performs targeting for all pooled Q's until convergence.
• Infinite-dimensional TMLE (iTMLE) for longitudinal data. Fits sequential G-Computation and performs additional infinite-dimensional targeting to achieve sequential double robustness.

Citation

To cite stremr in publications, please use:

Sofrygin O, van der Laan MJ, Neugebauer R (2017). stremr: Streamlined Estimation for Static, Dynamic and Stochastic Treatment Regimes in Longitudinal Data. R package version x.x.xx

Funding

This work was partially supported through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1403-12506). All statements in this work, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee. This work was also supported through an NIH grant (R01 AI074345-07).

Copyright

The contents of this repository are distributed under the MIT license.

The MIT License (MIT)

Copyright (c) 2015-2017 Oleg Sofrygin 

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

---

References

[1] L. Breiman. "Stacked regressions". In: Machine learning 24.1 (1996), pp. 49-64.

[2] H. Bang and J. Robins. "Doubly robust estimation in missing data and causal inference models". In: Biometrics 61 (2005), pp. 962-972. DOI: 10.1111/j.1541-0420.2005.00377.x.

[3] M. J. van der Laan, E. C. Polley and A. E. Hubbard. "Super learner". In: Statistical applications in genetics and molecular biology 6.1 (2007).

[4] M. van der Laan and S. Gruber. "Targeted minimum loss based estimation of causal effects of multiple time point interventions". In: The International Journal of Biostatistics 8 (2012). DOI: 10.1515/1557-4679.1370.

[5] A. R. Luedtke, O. Sofrygin, M. J. van der Laan, et al. "Sequential Double Robustness in Right-Censored Longitudinal Models". In: arXiv preprint arXiv:1705.02459 (2017). arXiv: 1705.02459.