R Code for Using Q-Learning to Construct Adaptive Treatment Strategies Using Data from a SMART

(Software available free to members. Register.)

The Q-learning package implements the Q-learning method described in Nahum-Shani et al. (2010b).

Instructions for Using the Q-learning Package in R

In order to use the Q-learning method to analyze data from the two-stage SMART described in Nahum-Shani (2010b), go to qlearning.zip if you have a Windows operating system and qlearning.tar if you have a Mac or Unix operating system, to download and install the qlearning package. Once installed, type command library(qlearning) in R to load the package. The HTML help file of the qlearning function can be found by typing the command qlearning in R. Run the qlearning code mentioned under the example section in the help file to get regression coefficients and confidence intervals for each stage and for different contrasts of the interventions.

The description of the data set used in the example section can be found in variables in data set or by typing the command DataEx in R.

Compare the obtained results with those available in results.

(Because the analysis in the current example was conducted on fake data, the results reported in the results file are different from those reported in the paper.)

Usage

qlearning(H10, H11, A1, ReRand, H20, H21, A2, Y, C1=diag(ncol(cbind(H10,H11))), C2=diag(ncol(cbind(H20,H21))), alpha=0.05, B=1000)

Arguments

H10	A matrix where each column represents a main effect term involved in the first stage regression (usually includes a column of ones as the intercept).
H11	A matrix where each column represents a term interacting with the first-stage intervention A1 (usually includes a column of ones as the main effect of A1).
A1	A vector representing the first-stage intervention (binary, coded as -1 or 1).
ReRand	Re-randomization indicator (1 if re-randomized, and 0 if not).
H20	A matrix where each column represents a main effect term involved in the second stage regression (usually includes a column of ones as the intercept).
H21	A matrix where each column represents a term interacting with the second stage intervention A2 (usually includes a column of ones as the main effect of A2).
A2	A vector representing the second stage intervention (binary, coded as -1 or 1).
Y	Outcome (coded so that larger values are desirable).
C1	The first-stage contrast matrix (default value is an identity matrix).
C2	The second-stage contrast matrix (default value is an identity matrix).
alpha	Significance level (default value is 0.05).
B	Number of bootstrap iterations (default value is 1000).

Details

At the first stage of the SMART, each subject is randomized to one of the two interventions. At the second stage, only non-responders (or responders) are re-randomized to one of the two interventions. The second-stage confidence intervals are constructed using the centered percentile bootstrap. The first-stage confidence intervals are constructed using the soft-threshold method with percentile bootstrap. See Nahum-Shani et al. (2010a) for the description of a SMART for use in developing adaptive interventions. See Nahum-Shani et al. (2010b) for the description of the Q-learning algorithm for constructing adaptive interventions. See Chakraborty et al. (2009) for the description of the soft-threshold method with percentile bootstrap for constructing the first-stage confidence intervals.

Value

A qlearning object is returned. Stage1coef and stage1df are used to get the first-stage regression coefficients and the degrees of freedom. Stage1CI is used to get the estimators of the first-stage contrasts (i.e., C1*stage1coef) and the associated confidence intervals. Similarly, stage2coef, stage2df and stage2CI are used to get the second-stage regression coefficients, degrees of freedom, and estimators of the second-stage contrasts (i.e., C2*stage2coef) and the associated confidence intervals.

Author(s)

Min Qian, Inbal Nahum-Shani, Amarpreet Kaur Ashkan Ertefaie, Daniel Almirall, and Susan A. Murphy.

Maintainer: Min Qian - minqian@umich.edu

References

Chakraborty, B., Murphy, S. A., & Strecher, V., (2009). Inference for non-regular parameters in optimal dynamic treatment regimes. Statistical Methods in Medical Research, 19(3), 317-343. View article

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W., Gnagy, B., Fabiano, G., … Murphy, S. A. (2012a). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17, 457-77. PMC Journal- In process

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W., Gnagy, B., Fabiano, G., … Murphy, S. A. (2012b). Q-learning: A data analysis method for constructing adaptive interventions. Psychological Methods, 17, 478-494. PMC Journal- In process

Examples

## load the example data set provided in the qlearning package
   data("DataEx")

## some data manipulation: center continues covariates, define others, etc.
## centering continues covariates makes it easier to contrast the effects of
## treatment

DataEx$ReRand <- 1 - DataEx$R ## R = 1 = responders are not ReRandomized
DataEx$O12c <- scale(DataEx$O12, center = TRUE, scale = FALSE)
DataEx$O21c <- scale(DataEx$O21, center = TRUE, scale = FALSE)

## construct design matrices to be used in the first-stage and the second-stage
## regression
H10 <- cbind(1, DataEx$O11, DataEx$O12c, DataEx$O13)
H11 <- cbind(1, DataEx$O13)
H20 <- cbind(1, DataEx$O11, DataEx$O12c, DataEx$O13, DataEx$A1, DataEx$O21c, DataEx$O22)
H21 <- cbind(1, DataEx$A1, DataEx$O22)

## Run qlearning function to get the regression coefficients and the associated
## confidence intervals
   object1 <- qlearning(
      H10=H10,
      H11=H11,
      A1=DataEx$A1,
      ReRand=DataEx$ReRand,
      H20=H20,
      H21=H21,
      A2=DataEx$A2,
      Y=DataEx$Y
)

## check the output
   object1

## Extract the first-stage regression coefficients, degrees of freedom and the ##confidence interval for each coefficient from “object1”
   stage1coef(object1)

## if you want to know the stage 1 regression degrees of freedom, use this:
   stage1df(object1)

## if you want to extract just the coefficients, use this:
   stage1CI(object1)

## Can do the same thing for the second-stage regression.

## Construct contrast matrices
C1 <- rbind(    c(1, 0, 0, 1, -1, -1),
                 c(1, 0, 0, 1, 1, 1),
                 c(0, 0, 0, 0, -2, -2)
             )

C2 <- rbind(     c(1, 0, 0, 0, -1, 0, 1, 1, -1, 1),
                 c(1, 0, 0, 0, -1, 0, 1, -1, 1, -1),
                 c(0, 0, 0, 0, 0, 0, 0, 2, -2, 2)
             )

## Run qlearning function to get estimates and confidence intervals for the ##contrasts
   object2 <- qlearning(
      H10=H10,
      H11=H11,
      A1=DataEx$A1,
      ReRand=DataEx$ReRand,
      H20=H20,
      H21=H21,
      A2=DataEx$A2,
      Y=DataEx$Y,
      C1=C1,
      C2=C2
)

   stage1CI(object2)
   stage2CI(object2)

Recommended Citation

If you use this code in your own research, please cite the tech report listed below.

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G., Waxmonsky, J., Yu, J., & Murphy, S. A. (2010). Q-Learning: A data analysis method for constructing adaptive interventions (Technical Report No. 10-107). University Park: The Methodology Center, The Pennsylvania State University.