What do we do as more studies come out?

Online dashboards, such as the COVID-19 clinical trials registry, provide near real-time tracking and categorization of findings accumulating across emerging research. As such one can update the robustness of the cumulative findings for a given COVID-19 treatment. We illustrate using a historical non-COVID example: the study-by-study accumulation of \(16\) estimated effects presented in a meta-analysis of randomized trials examining the impact of hypertension treatments on the probability of suffering a stroke (data from Collins et al., 1990).

Table 1 presents the outcome data and the Robustness of the Inference to Switches (RIS) on patient strokes for the first and second studies in the hypertension meta-analysis. A positive RIS represents switches from “no stroke” to “stroke” for treatment cases; a negative RIS, from “stroke” to “no stroke.” If \(6\) of the \(181\) no stroke cases in the treatment group instead had strokes, the probability difference would be statistically significant at conventional levels (\(RIS=-6\)) to favor a positive treatment effect. The second study found a statistically significant \(7.6\) percentage point decrease in stroke probability for the treatment group, an inference that would take \(5\) switches to invalidate \((RIS =+5)\). Combining the result from the first study with the finding from the second lowers the treatment benefit reporting in the second study from \(7.6%\) to \(5.9%\) but leaves the robustness of the inference from the second study largely unchanged (Year \(1967\), Combined \(N=467\), \(RIS = +4\) instead of \(+5\)).

Table 1. Association between antihypertensive treatment and stroke. Control, treatment, and total contain number of cases from Table II of Collins et al.3 Remaining columns based on authors’ calculations. RIS=Robustness of Inference to Switches.

Table 1. Association between antihypertensive treatment and stroke. Control, treatment, and total contain number of cases from Table II of Collins et al.3 Remaining columns based on authors’ calculations. RIS=Robustness of Inference to Switches.

In Figure 1 we present a series of “robustness” updates as each study is added, where each subsequent point presents an updated estimated effect as well as corresponding RIS. Note that even after updating the estimated treatment effect with multiple studies (Years \(1966\) to \(1977\)), the combined estimate still fluctuates by several percentage points until the 8th study is added (Year=\(1979\)). This again underscores the importance of considering the robustness of inferences beyond only examining p-values, even for accumulated estimates obtained from RCTs. Continuous updates to an analagous figure using COVID-19 studies would present decision-makers with an up-to-date and intuitive characterization of combined estimates as well as the robustness of the inferences drawn from scientific evidence.

Figure 1. Robustness of Inferences to Switches as Evidence Accumulates: Historical Case of Anti-hypertension Treatment and Stroke. Black squares indicate the size of the estimated treatment effect based on all studies available up to that point in time; blue dots, the effect size just below (above) statistical significance. Up arrow indicates direction of change necessary to make non-significant estimate, significant; down arrow, a significant estimate, not significant. Boxes label the corresponding RIS.

Figure 1. Robustness of Inferences to Switches as Evidence Accumulates: Historical Case of Anti-hypertension Treatment and Stroke. Black squares indicate the size of the estimated treatment effect based on all studies available up to that point in time; blue dots, the effect size just below (above) statistical significance. Up arrow indicates direction of change necessary to make non-significant estimate, significant; down arrow, a significant estimate, not significant. Boxes label the corresponding RIS.

[For advanced users, following we show some R code to reproduce Figure 1]

First we read in data where each row includes one row that one study was added. Each row includes the 4 cells in the cummulative 2 by 2 table. cumA is the cell for control failure, cumB for control success, cumC for treatment failure and cumD for treatment success.

data <- read.csv("data1.csv", header=T)
head(data)
##    study_name reference_num year X A._CF B_CS C_TF D._TS cumA cumB cumC cumD
## 1       Wolff            11 1966 3     1   41    2    43    1   41    2   43
## 2          Va            10 1967 2    20  174    5   181   21  215    7  224
## 3          VA            10 1967 3     3   60    1    67   24  275    8  291
## 4      Carter            41 1970 3    21   27   10    39   45  302   18  330
## 5 Barraclough            38 1973 3     0   58    0    58   45  360   18  388
## 6       HSCSG            36 1974 2    52  167   43   190   97  527   61  578
##    cum_odds
## 1 0.5243902
## 2 3.1255814
## 3 3.1745455
## 4 2.7317881
## 5 2.6944444
## 6 1.7440508

Now we can do analysis and plot the figure.

#install and library konfound
install.packages("devtools")
devtools::install_github("jrosen48/konfound")
# run the analysis for each study 
## the first study is speacial here.
## Because the observed effect is negative, accordingly the default of the function getswitch_chisq is to change it to negative significant. 
## But we want to see how far it is from the positive significant.
## We use tkonfound_fig(1, 41, 2, 43) to get the solution and plug in the results here
data$switch[1] <- -6
data$perc[1] <--6/(data$cumC[1]+data$cumD[1])
data$pdif_thr[1] <- 44/45-36/42
## run analysis for all other accumalative studies
for (i in 2:16){
 solution <- getswitch_chisq(data$cumA[i], data$cumB[i], data$cumC[i], data$cumD[i])
 data$switch[i] <- solution$total_switch
 data$pdif[i] <- data$cumD[i]/(data$cumD[i]+data$cumC[i])-data$cumB[i]/(data$cumA[i]+data$cumB[i])
 data$perc[i] <- data$switch[i]/(data$cumC[i]+data$cumD[i])
 final_table <- solution$Transfer_Table
 data$pdif_thr[i] <- final_table[2,2]/(final_table[2,1]+final_table[2,2])-final_table[1,2]/(final_table[1,1]+final_table[1,2])
 data$oder[i] <- i
 data$samplesize[i] <- data$cumA[i] + data$cumB[i] + data$cumC[i] + data$cumD[i]
}
# now we plot the figure
data$xaxis <- paste(data$year,"\n(",data$samplesize,")",sep = "")
library(ggplot2)
ggplot(data, aes(x=oder,y=pdif))+
  geom_line(aes(y=pdif), size = 1) +
  geom_point(aes(y=pdif), shape = 16, size = 4)+
  ggplot2::geom_linerange(ggplot2::aes(ymin=pdif_thr,ymax=pdif), linetype = "dashed")+
  ggplot2::geom_point(aes(y=pdif_thr),shape = 21, fill="blue",size=4)+
  #ggplot2::scale_fill_manual(values="blue")+
  ggplot2::geom_label(ggplot2::aes(label=switch), nudge_y =- (data$pdif-data$pdif_thr)/2)+
  scale_x_continuous(name="Year of Update (Cumulative Sample Size)", breaks= seq(1,16,by=1),labels=data$xaxis) +
  ggplot2::scale_y_continuous(name="Decrease in the Probability of Stroke due to Treatment")+
  theme(#axis.title = element_text(size = 18),
        #axis.text = element_text(size = 16),
        panel.grid.major = element_blank(), 
        panel.grid.minor = element_blank(),
        panel.background = element_blank(), 
        axis.line = element_line(colour = "black"),
        legend.position = "none")
 
comments powered by Disqus