Using the contribution matrix to evaluate complex study limitations in a network meta-analysis: a case study of bipolar maintenance pharmacotherapy review

The problem of “enrichment design” in bipolar maintenance pharmacotherapy studies

The appraisal of the impact of study limitations in the NMA of the maintenance pharmacotherapy of bipolar disorder presents an additional interesting feature that renders this assessment even more challenging.

Bipolar disorder is a psychiatric disease in which patients typically show recurrent episodes of both manic and depressive episodes. While acute treatment is aimed at treating the acutely manic or depressive symptoms, long-term maintenance treatment is usually necessary to minimise the risk of recurrence of both manic and depressive episodes. Bipolar patients recruited into maintenance or prophylactic studies are usually in an euthymic phase, without acute symptoms. In some of these clinical trials, however, only the participants who had achieved remission of the index acute manic or depressive episode by treatment with a certain drug were included in the maintenance phase of the trial and then were randomised to continue the same drug or switch to another active drug (or placebo). Such a study design is called ‘enrichment design’, as it is ‘enriched’ by patients whose acute manic or depressive episode had responded to the drug used in the acute phase.

This study design has many limitations [5]. In particular, its results will tend to favour the drug that was effective in the acute phase mainly in the prevention of future episodes of the same polarity as the index episode and not necessarily in the prevention of episodes of the opposite polarity. The risk of bias due to the enrichment design therefore has a direction. For example, if a study included only those who had remitted from a manic episode on drug X and randomised them to continue on drug X or to switch to drug Y in order to compare these interventions’ efficacy in preventing a new manic or depressive episode, it is easy to foresee that such patients’ future manic episodes would be relatively responsive to drug X but possibly not their depressive episodes. On the other hand, drug Y is clearly not favoured in any direction as the patients had been originally selected as responders to drug X.

In the present article we use a published NMA as a working example and present a transparent and systematic method to assess how study limitations of individual randomised controlled trials (RCTs), including those due to the enrichment design, impact on the quality of evidence in the NMA. In NMA, it is almost certain that confidence in estimates will vary from comparison to comparison. We therefore essayed to appraise the quality of evidence for each comparison contained in the network. In the following we will illustrate how study limitations without direction (i.e. risks of bias usually assessed according to the Cochrane Handbook) and then those with direction (i.e. risk of bias due to the enrichment design) can be quantitatively summarised and evaluated to characterise each network estimate.

Materials

The NMA in question represents a systematic review of randomised controlled trials that compared active treatments for bipolar disorder (or placebo), either as monotherapy or as add-on treatment, for at least 12 weeks [4]. The primary outcome was the number of participants with recurrence of any mood episode this primary outcome was a combination of two secondary outcomes, namely the number of participants with recurrence of a manic episode and those with recurrence of a depressive episode. All in all we identified and included 33 randomised controlled trials that examined 17 maintenance pharmacotherapies for bipolar disorder in 6846 participants. Figure 1 shows the network formed by the identified comparisons in this NMA. We conducted a random-effects network meta-analysis within a Bayesian framework using Markov chain Monte Carlo in OpenBUGS 3.2.2. [6].

Assessment of risk of bias of each study and of each direct comparison

Two assessors rated the risk of bias (RoB) of each RCT according to the Cochrane Handbook risk of bias tool [7]. The RoB examines the key methodological issues in a randomised trial, such as generation of random sequence, concealment of allocation, blinding of participants, blinding of therapists, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. We also assessed whether the definitions of the mood episode relapse or recurrence were explicit/operationalised or not in the primary studies, and the sponsorship bias. We rated an item at unclear risk of bias when we did not find sufficient information to judge it at either high or low risk.

We examined the validity of this classification by pooling and comparing RR for studies rated as low, moderate or high risk of bias in a comparison if this comparison had an enough number of included trials to enable such validation.

The above method of summarising RoBs of various domains into RoB of a study and then summarising study RoBs into RoB of a comparison is admittedly to a certain extent arbitrary. However, it must be noted that we can use the same logic and calculations, as we demonstrate below, to synthesise these characteristics at the level of each pairwise comparison into those at the level of each network estimate. In the following we shall therefore use the definitions above to illustrate our method.

Assessment of ‘enrichment design’ for each study and for each direct comparison

We also evaluated whether each study used the enrichment design in relation with the polarity of the mood episode. The influence of the enrichment design was assessed separately for the two secondary outcomes: prevention of depressive episodes and prevention of manic episodes. Participants were considered to be enriched for a certain drug for depressive episode relapse (depressive enrichment) when they had been recruited at an acute depressive episode and investigated for the depressive episode relapse after being stabilised by that drug, and participants were considered to be enriched for a drug for manic episode relapse (manic enrichment) when they had been recruited at an acute manic episode and investigated for the manic episode relapse after being stabilized by that drug.

We first calculated the percentages of both depressive and manic enrichment for each study according to the number of participants in acute depressive or manic episode at recruitment, and then we estimated the corresponding percentages for each direct comparison consisting of one or more studies with consideration of the direction of enrichment for each study. For example, if a direct comparison A vs B consisted of two studies, one of which (n = 100) did not use the enrichment design but the other (n = 200) recruited patients at their depressive episodes and treated them with drug A, then this direct comparison would have 67 % (200/300) of participants enriched for depressive relapse in favour of drug A, 33 % not enriched for depressive relapse and 100 % not enriched for manic relapse.

Using the contribution matrix to quantify the influence of RoB and of enrichment design in each network estimate

We used a recently developed tool for NMA, called the contribution matrix, that quantifies how much each direct comparison in the network contributes to each network estimate in the NMA [8, 9].

Let’s take a simple, triangular network ABC. We first calculate the direct estimate comparing A vs B, A vs C and B vs C by pooling trials comparing A vs B, A vs C, and B vs C, respectively. We denote these as D_AB, D_AC and D_BC. In the NMA of the full triangle, the mixed or network estimate comparing A vs B comes from the direct comparison D_AB and the indirect comparison I_AB consisting of D_BC and D_CA via C. For the simple situation in which each of the direct estimates has the same variance, the network estimate N_AB is (2*D_AB + (D_AC−D_BC))/3. Thus, for the mixed estimate (or also called network estimate) N_AB, the three direct estimates D_AB, D_AC and D_BC makes contributions of 50, 25 and 25 %, respectively.

When the network structure is complex and when variances are not equal, calculating the contribution of each direct estimate to each network estimate in the NMA is more complicated. In general more weight is given to direct comparisons with more precision and to those that are more central to the network and thus contribute to more indirect comparisons. Using the netweight command in Stata [10], we calculated the contribution matrix showing contributions from each direct comparison to the network comparisons. The weight that each direct comparison contributes to the network estimates is a combination of the variance of the direct comparison and the network structure: a comparison with much direct information not only contributes much to the network estimate of that comparison but also is more influential on its neighboring comparisons than its remotely placed comparisons, and a comparison for which little direct evidence exists benefits most from the rest of the network. Using netweight, ¹ the percentage contribution of each direct comparison to each network estimate is summarised in a matrix with rows representing network estimates and columns representing the available direct comparison in the network.

In order to characterize the RoB of each network estimate, we multiplied the contributions from direct comparisons at low, moderate or high risk of bias, respectively, by the contribution percentage that each direct estimate is making to the network estimate. This calculation provided the percentage of contributions from direct estimates rated at low, moderate or high risk of bias, respectively, to each network estimate.

In order to quantify the contribution from enrichment design to each network estimate, we multiplied the percentage of enrichment for each direct comparison by the contribution percentage that each direct estimate is making to the network estimate. For a particular network estimate of A vs B, this calculation provided the percentage of contributions from enriched studies favouring A, those favouring B, those dis-favouring A (i.e. favouring another drug C over A), those dis-favouring B, and those that involve neither A nor B (enrichment of unknown direction). The remaining came from non-enriched studies. We summed up the percentage of contributions from studies favouring A and those dis-favouring B as the percentage of enrichment favouring A. In the same manner, the percentage of enrichment favouring B was calculated by summing up the percentage of contributions from studies favouring B and those dis-favouring A.

RoB of network estimates

For example, 0.2, 22.5 and 77.6 % of the contributions to the network estimate for placebo vs lithium in preventing any mood episode come from direct comparisons with low, moderate and high, respectively, risks of bias. Figure 2 graphically presents the respective contributions for major comparisons in the network.

Thus the network estimate of efficacy of lithium over placebo to prevent any mood episode was based nearly 80 % on studies at high risk of bias and nearly 20 % on studies at moderate risk of bias. This estimate would then be considered quite likely to be biased, either in the direction of under- or over-estimation.

Contribution of the enrichment design to network estimates

We graphically showed the percentages of contributions of enriched vs non-enriched studies to effect estimates of main comparisons against placebo in the network (Fig. 3).

Thus, the network estimate of the efficacy of lithium vs placebo to prevent a depressive episode received a small contribution from studies enriched in favour of lithium, and a similarly small contribution from studies enriched in favour of placebo but the bulk of the evidence was from non-enriched studies. By contrast, the network estimates of fluoxetine or lamotrigine in the prevention of depressive episodes received nearly half or more contribution from studies enriched in favour of the active drugs: it is quite possible that the network estimates for these drugs are overestimated.