A pilot study to evaluate the application of a generic protein standard panel for quality control of biomarker detection technologies
© Pang et al; licensee BioMed Central Ltd. 2011
Received: 7 March 2011
Accepted: 11 August 2011
Published: 11 August 2011
Protein biomarker studies are currently hampered by a lack of measurement standards to demonstrate quality, reliability and comparability across multiple assay platforms. This is especially pertinent for immunoassays where multiple formats for detecting target analytes are commonly used.
In this pilot study a generic panel of six non-human protein standards (50 - 10^7 pg/mL) of varying abundance was prepared as a quality control (QC) material. Simulated "normal" and "diseased" panels of proteins were prepared in pooled human plasma and incorporated into immunoassays using the Meso Scale Discovery® (MSD®) platform to illustrate reliable detection of the component proteins. The protein panel was also evaluated as a spike-in material for a model immunoassay involving detection of ovarian cancer biomarkers within individual human plasma samples. Our selected platform could discriminate between two panels of the proteins exhibiting small differences in abundance. Across distinct experiments, all component proteins exhibited reproducible signal outputs in pooled human plasma. When individual donor samples were used, half the proteins produced signals independent of matrix effects. These proteins may serve as a generic indicator of platform reliability.
Each of the remaining proteins exhibit differential signals across the distinct samples, indicative of sample matrix effects, with the three proteins following the same trend. This subset of proteins may be useful for characterising the degree of matrix effects associated with the sample which may impact on the reliability of quantifying target diagnostic biomarkers.
We have demonstrated the potential utility of this panel of standards to act as a generic QC tool for evaluating the reproducibility of the platform for protein biomarker detection independent of serum matrix effects.
Protein biomarkers for diagnosis of disease have formed the basis of clinical research proteomics for several decades [1–3]. In spite of FDA approval of various disease protein biomarkers, including CA-125 for ovarian cancer and prostate specific antigen for prostate cancer, few biomarkers are adopted in standard clinical practices . The FDA highlighted this issue as a major challenge to developing new medicinal products . A key hindrance identified was the lack of assay robustness, which may be improved using appropriate measurement standards and control materials. These reference standards ensure robust comparability of a diagnostic test for the same patient between distinct test sites, or for tests after significant time intervals.
Many protein-based detection methods suffer from a lack of standardisation with the reagents and methods employed , in a similar way to microarray assays prior to the advent of the MIAME checklist . With conventional immunoassays, significant variability may exist by using finite sources of polyclonal antibodies which differ in immunogenicity . Variable performance from distinct platforms may arise from differences in reagent quality or platform bias. Commercial immunoassay kits lack standardisation to ensure the traceability of measurements. Often the source or identity (e.g. clone number for monoclonal antibodies) of capture and detection antibodies used in kits are not stipulated . Improved standardisation may be achieved through the use of generic protein standards, demonstrating the reproducibility of the platform function. Such generic standards are emerging for mass spectrometry analysis of proteins, though they are specific to this platform rather than for broad stream applications including immunoassays .
For most protein biomarker assays, the diagnosis of diseases may be achieved by detecting the appropriate protein biomarker(s) above specified thresholds, alongside the generic QC proteins to indicate platform functionality. The change in the collective signal output profile of these QC proteins may indicate the presence of inhibitors within the biological matrix, and may infer that the robustness of detection of the target diagnostic biomarker(s) is also adversely affected.
In this paper we have prepared a panel of generic protein standards and evaluated its utility as a quality control (QC) material using the to MSD® platform. The scope of detecting each protein amidst the full panel of proteins was assessed, as well as the ability to identify small known changes in the protein composition. The panel of protein standards was also evaluated as a spike-in material, by supplementing individual donor plasma (ovarian cancer diseased and non-diseased) samples with the QC material. This pilot study revealed the value of the QC material as an indicator of platform robustness, as well as for highlighting any matrix effects associated with individual samples that may influence the reliability of detecting the target analytes within the test samples.
Preparation and storage of the generic panels of protein standards
The components and parameters of the proteins within the spike panels
(with UniProt accession number)
Spike concentration (pg/mL)
Normal panel of generic standards
"Simulated diseased" panel of standards
3 × 105
6.67 × 105
Biological test samples were spiked with the QC material, extending over a broad dynamic range (50 - 107 pg/mL) emulating the natural abundance of proteins in serological samples . If any of these component proteins provide a robust signal independent of biological matrix effects, they may be indicative of a platform being fit-for-purpose, as well as allowing assay performance comparisons between different platforms. Additionally, if a subset of the spike proteins is influenced by matrix effects associated with individual serological samples, this may bring into question the reliability of the detection of the target analytes, which may also be adversely affected by differential matrix effects.
Recombinant (mouse CCL6, mouse lungkine, chicken caronte/Fc chimera, mouse soggy from R&D Systems, Abingdon, UK) and purified proteins (firefly luciferase and chicken egg lysozyme from Sigma, Poole, UK) were reconstituted in PBS prior to gravimetric preparation of the 10× stock QC material (also termed the normal panel of generic standards) and stored at - 80°C. Refer to Additional file 1 for details of characterisation of the homogeneity, Additional file 2 for short-term stability data and Additional file 3 for long-term stability of the QC material.
Direct MSD®-based assays were constructed for each of the six spike proteins and four candidate ovarian cancer biomarkers, either alone or within serological samples. Capture antibodies (rat anti-CCL-6, rat anti-lungkine, rat anti-soggy, rat anti-epidermal growth factor receptor (EGFR), mouse anti-osteopontin, and mouse anti-interleukin-8 (IL-8) monoclonal antibodies (mAbs) and goat anti-caronte polyclonal antibody (pAb) (R&D Systems); anti-luciferase and anti-CA-125 mouse mAbs (AbCam, Cambridge, UK) and mouse anti-lysozyme mAb (Cosmo Bio, Tokyo, Japan) diluted in PBS (at 1-8 μg/mL antibody), were added to wells on either MULTI-ARRAY™ standard bind or high bind MSD® plates (Meso Scale Discovery®, Gaithersburg, USA) and incubated overnight at 4°C. The plate was decanted and each well incubated with 150 μL blocking buffer (phosphate-buffered saline (PBS), pH 7.4; 1% (w/v) BSA, 0.05% (w/v) sodium azide) on a shaker for 1 hour at room temperature. After three washes with PBS, 0.05% (w/v) polyoxyethylenesorbitan monolaurate (Tween20), 25 μL of samples and standards (six spike proteins, and human recombinant EFGR/Fc chimera, IL-8, and osteopontin (R&D Systems)) diluted in either blocking buffer, pooled normal human plasma (Firefly Scientific Limited, Manchester, UK), or single donor normal and ovarian cancer diseased plasma (Sera Laboratories International, Haywards Heath, UK) were added to each well and incubated for 1 hour at room temperature on the shaker. After three washes, 25 μL MSD® SULFO-TAG detection antibody (1 or 2 μg/mL; prepared using the manufacturer's protocol) was added to each well and incubated for 1 hour at room temperature on a shaker.
The unreacted SULFO-TAG N-hydroxysuccinimide (NHS)-ester has molecular weight of 1141 Daltons, and after the labelling reaction, the conjugated SULFO-TAG adds 1027 Daltons to the protein. As the SULFO-TAG is a small hydrophilic molecule (approximately 1 kDa), it is not expected to affect the function of large protein conjugation partners such as antibodies, especially as the SULFO-TAG is much smaller than biotin (13 kDa). Biotin NHS-ester is the most commonly used method to label antibodies, which also binds to the antibody via primary amines in the same manner as the SULFO-TAG NHS-ester and is generally not anticipated to interfere with the binding of the antibody to the cognate target antigen.
For labelling, lyophilised antibodies were directly reconstituted in PBS, pH 7.9. For antibodies sourced in solution, buffer exchange was performed with PBS, pH 7.9 using Zeba Desalt Spin columns with a 7000 Dalton molecular weight cut-off threshold (Thermo Scientific). The concentration of the antibody was then determined by the BCA protein assay (Thermo Scientific, Massachusetts, USA) using the microplate procedure. 3 nmol/μL Sulfo-Tag NHS-Ester tag (Meso Scale Discovery) was added to the antibody for the labelling step using a molar challenge ratio of 12:1, and the sample was mixed for 2 hours in the dark at room temperature. The antibodies were then buffer exchanged into PBS, pH 7.4/0.05% sodium azide using Zeba Desalt Spin columns. The concentration of the labelled antibody was then determined by BCA protein assay using the microplate procedure. Absorbance of protein conjugate at 455 nm was measured by NanoDrop (Thermo Scientific) to establish labelling ratio of Sulfo-tag to antibody. The detection antibodies subjected to labelling were goat pAbs: anti-luciferase (Promega, Southampton, UK), anti-caronte, anti-CCL-6, anti-lungkine, anti-soggy, anti-osteopontin, anti-IL-8 and anti-EGFR (R&D Systems). Rabbit detection pAbs were anti-lysozyme (Millipore, Watford, UK), and anti-CA-125 (Bioquote Limited, York, UK). Wells were washed before addition of 150 μL of MSD® Read Buffer T (with surfactant).
A voltage was applied to the carbon electrodes integrated in the plate, initiating a redox reaction involving ruthenium chemistry, resulting in the emission of light detected by the cooled charge coupled device (CCD) camera. The raw data output was analysed by MSD® Discovery Workbench 3.0 software.
Evaluating the detection of each analyte within the protein mixture
Each of the spike mixtures with five proteins was prepared in pooled normal human plasma, maintaining the designated concentrations for each analyte as outlined in Table 1 for the full complement of proteins. The six combinations of five spike proteins were assayed alongside the full panel of six spike proteins in pooled plasma and the diluent (pooled normal human plasma) as the negative control for each of the six uniplexed assays. Three separate experiments were performed to evaluate the scope of detecting each analyte amidst the complete protein mixture. Triplicate determinants were incorporated within each experiment, and the data for each separate experiment was shown.
Evaluating the ability to discriminate between two panels with variable abundance in spike proteins within pooled normal human plasma
Two distinct panels of the six protein spikes were prepared, with the composition outlined in Table 1. One panel was termed as the "normal" panel of generic standards. A second panel comprising of 1.5-3 fold changes in the composition of three component proteins (CCL6, soggy and luciferase) with all concentrations remaining within the linear working range of the assay was designated as the "simulated diseased" panel of proteins.
Uniplexed assays for each of the six proteins were performed with both panels of protein standards, incorporating triplicate determinants for all three separate experiments. The reliability of detecting known fold changes in concentrations of selected spike proteins diluted in pooled normal human plasma as the biological matrix, was evaluated by comparing the fold changes between the two protein panels. 7-point internal calibration curves were incorporated for the assay of each analyte.
Ovarian cancer model spike-in study
Single donor plasma samples (six normal and six ovarian cancer diseased) were supplemented with the QC material to incorporate a 1× working concentration of the six spike proteins and assayed for the six spike proteins and four putative ovarian cancer biomarkers, CA-125, EGFR, IL-8 and osteopontin. Experiments were performed with internal calibration curves (except for CA-125, as the recombinant protein could not be sourced) with triplicate determinants, in a randomised plate format for three separate experiments.
The MSD Discovery Workbench analysis software version 3.0 was applied to process the data. A 4-parameter logistic model was used for curve-fitting. PCA was performed using SIMCA-P (Umetrics). Linear mixed-effects models were fitted by residual (or restricted) maximum likelihood using the program "R".
Evaluating the scope to detect each component protein within the mixture
Evaluation of the robustness in detecting each component protein within the spike material comprising of six proteins in pooled normal human plasma
Detection of compositional fold-changes in selected analytes between two panels of spike proteins
Ratio of signal outputs from the "normal" to "simulated diseased" panels of protein standards (n = 3)
Ratio of interpolated concentrations from the "normal" to "simulated diseased" panels of protein standards (n = 3)
1: 1.26 ± 0.06
1: 1.72 ± 0.28
1: 0.94 ± 0.05
1: 1.71 ± 1.21
1: 1.11 ± 0.21
1: 1.20 ± 0.29
3: 1.02 ± 0.11
3: 0.96 ± 0.11
3: 1.80 ± 0.19
3: 1.86 ± 0.43
1: 0.96 ± 0.06
1: 0.88 ± 0.32
Generally, interpolation increased the variability in the ratios between the "normal" and "simulated diseased" panels, with CVs of 5.91 - 70.76%, compared with the CV range of 0.19 - 10.78% associated with the ratios between the two panels derived from the mean signal output data. Hence the material may be used to evaluate the robustness the platform by evaluating the performance of these assays in terms of the signal output, rather than by interpolation from the internal standard curves.
Implementation of the QC spike protein in a model system
The QC material was spiked into six single donor ovarian cancer plasma samples (termed as OC samples) and six single donor normal plasma samples. Three separate experiments were performed to evaluate the reproducibility of detecting the six spiked analytes, in addition to three putative ovarian cancer biomarkers, IL-8, EGFR, and osteopontin, alongside the FDA-approved marker CA-125. If all four candidate biomarkers are appropriate markers for ovarian cancer, differential expression of these cancer markers is anticipated between the normal and OC samples. It is anticipated that the spiked proteins are detectable at the same abundance if all samples are supplemented with the same quantity of QC material.
To demonstrate platform robustness, the signal output of the component proteins should not exhibit significant statistical difference between the individual donor plasma samples, or between technical replicates and separate experimental runs. The component proteins exhibiting differential signals between distinct samples would be indicative of the existence of matrix effects that may also influence credibility in the data derived for the detection of the target analytes. In this pilot study, the purpose of the generic QC material was illustrated by immunoassay using a single set of antibodies. These data form a preliminary finding, and all aspects of variability inducing parameters should be subjected to a full validation. A larger scale study with a greater number of individual donor plasma samples, platforms and antibodies may be required to determine the true criteria for acceptable variability in the signal outputs for the component proteins. Generic protein standards have not previously been applied to immunodetection methods, however, it is anticipated that this QC material may be applied to all protein-based detection methods, including mass spectrometry and other immunoassays.
One limitation with immunoassays is the potential for cross-reactivity between proteins and non-cognate antibodies that may impair the detection of the target analyte. We have shown that each protein was detectable amidst the presence of the five non-cognate target proteins. However, with the ovarian cancer pilot study involving single donor plasma samples, the CCL6, lungkine and luciferase assays exhibited differential degrees of cross reactivity, giving rise to the variable signal outputs observed when the QC material was spiked in different patient samples. Interestingly, there was concordance among these three assays in terms of their signal profiles across the twelve donor samples, in spite of the concern regarding the robustness of the CCL6 assay. As this cross-reactivity was not due to the presence of the five other proteins within the spike panel per se, it was most probable that the variable protein composition of the biological matrix bound non-specifically to some of the antibodies within these assays.
Lungkine as well as CCL6 and luciferase, at their current designated spike concentrations could not be used as the QC material of platform reliability for immunoassays with single donor plasma samples, with the current set of antibody pairs. However, this does not exclude their use as spike materials using distinct antibody pairs or for assays without antibodies (e.g. mass spectrometry), or as a spike material for other biological samples. These alternative uses will require further investigation on a case-specific basis. Nonetheless, with the existing assay conditions for these three component proteins, their signal outputs are valuable indicators of matrix effects associated with individual samples. Identifying matrix effects is of importance as this phenomenon may also adversely affect the accuracy of quantifying the presence of the target analyte.
Hence, this subset of proteins may collectively highlight the need for caution when evaluating the level of robustness for the test analyte data. Conversely, soggy, lysozyme and possibly caronte do not exhibit sensitivity to sample matrix effects. Hence these latter three proteins may serve as indicators of the robustness of the platform.
Another limitation of many antibody-based assays is the narrow linear range of the working assays, especially when calibrants are spiked into a biological matrix (e.g. plasma) rather than a buffer. Assays may not always exhibit a sufficiently broad linear range encompassing the full concentration coverage for physiological status, leading to interpolation of some data from a non-linear portion of the fitted curve. Albeit curve-fitting within an experiment may be robust, calibration curves may be susceptible to changes in the trendline thus increasing variability between distinct experiments. This subsequently reduces the capacity for robust inter-experiment comparison. High variability with the interpolated analyte concentrations via internal calibration curves from different experimental runs is observed when evaluating the the normal and "simulated disease" protein panels, as well as for IL-8 detection.
In this instance, IL-8 concentration in the OC samples coincided with the linear portion of the IL-8 standard curve, whereas the concentrations of IL-8 associated with normal plasma fell outside this robust range. This brings into question the need for a standard curve, given that detection above an assigned cut-off threshold (e.g. the mean of the normal IL-8 plasma concentration ± 3 SD) may suffice for the diagnosis of disease. The incorporation of internal standard curves also utilises numerous additional wells to ensure there are sufficient datapoints for robust curve fitting, with adequate technical replication. However, using the QC material in lieu of internal calibrants for each analyte may demonstrate the platform is fit-for-purpose and improve on assay throughput by reducing the number of wells consumed by calibrants.
We have shown the utility of the generic protein QC material as an indicator of platform performance and matrix effects, using immunoassays on the MSD Sector 6000 Imager platform. The QC material may also be used alone to evaluate bias introduced by variable instruments, operators or reagents. This panel of proteins has also exhibited suitable homogeneity and stability for this application. The material can be incorporated into a "quality metrics" toolkit for assessing protein biomarker platform performance, addressing issues associated with insufficient assay robustness delineated in the FDA's Critical Path Initiative in 2004.
Bovine serum albumin
epidermal growth factor receptor
Meso Scale Discovery®
median fluorescent intensity
phosphate buffered saline
The work described in this paper was funded by the UK National Measurement System.
- Anderson NL, Anderson NG: The human plasma proteome: history, character, and diagnostic prospects. Mol Cell Proteomics. 2002, 1: 845-867. 10.1074/mcp.R200007-MCP200.PubMedView Article
- Schrohl AS, Wurtz S, Kohn E, Banks RE, Nielsen HJ, Sweep FC, Brunner N: Banking of biological fluids for studies of disease-associated protein biomarkers. Mol Cell Proteomics. 2008, 7: 2061-2066. 10.1074/mcp.R800010-MCP200.PubMedPubMed CentralView Article
- Polanski M, Anderson NL: A list of candidate cancer biomarkers for targeted proteomics. Biomark Insights. 2007, 1: 1-48.PubMedPubMed Central
- Amur S, Frueh FW, Lesko LJ, Huang SM: Integration and use of biomarkers in drug development, regulation and clinical practice: a US regulatory perspective. Biomark Med. 2008, 2: 305-311. 10.2217/175203126.96.36.1995.PubMedView Article
- US Food and Drug Administration: Challenge and Opportunity on the Critical Path to New Medicinal Products. 2004, USA, [http://www.who.int/intellectualproperty/documents/en/FDAproposals.pdf]
- Shulman G: Quality of commercially available controls in laser immunonephelometry. Ann Clin Biochem. 1980, 17: 178-182.PubMedView Article
- Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C, Aach J, Ansorge W, Ball CA, Causton HC, Gaasterland T, Glenisson P, Holstege FC, Kim IF, Markowitz V, Matese JC, Parkinson H, Robinson A, Sarkans U, Schulze-Kremer S, Stewart J, Taylor R, Vilo J, Vingron M: Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat Genet. 2001, 29: 365-371. 10.1038/ng1201-365.PubMedView Article
- Hoofnagle AN, Wener MH: The fundamental flaws of immunoassays and potential solutions using tandem mass spectrometry. J Immunol Methods. 2009, 347: 3-11. 10.1016/j.jim.2009.06.003.PubMedPubMed CentralView Article
- Richens JL, Urbanowicz RA, Metcalf R, Corne J, O'Shea P, Fairclough L: Quantitative Validation and Comparison of Multiplex Cytokine Kits. J Biomol Screen. 2010, 15: 562-568. 10.1177/1087057110362099.PubMedView Article
- Kolker E, Hogan JM, Higdon R, Kolker N, Landorf E, Yakunin AF, Collart FR, van Belle G: Development of BIATECH-54 standard mixtures for assessment of protein identification and relative expression. Proteomics. 2007, 7: 3693-3698. 10.1002/pmic.200700088.PubMedView Article
- Liu T, Qian WJ, Gritsenko MA, Xiao W, Moldawer LL, Kaushal A, Monroe ME, Varnum SM, Moore RJ, Purvine SO, Maier RV, Davis RW, Tompkins RG, Camp DG, Smith RD: High dynamic range characterization of the trauma patient plasma proteome. Mol Cell Proteomics. 2006, 5: 1899-1913. 10.1074/mcp.M600068-MCP200.PubMedPubMed CentralView Article
- Chowdhury F, Williams A, Johnson P: Validation and comparison of two multiplexed technologies, Luminex and Mesoscale Discover, for cytokine profiling. J Immunol Methods. 2009, 55-64.
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.