Computer programs for the concordance correlation coefficient

doi:10.1016/j.cmpb.2007.07.003

Computer Methods and Programs in Biomedicine

Volume 88, Issue 1, October 2007, Pages 62-74

https://doi.org/10.1016/j.cmpb.2007.07.003 Get rights and content

Abstract

The CCC macro is presented for computation of the concordance correlation coefficient (CCC), a common measure of reproducibility. The macro has been produced in both SAS and R, and a detailed presentation of the macro input and output for the SAS program is included. The macro provides estimation of three versions of the CCC, as presented by Lin [L.I.-K. Lin, A concordance correlation coefficient to evaluate reproducibility, Biometrics 45 (1989) 255–268], Barnhart et al. [H.X. Barnhart, J.L. Haber, J.L. Song, Overall concordance correlation coefficient for evaluating agreement among multiple observers, Biometrics 58 (2002) 1020–1027], and Williamson et al. [J.M. Williamson, S.B. Crawford, H.M. Lin, Resampling dependent concordance correlation coefficients, J. Biopharm. Stat. 17 (2007) 685–696]. It also provides bootstrap confidence intervals for the CCC, as well as for the difference in CCCs for both independent and dependent samples. The macro is designed for balanced data only. Detailed explanation of the involved computations and macro variable definitions are provided in the text. Two biomedical examples are included to illustrate that the macro can be easily implemented.

Introduction

In the health sciences, it is often necessary to study the reproducibility of continuous measurements made using a certain diagnostic tool or method. As technology brings forth new tools and methods, we are interested in evaluating the consistency of evaluations made using the new method as well as comparing this measure to the current gold standard if one exists. The concordance correlation coefficient (CCC) provides a means for examining the reproducibility of continuous measurements made by multiple raters using a single method or by two or more raters using two methods. Several other reproducibility measures are available, such as the Pearson correlation coefficient, the intraclass correlation coefficient [4], [5], and the within-subject coefficient of variation [6]. In general, these measures do not address both precision and accuracy as does the concordance correlation coefficient; however, the equivalency and similarities of the intraclass correlation coefficient to the concordance correlation coefficient under certain scenarios has been discussed by Nickerson [7], Carrasco and Jover [8], and Barnhart et al. [9]. The CCC measures how far the fitted linear relationship of two variables X and Y deviates from the concordance line (accuracy) and how far each observation deviates from the fitted line (precision).

There are several forms of the concordance correlation coefficient (CCC). The CCC for two raters evaluating a single method was presented by Lin [1], [10]. Barnhart et al. [2] considered an overall CCC for multiple raters evaluating a single method, which is equivalent to the functions presented by Lin [1], [11] and King and Chinchilli [12]. Williamson et al. [3] examined the agreement between two methods for multiple raters. The estimation of these forms of the CCC can be conducted by estimating the means, variances, and covariances for the ratings. As an alternative to estimation by the method of moments, Carrasco and Jover [8] propose estimating the CCC using variance components from a mixed model. Several methods have been proposed for inference regarding the CCC. For two raters evaluating a single method, Lin [1] proposed an asymptotic approach for computing variance estimates. For the overall CCC for multiple raters evaluating a single method, King and Chinchilli [12] conducted inference using a U-statistics approach, while Barnhart et al. [2] explored both a GEE and bootstrap approach. Williamson et al. [3] explored permutation testing and the bootstrap for agreement between two methods for multiple raters.

Here we describe the CCC macro written in SAS [13] which is designed to estimate all three forms of the CCC. The macro also provides confidence intervals for these estimates as well as for the difference in two CCCs. Where applicable, an asymptotic confidence interval is computed for both the estimation of the CCC as well as for the estimation of the difference in CCCs [1]. Otherwise, bootstrap confidence intervals are computed with the CCC macro [2], [3], [14]. The CCC macro was also written in R with the same input and output [15]. General examples for the macro call in R are included in Section 3.3 detailing the required parameters and output for each analysis, but all of the practical examples are presented in SAS.

Section snippets

Estimation of concordance correlation coefficients

A major component of the CCC macro is to provide an estimate of the concordance correlation coefficient (CCC). The formula used to compute the CCC is dependent upon the number of raters and the number of methods specified by the user. When a single method is evaluated by two raters, the CCC proposed by Lin [1] is used: $ρ^{c} = \frac{2 σ_{12}}{σ_{1}^{2} + σ_{2}^{2} + {(μ_{1} - μ_{2})}^{2}},$ where μ₁ and $σ_{1}^{2}$ represent the mean and variance for the first rater, μ₂ and $σ_{2}^{2}$ represent the mean and variance for the second rater, and σ₁₂ is the

Macro overview

The CCC macro is designed to perform a variety of analyses pertaining to the concordance correlation coefficient. The macro can provide estimates of the overall CCC along with confidence intervals, either asymptotic or bootstrap, for one method with multiple raters, or for two methods with multiple raters (i.e. an experimental method and a gold standard). It will also perform estimation of the difference in two CCCs under the assumptions of both independence and dependence, and calculate

Biochemical in vitro assays

In a study of biochemical in vitro assays, researchers were interested in the reproducibility of toxicity measurements made by two different assays: cellular adenosine triphosphate activity using cell line 76 (ATP-76) and cellular adhesion using cell line 74 (CLA-74) [1]. The percent cell function measured by each assay at two independent trials conducted 1 week apart was recorded for 10 materials of varying toxicity. We are interested in assessing the agreement of the measurements produced

Macro availability and run time

The CCC macro written in SAS [13], R [15], or S-PLUS [18] can be obtained by directly contacting the authors or by accessing the following websites: http://www.statisticaldisplays.org or http://www.personal.psu.edu/hxl28/research/CCCprogram. The SAS macro was written in v9, but because some of the IML functions available in SAS v9 are not available in SAS v8, a v8 macro also was created. This macro performs the same functions, but requires more computing time, and can be obtained following the

Acknowledgements

Sara Crawford's research is supported by an appointment to the Research Participation Program at the Centers for Disease Control and Prevention, National Center for Infectious Diseases, Division of Parasitic Diseases administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and CDC. The research of Andrzej Kosinski and Huiman Barnhart is supported by the National Institutes of Health Grant R01 MH70028. We thank two

References (18)

J. Lee et al.
Statistical evaluation of agreement between two methods for measuring a quantitative variable
Comput. Biol. Med.
(1989)
L.I-K. Lin
A concordance correlation coefficient to evaluate reproducibility
Biometrics
(1989)
H.X. Barnhart et al.
Overall concordance correlation coefficient for evaluating agreement among multiple observers
Biometrics
(2002)
J.M. Williamson et al.
Resampling dependent concordance correlation coefficients
J. Biopharm. Stat.
(2007)
J.L. Fleiss
The Design Analysis of Clinical Experiments
(1986)
H. Quan et al.
Assessing reproducibility by the within-subject coefficient of variation with random effects models
Biometrics
(1996)
C.A.E. Nickerson
A note on “a concordance correlation coefficient to evaluate reproducibility”
Biometrics
(1997)
J.L. Carrasco et al.
Estimating the generalized concordance correlation coefficient through variance components
Biometrics
(2003)
H.X. Barnhart et al.
An overview on assessing agreement with continuous measurement
J. Biopharm. Stat.
(2007)

There are more references available in the full text version of this article.

Cited by (61)

Development of a blood calcium test for hypocalcemia diagnosis in dairy cows
2022, Research in Veterinary Science
Citation Excerpt :
Bland-Altman plots depict the difference between methods (1-step colorimetric assay and CalTreAx), in which good agreement is shown by values that lie close to the 0 mean difference line and between the 95% confidence interval limits of agreement (Bland and Altman, 2007). Lin's concordance correlation coefficients (CCC) were also included to determine agreement among results of the 1-step colorimetric assay and CalTreAx; the coefficient ranges from −1 to 1 with −1, 0 and 1 indicating perfect disagreement, independence and perfect agreement, respectively (Crawford et al., 2007). Contingency tables based on the tCa concentration categories were created to calculate a Cohen's Kappa (weighted) coefficient to indicate the agreement of subclinical hypocalcemia (1.37 to 2.12 mmol/L) and clinical hypocalcemic (< 1.37 mmol/L) diagnosis beyond chance between the 1-step colorimetric assay and CalTreAx.
Hypocalcemia, defined as total blood calcium concentrations below 2.1 mM, has detrimental impacts on welfare, production and reproduction in dairy cattle. Yet, no cow-side test exists for testing total bovine blood calcium. Here, we modified the split trehalase complementation assay to detect total calcium in serum by incorporating calmodulin and the M13 peptide as fusion partners to the trehalase fragments. In the presence of calcium, calmodulin undergoes a conformation change and gains strong affinity for M13 peptide. A high reactive assay for calcium was developed with detection threshold of 10 uM and dynamic range between 1 uM and 1 mM. The addition of a specific concentration of calcium chelator, EDTA, in mild acidic conditions, shifted the dynamic range to physiological calcium concentrations and transformed the sensor from ionized calcium sensor to total calcium sensor. The sensor was validated on a collection of 213 bovine serum samples by comparison with quantitative colorimetric calcium test. A correlation coefficient of 0.81 was achieved and the accuracy of detecting subclinical hypocalcemia was 0.86 and specificity of 100%. The area under the Receiver Operating Characteristic curve was 0.93. The concordance correlation coefficient (0.80), Bland-Altman plot and weighted Kappa coefficient (0.71) demonstrated a substantial agreement between both methods. In conclusion, a novel total calcium test was developed that can be used as a convenient high throughput laboratory test and with potential to be incorporated into a version compatible with on-farm testing.
Validation of 2 urine pH measuring techniques in a prepartum negative dietary cation-anion difference diet and the relationship with production performance
2022, JDS Communications
Negative dietary cation-anion difference (DCAD) diets have been implemented to combat hypocalcemia, a common peripartal disease in dairy cows; however, the extent of compensatory metabolic acidosis necessary and the subsequent effects on performance are still debated. Additionally, there is a need for an inexpensive, accurate method to measure urine pH on farm during the prepartum period to assess the extent of metabolic acidosis achieved by negative DCAD diets. Therefore, this experiment was conducted to determine the accuracy of Fisher pH sticks (pHF; ThermoFisher Scientific) and pHion balance test strips (pHI; pHion Balance) compared with a portable pH meter (pHP; Accumet AP115, ThermoFisher Scientific) in measuring urine pH (UpH) and the effect of UpH on pre- and postpartum dry matter intake (DMI), milk, and milk composition yields. Cows consumed a total mixed ration with a DCAD of −118 mEq/kg for 4 wk prepartum and 397 mEq/kg for 4 wk postpartum. Prepartum UpH measurements (n = 75) for each cow were averaged and used to classify cows in terms of urine pH as low (UpH ≤5.54; mean ± standard deviation; 5.44 ± 0.07), medium (UpH >5.54 and ≤5.90; 5.67 ± 0.09), or high (UpH >5.90; 6.42 ± 0.36). Cows were milked twice a day, and milk samples were taken on d 7 ± 1.3, 14 ± 1.4, and 28 ± 1.1 relative to calving. Milk yield and DMI were recorded daily and averaged weekly. Bland-Altman plots and Lin's concordance correlation coefficient (CCC) were used to assess the agreement between pHP and pHF or pHI (n = 375). Receiver operating characteristic curves were used to determine the threshold with pHF and pHI that best discriminated between UpH >5.75 and ≤5.75 compared with pHP, and area under the curve (AUC) was used to assess the accuracy. At the UpH threshold of 5.75 for pHF and pHI, the sensitivity, specificity, and AUC were 89.5 and 87.4, 99.1 and 97.0, and 0.94 and 0.92, respectively. The CCC was 0.93 for pHF and pHI, indicating near-perfect agreement with pHP. The UpH did not affect pre- or postpartum DMI. There was a tendency for a UpH × week interaction for milk yield, in which milk yields were less for cows in the low and medium groups. In conclusion, pHI and pHF are accurate measurements for UpH, and UpH did not affect DMI; however, when UpH was low or medium, milk yield was decreased at wk 1 postpartum.
Validation of a point-of-care handheld blood total calcium analyzer in postpartum dairy cows
2021, JDS Communications
Our objective was to validate a point-of-care handheld blood total calcium analyzer (Ca meter, CM; TD-5220 Vet Ca²⁺, TaiDoc, New Taipei, Taiwan) to estimate circulating Ca concentrations in postpartum dairy cows. Whole blood was collected from 251 multiparous cows between 1 and 4 d in milk from 2 commercial dairy herds in Ontario, Canada. Blood total calcium concentration (tCa) was analyzed in whole blood, fresh plasma, and thawed plasma, and compared with tCa results from thawed serum analyzed in a diagnostic laboratory (using a Cobas Calcium Gen 2 kit, Roche Diagnostics, Indianapolis, IN) as the reference test (RT). Lin's concordance correlation coefficient (βrho;) and Bland-Altman (B-A) plots were assessed to evaluate the agreement between the RT and CM results in each type of sample. Receiver operating characteristic curve analyses were used to describe the accuracy of each test against the categorized RT results (at a cut-point of ≤2.14 mmol/L). Samples where the meter gave a nonquantitative result (“high” or “low”; thawed plasma: 3/247; fresh plasma: 6/100; and whole blood: 20/98) were not included in the βrho; and B-A analyses. Lin's correlation coefficients demonstrated poor agreement between tests (thawed plasma: βrho; = 0.16; fresh plasma: βrho; = 0.21; and whole blood: βrho; = 0.23). Fresh plasma (using a cut-point of 2.55 mmol/L as measured on the CM) had the greatest diagnostic sensitivity (72%), specificity (86%), and accuracy (77%) for determining subclinical hypocalcemia, but that would still misclassify 23% of samples. In addition to substantial variability, the B-A plots revealed bias with changing concentrations of calcium. Because of low sensitivity on whole blood (58%) or thawed plasma (56%), measurement with the CM is not recommended on these types of samples. This rapid and low-cost meter was not sufficiently accurate to quantify blood Ca concentration, but when used with fresh plasma it might be useful as a screening tool for subclinical hypocalcemia.
MR fingerprinting with simultaneous T <inf>1</inf> , T <inf>2</inf> , and fat signal fraction estimation with integrated B <inf>0</inf> correction reduces bias in water T <inf>1</inf> and T <inf>2</inf> estimates
2019, Magnetic Resonance Imaging
MR fingerprinting (MRF) sequences permit efficient T₁ and T₂ estimation in cranial and extracranial regions, but these areas may include substantial fat signals that bias T₁ and T₂ estimates. MRI fat signal fraction estimation is also a topic of active research in itself, but may be complicated by B₀ heterogeneity and blurring during spiral k-space acquisitions, which are commonly used for MRF. An MRF method is proposed that separates fat and water signals, estimates water T₁ and T₂, and accounts for B₀ effects with spiral blurring correction, in a single sequence.
A k-space-based fat-water separation method is further extended to unbalanced steady-state free precession MRF with swept echo time. Repeated application of this k-space fat-water separation to demodulated forms of the measured data allows a B₀ map and correction to be approximated. The method is compared with MRF without fat separation across a broad range of fat signal fractions (FSFs), water T₁s and T₂s, and under heterogeneous static fields in simulations, phantoms, and in vivo.
The proposed method's FSF estimates had a concordance correlation coefficient of 0.990 with conventional measurements, and reduced biases in the T₁ and T₂ estimates due to fat signal relative to other MRF sequences by several hundred ms. The B₀ correction improved the FSF, T₁, and T₂ estimation compared to those estimates without correction.
The proposed method improves MRF water T₁ and T₂ estimation in the presence of fat and provides accurate FSF estimation with inline B₀ correction.
Accuracy of a cow-side test for the diagnosis of hyperketonemia and hypoglycemia in lactating dairy cows
2017, Research in Veterinary Science
Citation Excerpt :
Bland-Altman plots depict the difference between methods, in which good agreement is shown by values that lie close to the 0 mean difference line and between the 95% confidence interval limits of agreement (Bland and Altman, 2007). Lin's concordance correlation coefficients (CCC) were also included to determine agreement amongst results; the coefficient ranges from − 1 to 1 with − 1, 0 and 1 indicating perfect disagreement, independence and perfect agreement, respectively (Crawford et al., 2007). Contingency tables based on the BHBA and glucose concentration categories were created to calculate a Cohen's Kappa (weighted) coefficient to indicate the agreement of ketosis and hypoglycemia diagnosis beyond chance.
The objective of this study was to evaluate the use of a cow-side device (FreeStyle Precision Neo™) to diagnose ketosis and hypoglycemia based on measures of blood β-hydroxybutyrate (BHBA) and glucose. Eleven commercial dairy farms were visited and blood samples were taken from Holstein cows between 2 and 14 days in milk, yielding 441 samples for BHBA analysis and 308 samples for glucose analysis. Concentrations of BHBA and glucose were measured in two ways, 1) using the cow-side device with whole blood immediately after sampling and 2) serum samples analyzed with a standard laboratory assay (Animal Health Laboratory, University of Guelph, Canada). The accuracy of the device was determined by comparing the results to the laboratory method as well as the ability to diagnose ketosis (BHBA ≥ 1.2 mmol/L) and hypoglycemia (glucose < 2.5 mmol/L). The concordance correlation coefficient (CCC), Bland-Altman plot and Kappa coefficient were calculated to evaluate agreement between the 2 methods using SAS (version 9.3). The CCC was 0.92 for BHBA and 0.56 for glucose measurements. The 95% confidence intervals of the Bland-Altman plot encompassed 97% and 95% of the mean difference between methods for BHBA and glucose measurements, respectively. The Kappa coefficients were 0.78 for BHBA and 0.23 for glucose measurements. These results indicate that the cow-side device is accurate for rapid measurement of blood BHBA and diagnosis of ketosis on farms but is not accurate for measurement of blood glucose concentrations and diagnosis of hypoglycemia.
Multi-frequency interpolation in spiral magnetic resonance fingerprinting for correction of off-resonance blurring
2017, Magnetic Resonance Imaging
Magnetic resonance fingerprinting (MRF) pulse sequences often employ spiral trajectories for data readout. Spiral k-space acquisitions are vulnerable to blurring in the spatial domain in the presence of static field off-resonance. This work describes a blurring correction algorithm for use in spiral MRF and demonstrates its effectiveness in phantom and in vivo experiments. Results show that image quality of T1 and T2 parametric maps is improved by application of this correction. This MRF correction has negligible effect on the concordance correlation coefficient and improves coefficient of variation in regions of off-resonance relative to uncorrected measurements.

View all citing articles on Scopus

View full text

Computer programs for the concordance correlation coefficient

Abstract

Introduction

Section snippets

Estimation of concordance correlation coefficients

Macro overview

Biochemical in vitro assays

Macro availability and run time

Acknowledgements

Comput. Biol. Med.

A concordance correlation coefficient to evaluate reproducibility

Biometrics

Overall concordance correlation coefficient for evaluating agreement among multiple observers

Biometrics

Resampling dependent concordance correlation coefficients

J. Biopharm. Stat.

The Design Analysis of Clinical Experiments

Assessing reproducibility by the within-subject coefficient of variation with random effects models

Biometrics

A note on “a concordance correlation coefficient to evaluate reproducibility”

Biometrics

Estimating the generalized concordance correlation coefficient through variance components

Biometrics

An overview on assessing agreement with continuous measurement

J. Biopharm. Stat.