Article Text
PDF
Abstract
Objective Several definitions have been proposed to distinguish the daughter sac when treating unruptured intracranial aneurysms. The aim of this study was to evaluate interobserver variability of aneurysm morphology, including the daughter sac, using criteria from the International Study of Unruptured Intracranial Aneurysms (ISUIA) and the Unruptured Cerebral Aneurysm Study of Japan (UCAS).
Materials and methods After approval by the institutional review board, we analyzed three morphological features (daughter sac, lobulation, and irregular margin) from the ISUIA and UCAS using angiographic images from 102 saccular aneurysms. Four independent readers interpreted each morphological criterion using dichotomized scales (existence or not). The κ statistic was used to measure interobserver agreement, and κ>0.6 was considered substantial agreement.
Results For discrimination of the daughter sac, interobserver agreement among the four readers was substantial using the UCAS criteria (k=0.626 for two-dimensional (2D) and 0.659 for three-dimensional (3D) images) but not for the ISUIA criteria (k=0.487 for 2D and 0.473 for 3D images; significant difference). Irrespective of the images used, pairwise pooled κ values for the UCAS were >0.6, except for one case (score of 0.54 between readers A and B). Regarding the proportion of positive reads, there was a significant difference between reads for the daughter sac using the UCAS and ISUIA criteria.
Conclusions For discrimination of the daughter sac, the UCAS definition showed a higher reliability than the ISUIA. However, a further prospective study is necessary to validate this definition as the treatment standard for unruptured intracranial aneurysms.
- Aneurysm
- Angiography
Statistics from Altmetric.com
Introduction
With the advance of imaging technology, unruptured intracranial aneurysms (UIAs) can be detected more frequently, and treatment of these previously undetected aneurysms is rapidly increasing. However, the natural course of UIAs is still poorly understood and their management remains controversial. The International Study of Unruptured Intracranial Aneurysms (ISUIA)1 has proposed the size and location of the aneurysm as independent risk factors predicting rupture, and recently the Unruptured Cerebral Aneurysm Study of Japan (UCAS)2 has also reported that the natural course of UIAs varies with aneurysm morphology, especially the presence of a daughter sac (DS). However, the definition of aneurysm morphology is still not clearly defined and is subject to interobserver variation. Although some quantitative or mixed descriptions have been used in previous studies3–7 to overcome this ambiguity, these descriptions have yet to be standardized.
Therefore, the purpose of this study was to assess interobserver agreement of aneurysm morphology, including the presence of a DS, using the different morphological criteria from the UCAS and ISUIA.
Materials and methods
Patients
After approval by the institutional review board for this retrospective study, 100 patients with 102 saccular aneurysms (men:women 20:80; mean age 55.6 years, range 31–89 years), who underwent aneurysm coiling from January 2001 through March 2010, were randomly selected from the institutional database of the Mayo Clinic (Rochester, Minnesota, USA). All selected patients had undergone two-dimensional (2D) DSA and three-dimensional (3D) rotational angiography (3DRA) during their management.
Of 102 aneurysms, the location was the anterior cerebral artery in 18 (18%), the internal carotid artery–posterior communicating artery in 21 (21%), the cavernous internal carotid artery in 40 (39%), the middle cerebral artery in 7 (7%), and the vertebrobasilar artery in 16 (15%). All aneurysms were saccular type.
Image acquisition
Acquisition of 2D images was accomplished with angiograms of the internal carotid artery or vertebral artery using a 5 or 6 F catheter via femoral artery access. Three-dimensional as well as 2D images, including conventional anteroposterior and lateral projection and working projection, were obtained before the procedure. The working projection was decided based on the image that provided the best discrimination between the aneurysm and parent artery after 3DRA.
Three-dimensional images were acquired using a biplane C arm digital angiography suite (Integris; Philips Medical Systems, Best, The Netherlands) with an FOV of 17.78 cm and a frame rate of 30 f/s. Images were obtained with a head end propeller C arm orientation at a rotational speed of 55°/s, covering +120° to -120°. Image acquisition was started 1–3 s after contrast injection (16 mL of non-ionic contrast medium via an injector with a velocity of 4 mL/s); image acquisition time was 4.4 s. Volume rendered reconstruction was performed with 100% magnification (a field of 37.56 cm2) and a matrix of 256 pixels3 using the 3DRA volumetric measurement of the system software (Philips Medical Systems). The threshold for the volume rendered image was fixed at the default value provided by the manufacturer.
In all cases, the 2D and 3D images were obtained at the initial diagnosis. A total of 618 images (314 3D and 304 2D images) in JPEG format were selected from a Mayo Clinic image database by one interventional neuroradiologist (SHS). A file for online review and evaluation was made in Adobe PDF format, where 2D and 3D images of the appropriate projections were arrayed side by side for comparison. Two different files, one of 2D images and the other of 3D images, were made for interpretation.
Readers and image interpretation
All images of 102 aneurysms were evaluated by four experienced readers who worked in two different centers in different countries (USA and Korea). There were three interventional neuroradiologists (HJC and DFK, with more than 10 years of experience; SHS, with 8 years of experience) and one neuroradiologist (JH, with more than 10 years of experience).
For aneurysm morphology, three morphological features were identified from the UCAS2 and ISUIA8: discrimination of a ‘DS’, ‘lobulation’, and ‘margin’ of the aneurysm. ‘DS’ was defined as “an irregular protrusion of the aneurysm wall on 2D or 3D images from MR angiography (MRA), 3D CT angiography (CTA), or DSA” in the UCAS (DS-UCAS), and as “a separate protuberance arising from the margin sac that is less than 25% of the total volume of the sac” in the ISUIA (DS-ISUIA). ‘Lobulation’ was defined as “a protuberance arising directly from the primary neck of the aneurysm or from the main body and representing 25% or more of the apparent volume of the main sac”. The ‘margin’ of the aneurysm was defined as a ‘smooth’ or ‘irregular’ shape of the aneurysm.
Each reader made his own morphological assessment independently using a dichotomized scale (present or not) for the three features, and repeated the second assessment using the same criteria and materials 2 months after the first reading for measurement of intraobserver variability. No additional training or knowledge was provided to the readers.
Statistical analysis
Statistical analyses were performed using SAS (V.9.2; SAS Institute Inc, Cary, North Carolina, USA). To determine the degree of intraobserver and interobserver agreement, the κ statistic was used for each morphological feature, using 2D and 3D images. The κ value was interpreted as follows9: slight agreement, <0.20; fair agreement, 0.21–0.40; moderate agreement, 0.41–0.60; substantial agreement, 0.61–0.80; and perfect agreement, >0.81. Comparison of the κ value between two different features was performed using 95% CI for the difference between the κ statistics with 1000 bootstrapped samples. The difference was statistically significant if 95% CI did not include zero. A pooled κ value, as a weighted average of the κ value for the 2D and 3D images, was also applied between two readers by integration of both reads using 2D and 3D images.
Results
Table 1 summarizes the k values for each category for the 2D and 3D images. There was moderate to substantial agreement between the four readers. While the k value for DS-ISUIA was <0.6, it was >0.6 in DS-UCAS (significant difference) (table 2). Two readers (SHS and DFK) participated in the second reading, and intraobserver agreement was also moderate to substantial.
Results of interobserver agreement among the four readers
Pairwise difference in κ values between the morphological features
Table 3 shows the pooled k values for each interobserver pair of readers according to DS-UCAS and DS-ISUIA. Although the k values were <0.6 in most cases for DS-ISUIA, all pooled k values for DS-UCAS showed substantial agreement, except for one case (score of 0.54 between readers A and B).
Pairwise pooled k values (SE) for DS-UCAS and DS-ISUIA by integration of reads using two-dimensional and three-dimensional images
Table 4 shows the raw data for the positive ratings for the four readers for each morphological group. Irrespective of the images used, there was a significant difference in the proportion of positive reads between DS-UCAS and DS-ISUIA (p<0.05 for all readers). Positive reads of DS-UCAS showed a similar proportion with those for ‘irregular margin’ and were comparable with the sum of positive reads for ‘lobulation’ and DS-ISUIA. Between the 2D and 3D images, there was a significant difference only in the proportion of positive reads of DS-ISUIA (p<0.05 for all readers).
Proportion in the ratings of the four readers of ‘yes’ by categories and readers
Discussion
In this study, we have demonstrated the followings: (1) although there was moderate agreement for DS-ISUIA, interobserver agreement for DS-UCAS was substantial; (2) irrespective of the images used, DS-UCAS also showed higher κ values than DS-ISUIA; and (3) for the proportion of positive reads, there was a significant difference between DS-UCAS and DS-ISUIA, and the former was closely analogous to ‘irregular margin’ as well as the total ‘lobulation’ and DS-ISUIA.
For DS, there was a significant difference in interobserver reliability between the two sets of criteria, with DS-UCAS being superior to DS- ISUIA. Possible explanations for this discrepancy include the following. Unlike DS-UCAS, DS-ISUIA was defined by a specific volume per cent, which may have triggered interobserver variability between readers. In the ISUIA, Forbes et al8 reported interobserver variability for size, location, and angiographic morphologies for 55 aneurysms by two independent readers interpreting a set of 2D angiographies. Although excellent correlation was found for detection and measurement of aneurysms, they showed poor reliability for morphological classification, especially for ‘DS’ and ‘lobulation’. In contrast, board certified neurosurgeons or radiologists in the UCAS interpreted DS using images from MRA, CTA, or DSA. However, these studies did not provide definite results for interobserver variation.
In this study, it is possible that interobserver agreement for the same aneurysms was affected by the images used (2D or 3D images). This disparity may be related to selection bias, where only limited 3D images were available to assess aneurysm morphology. This does not reflect the usual clinical situation where the full 3D dataset is generally available and can be manipulated. While 3D images had some fundamental variability in image quality due to technical and physiological limitations,10 the 3D images are generally superior as they have the advantage of a 360° view of the aneurysm using a volume rendered reconstruction method. Despite the limitations of the 3D images, the proportion of positive ratings on 3D images was higher than for the 2D images for all of the morphological features, except ‘lobulation’. We demonstrated that DS-UCAS could be reliable for the presence of DS using pooled κ values, regardless of the images used (table 3).
As the descriptions for aneurysm morphology are still not fully agreed, there is substantial variability in the definitions and applications. In contrast with DS-ISUIA or DS-UCAS, a recent consensus for UIA treatment decision reported another definition for aneurysm morphology.7 They used ‘lobulation’, which was specified as “irregular, daughter sack-like protrusion of the aneurysm wall on 3D images”. In the ISUIA, aneurysm morphology was subdivided into three different groups, of which ‘lobulation’ was distinguished numerically from ‘DS’, whereas ‘irregular margin’ was similar to DS-UCAS. Interestingly, this study showed that the sum of the presence of ‘lobulation’ and DS-ISUIA was comparable with positive reads for DS-UCAS or ‘irregular margin’ in the ISUIA (table 4).
This study has some limitations. As this was a retrospective study, we did not carry out a formal power calculation. The primary intent of the study was to determine descriptive statistics regarding the prevalence of aneurysm features using different grading schemes. Another limitation was that the window settings of the 3D images in this study could not be adjusted by the readers. In fact, the threshold value for 3D reconstruction had a considerable influence on image quality.10 Finally, although DS-UCAS showed substantial agreement among readers, we did not demonstrate the clinical impact on the treatment strategy for UIAs.
Conclusion
For discrimination of the DS, DS-UCAS showed a higher reliability than DS-ISUIA. However, a further prospective study is necessary to define and validate the optimal definition in treatment decisions for the management of UIAs.
References
Footnotes
Contributors All authors met the requirements for authorship. The corresponding author takes full responsibilities for the data, analyses, and interpretation, and the conduct of the research. The corresponding author had full access to all of the data and has the right to publish any and all of the data separately and apart from any sponsor. SHS: substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of the data; drafting the work or revising it critically for important intellectual content; and final approval of the version to be published. HJC, JH, and DFK: contributions to the conception or design of the work; acquisition of the data; and final approval of the version to be published. KHH: contributions to the analysis of the data.
Funding This study was supported by a faculty research grant from Yonsei University College of Medicine (6-2013-0051).
Competing interests None.
Patient consent Obtained.
Ethics approval The study was approved by the institutional review board.
Provenance and peer review Not commissioned; externally peer reviewed.