Paper is not enough: Crowdsourcing the T₁ mapping common ground via the ISMRM reproducibility challenge

Purpose: T₁ mapping is a widely used quantitative MRI technique, but its tissue-specific values remain inconsistent across protocols, sites, and vendors. The ISMRM Reproducible Research and Quantitative MR study groups jointly launched a challenge to assess the reproducibility of a well-established inversion recovery T₁ mapping technique, with acquisition details published solely as a PDF, on a standardized phantom and in human brains.

Methods: The challenge used the acquisition protocol from Barral et al. 2010. Researchers collected T₁ mapping data on the ISMRM/NIST phantom and/or in human brains. Data submission, pipeline development, and analysis were conducted using open-source platforms. Inter-submission and intra-submission comparisons were performed.

Results: Eighteen submissions (39 phantom and 56 human datasets) on scanners by three MRI vendors were collected at 3T (except one, at 0.35T). The mean coefficient of variation (COV) was 6.1% for inter-submission phantom measurements, and 2.9% for intra-submission measurements. For humans, inter-/intra-submission COV was 5.9/3.2% in the genu and 16/6.9% in the cortex. An interactive dashboard for data visualization was also developed: https://rrsg2020.dashboards.neurolibre.org.

Conclusion: The T₁ inter-submission variability was twice as high as the intra-submission variability in both phantoms and human brains, indicating that the acquisition details in the selected paper were insufficient to reproduce a quantitative MRI protocol. This study reports the inherent uncertainty in T₁ measures across independent research groups, bringing us one step closer to a practical clinical baseline of T₁ variations in vivo.

Dashboard: Challenge Submissions

+++

1| introduction¶

Significant challenges exist in the reproducibility of quantitative MRI (qMRI) Keenan et al., 2019. Despite its promise of improving the specificity and reproducibility of MRI acquisitions, few qMRI techniques have been integrated into clinical practice. Even the most fundamental MR parameters cannot be measured with sufficient reproducibility and precision across clinical scanners to pass the second of six stages of technical assessment for clinical biomarkers Fryback & Thornbury, 1991Schweitzer, 2016Seiberlich et al., 2020. Half a century has passed since the first quantitative T₁ (spin-lattice relaxation time) measurements were first reported as a potential biomarker for tumors Damadian, 1971, followed shortly thereafter by the first in vivo T₁ maps Pykett & Mansfield, 1978 of tumors, but there is still disagreement in reported values for this fundamental parameter across different sites, vendors, and measurement techniques Stikov et al., 2015.

Among fundamental MRI parameters, T₁ holds significant importance Boudreau et al., 2020. T₁ represents the time constant for recovery of equilibrium longitudinal magnetization. T₁ values will vary depending on the molecular mobility and magnetic field strength Bottomley et al., 1984Wansapura et al., 1999Dieringer et al., 2014. Knowledge of the T₁ values for tissue is crucial for optimizing clinical MRI sequences for contrast and time efficiency Ernst & Anderson, 1966Redpath & Smith, 1994Tofts, 1997 and to calibrate other quantitative MRI techniques Sled & Pike, 2001Yuan et al., 2012. Inversion recovery (IR) Drain, 1949Hahn, 1949 is considered the gold standard for T₁ measurement due to its robustness against effects like B₁ inhomogeneity Stikov et al., 2015, but its long acquisition times limit the clinical use of IR for T₁ mapping Stikov et al., 2015. In practice, IR is often used as a reference for validating other T₁ mapping techniques, such as variable flip angle imaging (VFA) Fram et al., 1987Deoni et al., 2003Cheng & Wright, 2006, Look-Locker Look & Locker, 1970Messroghli et al., 2004Piechnik et al., 2010, and MP2RAGE Marques et al., 2010Marques & Gruetter, 2013.

In ongoing efforts to standardize T₁ mapping methods, researchers have been actively developing quantitative MRI phantoms Keenan et al., 2018. The International Society for Magnetic Resonance in Medicine (ISMRM) and the National Institute of Standards and Technology (NIST) collaborated on a standard system phantom Stupic et al., 2021, which was subsequently commercialized (Premium System Phantom, CaliberMRI, Boulder, Colorado). This phantom has since been used in large multicenter studies, such as Bane et al. Bane et al., 2018 which concluded that acquisition protocols and field strength influence accuracy, repeatability, and interplatform reproducibility. Another NIST-led study Keenan et al., 2021 found no significant T₁ discrepancies among measurements using NIST protocols across 27 MRI systems from three vendors at two clinical field strengths.

The 2020 ISMRM reproducibility challenge ^[1] posed a slightly different question: can an imaging protocol, independently implemented at multiple centers, consistently measure one of the fundamental MRI parameters (T₁)? To assess this, we proposed using inversion recovery on a standardized phantom (ISMRM/NIST system phantom) and the healthy human brain. Specifically, this challenge explored whether the acquisition details provided in a seminal paper on T₁ mapping Barral et al., 2010 is sufficient to ensure the reproducibility across independent research groups.

2| methods¶

2.1| Phantom and human data¶

The challenge asked researchers with access to the ISMRM/NIST system phantom Stupic et al., 2021 (Premium System Phantom, CaliberMRI, Boulder, Colorado) to measure T₁ maps of the phantom’s T₁ plate (Table 1). Researchers who participated in the challenge were instructed to record the temperature before and after scanning the phantom using the phantom’s internal thermometer. Instructions for positioning and setting up the phantom were devised by NIST and were provided to researchers through the NIST website ^[2]. In brief, the instructions explained how to orient the phantom and how long the phantom should be in the scanner room prior to scanning to achieve thermal equilibrium.

Researchers were also instructed to collect T₁ maps in healthy human brains, and were asked to measure a single slice positioned parallel to the anterior commissure - posterior commissure (AC-PC) line. Prior to imaging, the imaging subjects consented ^[4] to share their de-identified data with the challenge organizers and on the Open Science Framework (OSF.io) website. As the submitted data was a single slice, the researchers were not instructed to de-face the data of their imaging subjects. Researchers submitting human data provided written confirmation to the organizers that their data was acquired in accordance with their institutional ethics committee (or equivalent regulatory body) and that the subjects had consented to data sharing as outlined in the challenge.

2.2| MRI Acquisition Protocol¶

Researchers followed the inversion recovery T₁ mapping protocol optimized for the human brain as described in the paper published by Barral et al. Barral et al., 2010, which used: TR = 2550 ms, TIs = 50, 400, 1100, 2500 ms, TE = 14 ms, 2 mm slice thickness and 1×1 mm² in-plane resolution. Note that this protocol is not suitable for fitting models that assume TR > 5T1. Instead, the more general Barral et al. Barral et al., 2010 fitting model described in Section 2.4 can be used, and this model is compatible with both magnitude-only and complex data. Researchers were instructed to closely adhere to this protocol and report any deviations due to technical limitations.

2.3| Data Submissions¶

Data submissions for the challenge were handled through a GitHub repository (https://github.com/rrsg2020/data_submission), enabling a standardized and transparent process. All datasets were converted to the NIfTI format, and images for all TIs were concatenated into a single NIfTI file. Each submission included a YAML file to store additional information (submitter details, acquisition details, and phantom or human subject details). Submissions were reviewed ^[5], and following acceptance the datasets were uploaded to OSF.io (osf.io/ywc9g/). A Jupyter Notebook Kluyver et al., 2016Beg et al., 2021 pipeline using qMRLab Karakuzu et al., 2020Cabana et al., 2015 was used to process the T₁ maps and to conduct quality-control checks. MyBinder links to Jupyter notebooks that reproduced each T₁ map were shared in each respective submission GitHub issue to easily reproduce the results in web browsers while maintaining consistent computational environments. Eighteen submissions were included in the analysis, which resulted in 39 T₁ maps of the NIST/system phantom, and 56 brain T₁ maps. Figure 1 illustrates all the submissions that acquired phantom data (Figure 1-a) and human data (Figure 1-b), the MRI scanner vendors, and the resulting T₁ mapping datasets. Some submissions included measurements where both complex and magnitude-only data from the same acquisition were used to fit T₁ maps, thus the total number of unique acquisitions is lower than the numbers reported above (27 for phantom data and 44 for human data). The datasets were collected on systems from three MRI manufacturers (Siemens, GE, Philips) and were acquired at 3T ^[6], except for one dataset acquired at 0.35T (the ViewRay MRidian MR-linac).

Figure 1. List of the datasets submitted to the challenge. Submissions that included phantom data are shown in a), and those that included human brain data are shown in b). For the phantom (panel a), each submission acquired its data using a single phantom, but some researchers shared the same physical phantom with each other. Green indicates submissions used for inter-submission analyses, and orange indicates the sites used for intra-submission analyses. T₁ maps used in the calculations of inter- (green) and intra- (orange) submission coefficients of variation (COV) are indicated with asterisks. Images c) and d) illustrate the ROI choice in phantoms and humans.

2.4| Fitting Model and Pipeline¶

A reduced-dimension non-linear least squares (RD-NLS) approach was used to fit the complex general inversion recovery signal equation:

where a and b are complex constants. This approach, developed by Barral et al. Barral et al., 2010, offers a model for the general T₁ signal equation without relying on the long-TR approximation. The a and b constants inherently factor TR in them, as well as other imaging parameters such as excitation pulse angle, inversion pulse flip angles, TR, TE, TI, and a constant that has contributions from T₂ and the receive coil sensitivity. Barral et al. [31] shared their MATLAB (MathWorks, Natick, MA) code for the fitting algorithm used in their paper ^[7]. Magnitude-only data were fitted to a modified version of Eq. 1 (Eq. 15 of Barral et al. 2010) with signal-polarity restoration by finding the signal minima, fitting the inversion recovery curve for two cases (data points for TI < TI_minimum flipped, and data points for TI ≤ TI_minimum flipped), and selecting the case that resulted in the best fit based on minimizing the residual between the model and the measurements ^[8]. This code is available as part of the open-source software qMRLab Karakuzu et al., 2020Cabana et al., 2015, which provides a standardized application program interface (API) to call the fitting in MATLAB/Octave scripts.

A data processing pipeline was written using MATLAB/Octave in a Jupyter Notebook. This pipeline downloads every dataset from OSF.io (osf.io/ywc9g/), loads its configuration file, fits the T1 maps, and then saves them to NIfTI and PNG formats. The code is available on GitHub (https://github.com/rrsg2020/t1_fitting_pipeline, filename: RRSG_T1_fitting.ipynb). Finally, T₁ maps were manually uploaded to OSF (osf.io/ywc9g/).

2.5| Image Labeling & Registration¶

The T₁ plate (NiCl₂ array) of the phantom has 14 spheres that were labeled as the regions-of-interest (ROI) using a numerical mask template created in MATLAB, provided by NIST researchers (Figure 1-c). To avoid potential edge effects in the T₁ maps, the ROI labels were reduced to 60% of the expected sphere diameter. A registration pipeline in Python using the Advanced Normalization Tools (ANTs) Avants et al. (2009) was developed and shared in the analysis repository of our GitHub organization (https://github.com/rrsg2020/analysis, filename: register_t1maps_nist.py, commit ID: 8d38644). Briefly, a label-based registration was first applied to obtain a coarse alignment, followed by an affine registration (gradientStep: 0.1, metric: cross correlation, number of steps: 3, iterations: 100/100/100, smoothness: 0/0/0, sub-sampling: 4/2/1) and a BSplineSyN registration (gradientStep:0.5, meshSizeAtBaseLevel:3, number of steps: 3, iterations: 50/50/10, smoothness: 0/0/0, sub-sampling: 4/2/1). The ROI labels template was nonlinearly registered to each T₁ map uploaded to OSF.

For human data, manual ROIs were segmented by a single researcher (M.B., 11+ years of neuroimaging experience) using FSLeyes McCarthy (2019) in four regions (Figure 1-d): located in the genu, splenium, deep gray matter, and cortical gray matter. Automatic segmentation was not used because the data were single-slice and there was inconsistent slice positioning between datasets.

2.5.1| Analysis and Statistics¶

Analysis code and scripts were developed and shared in a version-controlled public GitHub repository ^[9]. The T₁ fitting and data analysis were performed by M.B., one of the challenge organizers. Computational environment requirements were containerized in Docker Merkel, 2014Boettiger, 2015 to create an executable environment that allows for analysis reproduction in a web browser via MyBinder ^[10] Project Jupyter et al., 2018. Backend Python files handled reference data, database operations, ROI masking, and general analysis tools. Configuration files handled dataset information, and the datasets were downloaded and pooled using a script (make_pooled_datasets.py). The databases were created using a reproducible Jupyter Notebook script and subsequently saved in the repository.

The mean T₁ values of the ISMRM/NIST phantom data for each ROI were compared with temperature-corrected reference values and visualized in three different types of plots (linear axes, log-log axes, and error relative to the reference value). Temperature correction involved nonlinear interpolation ^[11] of a NIST reference table of T₁ values for temperatures ranging from 16 °C to 26 °C (2 °C intervals) as specified in the phantom’s technical specifications. For the human datasets, the mean and standard deviations for each tissue ROI were calculated from all submissions across all sites. Two submissions (one of phantom data – submission 6 in Figure 1-a, and one of human data – submission 18 in Figure 1-b) were received that measured large T₁ mapping datasets. Submission 6 consisted of data from one traveling phantom acquired at seven Philips imaging sites, and submission 18 was a large cohort of volunteers that were imaged on two 3T scanners, one GE and one Philips. These datasets (identified in orange in Figures 1, 3, and 4) were used to calculate intra-submission coefficients of variation (COV) (one per scanner/volunteer, identified by asterisks in Figure 1-a and 1-b), and inter-submission COVs were calculated using one T₁ map from each of these (orange) along with one from all other submissions ^[12] (identified as green in Figures 1, 3, and 4, and the T₁ maps used in those COV calculations are also indicated with asterisks in Figure 1-a and 1-b). All quality assurance and analysis plot images were stored in the repository. Additionally, the database files of ROI values and acquisition details for all submissions were also stored in the repository.

2.5.2| Dashboard¶

To widely disseminate the challenge results, a web-based dashboard was developed (Figure 2, https://rrsg2020.dashboards.neurolibre.org). The landing page (Figure 2-a) showcases the relationship between the phantom and brain datasets acquired at different sites/vendors. Selecting the Phantom or In Vivo icons and then clicking a ROI will display whisker plots for that region. Additional sections of the dashboard allow for displaying statistical summaries for both sets of data, a magnitude vs complex data fitting comparison, and hierarchical shift function analyses.

Figure 2 Dashboard. a) Welcome page listing all the sites, the types of subject, and scanner vendor, and the relationship between the three. b) The phantom tab for a selected ROI, and c) The in vivo tab for a selected ROI. Link: [https://rrsg2020.dashboards.neurolibre.org](https://rrsg2020.dashboards.neurolibre.org)

3| results¶

Figure 3 presents a comprehensive overview of the challenge results through violin plots, depicting inter- and intra- submission comparisons in both phantoms (a) and human (b) datasets. For the phantom (Figure 3-a), the average inter-submission COV for the first five spheres, representing the expected T₁ value range in the human brain (approximately 500 to 2000 ms) was 6.1%. By addressing outliers from two sites associated with specific challenges for sphere 4 (signal null near a TI), the mean inter-submission COV was reduced to 4.1%. One participant (submission 6, Figure 1) measured T₁ maps using a consistent protocol at 7 different sites, and the mean intra-submission COV across the first five spheres for this submission was calculated to be 2.9%.

For the human datasets (Figure 3-b), inter-submission COVs for independently-implemented imaging protocols were 5.9% for genu, 10.6 % for splenium, 16 % for cortical GM, and 22% for deep GM. One participant (submission 18, Figure 1) measured a large dataset (13 individuals) on three scanners and two vendors, and the intra-submission COVs for this submission were 3.2% for genu, 3.1% for splenium, 6.9% for cortical GM, and 7.1% for deep GM. The binomial appearance for the splenium, deep GM, and cortical GM for the sites used in the inter-site analyses (green) can be explained by an outlier measurement, which can be seen in (Figure 4 e-f, site 3.001).

A scatterplot of the T₁ data for all submissions and their ROIs is shown in Figure 4 (phantom a-c, and human brains d-f). The NIST phantom T₁ measurements are presented in each plot for different axes types (linear, log, and error) to better visualize the results. Figure 4-a shows good agreement for this dataset in comparison with the temperature-corrected reference T₁ values. However, this trend did not persist for low T₁ values (T₁ < 100-200 ms), as seen in the log-log plot (Figure 4-b), which was expected because the imaging protocol is optimized for human brain T₁ values (T₁ > 500 ms). Higher variability is seen at long T₁ values (T₁ ~ 2000 ms) in Figure 4-a. Errors exceeding 10% are observed in the phantom spheres with T1 values below 300 ms (Figure 4-c), and 3-4 measurements with outlier values exceeding 10% error were observed in the human brain tissue range (~500-2000 ms).

Figure 4 d-f displays the scatter plot data for human datasets submitted to this challenge, showing mean and standard deviation T₁ values for the WM (genu and splenium) and GM (cerebral cortex and deep GM) ROIs. Mean WM T₁ values across all submissions were 828 ± 38 ms in the genu and 852 ± 49 ms in the splenium, and mean GM T₁ values were 1548 ± 156 ms in the cortex and 1188 ± 133 ms in the deep GM, with less variations overall in WM compared to GM, possibly due to better ROI placement and less partial voluming in WM. The lower standard deviations for the ROIs of human database ID site 9 (by submission 18, Figure 1, and seen in orange in Figure 4d-g) are due to good slice positioning, cutting through the AC-PC line and the genu for proper ROI placement, particularly for the corpus callosum and deep GM.

3.1| discussion¶

This challenge focused on exploring if different research groups could reproduce T₁ maps based on the protocol information reported in a seminal publication Barral et al., 2010. Eighteen submissions independently implemented the inversion recovery T₁ mapping acquisition protocol as outlined in Barral et al. Barral et al., 2010, and reported T₁ mapping data in a standard quantitative MRI phantom and/or human brains at 27 MRI sites, using systems from three different vendors (GE, Philips, Siemens). The collaborative effort produced an open-source database of 94 T₁ mapping datasets, including 38 ISMRM/NIST phantom and 56 human brain datasets. The inter-submission variability was twice as high as the intra-submission variability in both phantom and human brain T₁ measurements, demonstrating that written instructions communicated via a PDF are not enough for reproducibility in quantitative MRI. This study reports the inherent uncertainty in T₁ measures across independent research groups, which brings us one step closer to producing a practical baseline of variations for this metric.

Overall, our approach did show improvement in the reproducibility of T₁ measurements in vivo compared to researchers implementing T₁ mapping protocols completely independently (i.e. with no central guidance), as literature T₁ values in vivo vary more than reported here (e.g., Bojorquez et al. Bojorquez et al., 2017 reports that reported T₁ values in WM vary between 699 to 1735 ms in published literature). We were aware that coordination was essential for a quantitative MRI challenge, which is why the protocol specifications we provided to researchers were more detailed than any guidelines for quantitative MR imaging that were available at the time. Yet, even in combination with the same T₁ mapping processing tools, this level of description (a PDF + post-processing tools) leaves something to be desired.

This analysis highlights that more information is needed to unify all the aspects of a pulse sequence across sites, beyond what is routinely reported in a scientific publication. However, in a vendor-specific setting, this is a major challenge given the disparities between proprietary development libraries Gracien et al., 2020. Vendor-neutral pulse sequence design platforms Layton et al., 2017Cordes et al., 2020Karakuzu et al., 2022 have emerged as a powerful solution to standardize sequence components at the implementation level (e.g., EF pulse shape, gradients, etc.). Vendor neutrality has been shown to significantly reduce the variability of T₁ maps acquired using VFA across vendors Karakuzu et al., 2022. In the absence of a vendor-neutral framework, a vendor-specific alternative is the implementation of a strategy to control the saturation of MT across TRs A G Teixeira et al., 2020. Nevertheless, this approach can still benefit from a vendor-neutral protocol to enhance accessibility and unify implementations. This is because vendor-specific constraints are recognized to impose limitations on the adaptability of sequences, resulting in significant variability even when implementations are closely aligned within their respective vendor-specific development environments Lee et al., 2019.

The 2020 Reproducibility Challenge, jointly organized by the Reproducible Research and Quantitative MR ISMRM study groups, led to the creation of a large open database of standard quantitative MR phantom and human brain inversion recovery T₁ maps. These maps were measured using independently implemented imaging protocols on MRI scanners from three different manufacturers. All collected data, processing pipeline code, computational environment files, and analysis scripts were shared with the goal of promoting reproducible research practices, and an interactive dashboard was developed to broaden the accessibility and engagement of the resulting datasets (https://rrsg2020.dashboards.neurolibre.org). The differences in stability between independently implemented (inter-submission) and centrally shared (intra-submission) protocols observed both in phantoms and in vivo could help inform future meta-analyses of quantitative MRI metrics Mancini et al., 2020Lazari & Lipp, 2021 and better guide multi-center collaborations.

By providing access and analysis tools for this multi-center T₁ mapping dataset, we aim to provide a benchmark for future T₁ mapping approaches. We also hope that this dataset will inspire new acquisition, analysis, and standardization techniques that address non-physiological sources of variability in T₁ mapping. This could lead to more robust and reproducible quantitative MRI and ultimately better patient care.