3.1.

Images Menu

The QIFP maintains its own image repository and can connect to other image sources. Figure 2 shows the QIFP user interface when the “Images” menu (along the top row) and the “Local” source are selected, revealing a list of available cohorts in the left-hand panel and the available patient images/annotations in the right-hand panel. Other available sources include: The Cancer Imaging Archive (TCIA),²² an instance of the ePAD²¹ image annotation and storage system, and either the Google or Amazon S3 cloud services [Fig. 2(B)]. QIFP can also be configured to query data from any DICOM compliant local PACS. When QIFP is running on a local server, local cohorts can be created by transferring them from the other sources to the QIFP, or by manually uploading images, segmentations, and/or annotations by clicking on the upload button next to cohort name [Fig. 2(E)]. This will allow users to select a file type to upload and browse to files on their local computer. For example, users can upload a zip file of DICOM or Neuroimaging Informatics Technology Initiative (NIfTI) images and segmentations. Finally, owners of a local cohort can add other QIFP users to the cohort to allow them to access the data and workflows associated with it [Fig. 2(F)].

Once a cohort has been selected, the QIFP shows the list of DICOM images available in that cohort in the main right-hand panel and lists information on the patient, study, and series. A link to an ePAD image viewer (a freely available open-source DICOM viewer²¹) is also provided to quickly visualize an image series [Fig. 2(K)] and annotate images. Any new annotations (e.g., segmentation seed points) created in ePAD will then be available in the QIFP. Any annotations associated with a specific series are also displayed. Users can select individual series, studies, or patients to process by selecting the box next to each one. Otherwise, users can select the whole cohort by clicking the box next to the cohort name at the top of the screen, or a subset of patients by clicking on each patient and holding down the shift key to select all patients in between. Selected cohorts can then be processed by one of many processing tools and pipelines, described in Secs. 3.5 and 3.6, respectively.

3.2.

Annotations Menu

The “Annotations” menu lists all the available image annotations or segmentations for a given cohort. Annotation files can be stored as annotation and image markup (AIM)²³ files or DICOM segmentation objects (DSO). Users can also upload other segmentation file types, e.g., NIfTI file format as described above [Fig. 2(E)],²⁴ and save them to the local cohort. Only users who are members of the local cohort have access to these segmentations.

3.3.

Models Menu

Whenever a new predictive model is created through a machine learning workflow, the user has the option to save it and include it in future workflows. The “Models” menu contains a list of previously constructed predictive models that are available in the QIFP system.

3.4.

Pipeline Results Menu

Users can see results for past and progress for currently running workflows under the “Pipeline Results” menu. Figure 3 shows an example of a workflow in progress. The cohort name is listed on the top left corner of the page and actively running workflows are displayed under “Active Docker Tools.”

Fig. 3

Example of the “Pipeline Results Menu” with a feature extraction workflow in progress. This menu shows all active and completed tools and workflows for the cohort selected prior to invoking this menu. (A) Currently active Docker tool (red box) and the results in progress, including status, elapsed time, parameters used, and access to a log file, pointed to by red arrow. Workflows shown below “Active Docker Tools” are organized by type; for example, clicking the arrow (B) will show all workflows using the PyRadiomics Tool completed or in progress. Clicking on the workflow name, as shown in the red box, will open a new webpage with additional details on that workflow.

While a workflow is running, the Pipeline Results page displays relevant information, such as the elapsed time, the parameters used, log-file entries, and overall status of the workflow. All workflows have an ID name which is based on the tool name (e.g., QIFE), and the date and time that the workflow was started. Users receive an email once the workflow has completed. A log file is also available to provide information on the completed workflow, if it finds an error, and why it may have failed.

3.5.

Docker Tools Menu

The “Docker Tools” menu provides access to all available Docker tools in the system, as well as a “Tool Help.” Currently, the QIFP system has a range of tools available for quantitative image analysis, including preprocessing, segmentation, feature extraction, and machine learning prediction models (Table 1). What follows provides a detailed explanation of the tools currently available on the QIFP.

Table 1

Docker tools currently available on the QIFP.

3.5.3.

Feature extraction

There are several different feature extraction modules that are currently available within the QIFP system. Stanford’s Quantitative Image Feature Extraction (QIFE) tool allows for the extraction of size, shape, intensity, texture, and law’s features.²⁹ Figure 4 shows an example of how to configure a workflow containing this tool. Users can view, edit, and upload their own configuration file or manually select workflow options through the checkboxes provided. Figure 5 shows a completed workflow. All the files produced by the workflow are available for download through a link provided at the bottom of the results, including the log file, the resultant feature file in a comma-separated values (CSV) format, and the configuration file used for that run.

Fig. 4

Example feature extraction workflow using the QIFE. Users can upload a configuration file or manually select configuration options in the interface shown.

Fig. 5

Example of a completed feature extraction workflow. The output components are displayed at the bottom. The files for this workflow include (A) the extracted features, (B) the log file describing the results of the workflow, and (C) the configuration file used to run the workflow. Clicking on any of the file names will allow the user to view and/or download them.

Fig. 6

Example clinical file required in all LASSO training workflows.

The PyRadiomics tool is another feature extraction engine that has the option to extract higher order wavelet features along with the traditional features on the original images.¹⁶ Additional feature extraction tools include 2-D Riesz features³⁰ and scale-invariant feature transform (SIFT) features.³¹ In general, each feature extraction module has its own set of user-configurable parameters to ensure the workflow is configured to best suit the required data type and analysis.

Different feature extraction modules may compute radiomic features differently, and for this reason may arrive at different values for what might appear to be the same feature.³² Common differences may be specifications for directional sampling of voxels for texture features, algorithms used for surface area calculations, and intensity discretization. We refer the user to the manuscripts describing QIFE²⁹ and PyRadiomics¹⁶ for feature definitions. Also, since each Docker tool will contain a version of the tools from a specific point of time, the version code and Docker ID for each instance of the tool is recorded for each output workflow.

3.6.

Workflows Menu

This menu allows the user to visualize all currently available workflows, which are also categorized according to their type, including feature extraction, segmentation, or prediction workflows (Table 2). User can also create a new workflow from modifications of any existing workflow. Workflows have already been created to run individual or combinations of Docker tools. For example, there is a workflow to run only the PyRadiomics feature extractor and another that will first run PyRadiomics followed by a machine learning engine. Workflows can be customized to include any of the tools listed in Table 1.

Table 2

Workflows currently available on the QIFP.

3.7.

Other Menus

The remaining buttons will provide access to user profile information and preferences (Profile) as well as usage statistics and event logs for completed QIFP actions (System Status/Statistics).

Fig. 7

Example of how to add a new Docker Tool. After selecting the Docker Tool menu, click on the “New” option at the top of the left-hand (indicated by the arrow) to add the requirements for a user-supplied tool.

4.1.

Creating and Uploading Tools

Users are able to upload their own tools to the QIFP by encapsulating them in a Docker container and storing them on DockerHub. For each Docker tool added to the QIFP, a Linux command should be indicated to describe the required inputs and outputs of the tool. For ease of incorporation into the QIFP system, tools can be created in two formats. The first being a tool that works on a single patient/series and a single segmentation or annotation. This type is simpler to implement, since the program does not need to figure out which segmentation refers to which series or to aggregate features/results for all patients. When using this type of tool on a whole cohort, the QIFP system will call the Docker image multiple times and run a separate Docker image for each case. The second option is to create a tool that processes multiple series/segmentations (i.e., an entire cohort). Each patient could have one or more series, and each series could have one or more segmentations/annotations. This type is more difficult to implement and requires using index files that contains file references and the feature results, if any, need to be aggregated into a single file. However, this type of tool is more efficient since only one Docker image is run for the entire data set. Figure 7 demonstrates how to upload a Docker tool and what information is required.

5.1.

Selecting the Cohort and Workflow

To run a workflow, the user must first select the cohort that they would like to analyze by clicking on “Images” menu at the top, choosing the “TCIA” source, and then clicking on the “tcia:NSCLC Radiogenomics” cohort. This example will process 75 patients (R01-001 through ROI-075). To select these, click on the checkbox next to R01-001 then scroll down to R01-075 then press and hold SHIFT while clicking on the checkbox next to R01-075. Next, click on “run workflow” at the top left-hand corner of the window and select the workflow of interest, in this example “PyRadiomics 3D Features → PyRadiomics, LASSO Train” workflow (Fig. 8). This workflow will first perform feature extraction using the PyRadiomics Docker tool and then perform LASSO training to build a predictive model.

Fig. 8

Example execution of a feature extraction and machine learning training workflow using the TCIA Radiogenomics cohort.

5.2.

Configuring and Running the Workflow

Figure 9 shows what information must be provided to successfully run the workflow. Specific text is shown to the left of the block diagram with arrows indicating which of these inputs are required for each specific tool. This includes uploading a file of clinical data, which includes the outcome of interest and a link to the image patient ID. For this example, extract columns A (case ID) and AE (recurrence) from the TCIA clinical data file. All rows for cases beginning with “AMC” can be deleted, so that only “R01” cases remain. The “Case ID” header must also be renamed to “Patient ID,” then these two columns can be saved as a new CSV file. This will be the clinical data file uploaded to run this workflow. In this case, the CSV file must be transposed, and the target feature “recurrence” can be selected from the drop-down menu provided. Configuration files for the feature extraction component and prediction engines can also be uploaded here; however, this example will use default configuration options. Users are also provided with output and processing options for each workflow. Selecting “retain data in local DB” will save the data in a local cohort named “tcia-NSCLC_Radiogenomics” and this will avoid having to redownload the images from TCIA for future processing. After all the selections have been made, clicking on “Upload and Run Workflow” will start the workflow. The status of the workflow can be tracked under the “Pipeline Results” menu. An email will be sent to the user once the workflow has completed with a direct link to the results of the workflow.

Fig. 9

Input screen for the feature extraction and machine learning training workflow applied to the TCIA cohort. The required input requirements are shown to the left of the block diagram. Users have the ability to upload these required files and are provided with example files to illustrate formatting and default files used.

5.3.

Saving the Prediction Model

The resultant model parameters can be found in the model.csv file provide as output from the training workflow (Fig. 10). In this example, LASSO selected six features for the final model. Figure 10 shows the output from the workflow, including the configuration files, feature extraction, and model results. To save the model and test it on a new cohort, users must go to the “Models” menu and click on “New” on the top of the left-hand panel. Users can name the model and select the appropriate workflow instance and tool instance from the training workflow that was just completed. Once a model is saved, it will appear under the “Models” menu on the left-hand side (Fig. 11).

Fig. 10

Output screen for the feature extraction and machine learning training workflow applied to the TCIA cohort (Fig. 9).

Fig. 11

Output screen allowing the saving of the model produced by the workflow applied to the TCIA cohort.

5.4.

Testing the Prediction Model

To test this model on a new cohort of patients, patients are selected as described in Sec. 5.1. For this example, testing will be done on 25 different patients from the same TCIA NSCLC radiogenomics cohort (patients R01-076 through R01-100) by running the workflow “PyRadiomics 3D Features → PyRadiomics and Lasso Test.” The same clinical data file can be uploaded that was used for testing, since it includes all patients in the TCIA dataset (Fig. 12). Once again, the default feature extraction and prediction configurations will be used. The model that was saved in Sec. 5.3 above can be selected from the drop-down menu and then the workflow can be started. Once the workflow has completed, the output files will be displayed (Fig. 13), including a list of the resulting model’s features and their coefficients, and an area under the receiver operating characteristic (ROC) curve describing performance (Fig. 14).