The Software Framework
The software framework developed for the CDS-QuaMRI project had to fulfill a number of requirements. It should serve as an data archive for the multi-modal data acquired in the project. Because of the size of the data and the heterogeneity of its source and time of acquisition, the platform should include a full-featured data management system paired with network access and transfer of data. The platform should also serve as a collaboration tool for developers in the project consortium, allowing organization of data in multiple sub-projects and secure and efficient sharing of data among the consortium members. Finally, the platform had to able to serve as data processing and analysis engine, integrating in a flexible manner a variety of processing modules.
Figure 1: CDS-QUAMRI Software Framework
The entire software system was implemented in a client-server environment and encompasses modules installed on a server and multiple clients. An overview of the framework is represented in Figure 1. The system builds on a data-centric design. Data is collected by data collectors or imported through a dedicated interface to the core system, where it is stored and cataloged. In a second step, client applications can access and process the data, and store processed data and results back into the database.
Server Core Component
The core component is implemented as a web server application and runs on the server. Hardware requirements for the server scale well with the amount of data to be stored in the system. The core component administrates the main file storage and stores any kind of data imported by data collectors or using the import interface. The file storage is data agnostic, and thus can hold any kind of files such as eg image files, raw data files, log files, excel files, or text files.
In addition, the core system also administrates the core database. The database stores descriptive, structural and administrative metadata. Where available, parameters are extracted from imported files and stored in the database. For example, public and private tags contained in Dicom files are extracted and stored in the database. The core component also manages users and access rights. Data is owned by and visible only to a specific user, who can grant access rights to other users. Finally, the core also implements a task manager and can invoke tasks involving data access and processing. This processing is organized in nodes and workflows, for which the system holds descriptions.
As a web server application, the core system provides a public representational state transfer (REST) interface that can be used by any kind of client application. The REST interface is the only interface to the storage and the database. Direct access to the storage and the database is not possible. Libraries to access the REST interface are available for C++, Python, Matlab, and as command line tool to integrate into other script languages. Windows, Linux and MacOS operating systems are supported.
Figure 2: Typical installation of the framework as a local cloud resource embedded in an existing research- or hospital environment.
Task Engine
To accommodate the analysis modules created in the CDS-QuaMRI project, the framework is capable of embedding so-called processing task. A task engine allows to execute these processing tasks on data selected from the database. Both task definition and execution is controlled from within the user interface of the framework, and accessed over a standard web browser. The task definition contains a description of the target execution machine, allowing processing to be executed on external hosts if necessary. The task definition contains either the calls to be executed on the host, or with the help of the containerization methods the entire processing environment itself. Either way both data and processing task are transmitted to the execution host, processing is initialized, and resulting data is collected. This versatile and flexible scheme allows to embed modules written in any script language or compiled for arbitrary architectures and operating systems, as long as a suitable host can be accessed from the framework server.
Finally multiple tasks can be concatenated to define entire processing workflows. Automatic triggering of workflow execution upon various events is also possible.
The development of the framework was financially supported by the Horizon 2020/CDS-QUAMRI/634541 project. This support is gratefully acknowledged. The framework is part of the Agora system developed and commercialized by the consortium partner GyroTools.