Towards implementing Hadoop/Spark-based next-generation NDB that stores health insurance claims data of all residents in Japan.

Points of the selection

Essential requirements: Flexible scalability and avoidance of vendor lock-in

Members of the project included professionals familiar with both medical and informatics fields from Kyoto University Hospital, Nara Medical University, and Graduate School of Medicine and Faculty of Medicine of The University of Tokyo. NTT DATA was selected to build the platform because of their rich track record in Hadoop/Spark platform development.
Professor Tomohiro Kuroda at Kyoto University, the coordinator of the project, says, "Adequate scalability was a key requirement in the public tender. If the system specification can only be enhanced through gradual steps involving server replacement as it was before, the system cannot cope with future increase in data and new data such as nursing care services data. Therefore, scalability with the future in mind is essential."
Another crucial requirement in the public tender was “a framework without vendor lock-in.” “After confirming the feasibility of the next-generation NDB through this project, new tenders will be called to build the production environment. Depending on the technology of a specific vendor tend to bring in disadvantages as a national project. Those disadvantages include unfairness, higher costs caused by lack of price competition, and restrictions in future system modifications,” continued Professor Kuroda.

Thus, the next-generation NDB had to solve the issues of the current NDB while satisfying the requirements as a public tender—flexible scalability and vendor lock-in avoidance. Two open-source software packages, Hadoop and Spark were selected as the core software of the next-generation NDB. Hadoop is capable of accumulating and processing large-scale data sets with distributed computing, and Spark is capable of efficiently processing iterative tasks that use large-scale data. These open-source software are based on scale-out design, thus the system performance can easily be increased by only adding commodity servers without using expensive high-performance servers.

"We had no choice but to use Hadoop as the software framework in order to solve the issues while satisfying the requirements of the public tender. Thus, we looked for a company that has technical competence and know-how on Hadoop/Spark, and found NTT DATA in the center of the community. Their leading worldwide track record of Hadoop-related development and frequent information sharing from the heart of the community proves that NTT DATA understands the core of technology. Also, in addition to the fineness of their work, corporate strength was also an important factor as a national project. In the light of these considerations, we decided that NTT DATA is the best company to handle this project," explains Professor Kuroda about the reasons for selecting Hadoop/Spark and NTT DATA.

Professor Tomohiro Kuroda
Medical Informatics, Graduate School of Medicine, Kyoto University
General Manager of Division of Medical Information Technology and Administration Planning, Kyoto University Hospital

Contents of the project

A dream team of specialists in each field with high technical competence

This project was launched in January 2017 and carried out with the following roles:

• - Kyoto University: Establishment of an environment that enables learning, trials, and operations related to the research using NDB data.
• - The University of Tokyo: Creation of data marts for analysis. Especially focus on advanced utilization of next-generation NDB with AI in mind.
• - Nara Medical University: Creation of data marts for analysis. Especially focus on improving the accuracy of patient search (ID matching).
• - NTT DATA: Building the high performing and scalable platform using Hadoop/Spark, and other required software.

Professor Kuroda who specializes in medical informatics supervised this project. He is responsible for managing digital medical data, utilizing those data in hospital operation and management. He is also familiar with analyzing medical data such as health insurance claims data. His past achievements and experiences largely contributed to the success of this project.

Professor. Kuroda says, "Specialists of different fields shared their goals and progress statuses in time, and this helped me a lot to manage the project. It was literally a dream team of aces. Among everything, I was amazed by the high technical competence of NTT DATA on Hadoop and Spark, their knowledge in the medical field and tasks related to health insurance claims, and their detailed project management skills."

Benefits of utilizing Hadoop/Spark

Realization of data research platform that enables high performance and flexible analysis

• Realization of high performance and scalability by utilizing Hadoop/Spark
  - Reduction of processing time from hours to minutes The project was started in January 2017. Construction of the next-generation NDB utilizing Hadoop/Spark was started from April 2017 after requirement definition phase. System tests using third-party data were started from November the same year, and data mart creation, analysis, and verification were carried out in January 2018. Data extraction, for example, which used to take four hours as an on-site job only takes several minutes in the next-generation NDB, thanks to the test environment and the preliminary data preparation. Researchers from different universities conducted analysis using the next-generation NDB and realized the effectiveness and speed of it.
  In addition, the scalability of the Hadoop/Spark platform was verified and it was confirmed that the system performance can be increased by adding commodity servers.
• Improvement of the learning efficiency by establishing "materials to learn the basics of health insurance claims" and "a learning environment where dummy data is available" Kyoto University prepared e-learning contents to learn the basics of health insurance claims for users who are not familiar with health insurance claims data and procedures. Kyoto University also prepared an NDB hands-on environment where users can search and do analysis using NDB dummy data, easily. With these enhancements, users can now efficiently learn and understand the basics of health insurance claims data by using the e-learning contents and by testing the analysis logic in the NDB hands-on environment. This helped beginners of NDB analysis to reduce the lead time until the actual analysis is started.
• Increase of the analysis efficiency and convenience by preparing data marts required for analysis The "ID problem" was particularly a critical issue among other issues such as data reorganization and data conversion when analyzing NDB data. NTT DATA successfully implemented the ID matching algorithm provided by Nara Medical University on the distributed platform. As a result, obstacles that blocked continuous data tracing were cleared, increasing the accuracy of patient ID-based data aggregation.
  NTT DATA also implemented the analysis algorithms provided by The University of Tokyo, and prepared frequently used data marts in advance. As a result, research and analysis activities can now be performed efficiently and conveniently using these existing data marts without creating new data marts.

(Service mentioned in this paper)

- Hadoop/Spark-based medical big data analysis platform

The next-generation-NDB data research platform is built on top of Hadoop/Spark, which is a scalable cost-efficient parallel distributed processing platform. This system is attracting worldwide attention as a system that promotes the use of large-scale medical data with over 13 billion records.