YOU CAN VIEW A RECORDING OF THE WEBINAR «How to visualAIze 5000 COVID CT scans daily with 15 minutes SLA?»
04.06.2020 18:00 GTM +1

Experiment

on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow

Artificial intelligence in radiology

Goal:

Scientific research on the possibility of using clinical decision support systems in the Moscow healthcare system with data analysis based on advanced innovative technologies

Location:

Moscow region, Russia

Base:

Unified Radiological Information Service (URIS)

3
use case

chest CT findings related to COVID-19; lung cancer

4
AI service

connected to URIS

147 579
studies

analyzed by AI

420
radiologists

evaluate the results of AI

Clinical application



Oncology
Lung cancer. CT/LDCT
С34
Breast cancer. MG
С50
Adrenal lesions. CT/LDCT
D44
Liver lesions. CT/LDCT
C22
Cardiology
Coronary calcium. CT/LDCT
I25
Aortic aneurysm. CT/LDCT
I79
Pericardial fat. CT/LDCT
I27
Pulmonary artery dilatation. CT/LDCT
I25
Chronic diseases
Vertebral fracture (osteoporosis). X-RAY/CT/LDCT
M80
Abnormal liver density. CT
K70-K77
Pulmonology
A spectrum of clinically relevant pathologies. X-RAY
J09-J18
Tuberculosis. X-RAY
A15
Emergencies
Limb fractures. X-ray/CT
S10-T07
Cerebral hemorrhage. CT
I60-I64
Skull fractures. CT
S10-T07
Neurology
Multiple sclerosis. MRI
G35
- Current
- Future

How to apply?

Preparation

Fill in and submit the questionnarie via e-mail

Download

Check the suitability of your AI service solution  (password mosmeddata)

SELF TEST

Check out quality standards

QA program

Apply for the experiment

Integration

Conclude an agreement


Check technical requirements

Integration guidelines for AI service providers AGFA DICOM conformance statement Kafka Apache

Integrate your service with URIS


Calibrate your service on the labeled dataset (retrospective data)

Basic requirements for AI services findings
Prospective study analysis

This is the main stage of the Experiment

Your AI service analyses studies in real time

The Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies provides for:

  • Diagnostic accuracy evaluation
  • Study audit by expert doctors
  • Analysis of identified discrepancies between the service and doctors
  • Doctors’ feedback collection and analysis

Constant selective monitoring for technical defects is carried out

Grant payment

Following 3 months of operation as part of the main stage of the Experiment, apply for the grant


The amount of grant to be paid is based on the number of studies analyzed by the AI service


Apply for the next grant payment every 3 months

Questions

Procedure and conditions to conduct the experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow (hereinafter, the Experiment) are regulated by Order of the Moscow Health Care Department No. 142 of 2/19/2020 (hereinafter, the Order No. 142 ) and Regulation of the Government of Moscow No. 1543-PP of November 21, 2019 “On Conducting the Experiment on the Use of Innovative Technologies in the Field of Computer Vision for the Analysis of Medical Images and Further Use in the Healthcare System of Moscow” (hereinafter, the Regulation ).

1
Who can participate in the Experiment?

Legal entities that provide services (software) based on computer vision technologies for the analysis of medical images of the following imaging modalities:
1. Computed tomography and low-dose computed tomography of the chest to detect lung cancer and/or coronavirus infection (COVID-19)  
2. Mammography to detect breast cancer
3. Chest X-ray to detect lung pathology (including the signs of coronavirus infection (COVID-19).

Experiment participant may not be a foreign legal entity or a Russian legal entity whose authorized (share) capital has more than a 50 percent share of foreign legal entities registered in a state or territory included in the list of states and territories that provide a preferential tax regime and/or do not disclose information on financial transactions (the offshore zones) in respect of such legal entities, approved by the Ministry of Finance of the Russian Federation.

2
What are the documents to be submitted to participate in the Experiment and where to apply?

The applicant for participation in the Experiment submits to the Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department (address: office 317, 28 Srednyaya Kalitnikovskaya St., bld. 1, 109029 Moscow, Russia; Phone: +7 (495) 276-04-36;
Contact person: Ekaterina Ilina
Tel.: +7 (937) 255-89-03.
Submission hours: 9:00 AM to 05:30 PM Moscow time.
An application to participate in the Experiment prepared in arbitrary form with the following documents attached.

Mandatory documents:

1. Certified copies of the legal entity's statutory documents.
2. Report on preliminary clinical and technical tests performed in accordance with Guidelines No. 43 “Clinical Acceptance of Software Based on Artificial Intelligence Technologies (Radiology)” (hereinafter – the Guidelines No. 43) recommended by the Expert Council for Science of the Moscow Health Care Department (Protocol No. 8 of June 25, 2019).

Additional documents:

1. Valid certificate of conformity of the quality management system to the ISO standard (if available).
2. Certified copies of the state certification as a medical device (if available).
3. Certified copies of certificates of quality compliance of the service proposed by the participant to local standards of FDA, CE, and their equivalents in other countries (if available).
4. Copies of scientific articles on evaluating the accuracy and efficacy of the proposed service (if available).
5. Originals or copies of reports on clinical implementation or application of the proposed service in medical centers of the Russian Federation or other countries (if available).
6. Other documents that the applicant considers necessary to provide.

3
In what form should the results of preliminary clinical and technical tests be provided?

In accordance with the Guidelines No. 43, preliminary clinical and technical tests (hereinafter, PCTT) must be conducted in collaboration with research and development or medical organizations under a simplified and adapted clinical test methodology to gain additional knowledge about the performance and limitations of the AI-based software.
PCTT is not a substitution or an equivalent to clinical trials and can either involve analytical validation and clinical acceptance, or consist of analytical validation alone. The recommended report form is given in Annex 7 to the Guidelines No. 43.
See the “Documents” section for Guidelines No. 43 and Annex 7.

4
Who and when is eligible to apply for a grant?

Who: the Experiment participants who successfully completed technical integration with ERIS EMIAS (Unified Radiological Information Service of the Unified Medical Information and Analytical System of Moscow), and whose operation during the main stage of the Experiment continued for at least 3 months.

When: The application is submitted on the basis of the predetermined number of analyzed images, not earlier than 3 months after entering the main stage of the Experiment, and then every 3 months.

5
What are the documents to be submitted to apply for a grant?

Application, filed in accordance with the Annex to the Order (Annex 4 to the Order):

Mandatory documents:

1. Certified copies of the legal entity's statutory documents.
2. Documents confirming the right to provide the proposed service based on computer vision for the analysis of medical images as well as describing the functions, technical architecture and characteristics, intended use scenarios of the service, and its hardware and software requirements.
3. Report on preliminary clinical and technical tests performed in accordance with Guidelines No. 43 “Clinical Acceptance of Software Based on Artificial Intelligence Technologies (Radiology)” (recommended by the Expert Council for Science of the Moscow Health Care Department, Protocol No. 8 of June 25, 2019) (if completed successfully, specify information on the place and date of the tests and attach a copy of the test report).

Additional documents:

4. Valid ISO certificate of conformity for the quality management system (if available).
5. Certified copies of the state registration certificate for medical device (if available).
6. Certified copies of certificates of quality compliance of the proposed service with the local FDA, CE standards and their equivalents in other countries (if available).
7. Copies of the scholarly articles evaluating the accuracy and efficacy of the service published in peer-reviewed journals indexed in Scopus and/or Web of Science (if available, provide references).
8. Original copies or scan copies of the reports on clinical implementation or use of the service in medical centers in the Russian Federation or other countries (if available).
9. Duration of acceptance testing of the proposed service in medical centers of the Russian Federation or other countries.
10. The number of and information on the medical centers where the acceptance testing of the service was carried out.
11. Availability and quality of implementation in the proposed service of the functionality of an automated analysis of medical images, indicating localization of pathological findings detected by the service and providing notification of the results (if available, attach a screenshot as an example).
12. Availability and quality of implementation in the proposed service of the possibility to form a DICOM series as a result of analysis, which contains information on the presence or absence of pathologies and analysis results visualizing the findings on the original images (if available, attach a screenshot as an example).
13. Support of the HealthLevel7 (HL7)/FHIR standard by the proposed service.
14.Availability and quality of implementation in the proposed service of the possibility to automatically provide information to prioritize studies in the radiologist worklist (if available, attach a screenshot as an example).
15. Availability and quality of implementation in the proposed service of the functionality of comparative analysis of studies of one patient performed at different times (evaluating the dynamics of pathological processes) (if available, attach a screenshot as an example).
16. Availability and quality of implementation in the proposed service of the functionality of automated preparation of the draft radiology report (if available, attach a screenshot as an example).
17. The time required to analyze one study using the proposed service.
18. Availability of the results of testing the diagnostic accuracy of the proposed service on depersonalized medical images of citizens of the Russian Federation and/or European and Asian subjects.
19. Diagnostic accuracy metrics of the proposed service, including classic ROC curve, area under the curve (AUC).
20. Diagnostic accuracy metrics of the proposed service, including false-negative rate.
21. Diagnostic accuracy metrics of the proposed service, including false-positive rate.
22. Diagnostic accuracy metrics of the proposed service, including other diagnostic accuracy parameters as per Guidelines No. 43.
23. Other documents that the applicant considers necessary to provide in order to receive the grant.

6
What are the requirements for completing the grant application?

See Order No. 142 for the application requirements.

The application and the documents attached to it should be submitted in Russian on paper, should be bound, numbered, and signed by an authorized person and stamped by the applicant. Corrections and erasures are not allowed. If the original document is issued in a foreign language, it should be accompanied by a translation in Russian certified by an authorized person and stamped by the applicant.

It is also recommended to attach a copy of the application and copies of the documents attached to it in electronic form in pdf format on a USB flash drive.

The copies of submitted documents (other than notarized ones) should be signed by an authorized person and stamped by the grant applicant.

7
Where to apply for a grant?

The application, along with the attached documents, are submitted to and registered by the Moscow Health Care Department .
The applications are accepted at 43 Oruzheinyi Lane, bld. 1, 1st floor, correspondence reception desk
Opening hours:
Mon, Thu: 8:00 AM to 05:00 PM
Fri: 8:00 AM to 03:45 PM
Lunch break: 12:30 to 01:15 PM
Sat, Sun: closed

8
What are the deadlines and procedure for application review?

The grant applications submitted by legal entities are reviewed by the Commission for Grant Applications from Legal Entities of the Moscow Health Care Department (paragraph 3.2 and 3.3 of the Order No. 142).
The term for the application review by the Commission does not exceed 10 calendar days from the date of receipt (paragraph 3.1 and paragraph 3.2 of the Annex to the Regulation).

9
How are the grant applications reviewed?

The applications are reviewed in accordance with paragraph 3 of the Annex to the Regulation and paragraphs 1, 2, and 3 of the Annex to the Resolution on the Commission for Grant Applications from Legal Entities of the Moscow Health Care Department of the Order No. 142.
As per paragraph 3.4 of the Annex to the Regulation the Commission forms a consolidated list of grant applicants and conclusions based on application review results.

10
Who makes the decision on grant allocation, what is the procedure for grant provision?

As per paragraph 3.5 of the Annex to the Regulation of the Department, based on the conclusions made by the Commission upon application review, the Department makes a decision on grant allocation, confirms the grant recipient status and the amounts of grants provided by adopting a legal act, and publishes the corresponding information on the official website of the Department on the Internet.

As per paragraph 3.6. of the Annex to the Regulation, grants are provided by the Department based on a grant contract (agreement) concluded between the grant recipient and the Department, which regulates the procedure and deadlines for the grant transfer.

11
How is the grant amount determined?

The grant amount shall be calculated as the product of the unit cost of analysis for one image of a certain imaging modality by the number of analyzed images.

Estimated cost of analysis required to determine the grant amount:

1. Computed tomography and low-dose computed tomography of the chest to detect lung cancer and/or coronavirus infection(COVID-19): 290 rubles per one processed study

2. Mammography to detect breast cancer: 175 rubles per one processed study

3. Chest X-ray to detect lung pathology (incl.COVID-19) 175 rubles per one processed study

12
How many studies will be subject to processing as part of the Experiment?

During the Experiment, the Services may process the following total number of studies, by imaging modality:

1. CT/LDCT of the chest to detect lung cancer and/or coronavirus infection(COVID-19)– 121,000 studies.
2. MG – 291,000 studies.
3. XR(incl. COVID-19) – 330,000 studies.

During the 3 months of participation in the Experiment, one Service can process the following minimum number of studies, by imaging modality:

1. CT/LDCT of the chest to detect lung cancer and/or coronavirus infection(COVID-19) – 4,800 studies.
2. MG – 8,200 studies.
3. XR(incl. COVID-19) – 11,000 studies.

 



 

For more information or if you have questions

For Press Inquiries, please contact Stanislav E. Samburskiy
e-mail: s.samburskiy@npcmr.ru