Experiment

on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow

Artificial intelligence in radiology

Goal:

Scientific research on the possibility of using clinical decision support systems in the Moscow healthcare system with data analysis based on advanced innovative technologies

Location:

Moscow region, Russia

Base:

Unified Radiological Information Service (URIS)

 

 

 

7
use case

Сhest computed tomography and/or low-dose computed tomography to detect lung cancer, COVID-19, osteoporosis of the spine, ischemic heart disease, including coronary calcium and pericardial fat, emphysema
Head computed tomography to diagnose various conditions, including strokes
Head magnetic resonance imaging to detect various conditions, including malignant neoplasms, multiple sclerosis, and Alzheimer's disease
Magnetic resonance imaging of the lumbar spine to detect various abnormalities, including hernias, protrusions, and stenosis
Mammography to detect breast cancer
Chest X-ray to detect various lung pathologies
X-ray of the musculoskeletal system to detect various abnormalities, including arthrosis, flat feet, bone and vertebral fractures

15
AI services

connected to URIS

1 031 363
studies

analyzed by AI

538
radiologists

evaluate the results of AI

Clinical application



Oncology
Lung cancer. CT/LDCT
С34
Breast cancer. MG
С50
Adrenal lesions. CT/LDCT
D44
Liver lesions. CT/LDCT
C22
Malignant neoplasm of brain. MRI
C71
Malignant neoplasm of prostate. MRI
C61
Cardiology
Coronary calcium. CT/LDCT
I25
Aortic aneurysm. CT/LDCT
I79
Pericardial fat. CT/LDCT
I27
Pulmonary artery dilatation. CT/LDCT
I25
Chronic diseases
Vertebral fracture (osteoporosis). X-RAY/CT/LDCT
M80
Abnormal liver density. CT
K70-K77
Osteoporosis. XR/CT
M80-M81
Arthroses. XR
M15-M19
Pulmonology
A spectrum of clinically relevant pathologies. X-RAY
J09-J18
Tuberculosis. X-RAY
A15
COVID-19
U07
Emphysema. CT
J43
Emergencies
Limb fractures. X-ray/CT
S10-T07
Cerebral hemorrhage. CT
I60-I64
Skull fractures. CT
S10-T07
Fractures/ XR/CT
S12–S92, T02
Infarctions and intracerebral hemorrhages. CT
I60–I64
Fracture of cervical vertebra. CT
S12
Neurology
Multiple sclerosis. MRI
G35
Alzheimer's disease. MRI
G30
Lumbar degenerative disc disease. MRI
M51
- Current
- Future

How to apply?

Preparation

Fill in and submit the questionnarie via e-mail

Download

Check the suitability of your AI service solution  (password mosmeddata)

SELF TEST

Check out quality standards

QA program

Apply for the experiment

Integration

Conclude an agreement


Check technical requirements

Integration guidelines for AI service providers AGFA DICOM conformance statement Kafka Apache

Integrate your service with URIS


Calibrate your service on the labeled dataset (retrospective data)

Basic requirements for AI services findings List of baseline functional requirements
Prospective study analysis

This is the main stage of the Experiment

Your AI service analyses studies in real time

The Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies provides for:

  • Diagnostic accuracy evaluation
  • Study audit by expert doctors
  • Analysis of identified discrepancies between the service and doctors
  • Doctors’ feedback collection and analysis

Constant selective monitoring for technical defects is carried out

Grants

Apply for a grant in 1 month of work at the main stage of the Experiment.


The amount of payments is determined based on the number of studies analyzed by the AI service. 


Please submit your invoices for payment on a monthly bases. 

Documents

Questions


Procedure and conditions to conduct the experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow (hereinafter, the Experiment) are regulated by Order of the Moscow Health Care Department No. 51 of 01/26/2021 (hereinafter, the Order No. 51 ) and Regulation of the Government of Moscow No. 1543-PP of November 21, 2019 “On Conducting the Experiment on the Use of Innovative Technologies in the Field of Computer Vision for the Analysis of Medical Images and Further Use in the Healthcare System of Moscow” (hereinafter, the Regulation ).

1
Who can participate in the Experiment?

Legal entities providing services (software) based on computer vision technologies designed for the analysis of medical images for the following types of studies:

1. Chest computed tomography and low-dose computed tomography for lung cancer detection

2. Chest computed tomography and low-dose computed tomography for COVID-19 detection

3. Chest computed tomography and low-dose computed tomography for detection of spinal osteoporosis

4. Chest computed tomography and low-dose computed tomography for detection of coronary artery disease (coronary calcium, pericardial fat)

5. Chest computed tomography and low-dose computed tomography for emphysema detection

6. Brain computed tomography for the diagnosis of various diseases, including strokes

7. Brain magnetic resonance imaging to detect various diseases, including malignant neoplasms, multiple sclerosis, Alzheimer's disease

8. Magnetic resonance imaging of the lumbosacral spine to detect pathologies, including hernias, protrusions, stenosis

9. Mammography for breast cancer detection

10. Chest X-ray and/or fluorography to detect various lung pathologies

11. Musculoskeletal X-ray to detect various abnormalities, including arthrosis, flat feet, bone and vertebral fractures  

Requirements for the Services:

1. Availability of functions in accordance with the basic functional requirements approved by the Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Healthcare Department (hereinafter referred to as the CDT).

2. Compliance with the basic requirements for the operation results approved by the CDT.

3. Compliance with the diagnostic accuracy indicators approved by the CDT.

4. Data obtained during a participation in the Experiment will be stored, analyzed and used only on the territory of the Russian Federation.

2
What are the documents to be submitted to participate in the Experiment and where to apply?

Applications for participation in the Experiment are sent to the Diagnostic and Telemedicine Center of the Moscow Healthcare Department at the following address: 24 Petrovka Street, office 240, Moscow, 127051, Russia.
Contact person: Larisa Rodionova.
Documents are accepted Monday to Friday from 10:00 AM to 5:00 PM Moscow time.

The application is submitted in accordance with the form in the Annexes to the Order No. 51. The following documents should be attached to the application:  

1. Certified copies of state registration documents of the applicant legal entity on the territory of the Russian Federation.

2. Documents confirming the development and/or the right to provide the proposed service based on computer vision for the analysis of medical images.

3. A report on preliminary clinical and technical tests in accordance with the Guidance  "Clinical acceptance of software based on artificial intelligence technologies (radiology)" (recommended by the Expert Council for Science of the Moscow Healthcare Department, Protocol No. 8 of June 25, 2019), and a report on testing on medical images of citizens of the Russian Federation and/or persons of the Caucasian and Mongoloid races.  

4. Documentation describing and confirming the availability of the service functions corresponding to the basic functional requirements, as well as describing the technical architecture and characteristics, provided scenarios for using the service, and its hardware and software requirements.  

5. A report on the clinical implementation and/or testing of the proposed service in medical organizations of the Russian Federation or other countries, indicating the duration of its testing, as well as a number of medical institutions where the proposed service was tested (if available).

 6. Certified copies of the certificate of conformity of the proposed service quality to FDA, CE standards and their regional analogues in other countries, or the state registration as a medical device (if available).

7. Copies of scientific articles on the assessment of accuracy and efficacy of the proposed service in peer-reviewed journals indexed in Scopus and/or Web of Science (if available).

8. A valid certificate of conformity of a quality management system to the ISO standard (if available).

9. Certified copies of the state registration as a medical device (if available).

10. Other documents that the applicant for participation in the Experiment considers necessary to submit.

3
In what format should the results of preliminary clinical and technical tests be provided?

In accordance with the Guidance, preliminary clinical and technical tests (hereinafter referred to as PCTT) are conducted by the company in collaboration with scientific research or medical organizations following a simplified and adapted methodology of clinical trials. These tests are necessary to get additional information about the capabilities and limitations of the AI-based software.    

PCTTs are not a substitution or equivalent of clinical trials. PCTT may include stages of analytical validation and clinical acceptance, or consist of analytical validation alone.   

1. A report on PCTT should indicate:  
- classic ROC - curve, area under the curve (AUC);
- indicators of sensitivity, specificity and accuracy;
- four-field table of comparing the results of the index and reference tests;
- duration of a study analysis.

2. A report on PCTT should be certified by a signature of the authorized person and a seal of the medical organization that prepared the report.

4
Who and when is eligible to apply for a grant?

Who: the Experiment participants who successfully completed a technical integration into URIS UMIAS (the Unified Radiological Information Service of the Uniform Medical Information and Analytical System of Moscow). 
When: no more than once a month, starting from the date of transferring studies from connected diagnostic devices to the industrial hardware-software complex URIS UMIAS (hereinafter referred to as IHSC URIS UMIAS) at the stages of testing and trial operation.

5
What are the documents to be submitted to apply for a grant?

A package of documents includes:

Application filed in accordance with Annex 4 to the Order № 51


Mandatory documents:

1. Certified copies of the legal entity's organisational documents.
2. Documents confirming the right to provide the proposed computer vision-based service for the analysis of medical images, as well as describing the functions, technical architecture and characteristics, intended scenarios for using the service, and its hardware and software requirements.
3. A report on preliminary clinical and technical testing in accordance with the Guidelines «Clinical acceptance of software based on artificial intelligence technologies (radiology)” (recommended by the Expert Council for Science of the Moscow Healthcare Department, Protocol No. 8 of June 25, 2019). 

Additional documents:

4. A valid certificate of conformity of the quality management system to the ISO standard (if available).
5. Certified copies of the state registration as a medical device (if available).
6. Certified copies of certificates of conformity of the proposed service quality to FDA, CE standards and their regional analogues in other countries (if available).
7. Copies of scientific articles on the assessment of accuracy and efficacy of the proposed service (if available).
8. Originals or copies of the reports on clinical implementation or use of the proposed service in medical organizations of the Russian Federation or other countries (if available).
9. Other documents that the applicant considers necessary to provide for the grant.

6
What are the requirements for completing the grant application?

Application requirements are described in Annex 4 to the Order № 51
The documents attached to the application should be submitted in Russian on paper, stitched up, numbered, signed by an authorized person and stamped by the applicant. Corrections and erasures are not permitted. If the original document is issued in a foreign language, its Russian translation, certified by an authorized person and stamped by the applicant should be attached to the document.
It is also recommended to attach electronic copies of the application and documents in pdf format on a USB flash drive.
Copies of the submitted documents (except for notarized ones) should be signed by an authorized person and stamped by the grant applicant.

In case of resubmission of the application, a provision of the original package is not required.

The application is submitted with a cover letter addressed to the Chairman of the Commission for the review of grant applications of legal entities of the Moscow Healthcare Department.

7
Where to apply for a grant?

The application, along with attached documents are submitted for registration to Larisa Rodionova, the Executive Secretary of the Commission for the review of grant applications of legal entities of the Moscow Healthcare Department.


Grant applications are sent to the following address: Diagnostic and Telemedicine Center of the Moscow Healthcare Department, 24 Petrovka Street, office 240, Moscow, 127051, Russia.
Documents are accepted Monday to Friday from 10:00 AM to 5:00 PM Moscow time.

8
What are the terms and procedure for reviewing a grant application?

Grant applications of legal entities are reviewed by the Commission for the review of grant applications of legal entities of the Moscow Healthcare Department (paragraphs 3.2 and 3.3 of the Order № 51). 
Meetings of the commission are held as applications are received (paragraph 3.1 and paragraph 3.2 of the Annex to the Regulation).

9
How are the grant applications being reviewed?

Applications are reviewed in accordance with the Regulation No. 1543-PP of the Moscow Government of November 21, 2019 and Order № 51.

10
Who makes the decision on grant allocation, what is the procedure for grant provision?

As per paragraph 3.5 of the Annex to the Regulation of the Department, based on the conclusions made by the Commission upon application review, the Department makes a decision on grant allocation, confirms the grant recipient status and the amounts of grants provided by adopting a legal act, and publishes the corresponding information on the official website of the Department on the Internet.

As per paragraph 3.6. of the Annex to the Regulation, grants are provided by the Department based on a grant contract (agreement) concluded between the grant recipient and the Department, which regulates the procedure and deadlines for the grant transfer.

11
How is the grant amount determined?

A grant amount is calculated as the product of a unit price for an image analysis of the certain type and a number of analyzed images.

The estimated cost of studies' analysis required to determine a grant amount is set up in the Annex to the Regulation № 47-PP of the Moscow Government of January 26, 2021.

12
How many studies will be subject to processing as part of the Experiment?

A total of 3 million studies can be analyzed in the Experiment.

 



 

For more information or if you have questions

YOU CAN VIEW A RECORDING OF THE WEBINAR «How to visualAIze 5000 COVID CT scans daily with 15 minutes SLA?»
04.06.2020 18:00 GTM +1

Email address of the Experiment: ai@npcmr.ru
Intergration into EMIAS URIS: eris@lvlmed.ru
Hotline support: +7 495 221-39-94

For Press Inquiries, please contact Stanislav E. Samburskiy
e-mail: s.samburskiy@npcmr.ru