Erasmus Project BeEmTel – “Advancing Telemedicine for Non-Communicable Diseases:

Simulation-Based Approaches

and e-Learning Platform Evaluation”

1 Manuela Bocchino

2 Giuseppe Di Lorenzo

1 Cecilia Damiano

3 Michela Barisone

1 Marco Silano

2 Valentina De Nicolò

3 Gianluca Aimaretti

4 Antonio Scalogna

3 Emiliano Loria

1 Department of Cardiovascular, Endocrine-metabolic Diseases and Aging, Italian National Institute of Health, ISS, Rome, Italy

2 La Sapienza University of Rome, School of Specialization in Hygiene and Public Health, Rome, Italy

3 University of Eastern Piedmont, Amedeo Avogadro, Department of Translational Medicine, Novara, Italy

4 University of Eastern Piedmont, SIMNOVA Centre, Novara, Italy

Article received: 31.10.2024.

Article accepted: 17.02.2025. https://doi.org/10.24141/2/9/1/3 Author for correspondence:

Manuela Bocchino ISS, Rome, Italy

E-mail: manuela.bocchino@iss.it

Keywords: e-health, medical pedagogy, medical simulation, telesimulation, e-learning platform

Abstract

Introduction. The key topics of e-health are education and training. The European Erasmus project BeEmTel aimed to create an innovative European curriculum dedicated to Telecare for chronic diseases and emer- gencies through educational tools based on an e-learn- ing platform and on remote simulation techniques.

Aim. To describe the experience regarding the intro- duction of an online telemedicine and telesimulation course applied to chronic diseases on some degree and academic career courses.

Methods. The online course has lasted from May 15, 2023 to January 15, 2024, on the e-learning Moodle platform managed by Simnova. The students’ popula- tion came from the nursing and medical faculties or departments of the four Partner Universities. The e- course has been structured into three sections: Non- Communicable Disease & Emergency, Health Simula- tion & Telecare, and New Forms of Proximity, with video lessons and resources, and a quiz area. Two immersive weeks have been reserved for a selected group of students. For the evaluation of the e-learn- ing platform and the simulation method, we have re- ferred to the first two levels of Donald Kirkpatrick’s model: user feedback, linked to the student’s reac- tion and level of satisfaction with the educational ac- tivity, and learning content, concerning learning and knowledge assessment.

Results. Among 394 participants, 38% have con- sistently attended the course, following a total of

57 theoretical lessons and 14 lectures using a tel- esimulation/telemedicine approach, and 31% have responded to the survey. Most of them have found the topics interesting (93.4%) and the lessons clear (81%), the platform interface has been considered user-friendly, although with certain practical prob- lems; 91% of learners have been satisfied with the course overall. In terms of the two immersive weeks, all participating learners have been satisfied with the general organization. When analyzing the meas- ures of association considering both degree course and nationality, only one association was statisti- cally significant (“Do you think telemedicine courses should be integrated into your degree program?”, Chi- squared test p-value=0.013).

Conclusions. The BeEmTel Erasmus+ Project has emerged as an innovative educational experience, effectively introducing the concept of telemedicine to healthcare students across different countries and academic disciplines, irrespective of their previ- ous background or attitudes. Students have demon- strated strong receptiveness to this approach. The student audience has found the type of instruction useful in preparing for careers in the healthcare field. Notably, both in-person medical simulation and “tel- esimulation” have been proved to be effective peda- gogical practices.

Introduction

The Covid-19 pandemic revealed the weaknesses of healthcare systems, especially in relation to vulner- able patients’ care, and vocational training system, which remains unable to tackle the significant chal- lenge of reforming the treatment and monitoring methods for chronic patients (1, 2). Various non-com- municable diseases (NCD) are associated with a pro- gressive reduction in functional capacity, leading to a persistent need for long-term care. At the European level, these diseases account for about 80% of mor- tality causes in people over 65 years old (3). Efforts to contain the Covid-19 pandemic have highlighted the need to improve the quality of care for NCD pa- tients and mitigate the negative impact they have not only on patients but also on their families and

society. To ensure consistent and high-quality sup- port for NCD patients, their families and caregivers, it is necessary to decentralize medical care at the ter- ritorial level (4). In this ever-approaching future sce- nario, the new profile of a healthcare and social care professional must include additional advanced skills. Doctors, nurses, psychologists and psychiatrists will increasingly require multidisciplinary training that in- cludes digital skills and more specific, practical and clinical competencies.

There are many definitions of competence. The most widely used formulation is the one proposed by G.O. Klempt in 1980 (5), taken up by Boyatzis (6) and then by Spencer&Spencer (7). Competence is “an intrinsic individual characteristic that is causally related to ef- fective and/or superior performance in a task or situ- ation and that is measured against an established criterion”. Being the result of knowing how to act, the production of a competent action derives from a shared responsibility between the individual him- self/herself, the management, the work context and the training programs. The main skills to be acquired for telehealth include: patient safety and appropriate use, access and equity, communication, data collec- tion and assessment, technology, ethical practices and legal requirements (8).

In recent years, literature has reported certain tel- emedicine teaching experiences among university students, which indicate that telemedicine is a valu- able experience and that an elective course in the medical school curriculum may be a useful way of providing future physicians with an understanding of telemedicine itself (9, 10, 11). Waseh and Dicker emphasized how undergraduate medical education preclinical years can be central to telemedicine train- ing and exposure (12); while many studies focused on the importance of telemedicine as a support to medical professionalism (13), there are still few ex- periences of its application in curricular teaching programs.

The majority of integration projects apply telemedi- cine during the clinical years, into the doctoring stream or during lecture time (14), or insert telemed- icine into objective structured clinical assessments

(15) through existing telemedicine hospital systems, rather than recreating new technology platforms (16). Two widely utilized teaching techniques in telemedi- cine are lectures, that provide the fundamental ideas, and patient interactions (17, 18), using video-based communication (19, 20), workshops (21), reflection

and reflective writing (20, 22). The instruments most frequently used to gauge customer satisfaction in this field are questionnaires and interviews (23), and pre- and post-test assessment scores (24).

Indeed, by incorporating telemedicine into students’ curricula, medical schools can augment their educa- tion and teach them competencies for patient care, giving them access to this form of training, which can significantly reduce healthcare costs and increase patient access to care (16).

Aim

The aim of this article is to describe the experience regarding the introduction of an online telemedicine and telesimulation course applied to chronic dis- eases on some degree and academic career courses. The BeEmTel project is an ongoing Erasmus K2 Ac- tion project, which took the opportunity to anticipate practical teaching to future assistance scenarios in chronic disease management.

Methods

The European project for the Erasmus+ Key Action 220 “Strategic Partnership Program BeEmTel - Be- yond the Emergency. Telecare for Non-Communica- ble Diseases through Simulation Techniques” comes from a partnership of five European Countries (Italy, Germany, Greece, Romania, and Croatia). The two specific objectives are the creation of:

1. the e-course;

2. the Digital Toolkit, available for free download from the BeEmTel website at the end of the project.

The e-course was initially designed for 250 partici- pants and was completely free, but due to high de- mand, we extended enrollment by the end of regis-

tration on April 25, 2023. The students’ population comes from the nursing and medical faculties or de- partments of the four Partner Universities; to enroll, students had to register on the Moodle platform by the deadline through the BeEmTel website (www. beemtel.eu). The Università del Piemonte Orientale (UPO) focused on the recognition of credits, Croatian, Greek, and Romanian students were easily engaged, as they came from active departments accustomed to Erasmus educational opportunities; however, Lud- wig-Maximilians-University (LMU) is a large universi- ty offering many structured projects and courses, so the accommodation for a non-recognized pilot course such as BeEmTel through official and compensatory means was not satisfactory.

The BeEmTel e-course engaged forty-six teachers from the partner countries and institutions: eight Croatians, five Greeks, twenty-five Italians, four Romanians, and four Germans. The development revolved around the three main areas: NCD & Emer- gencies (Area 1), Health Simulation & Telecare (Area 2), and New Forms of Proximity (Area 3). Fifty-seven theoretical lessons were delivered, with a total re- cording time of 35:02:26 hours. Additionally, four- teen lectures were prepared using a telesimulation/ telemedicine approach (Table 1), simulating clinical cases in virtual scenarios set in real-world environ- ments (patient’s home, outpatient clinic in a hospital, or general practitioner’s office).

There were three hundred ninety-four participants (Table 2): the majority were female (84.3%), while 31.8% were from Italy, followed by Romania (29.7%), Greece (22.1%), Croatia (14.6%) and Germany (1.8%); the majority (44.8%) was represented by students in

nursing, 44.0% in medicine, 2.7% in physiotherapy and 8.5% in other faculties (Computer Science, Phar- macies, Biomedical Laboratory Techniques, School of Engineering and Design - Human Factor Engineer- ing), while 20.6% were first year university students.

The online course designed for the project started on May 15, 2023 on the e-learning Moodle platform managed by Simnova, and ended on January 15, 2024. Simnova is the Interdepartmental Centre for Innovative Didactics and Simulation in Medicine and Health Professions of UPO. It relies on the external company Media Touch to manage the Moodle tech- nology, a widely used open source Learning Manage- ment System, ensuring usability, technical control, and the platform’s maximum security and reliability.

The e-course is structured into three sections: NCD & Emergency, Health Simulation & Telecare, and New Forms of Proximity; each section contains video les- sons and resources, including abstracts and supple- mentary materials, as well as a quiz section. Teach- ers prepared traditional video lessons and innovative telesimulation videos, since showcasing is a teaching method that gained prominence during the Covid-19 pandemic at leading universities worldwide (3, 25, 26, 27). The quiz area includes three self-assess- ment tests on each section, and a final quiz; passing the final quiz was required to finish the course.

The course held during two immersive weeks reserved for a selected group of approximately 50 students at two telesimulation centers in Novara and Munich took place between February and March 2024. Telesimula- tion refers to a new teaching methodology that com- bines telecommunication and healthcare simulation to provide effective educational support.

For the evaluation of the e-learning platform and the simulation method, we relied on Donald Kirkpatrick’s model (28), which analyzes and evaluates the results of training and educational programs. This frame- work identifies four levels of observation to assess the effectiveness of training:

• user feedback, linked to the student/resident’s reaction and the student’s level of satisfaction with the educational activity;

• learning content, concerning learning and knowledge assessment;

• workplace behavior, that looks into this topic and considers whether education has influ- enced behavior;

• impact on the organization, to assesses the ef- fect on outcomes by improving the quality of care and applying best practices.

We considered the first two levels of this model, sin- ce the last two are not fully applicable to our project data, using the following indicators:

1. User feedback: students received a survey, de- veloped through a review of relevant literature (29, 30, 31, 32, 33, 34) and a discussion process among the authors. The survey comprised 20 items, with an estimated time of five minutes to complete, six items were dichotomous ques- tions and 14 were multiple-choice questions with answers designed on a 5-point Likert scale.

   The survey is available in Appendix A, and it includes the following sections:

   (i.) integration of telemedicine in degree programs;

   (ii.) previous experience with telemedicine courses;

   (iii.) perception of telemedicine’s future applicability;

   (iv.) continued study of telemedicine;

   (v.) experience with e-learning course issues; (vi.) practical telesimulation experience;

   (vii.) relevance and interest of the course content;

   (viii.) course notifications and online platform; (ix.) clarity of the professors’ lessons;

   (x.) consistency in course attendance; (xi.) consistency in course attendance; (xii.) factors affecting participation; (xiii.) overall satisfaction;

   (xiv.) preference for learning mode.

   The questionnaire was available from January 4 to January 17, 2024; filling in the survey was manda- tory to complete the course.

2. Learning content: asynchronous distance learning has a great advantage for traceability, which allowed us to do an evaluation of this area based on the following indicators:

   • user-platform interaction (at least one access to the platform);

   • user-content interaction (at least once access to each lesson);

     
     

     • the percentage of completion of objectives (three modules);

     • the number of tests completed;

     • the average and total score for each test;

     • the percentage of those who would have passed the test by having 60% of the correct answers.

Ethics

The authors state that this it is not a study requiring approval by the ethics committee, according to the national regulations of the Italian Ministry of Health, Decree of 26 January 2023. At the time of registra- tion for the course, the participating students have given their informed consent for the processing of personal data, according to information pursuant to General Data Protection Regulation (EU) 2016/679, although the data from our study were anonymised.

Statistics

Considering the descriptive analysis, categorical vari- ables are reported as frequencies and percentages; for continuous variables, results are reported as median and interquartile range (IQR). We performed several measures of association, including the chi- square test and Cramer’s test (35, 36). Such tests an- alyzed the associations between the answers given by students in the e-learning survey and both their course study or their nationality. In all the analyses, a p value of <0.05 was considered as statistically significant. The software used for data management was Stata version 18.0 (Stata Corporation, College Station, TX, USA).

Results

Considering participation and proficiency, the major- ity (86.5%) declared willingness to participate to the courses, however, only a third of them completed the quizzes for the thematic areas (33.3% the test for NCD and Emergency and 32.3% for the other 2 ar- eas). Median scores were 79 (IQR 67-89), 77 (IQR 67-

84) and 77 (7-88), respectively. The final test was completed by 32.3% of the students and the median score was 87 (IQR 79-93). As a proxy, the association between achieving a passing grade on the final test and the aforementioned variables (degree course, nationality and willingness to participate) was stud- ied. It was found that passing the test was associ- ated with the student’s nationality (Table 3).

User feedback

One hundred and twenty-one students filled in the survey (30.7%). Figure 1 reports the user’s general opinion on telemedicine and their experience with the course. Only 16% of the students already attend- ed a telemedicine course (Figure 1c). Most of them consider telemedicine an important topic, noting that it should be integrated in their degree program (91%) and that they would continue independent study of the topic to strengthen their knowledge (88%) (Fig- ure 1a, Figure 1b). Students encountered problems in using the platform in 32% of cases (Figure 1d).

Figure 1. User’s general opinion on telemedicine and their experience (N=121)

Considering learners’ general opinion on the e-course (Figure 2), most of them found the topics covered in- teresting (93.4%) and in line with their knowledge (79.4%) (Figure 2c, Figure 2a). Students considered topics useful for their future professional life in 89% of cases (Figure 2d).

As for the user’s experience on the course’ organiza- tion (Figure 3), lessons were clear and easy to under-

stand (81%) (Figure 3c), the platform interface was useful and user-friendly for 88% of them, although 17% of the students encountered practical problems with it (Figure 3e).

During the entire year, thirty-eight percent of the learners consistently attended the e-course (Figure 4d). Among the aspects that positively influenced the e-course, the topics covered were indicated by

Figure 2. User’s general opinion on the BeEmTel course (N=121)

52.9%, 19.8% preferred the teachers and 12.4% the platform interface (Figure 4b). Conversely, among the aspects that negatively influenced the course, lack of time to devote to the course was indicated by 67.8% (Figure 4a). Finally, 91% of students were satisfied or very satisfied with the course overall (Figure 4c). When asked if they prefer face-to-face or e-learning lessons, more than half of the students (55.4%) opted for the latter (Figure 4e).

When analyzing the measures of association consid- ering both degree course and nationality (Tables 4 and 5), only one association was statistically signifi-

cant (“Do you think telemedicine courses should be integrated into your degree program?”). A univariate analysis was performed to verify the effect that area of study, nationality and willingness to participate in activities may have had on the final test score. The analysis did not provide significant findings (results not shown).

Figure 3. User’s experience on the course’ organization (N=121)

Discussion

Recent major advances in technology have made it easier and more cost-effective to connect patients and their family members/caregivers with remote health care providers. Many experiences tested the use of educational programs aimed at knowledge im- plementation, such as telesimulation and telemedi- cine, in the field of chronic diseases (8, 13), and the

recent Covid-19 pandemic period Covid-19 was a major driver. However, traditional university teach- ing across Europe has lagged significantly behind in incorporating even rudimentary elements of tel- emedicine and telenursing into medical and nursing curricula (37). To address this gap in university edu- cation, the Erasmus BemTel project proposed a pilot online course with an immersive experience lasting one week. The BeEmTel project aimed to highlight the potential of digital health education in foster- ing skills such as digital literacy, patient safety, and communication in telehealth settings. This study,

Figure 4. User’s overall opinion (N=121)

conducted as a final evaluation of the BeEmTel pro- ject’s outcomes, sought to determine whether in- tegrating telemedicine and telesimulation courses into academic curricula could help prepare healthcare professionals for evolving care models, particularly in managing chronic diseases. This approach is cru- cial for gaining a deeper understanding of how early exposure to telemedicine training may support stu- dents in addressing future healthcare challenges and

promoting more accessible, patient-centered care models. The success of this 10-month teaching and learning experience was measured by the satisfac- tion and approval of the primary targets, as well as by the breadth and diversity of the content focused on NCD, emergencies, telemedicine, and simulation. Additionally, thanks to the new and creative teaching methods of telesimulation, it was possible to deliver practical content even in a distance-learning format.

Are you satisfied with the course overall?

0.891 0.13

The positive overall impact of the BeEmTel project demonstrates that it is both possible and desirable to export and implement simulation-based teaching (both in-person and remotely) for telemedicine and, more broadly, for telecare.

BeemTel e-course has been analyzed from two dif- ferent perspectives: user’s feedback and learning of contents. Regarding the user’s feedback, only learn- ers who attended the e-course for its entire duration received the satisfaction survey and their responses appeared to be positive because, after the e-course was completed, most of them answered that tel- emedicine should be integrated into their degree programs and that they would continue to study this subject (Figure 1a, Figure 1b). This kind of interest appears to be an important result, since the major- ity of the responding students did not experience a telemedicine course before (Figure 1c). Other posi-

tive feedback was given regarding the topics covered in the lectures (Figure 2, Figure 3 b, Figure 4b) and the e-course organization (Figure 3). The overall ex- perience has been rated as positive by majority of the students who participated in the entire course (Figure 4c), despite some of them experienced some sort of problem (Figure 1d). Although the attending learners come from different faculties and countries, user’s feedback does not appear to be influenced by degree course or nationality (Table 2, Table 3).

Nevertheless, it has to be noticed that only 32.30% of enrolled students completed the e-course and passed the final test, meaning that over 2/3 of them dropped out of the course before it was over (January 2024). However, as shown in Table 2, the number of learners remained constant for the entire duration. This could suggest that the students who aban- doned the e-course did that at its beginning, while

those who started attending the lessons decided to do it until the end of the e-course.

Furthermore, it is worth noting that almost all en- rolled learners declared their willingness to attend the LTTA (Learning, Training and Teaching Activi- ties) before abandoning the e-course. Perhaps the perspective of a course based almost entirely on an e-learning platform, also suggested from the an- swers registered from the survey (Figure 1e, Figure 4e), could explain the observed dropout rate. The observed learning indicators confirm the positive feedback about the e-course, too. Lectures appear to have provided learners with well-established knowl- edge about all the subjects, as shown by the results of self-assessment tests and the final test (Table 2), although a small difference was observed among learners from different countries (Table 3).

However, despite the generally positive feedback from students, their willingness of integrating tel- emedicine topics into their study program had some heterogeneity and showed a statistically significant association with both the type of degree program and the respondents’ country of origin (Table 5). This finding suggests that, despite the overall favourable opinions, students’ willingness to add new subjects to their study path varies significantly depending on their specific program and the country where it is pursued. These differences may be attributed to the varying structure of academic programs across disci- plines and countries, leading to differing perceptions of workload among students.

No statistically significant differences were found re- garding the preference between online or in-person courses, suggesting that the uncertainty expressed on this topic does not appear to be influenced by the field of study or nationality. The same conclusion applies to questions addressing the overall course satisfaction, the perceived usefulness of the top- ics covered, or the factors influencing participation. None of these items show any association with the respondents’ academic or geographical background.

Limitations

This study has some limitations. The first limitation is that the course was voluntary and represented an extra step in comparison to the already demanding university work of the students. This has certainly limited the active and full participation of students.

The other limitation is linked to the methodology of analysis chosen, the four levels proposed by Kirkpat- rick: the structure of the project and the course has allowed us to evaluate only the first two levels, user feedback and learning content, leaving aside work- place behaviour, to consider whether education has influenced practice, and impact on the organization, to assess the effect on outcomes by improving the quality of care and applying best practices. Consider- ing only the first two levels of this model, since the last two are not fully applicable to our project data, our work surely needs to be completed.

Conclusion

The BeEmTel Erasmus+ Project has emerged as an innovative educational experience, effectively in- troducing the concept of telemedicine to healthcare students across different countries and academic disciplines, irrespective of their previous background or attitudes. Considering the growing importance of these fields, both now and in the future, it is crucial to impart this knowledge early in undergraduate edu- cation. Students have demonstrated strong receptive- ness to this approach. Multicultural initiatives such as the BeEmTel project offer significant opportunities to solidify these emerging disciplines at the European level, promoting community growth and standardi- zation. The experience of the BeEmTel Project lays a strong foundation for future efforts to assess the impact of these lessons and to explore how telemedi- cine and telesimulation modules can be integrated into university curricula, ensuring that future health- care professionals are well prepared for the real-world challenges that await them. The results of this project could be used as a basis for a subsequent one, struc- tured to assess its effectiveness by evaluating all four levels proposed by Kirkpatrick.

Author contributions

Conceptualization and methodology (MBo, EL, MBa); Data curation and formal analysis (GDL, CD); Data ex- traction (AS); Original draft (MBo, MBa, EL, GDL); Re- view and editing (MBo, GDL, CD, MBa, MS, VDN, GA, AS, EL); Supervison of the project (GA, MS).

Conflict of interest

The authors declare no conflicts of interest.

Acknowledgments

We would like to thank all the creators and imple- menters of the BeEmTel project, as well as all those who performed the online lessons and the telesimu- lation sessions and all participant students for the great experience.

Funding

This research received no external funding.

References

1. Sahu PK, Dalçik H. Editorial: Impact of COVID-19 on healthcare professions education. Front Med (Lau- sanne). 2023;10:1265811 https://doi.org/10.3389/ fmed.2023.1265811

2. Davis B, Bankhead-Kendall BK, Dumas RP. A review of COVID-19’s impact on modern medical systems from a health organization management perspective. He- alth Technol (Berl). 2022;12(4):815-24. https://doi. org/10.1007/s12553-022-00660-z

3. Council Of The European Union. Reflection pro- cess: Innovative approaches for chronic diseases in public health and healthcare systems. Brusse- ls; 2013. Available at: https://health.ec.europa.eu/ document/download/cdf1adad-a748-4146-b63a- 380ca79a4752_en

4. World Health Organization. Non communicable dise- ase facility-based monitoring guidance: framework, indicators and application. Geneva; 2022.

5. Klempt GO. The assessment of occupational compe- tence, report to the National Institute of Education. 1980.

6. Boyatzis RE. The competent manager: a model for effective performance. New York: Wiley & Sons, Inc; 1982.

7. Spencer LM Jr., Spencer SM. Competence at work: mo- dels for superior performance. Canada: John Wiley & Sons, Inc; 1993.

8. Cruz-Panesso I, Tanoubi I, Drolet P. Telehealth Com- petencies: Training Physicians for a New Reality? Healthcare. 2024;12(1):93. https://doi.org/10.3390/ healthcare12010093

9. Bulik, RJ, Shokar GS. Integrating telemedicine instruc- tion into the curriculum: expanding student perspec- tives of the scope of clinical practice. J Telemed Te- lecare. 2010;16(7):355-8. https://doi.org/10.1258/ jtt.2010.090910

10. Kuhn S, Jungmann F. [Medicine in the digital age: Telemedicine in medical school education]. Radi- ologe. 2018;58(3): 236-40. German. https://doi. org/10.1007/s00117-017-0351-7

11. Brockes C, Grischott T, Dutkiewicz M, Schmidt- Weitmann S. Evaluation of the Education “Clinical Telemedicine/e-Health” in the Curriculum of Medical Students at the University of Zurich. Telemed J E He- alth. 2017;23(11):899-904. https://doi.org/10.1089/ tmj.2017.0011

12. Waseh S, Dicker AP. Telemedicine training in un- dergraduate medical education: mixed-methods re- view. JMIR Med Educ. 2019;5(1):e12515. https://doi. org/10.2196/12515

13. Shawwa L. The Use of Telemedicine in Medical Edu- cation and Patient Care. Cureus. 2023;15(4):e37766. https://doi.org/10.7759/cureus.37766

14. Palmer RT, Biagioli FE, Mujcic J, Schneider BN, Spires L, Dodson LG. The feasibility and acceptability of admi- nistering a telemedicine objective structured clinical exam as a solution for providing equivalent educati- on to remote and rural learners. Rural Remote Health. 2015;15(4):3399.

15. Iancu AM, Kemp MT, Alam HB. Unmuting medical stu- dents’ education: utilizing telemedicine during the COVID-19 pandemic and beyond. J Med Internet Res. 2020;22(7):e19667. https://doi.org/10.2196/19667

16. Gareev I, Gallyametdinov A, Beylerli O, Valitov E, Alys- hov A, Pavlov V, et al. The opportunities and chall- enges of telemedicine during COVID-19 pandemic. Front Biosci (Elite Ed). 2021;13(2):291–8. https://doi. org/10.52586/E885

17. Poland S, Frey JA, Khobrani A, Ondrejka JE, Ruhlin MU, George RL, et al. Telepresent focused asse- ssment with sonography for trauma examination training versus traditional training for medical stu- dents: a simulation-based pilot study. J Ultrasound Med. 2018;37(8):1985-92. https://doi.org/10.1002/ jum.14551

18. Liu C, Lim RL, McCabe KL, Taylor S, Calvo RA. A web- based telehealth training platform incorporating automated nonverbal behavior feedback for te- aching communication skills to medical students: a randomized crossover study. J Med Internet Res. 2016;18(9):e246. https://doi.org/10.2196/jmir.6299

19. Bramstedt KA, Prang M, Dave S, Shin PN, Savy A, Fatica RA. Telemedicine as an ethics teaching tool for medical students within the nephrology curricu- lum. Prog Transplant. 2014;24(3):294-7. https://doi. org/10.7182/pit2014289

20. Newcomb AB, Duval M, Bachman SL, Mohess D, Dort

    
    

    J, Kapadia MR. Building rapport and earing the surgi- cal patient’s trust in the era of social distancing: te- aching patient-centered communication during video conference encounters to medical students. J Surg Educ. 2021;78(1):336-41. https://doi.org/10.1016/j. jsurg.2020.06.018

21. Iancu AM, Kemp MT, Gribbin W, Liesman DR, Nevarez J, Pinsky A, et al. Twelve tips for the integration of medical students into telemedicine visits. Med Teach. 2021;43(10):1127-33. https://doi.org/10.1080/014

    2159X.2020.1844877

22. AlDossary S, Martin-Khan MG, Bradford NK, Smith AC. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Inform. 2017;97:171-94. https:// doi.org/10.1016/j.ijmedinf.2016.10.012

23. Jonas CE, Durning SJ, Zebrowski C, Cimino F. An interdis- ciplinary, multi-institution telehealth course for third- year medical students. Acad Med. 2019;94(6):833-7. https://doi.org/10.1097/ACM.0000000000002701

24. Sharma D, Bhaskar S. Addressing the Covid-19 Bur- den on Medical Education and Training: The Role of Telemedicine and Tele-Education During and Beyond the Pandemic. Front Public Health. 2020;8:589669. https://doi.org/10.3389/fpubh.2020.589669

25. Gerstenberger JP, Hayes L, Chow CJ, Raaum S. Medical Student Experiential Learning in Telesimulation. J Med Educ Curric Dev. 2023;10:23821205231216067.

    https://doi.org/10.1177/23821205231216067

26. Honda R, McCoy CE. Teledebriefing in Medical Simu- lation. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024. Available from https:// www.ncbi.nlm.nih.gov/books/NBK546584/

27. Rosa A. Kirkpatrick’s four-level model for evaluating training programs: the weak links in a “chain of evi- dence”. Lifelong, Lifewide Learning. 2018;14(31):17-

    37. https://doi.org/10.19241/lll.v14i31.113

28. Kirkpatrick DL. Evaluating training programs: The four levels. San Francisco: Berrett-Koehler; 1994.

29. Taveira-Gomes T, Saffarzadeh A, Severo M, Guimarães MJ, Ferreira MA. A novel collaborative e-learning plat- form for medical students - ALERT STUDENT. BMC Med Educ. 2014;14:143. https://doi.org/10.1186/1472-

    6920-14-143

30. Falletta SV. Book Review: Evaluating Training Pro- grams: The Four Levels. American Journal of Evalua- tion. 1998;19(2):259-61. https://doi.org/10.1016/ S1098-2140(99)80206-9

31. Zimmerman BB, Clarke MC. Applying survey research methods to gather customer data and to obtain user feedback. In: Albers MJ, Mazur MB, editors. Content and Complexity. New York: Routledge; 2014.

32. Shee DY, Yi-Shun W. Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Compu- ters & Education 2008;50(3):894-905. https://doi. org/10.1016/j.compedu.2006.09.005

33. Santoso HB, Schrepp M, Isal RYK, Utomo AY, Priyogi

    B. Measuring user experience of the student-cente- red e-learning environment. The Journal of Educators Online 2016;13(1):58-79. https://doi.org/10.9743/ JEO.2016.1.5

34. Panagiotis Z, Poylymenakou A. Developing a usabi- lity evaluation method for e-learning applications: Beyond functional usability. Intl. Journal of Human– Computer Interaction 2009;25(1);75-98. https://doi. org/10.1080/10447310802546716

35. Cramér H. Mathematical Methods of Statistics. Prince- ton: Princeton University Press; 1946.

36. Pearson K. On the criterion that a given system of deviations from the probable in the case of a corre- lated system of variables is such that it can be rea- sonably supposed to have arisen from random sam- pling. Philos. Mag. 1900;50(302):157-75. https://doi. org/10.1080/14786440009463897

37. Fradelos EC, Barisone M, Lora E, Valiakos E, Papatha- nasiou IV. Competencies and skills needed in the ma- nagement of chronic patients’ needs through teleca- re. Pol Merkur Lek. 2023;51(4):403-16. https://doi. org/10.36740/Merkur202304116

    
    

    
    

    SUPPLEMENTARY MATERIAL

    
    

    
    

    Appendix A.

    The appendix contains supplementary informa- tion which may be helpful in providing a more comprehensive understanding of the research subjects.

    
    

    BeEmTel USERS’ FEEDBACK SURVEY

    
    

    Course Topics

    • The subjects covered in the course were com- mensurate with your knowledge (5 points agreement Likert)

    • The subjects covered in the course were inter- esting (5 points agreement Likert)

    • The course covered all the relevant subjects about Telemedicine (5 points agreement Lik- ert)

    • The subjects covered in the course were use- ful for my future professional life (5 points agreement Likert)

    • In your opinion, Telemedicine will find greater applicability in the future? (Yes/No)

    • After this course, you will continue to study Telemedicine to strengthen your knowledge on this topic? (Yes/No)

      
      

      Structure and Platform

      Did you experience any type of issues attending to this e-learning course? (Yes/No)

      I found e-mail notifications about the course useful for attending the course (5 points agreement Likert)

      How often did you experience some problems with the online platform (slow responses, server crashes, account issues, etc.)? (5 points frequency Likert)

      I found the online platform interface very useful and user-friendly (5 points agreement Likert)

      I found the subjects division of the course very easy to understand (5 points agreement Likert)

      Teachers

    • Professor’s lessons were clear and easy to un- derstand (5 points agreement Likert)

      
      

      Participation and appreciation

    • I attended the course consistently during the entire year (5 points agreement Likert)

    • Which of the following factors negatively af- fected your participation in the course? (Top- ics; Platform interface; Lessons breakdown; Teachers; Lack of time to devote to the course; None)

    • Which of the following factors positively influ- enced your participation in the course? (Top- ics; Platform interface; Lessons breakdown; Teachers; Lack of time to devote to the course; None)

    • Are you satisfied with the course overall? (5 points satisfaction Likert)

    • Do you prefer face-to-face or e-learning mode? (Face-to-face; e-Learning)

    • Do you think that in the future the practical experience of telesimulation should be pro- vided to all course users? (Yes/No)