The design of the National Medical Science Olympiad in Iran is focused on critical thinking among medical students. The first Olympiad was held in Isfahan in 2009, and the second in Shiraz in 2010 [12, 13]. The specific goals of the Olympiad were competition to achieve the following: identifying and encouraging scientifically talented students; generating scientific morale; helping students gain closer familiarity with scientific culture; cultural exchanges among students; encouraging teamwork to develop creative and critical thinking; reinforcing the goals and objectives of the health system; further the development of interdisciplinary activities [12].
The Ethics Committee of the Olympiad approved the protocol adopted in the present study, and written, informed consent was obtained from all participants. Currently enrolled medical students who were interested in the study topic and who had an average grade of 16/20 (equivalent to a GPA of about 3.2 in the United States or a class of about 60 in the United Kingdom) or higher were able to register for the study. They then took part in an intensive training course in the subject area of their choice at their own university. Enrollees were tested, and those with the highest grades were allowed to participate in the study.
Iran has 46 medical universities, and each university is allowed to send only three students in the field of basic science to the Olympiad. A total of 133 undergraduate students took the test in basic science. In this study, we analyzed the examination results in basic science.
Development of the Olympiad examination by the reference panel
To prepare the examination used in the Olympiad, 10 experts from different universities in Iran were chosen for the reference panel. These experts held PhDs in such areas of basic science as biochemistry, hematology, physiology, immunology, pharmacology, and pathology, though they had different levels of professional experience. Each member of the panel took each of the four tests. On the morning of the first day, the drawing up of a concept map was completed; in the afternoon, the hypothesis writing was completed. On the morning of the second day, the hypothesis testing was evaluated; in the afternoon, the measuring of scientific thought was investigated. Each of the four examination periods lasted 4 h.
Examinees
All the medical students were invited to participate in this program. The selected examinees were 133 undergraduate medical students from 45 medical schools in Iran who had grades ≥ 16/20. The length of medical education in Iran is 7 years. Forty-three percent of the participants were male, and 57% were female. The mean age of the participants was 21.3 years, and their mean average grades were 18.3/20. The examinees were asked to complete all the Olympiad examination items in their own time without using textbooks, websites, or personal consultations.
Exam I: drawing a concept map
On the morning of the first day, the students were asked to draw a concept map based on three new articles in the field of stem cell research [14–16]. The examination period lasted 4 h.
To learn how to make concept maps, each participant previously completed a standardized concept map training session. Briefly, training included an introduction to concept mapping, followed by practice making concept maps on medical topics. The raters also received training about the concept-mapping process in the National Medical Science Olympiad.
Participants were given 90 min to complete their maps. The maps were coded, so that the identity, specialty, and level of training of the participants would be unknown to the raters. Participants created the maps on 20 × 30-cm sheets of paper. In constructing their concept maps, the participants drew concepts related to a certain domain and then indicated links using arrowed lines with a proposition written above the line, describing how the concepts were related (concept link), [8, 9]. Map hierarchy is an important part of the process; it is indicated by the direction of the arrow in the concept link and in the arrangement of concepts in the map with more general concepts at the top and more specific concepts below. Scoring of each concept map was based on the following criteria, using previously published reports [8, 9]:
-
1)
Valid and meaningful selection of concepts from the papers (score, 25%).
-
2)
Hierarchical arrangement of concepts with more general concepts at the top and more specific ones below (score, 20%).
-
3)
Meaningful integration among concepts in the map (score, 10%). Incorrect concept relationships were given zero points. A closely related concept was given the highest number of points. Less important, but correct concept relationships were given an intermediate score.
-
4)
Accuracy and depth of understanding of the relationship (score, 20%).
-
5)
Degree of student creativity showing more sophisticated understanding (score, 25%).
Propositions or cross-links that lacked linking phrases above the connecting line were counted separately and given less credit. No credit was given for an invalid (i.e., incorrect or wrong) proposition or cross-link. Before scoring the maps, the raters discussed problems with each other. Raters scored a random subset (40%) of the same maps to assess reliability. Three different raters, with experience in stem cell field research and blinded to the identity of the map author, independently scored each map. The raters' total scores were added to create a final score for each map. The use of raters who were familiar with both the Olympiad process and the assessment of the different steps involved in making concept maps was believed to be essential in obtaining low inter-rater variability.
Exam II: hypothesis generation
In the afternoon of the first day, the measuring of scientific thoughts was investigated in terms of hypothesis generation. After drawing their concept maps, the students were asked to write a hypothesis. A sample of a concept map that was drawn by the scientific committee was also available during the exam period (see Additional file 1 and Figure 1). This exam lasted 1 h.
The written hypothesis was evaluated by these criteria:
-
1)
Simplicity (score, 15%)
-
2)
Accuracy (score, 20%)
-
3)
Precision (score, 10%)
-
4)
Explaining the connection between events (score, 15%)
-
5)
Logical coherence (score, 10%)
-
6)
Fruitfulness (score, 10%)
-
7)
Creativity (score, 20%)
Exam III: choosing variables based on the hypothesis
On the morning of the second day, the participants were asked to choose variables that were important in hypothesis testing. This step lasted 2 h. It was made clear to the participants that they should choose variables that helped prove the hypothesis they had detailed. The credit given for the best answer was 50%, and scientific explanation was scored up to an additional 50%.
Exam IV: measuring scientific thought
In the final step, on the afternoon of second day, the students were presented with a scientific finding. To measure the students' scientific thinking, they were asked a series of questions related to the finding. They were also asked to make suggestions for future research. This step also lasted 2 h. The task for the examinees was to evaluate their suggestions in terms of direction (positive, negative, or neutral) and intensity. This effect was captured with a Likert scale. A sample of each examination is given in Additional file 1.
Total final score
The total exam score was assessed using the sum of the four exam grades, with each exam accounting for 25% of the total score.
