Share on facebook
Save on delicious
RSS Feed
Grid-enabled Data Collection and Analysis - Semantic Annotation in Skills-based Learning
McDonald (1), John; Gobbi (2), Mary; Michaelides (3), Danius; Monger (2), Eloise; Weal (3), Mark; De Roure (3), David
- (1) Institute of Education, University of London
- (2) School of Nursing and Midwifery, University of Southampton
- (3) Electronics and Computer Science, University of Southampton
This paper describes how Semantic Grid technologies can be used to generate enhanced tools for data collection that provide enabling technologies for interdisciplinary work, thereby enhancing the capacity to address substantive social science research. A feasibility study is presented in which semantic annotation (i.e. machine-processable annotation using Semantic Web technologies) is used to capture and work with the digital record, in support of subsequent qualitative and quantitative analysis. This case study provides a proof of concept, helping to develop the agenda for future work in this area.
The case study focuses on the research and practice of skills-based learning in the context of health care education (Greenberg, Loyd and Wesley, 2002). To ensure that there is a competent workforce which is "for practice" requires not only an increase in the number of Healthcare workers but significant changes in their roles, competence levels and working practices (Dept. of Health, 2002). Simulation is being used increasingly within the field of healthcare to help students develop their clinical skills (Alnier, 2006). Our work utilises a significant skills-based learning facility in the School of Nursing and Midwifery in Southampton, where a simulated ward provides a powerful testbed for the study, including extensive data collection and replay capabilities. Student nurses carry out simulation activities which are recorded, with post activity feedback given by the facilitating mentors. The activities build on the students' previous use of on-line learning resources developed by the School of Nursing. This provides a social science research context in which to investigate the new techniques.
The simulations in the skills-based learning lab allow for the acquisition of practical skills and decision-making, team working, communication and problem-solving (Gobbi et al., 2004). To use these facilities for assessment and feedback purposes, they need to be sound, valid, reliable, feasible, educational, and acceptable to practitioners (Ram et al., 1999). The annotations therefore need to cope with these forms of activity and provide opportunities for meaningful, timely feedback to student, teacher and 'real time' researchers. The approach taken was to use naturalistic time sequenced observation, followed by clustering of themes according to discipline specific relationships, for example `taking a pulse' was situated under a heading of `taking and recording vital signs'. Through a series of workshops, observational sessions and discussion groups, different types of annotations were identified including those made real-time during the exercise and annotations made post-exercise. The annotation ontology was constructed through a co-design exercise allowing the vocabulary to be controlled by the domain experts and aligned with the vocabulary used within the on-line resources.
Immediately after the exercises, the students receive feedback from the mentors. The standard mechanism for this is purely verbal, as although the session is videoed, video use is problematic as attempting to find specific points of interest would require going forward or backwards through the video and visually identifying the relevant points. Annotations captured real-time can give an index into this video and enable the quick identification of points of interest based on the events captured. Our approach builds on previous work in using Semantic Web in the capture and replay of meetings (Page et al., 2005). Through the use of the captured annotations more specific aspects of the scenario can be scrutinised and post-hoc reflection in, and upon action, can be facilitated through viewing back events marked as important by the mentor running the debriefing session. These annotations, although possibly using medical terminology, are more naturalistic observations, and the Semantic Web ontologies developed are not intended as a mechanism for sharing clinical knowledge as is supported by other systems (Gustafsson et al., 20006)(Kashyap et al., 2005). Post session study by the students could incorporate personal as well as teacher-led video annotations as an index into relevant on-line material that share the common vocabulary of the annotations. Video annotation systems for other domains exist, for example news production (Nack and Putz, 2001) where the focus is on more explicit description of content or for collaborative annotation of video (Fraser et al., 2006).
By more closely linking the skills based sessions with the on-line learning activities we may increase the ability of students to better reflect on their practical learning and capitalise on their individual study interests or needs.
The analysis of the students interacting with the on-line Web resources can provide insights into their learning strategies and achievements. By processing the logs created by the Web server we can answer questions such as: did the students cover the material adequately, how deeply did the students engage with the material and what do the paths students took through the material suggest about the content and its organisation? We transform the logs into a collection of sequences representing each student's session and apply sequence analysis techniques.
We also use the sequences to create visualisations to help the researchers understand the data. Directed graphs show the popular pages and routes that students take through the material, whilst event charts plot the sessions in the time domain and show durations of page viewing and total session time. These visualisations promote exploratory analysis but require some manipulation of the data. To help facilitate this analysis we provide workflows (Gil et al., 2007) to the statisticians encapsulating this processing, which allows them to select and refine the views they require easily. The high-level workflow language also allows the authoring of new visualisations by allowing components of the analysis to use different parameters and also be composed in different arrangements. These workflows are launched from a tool directly on their personal computer but may interact with remote databases and services.
Similar to the ability to search and mark video for student feedback purposes, the researcher can use this facility to guide the direction of any subsequent transcriptions required for analysis. This raises questions and options for the future direction of the traditional 'transcriber' role in qualitative research. For example, the researcher can annotate and bookmark in real-time or later before it is given to the transcriber. This can provide technical commentary, guides, instructions or explanations to the transcriber. Second, it may be possible to use or train domain specific transcribers so that they can do 'real time' work for research or other purposes. Third, it is worth considering whether parallel annotation of the researcher and transcriber might have a role in the first or second stage of video analysis. Fourth, the potential to use these technologies to guide therapeutic interventions in interpersonal therapies has yet to be explored.
Our initial experiments have highlighted the necessity for the nursing staff to create their own ontology for their annotations. This is especially important for accurate and efficient real-time annotation where familiarity with the annotation hierarchy is vital. It also allows them to include events pertinent to their own specialisms and to integrate more fully with the on-line teaching resources. Through the use of a Semantic infrastructure, we are better able to link the activities across disciplines allowing the statistical analysis of the Web logs to feedback into the teaching activity both in the modification of the on-line Web resources and providing a more comprehensive overview of the student learning process for the educators in nursing.
The capturing of detailed annotation of student activities during skills-based sessions is also allowing researchers in nursing to look it more detail at the teaching process itself, and providing a record of what nursing educators 'see' when they watch students carry out the scenarios. Analysis of this record may provide some insight into deeper research questions around the assessment and education of students in such sessions.
References
G. Alinier. Nursing students' and lecturers' perspectives of objective structured clinical examination incorporating simulation. Nurse Education Today 23[6], 419-426. 2004.
The Department of Health. Delivering 21st century IT support for the NHS: national strategic programme. Technical report, The Stationery Office, London, June 2002.
M. Fraser, J. Hindmarsh, K. Best, C. Heath, G. Biegel, C. Greenhalgh and S. Reeves. Remote collaboration over video data: Towards real-time e-social science. Computer Supported Cooperative Work (CSCW), 15(4):257-279, August 2006.
Y. Gil, E. Deelman, M. Ellisman, T. Fahringer, G. Fox, D. Gannon, C. Goble, M. Livny, L. Moreau and J. Myers (2007). Examining the Challenges of Scientific Workflows. IEEE Computer 40(12): 24-32.
M. Gobbi, E. Monger, G. Watkinson, A. Spencer, M. Weaver, J. Lathlean, and Bryant S. Virtual interactive practiceTM: A strategy to enhance learning and competence in health care students. In M. et al Fieschi, editor, Medinfo 2004, pp. 874-878, 2004.
R. Greenberg, G. Loyd, and G. Wesley. Integrated simulation experiences to enhance clinical education. Medical Education, 36:1109-1110, 2002.
M. Gustafsson, G. Falkman, F. Lindahl and O. Torgersson. Enabling an online community for sharing oral medicine cases using semantic web technologies. In Isabel F. Cruz, Stefan Decker, Dean Allemang, Chris Preist, Daniel Schwabe, Peter Mika, Michael Uschold, and Lora Aroyo, editors, ISWC 2006, 5th International Semantic Web Conference, pp. 820-832, 2006.
V. Kashyap, A. Morales, T. Hongsermeier and Q. Li. Definitions management: A semantics-based approach for clinical documentation in healthcare delivery. In ISWC'05 International Semantic Web Conference, pp. 887-901, 2005.
F. Nack and W. Putz. Designing annotation before it's needed. In MULTIMEDIA '01: Proceedings of the ninth ACM international conference on Multimedia, pp. 251-260, New York, NY, USA, 2001. ACM Press.
K.R. Page, D.T. Michaelides, S.J. Buckingham Shum, Y. Chen-Burger, J. Dalton, D. De Roure, M. Eisenstadt, S. Potter,N.R. Shadbolt, A. Tate, M. Bachler and J. Komzak. Collaboration in the semantic grid: A basis for e-learning. Applied Artificial Intelligence, 19(9):881-904, 2005.
P. Ram, R. Grol, J. Rethans, B. Schouten, C. van der Vleuten and A. Kester. 1999, "Assessment of general practitioners by video observation of communicative and medical performance in daily practice: issues of validity, reliability and feasibility.", Medical Education, vol. 33, no. 6, pp. 447-454.
