Vernon Curran, PhD^*, Fran Kirby, MEd^*, Michael Allen, MD^†, Joan Sargeant, MEd^†

     Participation in traditional face-to-face CME has always been a challenge for rural physicians because of the difficulties in arranging practice or hospital coverage, spending time away from family, and the cost of travel and attendance. The World Wide Web (WWW) offers exciting opportunities for self-directed CME, providing physicians with the opportunity to address their individual learning preferences and needs. It has been reported that Web-based CME facilitates greater access to current information on new clinical developments and enables physicians to learn at a pace, place, and time which is more convenient for them.^1-6 The WWW’s multimedia capability and the variety of formats in which Web-based CME may be delivered are also believed to be advantages of this form of learning.^7-15

     It has been suggested that CME in the 21st century must move toward “a fully integrated practice-learning environment”¹ in which a variety of educational resources and technologies are available to address the individual learning needs and preferences of physicians. One aspect, which appears to be missing from the CME literature, is an examination of a mixed learning technology approach for providing CME at a distance. Moore & Kearsley¹⁶ have suggested that research is needed on the “synergistic effects of certain media — that is, the effectiveness of using one medium as a complement to another”. Such studies are pertinent today as there is a dramatic increase in the number of providers and technologies emerging in the CME field.¹⁷

    Audio teleconferencing, used since the 1960s, has proven to be effective for facilitating communication and interaction between practitioners.^18-24 Despite its reported effectiveness, one main limitation of audio teleconferencing is that participants cannot see each other, and it has been described as a very impersonal form of communications.²⁰ At Memorial University of Newfoundland, audio teleconferencing technology has been used since the 1960s to provide CME at a distance through the Wednesday at Noon program. This program entails the delivery of weekly, one-hour audio teleconference sessions using a province-wide network. Up to 34 dedicated audio teleconferencing sites can participate. The majority of these are located in hospitals and health clinics in rural and remote communities. Participants can also access the program from their office or home telephones.

     Our study team was interested in examining the usefulness of a mixed learning technology approach, which combined audio teleconferencing and Web-based learning to augment the CME learning experience. Physicians and nurse practitioners could access instructional materials before, during, and/or after an audio teleconference through PCs with Internet connectivity. The main components of the Web-based learning system included:

     The Web-based learning system was intended to be used as a medium for the real-time presentation of instructional materials related to live audio teleconferences, as well as a means for archiving instructional material and audio teleconference recordings. Presentation materials normally encompassed Power Point presentation files. The computer conferencing area was established as a resource to faculty and learners for facilitating asynchronous, text-based communications. There were no mandatory participation requirements for the computer-mediated conferencing area.

     The purpose of this study was to evaluate the utility and acceptability of a mixed learning technology approach for enhancing audio teleconferencing instruction. How useful was the mixed learning technology system for enhancing the distance CME experience? Was the mixed learning technology system an acceptable means for enhancing the distance CME experience? Specific evaluation questions included:

Methodology

     Our study population was comprised of 3 groups: 1) physicians and nurse practitioners who participated in the audio teleconferences and accessed the Web-based learning system (online users); 2) physicians and nurse practitioners who participated in the audio teleconferences and did not access the Web-based system (non-users); and 3) faculty who taught in the audio teleconference sessions. The online user and non-user learners were assigned to their study group designations based on whether they accessed or did not access the Web-based learning system. All learners practiced in Atlantic Canada (Newfoundland and Labrador and Nova Scotia) and were forwarded a letter of recruitment explaining the purpose of the study prior to being contacted.

      Web Site User Data - Through use of user-names and passwords we tracked the number of times different participants accessed a session’s Web site as well as the date and time when access took place.

     Online Participant Semi-Structured Telephone Survey - We conducted a semi-structured telephone survey with online users. All online users were contacted by letter and invited to participate. The survey questions were open-ended. The first section included questions about demographics (i.e., years of experience, practice type, physician type), self-reported computer experience, computer access, and teleconferencing attendance information. The second section asked respondents to comment on their experience with various aspects of the Web-based learning system, likes, dislikes, and suggested improvements.

     Non-User (or Audio-only) Participant Questionnaire Survey - A questionnaire survey was mailed to individuals who had participated in a live audio teleconference, but did not access the Web-based learning system. The survey collected information about demographics and respondents’ use of computers, computer experience, computer access, and location of participation. Open ended questions and five-point Likert scale questions (1 = strongly disagree to 5 = strongly agree) collected information on barriers and challenges to using the Web-based learning system.

     Faculty Questionnaire Survey - A questionnaire survey was mailed to faculty at the conclusion of the Wednesday at Noon program. The survey contained questions about years of practice experience, computer access, and computer skill. Open ended questions and five-point Likert scale questions (1 = strongly disagree to 5 = strongly agree) collected information on audio teleconferencing technology, design and navigability, system components, barriers and challenges to using the system, and likes and dislikes of the Web-based learning system and individual instructional Web sites.

Results

     Online Participation - The Web-based learning system could be accessed before, during or after a ‘live’ teleconference session. Thirty-one, one hour audio teleconference sessions were offered during the period of September 2000 to May 2001. The range of participating sites for the audio teleconferencing program was 9 to 21 with a mean of 14.4 sites. The range of online users accessing the Web-based learning system during a live teleconference was 1 to 6 with a mean of 2.3 users per session. The range of online users accessing the Web-based learning system before or after live teleconference times was 1 to 16 with a mean of 5.2 users. Figures 1 and 2 show the trend in total users of the Web-based learning system during the Fall and Winter offerings of the program. Generally, there was a gradual decrease in the number of users of the system, from the beginning to the end of the program in both the Fall and Winter semesters.

    Online Participant Evaluation - Thirty-five (35) participants were identified as online users and eighteen (18) telephone interviews were conducted, resulting in a response rate of 51%. Table 1 provides a summary of the demographic information, self-reported computer access and computer skill responses for the online users. The majority of respondents were family physicians (72%), reported less than 15 years of professional practice experience, and practiced in a group setting (72%). Sixty-seven percent (67%) of online respondents reported access to a computer at both work and home and approximately 50% of respondents reported a level of computer skill between medium-high and high.

     The most common reason which online respondents provided for accessing the Web-based learning system was to ‘view material’ (N = 7) and ‘to listen to previous teleconference sessions’ (N = 4). Fifty percent (50%) of online respondents indicated it was more useful to access the system prior to the live teleconference session, 28% indicated it was more useful after, while 22% reported it was more useful to access the site during the live teleconference.

     The most useful aspects of the Web-based learning system were reported to be the presentation materials (N = 16), audio recordings (N = 9), and the ability to complete and submit online evaluation forms (N = 4). Only 17% of participants accessed the computer conferencing area. Fifty-nine percent of online participants felt that the Web-based learning system enhanced the quality of the distance learning they experienced. The aspects of the Web-based learning system which participants enjoyed included ‘accessibility’, ‘convenience’, and its ‘ease of use’. Accessibility referred to being able to access CME information any time it was needed; ‘it is important for justin-time knowledge’. Some participants experienced difficulty navigating the Web site and one online user indicated that their ‘computer and teleconferencing equipment was not in the same place’ which presented an obvious challenge for viewing the Web site and participating in the teleconferencing presentation simultaneously. Downloading time was another dislike and participants suggested that the file size of information presented on the Web site should be considered for those using dial-up modems in rural areas where bandwidth is limited.

     Non-User Participant Evaluation - Thirty seven (37) participants were identified as potential non-user survey respondents and 16 participants returned surveys for a response rate of 43%. Table 1 presents the demographic information, self-reported computer access and computer skill responses from the survey. Fifty percent of the non-user respondents were family physicians, in practice less than 10 years (75%), and members of a group practice (71%). Thirty eight percent of respondents had access to computers at work only, 25% had access at home only, and the remaining 38% reported access at both home and work. Forty four percent of the respondents rated their computer skill level as medium and only a small number of respondents rated their skill level as medium-high (19%).

     Non-users were asked to rate a number of items related to barriers and challenges in using the Web-based learning system (Table 2). A majority of respondents (75%) agreed or strongly agreed that ‘difficult to find time from work’ was a challenge to using the Web-based learning system. Non-users also agreed or strongly agreed that the following factors presented barriers or challenges to using the learning system: ‘other personal activities and commitments left me no time’ (50%), ‘accessing a computer is often inconvenient in relation to where I work or practice’ (44%), and ‘I am not a skilled computer user’ (38%).

     Faculty Evaluation - Fifteen (15) faculty, of a total of 31, returned the evaluation survey for a response rate of 48%. The results suggest that faculty had very positive opinions of the Web-based learning system. The majority of respondents agreed or strongly agreed that the Web site for their session was ‘clearly laid out’ (83%); ‘instructional material was presented accurately and clearly’ (100%); and ‘Internet site links were relevant’ (75%). Faculty were also asked to rate the extent to which items were a challenge or barrier to their use of the Web-based learning system (Table 2). Thirty-five percent (35%) of the respondents agreed or strongly agreed that it was ‘difficult to find time from work’ and 39% agreed or strongly agreed that ‘other personal activities’ were significant barriers or challenges to their use of the Web-based learning system.

Discussion

     What was the level of usage of the mixed learning technology system? - The results indicate that the majority of users of the Web-based learning system accessed it at times other than the live delivery of an audio teleconferencing session. One of the main goals for developing the system was to enable the presentation of instructional material during live audio teleconference sessions. Therefore, it is questionable whether the system actually served an effective role as an instructional material presentation medium. Technical and communication system limitations (old computers, slow phone lines, modems, etc.) are considered to be a main disadvantage of programs which are offered over the Internet.^5,7 Our study revealed that a number of technical barriers were experienced by both participants and presenters attempting to use the Web-based learning system. The main challenge however to integrating a truly ‘mixed’ technology strategy into our distance CME program was computer hardware and connectivity access. Many sites simply did not have computers and Internet connectivity in the same room as the audio teleconferencing equipment.

     The mean number of users of the Web-based learning system was 7.5, while the mean number of participating sites was 14.4 per teleconference session. The average number of participants for each Wednesday at Noon session is 20 persons. This would suggest that approximately 38% of participants in the program had accessed the Web-based learning system. This usage is fairly positive in light of the technologic barriers which participants experienced at the teleconferencing sites. This estimate is higher than Canadian physicians’ overall reported use of online CME. The Canadian Medical Association’s 2001 Physician Resource Questionnaire²⁵ reported that approximately 23% of Canadian physicians used the Internet for participating in online CME during the year 2000.

     A major concern regarding the implementation of Web-based learning for continuing professional education has been that many health care professionals are simply not comfortable with the use of computers for learning. Kripilani et al.² in a survey of Texas physicians found that those with prior computer experience, greater access to computers and online services, and previous experience with computer-assisted CME rated this form of learning higher than counterparts with less computer experience. The findings from our evaluation support these findings. The participants who accessed the Web-based learning system reported a higher level of computer skill and a greater level of access to computers. Not surprisingly, computer access and experience would appear to be very important factors influencing use of Web-based CME.

     What were participants’ experiences and perceptions of the mixed learning technology system? - Generally, the respondents reported a high level of satisfaction with the Web-based learning system. Physicians indicated satisfaction with how the system was organized, the usefulness of the information which was supplied, and the impact the system had on the quality of instruction and learning they experienced. Ruskin et al.²⁶ believe that electronic publications which use multimedia components, such as sound and movies, are able to present health science information in a way that would almost be impossible in printed text. Computer-mediated learning also allows the adult learner quick and easy access to bibliographic and other reference materials. The majority of online participants in our study reported that their main reasons for accessing the system were to ‘view material’ and ‘listen to recorded teleconference sessions’. The mixed learning technology approach appeared to be successful in providing greater access to self-directed, multimedia learning resources, even if the material was not used by the participant to augment live teleconference presentations. Location and time independence (being able to log on and participate from wherever and whenever) are often two of the greatest advantages of computer-mediated learning reported by adult learners.^3,27 A number of respondents indicated that the 12:00 pm teleconference time was inconvenient, either due to time constraints or scheduling conflicts, to participate in the audio teleconferencing program. A main benefit to the incorporation of computers into the continuing education process is the transmission of information directly to the learner whenever and wherever he/she chooses to learn.

      What were faculty experiences and perceptions of the mixed learning technology system? - An analysis of the open-ended responses from faculty suggested a high level of satisfaction with the system and the way in which their instructional materials were presented. Unfortunately, a number of faculty did not access the system to an extent which might have had a positive influence on participants’ use of the system. The success of the asynchronous computer conferencing area was dependent on the faculty member’s integration of collaborative discussion activities as part of their instructional strategy. A majority of faculty did not incorporate online discussion activities as part of their instruction and this influenced the use of the online discussion area by participants.

      Educators who have traditionally enjoyed teaching in the face-to-face setting are not required to perform in the same manner in online learning environments. The successful implementation of computer-mediated learning calls for a change in the role of the educator as he or she must learn new skills for becoming a manager of online learning. For some, this is a major paradigm shift, therefore faculty development and support activities which assist faculty in how to lead and facilitate online discussions are very important.

      How effective was the mixed learning technology approach in fostering online collaboration between learners and faculty? - The majority of the online respondents indicated they did not access the computer conferencing area of the Web-based learning system. A review of postings to the conferencing system confirmed the low participation and usage rate. Several participants reported that they did not need to access the computer conferencing area due to being able to access instructional materials or media in other areas of the system. In other words, the participants did not perceive an educational benefit from accessing the conferencing area. There was also no direct instructional link between the live teleconference sessions and the use of the computer conferencing area.

     The main limitation of this study is related to the opinions, which were reported by the evaluation respondents. It is possible that these views are mainly representative of physicians holding positive attitudes towards technology-based learning. Despite this limitation, the results from this exploratory evaluation study do provide a basis for further investigation into mixed learning technology approaches in CME. A number of questions have been raised as a result of the findings from this study. What are the effects of mixed learning technology approaches on cognitive, attitudinal, or performance-based outcomes? What combination of distance learning technologies are most effective in CME delivery? What are the specific features of distance learning technologies which best lend themselves to mixed technology approaches? What is the nature of the instructional strategies which should be used in mixed technology approaches?

     The findings from the evaluation study indicate that the Web-based learning system was used by the adult learners. Most of these users appeared to be using it as a resource for supporting their self-directed learning efforts. We have made several modifications to the system to make it more user-friendly, such as improving the user registration process, navigational features, and replacing audio streaming files with video. Greater efforts have also been made to promote the Web-based learning system to administrators and decision-makers at the health care board level where decisions concerning expenditures for computer equipment and connectivity infrastructure are made. The most important lessons which we have learned relate to computer hardware and connectivity access at teleconferencing sites and the provision of faculty development support for presenters.

Acknowledgments

     Funding for this project was provided through a grant from the Office of Learning Technologies, Human Resources Development Canada. We would like to acknowledge the contributions from the following partners: IDON East Corporation, Telehealth and Educational Technology Resource Agency (TETRA), the Health Sciences Library of Memorial University of Newfoundland, Glaxo Smith Kline, and the Office of Continuing Medical Education, Dalhousie University.

Lessons for Practice

     A mixed learning technology approach enables CME providers to combine the synergistic benefits of different technologies and media to enhance the distance learning experience.

     The features of the mixed learning technologies must be combined and utilized in a manner which facilitates a seamless learning experience for the adult learner.

     In organizing and planning a mixed learning technology approach, CME providers need to be attentive to technology access at distant sites and adult learners’ experience and comfort with the particular technologies.

     Faculty development is required to support faculty with the design of instructional strategies which draw on the synergistic benefits of the different technologies and media.

     Further studies are needed to explore which combinations of learning technologies can be best used to facilitate effective distance CME learning environments.

References

Correspondence

Vernon Curran, PhD
Assistant Professor (Medical Education)
Centre for Collaborative Health Professional Education
Faculty of Medicine
Memorial University of Newfoundland\
St. John’s, NF
A1B 3V6