Consulting Services

An evaluation of the non-clinical aspects of the South Australian Mental Health Services Telemedicine Pilot Project June - October 1994

John Mitchell, BA (Hons), Dip Ed, M Ed Admin, AFAIM Benjamin Mitchell, BA (Psych.), December 1994

1. Project brief and methodology

Background

A pilot project has been operating in South Australia since mid-1991, aimed at assessing the ability of a wide range of clinicians and other health workers to conduct diagnostic, therapeutic and teaching and supervision sessions at remote sites in a clinically and economically effective manner. The project was funded $250,000 by the Health Communication Network (HCN) as one of a number of demonstration projects to evaluate the benefits and uses of telecommunications in health care. The HCN grant enabled the South Australian Mental Health Service to extend the project to the following sites in 1994: Berri, Mount Gambier and Glenside Hospital.

The application areas for the pilot were as follows:

• Tele-psychiatry: professional psychiatric support for clinical services delivered over the tele-mental health link.
• Mental health team training and development: specific training and skill enhancement for mental health practitioners- viz. mental health nurses, psychologists, allied health workers and social workers currently located in rural areas.
• Video-conferencing: for case management reviews, problem solving, etc

The objectives of the project and the foci of the two evaluators, Dr Michael Baigent (Dr MB) and John Mitchell (JM), were as follows:

1. to design an evaluation process to assess the impact of the tele- mental health system on performance indicators (see 4 below) (Dr MB; JM)
2. to identify clinical consultancy correlations between tele-mental health and face-to-face visits (Dr MB)
3. to implement and co-ordinate the evaluation process for
   • clinical applications (Dr MB)
   • administration and education (JM)
4. to analyse the following performance indicators:
   1. technical-e.g. call completion rates and call connect time (JM)
   2. economic, to measure cost effectiveness e.g. variation in frequency of patient travel to Adelaide, variation in frequency of hospital visits by Visiting Medical Officers, variation in staff travel (JM)
   3. usage and acceptance of the system(Dr MB)- e.g. number of user groups, types of user groups, frequency of use, hours in operation (JM)
   4. productivity gains resulting from the freeing of time by the users (JM)
   5. service delivery- e.g. care delivery (JM)
   6. education and training- e.g. number of skill enhancement sessions, number of participants in education and training sessions (JM)

The Steering Committee for the Project consists of Ross Hummerstone, (South Australian Health Commission-SAHC), Robin Michael (SAHC), Jennifer Bowers (SAMHS), Len Payne (SAHC), Jon Blackwell (SAHC), Peter Yellowlees (SAMHS), Dr Andy Czechowicz (SAMHS), Dr John Wallace (Rural Doctors' Association), Bill McCoy (Foundation SA), Ian Ross (Commonwealth Health Commission, CHN) and Antonia Lehn (HCN).

The Project Co-ordinator is Steve Kavanagh and clerical and administrative assistance is provided by the following personnel: Tina Bull (Glenside Hospital), Bronwyn Smith (Berri Hospital) and Joan Ellett (Mt Gambier).

Both qualitative and quantitative research techniques were used in the evaluation of the project. Qualitative techniques included structured and unstructured interviews, observations, verbal reports and group discussions. Quantitative techniques included a questionnaire which provided statistical information and collection of statistical records of the log of uses.

The use of qualitative techniques is a recognition that there are many factors which will affect the processes in the tele-psychiatry environment, for example, the experience or inexperience of the user. Research shows that there are many social, cultural and other factors which will affect users of new technologies.

While qualitative evaluation is concerned with process, quantitative evaluation is concerned with measurement. Quantitative results from the User Survey provided useful statistical information which will be valuable for the Steering Committee in analysing such topics as the frequency and duration of uses; the number and backgrounds of users; and the ratings that users attribute to various aspects of the medium.

The major evaluation activities occurred at the following times:

• June: participation in project planning with SAMHS personnel and other evaluators
• June: production of a project Gantt chart
• July: development of a framework for the research, based on a review of the international literature on videoconferencing and telemedicine, in relation to the performance measures set out in the Project Specification
• July: production of an interview schedule and survey instrument
• July-September: field research, including interviews, surveys, observations and collection of statistical data
• September-October: analysis of the data
• October: production of the written report.