This is a text written by Jon Berg <jon.berg|a|turtlemeat.com> spring 2005 in the Computer Science course Medical Informatics at Tromsø University, Norway.
Current and future possibilities of Medical Informatics
8. Evaluating Medical Informatics Systems
The purpose of an evaluation is to get answers to a question or to help make decisions. Evaluation in medical informatics is done by having a study of the problem. This evaluation should be as objective as possible to have the most credibility. In medical informatics what are studied are often the collection, processing and dissemination of healthcare information these results in Information resources.
The things that can be studied:
Comparison-based approach is the most common objectivist study. In comparison based studies a contrasting conditions is often studied. The goal for this is to relate cause and effect of a phenomenon. The subjects in the study are often assigned to one of the conditions by random. Some variable of interest is measured for each subject, and then the aggregated values are compared across the conditions.
Measurement is the process of assigning a value corresponding to the presence, absence or degree of a particular attribute in a specific object. The measurement process can not be done without error. In most studies it shows that it is difficult to get good measurement. Studies that are undertaken to develop new methods for measurements are called measurement studies. The procedures for measurement developed in measurement studies are the basis for the measurement done in demonstration studies. Demonstration studies are studies that directly address concrete and practical problems.
The challenge when doing experiments is to control the results for changes that are not produced by the information resource. In rare cases there exist predictive models that easily allows for predicting patient outcomes from a small group of clinical findings. The strategies for controlling the study is descriptive (uncontrolled) study, historically controlled experiments, simultaneously nonrandomized controls and simultaneous randomized controls.
In descriptive (uncontrolled) studies the experiment is done and measurements are taken. It is hard to make an interpretation of the values without comparison, since the system has not been changed.
Historically controlled experiments are often called before-after study. The investigator makes a baseline measurement, and then a measurement with a modified system. This gives a value before and after that can be compared, however there is also a chance that many other parameters may have changed between the two samples, for example as new staff coming on.
In simultaneously randomized controls a parallel control group that is not influenced by the stimuli. This is used to capture eventual changes that happen between the two time periods. In this type of experiment there is also a chance that there may be sufficient differences between the groups since they are separated. It could be a different physical location and different staff.
In simultaneously randomized controls the two parallel groups are mixed such that both groups contain a mix of both persons being in the control persons and persons taking the stimuli. Also the personnel working are randomized between the two groups. This method should possible be able to filter out any noise that may happen from other sources than the stimuli to the measured values.
Validation and interference with the results
Important effects that can have an influence on the validity of the results are:
It must be assured that the people in the study do not have any both positive and negative prior feelings or believes about an information resource.
Studies performed in proximity of the environments that have developed them will often result in the subjects having strong positive or negative feelings.
The Hawthorne effect
In essence the Hawthorne effect is the tendency humans have to improve their performance if they know they are being studied.
The checklist effect
The checklist effect is improvement seen when structured input such as paper-based forms or computer forms are used.
The placebo effect
The placebo effect is improvement of some clinical variables such as sleep, well being, and exercise tolerance. This is largely a psychological effect due to persons receiving attention and potentially useful medication. This effect may be stronger than the effect on the actual drugs.
The subjective studies are not less bound to the method, but use different methods for investigating the area. The goal of subjective studies is to illuminate the field rather than to judge it. It can be seen as “the whys according to whom”. The way subjective studies are done in the environment where the informational resource is and collects data through observations, interviews and reviews of documents.
Subjective studies are emergent. In Objective studies the study often starts with a hypothesis that the study is going to build up under. A subjective study often begins with orienting issues that is going to stimulate the beginning of the study. Subjective studies are often incremental, planning is done from day to day and the work plan has a high tolerance for ambiguity. Subjective studies are often natural in the way that they do not want to manipulate the settings of the study.
The data collection methods used in subjective studies are; observation, interviews, document and artifact analysis and other things that may be relevant.
Web (webmastering) Mix Computer related text
Mix Computer related text