Is the trial valid?
Is low-energy laser an effective treatment for lateral epicondylitis? Do stretching programs prevent the development of contracture following stroke? Can the use of the flutter valve reduce postoperative respiratory complications? Rigorous answers to these questions can only be provided by properly designed, properly implemented clinical trials. Unfortunately the literature contains both well performed trials which draw valid conclusions and badly performed trials which draw invalid conclusions. The reader must be able to distinguish between the two. This tutorial describes key features of clinical trials (or “methodological filters”) which confer validity.
Some studies which purport to determine the effectiveness of physiotherapy treatments simply assemble a group of subjects with a particular condition and take measures of the severity of the condition before and after treatment. If subjects improve over the period of treatment, the treatment is said to have been effective. Studies which employ these methods rarely provide satisfactory evidence of treatment effectiveness because it is rarely certain that the observed improvements were due to treatment, and not to extraneous variables such as natural recovery, statistical regression (a statistical phenomena whereby people become less “extreme” over time simply as a result of the variability in their condition), placebo effects, or the “Hawthorne” effect (where subjects report improvements because they think this is what the investigator wants to hear). The only satisfactory way to deal with these threats to the validity of a study is to have a control group. Then a comparison is made between the outcomes of subjects who received the treatment and subjects who did not receive the treatment.
The logic of controlled studies is that, on average, extraneous variables should act to the same degree on both treatment and control groups, so that any difference between groups at the end of the experiment should be due to treatment. By way of example, it is widely known that most cases of acute low back pain resolve spontaneously and rapidly, even in the absence of any treatment, so simply showing that subjects improved with a course of a treatment would not constitute evidence of treatment effectiveness. A controlled trial which showed that treated subjects fared better than control subjects would constitute stronger evidence that the improvement was due to treatment, because natural recovery should have occurred in both treatment and control groups. The observation that treated subjects fared better than control subjects suggests that something more than natural recovery was making subjects better. Note that, in a controlled study, the “control” group need not receive no treatment. Often, in controlled trials, the comparison is between a control group which receives conventional therapy and an experimental group which receives conventional therapy plus treatment. Alternatively, some trials compare a control group which receives conventional treatment with an experimental group that receives a new therapy.
Importantly, control groups only provide protection against the confounding effects of extraneous variables in so far as treatment and control groups are alike. Only when treatment and control groups are the same in every respect that determines outcome (other than whether or not they get treated) can the experimenter be certain that differences between groups at the end of the trial are due to treatment. In practice this is achieved by randomly allocating the pool of available subjects to treatment and control groups. This ensures that extraneous factors such as the extent of natural recovery have about the same effect in treatment and control groups. In fact, when subjects are randomly allocated to groups, differences between treatment and control groups can only be due to treatment or chance, and it is possible to rule out chance if the differences are large enough – this is what statistical tests do. Note that this is the only way to ensure the comparability of treatment and control groups. There is no truly satisfactory alternative to random allocation.
Even when subjects are randomly allocated to groups, it is necessary to ensure that the effect (or lack of effect) of treatment is not distorted by “observer bias”. This refers to the possibility that the investigator’s belief in the effectiveness of a treatment may subconsciously distort the measurement of treatment outcome. The best protection is provided by “blinding” the observer – making sure that the person who measures outcomes does not know if the subject did or did not receive the treatment. It is generally desirable that patient and therapists are also blinded. When patients have been blinded, you can know that the apparent effect of therapy was not produced by placebo or Hawthorne effects. Blinding therapists to the therapy they are applying is often difficult or impossible, but in those studies where therapists are blind to the therapy (as, for example, in trials of low-energy laser where the device emits either laser or coloured light, but the therapist is not informed which), you can know that the effects of therapy were not produced by the therapist’s enthusiasm with the therapy, rather than by the therapy itself.
It is also important that few subjects discontinue participation (”drop-out”) during the course of the trial. This is because dropouts can seriously distort the study’s findings. A true treatment effect might be disguised if control subjects whose condition worsened over the period of the study left the study to seek treatment, as this would make the control group’s average outcome look better than it actually was. Conversely, if treatment caused some subjects’ condition to worsen and those subjects left the study, the treatment would look more effective than it actually was. For this reason dropouts always introduce uncertainty into the validity of a clinical trial. Of course the more dropouts, the greater the uncertainty – a rough rule of thumb is that if more than 15% of subjects drop out of a study, the study is potentially seriously flawed. Some authors simply do not report the number of dropouts. In keeping with the established scientific principal of guilty until proven innocent, these studies ought to be considered to be potentially invalid.
To summarise, valid clinical trials:
- randomly allocate subjects to treatment and control groups
- blind observers, and preferably patients and therapists as well
- have few dropouts.
The next time you read a clinical trial of a physiotherapy treatment, ask yourself if the trial has these features. As a general rule, those trials which do not satisfy these criteria could be invalid and should not be considered to constitute strong evidence of treatment effectiveness (or ineffectiveness). Those trials which do satisfy these criteria should be read carefully and their findings should be committed to memory!
If you want to read further about assessing trial validity, try:
Guyatt GH, Sackett DL, Cook DJ (1993). User’s guide to the medical literature: II. How to use an article about therapy or prevention: A. Are the results of this study valid? JAMA 270:2598-2601.
