Why You Should Be Wary of so-called 'Success Rates' in Therapy (Part 1)

  • Posted on: 25 August 2015
  • By: Dr Emma

Why Be Wary of So-called 'Success Rates'

As you may be aware, I don't publish personal success rates or make statistical claims as to the effectiveness of my therapy in advertising material. You will rarely find a legitimate psychologist, psychotherapists or ethical hypnotherapist who does.

Only occasionally, thankfully, you might come across a hypnotherapy practitioner or clinic claiming so called success rates for their own treatments, or making claims using figures and percentages on their websites or advertisements in an attempt to demonstrate effectiveness of their own therapies.

Personally, I cring when I see this. To me, it can mean only one of two things:

 

1. That practitioner is ignorant as to what constitutes good statistical research. or

2. They are aware that using their figures in this way is highly misleading to the public and against Advertising Standards Authority guidance, but they are willing to do it anyway.

 

Firstly, check out how they have come to such figures

Often these figures that claim to demonstrate effectiveness are cobbled together using

  • statistically unvalidated client surveys or questionnaires before and after treatment.
     
  • without a control treatment group or randomisation process to control for the wealth of ways bias can come into psychological research
     
  • with no mention of the numbers of clients used to come up with these figures (even hair spray ads have to state the number of people they surveyed if they're going to attempt to show effectiveness)
     
  • with little or no mention at all of their scientific method
     
  • often they can involve the therapist contacting ex clients via phone or email months or years after treatment; a wholly unethical practice
     
  • often there are crazy, false comparisons made between data from their figures, and high quality psychological research or other psychological services
     

This stuff is not research or evidence and it does not demonstrate effectiveness; it's persuasion tactics.

Anyone publishing figures and statistics to promote their services, if it’s to be anything other than wholly misleading and meaningless, needs to publish fully and clearly within their advertising, their methodology including numbers of participants, and tick as many of the above boxes as possible.

If they don't do this, you need to be very wary. The Advertising Standards Authority does not support such baseless claims. 

 

Contacting Ex-Clients after Treatment

Did you know it’s not ethical or valid research to just call up ex-clients to see how they got on?

It’s widely known that methods such as this can leave a therapists falsely basking in their own glory, because its well-known ex clients are likely to overestimate their success to their ex therapist on the phone. It’s also not ethical within the client- therapist relationship in any sense, in my view, to intrude into a ex-clients life months or years later, with questions for the therapists own purpose.

It also shows the lack of a poorly planned study. Any client taking part in a well-planned study should be fully aware of being contacted by a third party at a later date.

 

So What is the 'Gold Standard' of Research Evidence

The 'Gold Standard' of clinical or healthcare research is the Randomised Controlled Trial (RCT). These are challenging and difficult to get right. Even the NHS struggles at times to demonstrate effectiveness in psychological therapies of all types. What's the likelihood an individual clinic or practitioner near you is somehow doing it and coming up with figures that are anything other than misleading?

So to produce meaningful figures that are not misleading, research studies must be:
 

1. Large Enough to be statistically meaningful and have enough power to be able to prove whatever is being researched in a statistically significant way. Often the kind of numbers of clients needed for this comes well into the thousands, or tens of thousands or clients. Even with meta-analysis studies, large client numbers are needed.

Think for a moment, how many the average therapist might see in a week, anywhere from 5-15. And how many of those will be for a particular issue, say stop smoking?

Even therapists such as myself, seeing maximum numbers of clients every week, 30 plus, its going to take a very long long time to get adequate numbers for meaning full figures.

How often is an individual clinic or practitioner going to be able to generate any statistically meaningful figures with low-medium numbers of participants? Very rarely.

 

2. Well Designed.

Research studies have to be well designed, ideally with the help of statisticians, to ensure that the study has the power to be able to prove effect and to ensure it is in fact researching what the practitioner is trying to research.

Practitioners are often too close to their own work to always be able to step back and make sure the science is sound.

Often individual practitioner studies in hypnotherapy are poorly designed, in truth, because they don't have the awareness, help, support or funding to design research trials that produce figures and statistics that are meaningful.

 

3. Randomised

This means participants must be randomly allocated to treatment group (hypnotherapy) or control group.

For example, if a study compares stop smoking hypnosis against willpower alone, then the patients should be allocated to either stop smoking hypnosis or to the willpower alone control using randomisation. Participants can't choose the group they go into. As you can imagine, generally hypnotherapy practitioners are not doing studies like this with their clients.

This randomisation process is important as it reduces confounding. What I mean by that is that is equalises out the effect of extraneous factors (independent variables) that may not been accounted for in the experimental design. It makes sure the experimenter has reasonably eliminated any plausible alternative explanations for the results they find. To produce any useful statistics, randomisation is important.

 

4. Controlled.

Surveys or studies showing so called 'success rates', or claiming 'demonstrated effectiveness', need to be controlled.

There are all kinds of forms of bias that come into psychological research, therefore a Control Group is vitally important, to neutralise these biasing effects as far as possible. A Control group is a group of participants who are treated identically at every part of the study, except that they receive a different treatment other than the one being investigated. For example, in trial looking at a smoking cessation hypnosis programme, the control group must be treated identically, other than they receive another treatment other than hypnosis, such as relaxation.

As you can imagine, Randomised Controlled Trials are very challenging to undertake and there are often ethical challenges, however, they remain the ideal method by which all healthcare interventions should be evaluated.

 

Watch out for Part 2 of this Blog Why You Should Be Wary of so-called 'Success Rates' in Therapy, examining the kinds of figures and statistics therapists can ethically use in their marketing.

 

Dr Emma Rae Rhead. Providing Ethical Hypnotherapy, Hypno-Psychotherapy and Medical Hypnosis you can trust in Chester, Cheshire. You can contact Emma on 01244 470181 or by email at emma@chesterhypnotherapy.co.uk.