(Time for another story...)
Once upon a time, I attended a convention. At this convention, I attended a talk. Prior to the talk, I conversed with two individuals sitting next to me. They asked what I did, I told them about what I was studying and, like clockwork, their eyebrows went up in the fashion of skeptics everywhere. "A demonstration is in order," thought I.
There were roughly 100 people attending this talk, and two fairly substantial and desirable door prizes were to be given away before the talk ended. Skipping to the end of the story... Yes, I walked away with one of the door prizes. The interesting part of this story is HOW I plotted a trajectory through Smearland that led a universe where I won one of the door prizes.
To be a truly impressive feat in keeping with my claims, the door prizes would have to be given away in some random fashion. (As opposed to as prizes for a competition which involved knowledge or skill.) Task #1 - Make sure we end up in the universe where the prizes are given away via some random or quasi-random system. (Done.)
Task #2 involved the framework problem. The gist of my problem was this... Under these circumstances, it would have been more difficult for me to navigate to the winning universe if only one critical observation were involved. Ex: If every person had a unique number, all numbers were put into a bin, and two were selected at random. Task #2 - Secure a more favorable framework of observations. (Done. I found the universe where a subgroup of the audience was selected, and then from the subgroup, a winner. Much easier course to navigate for reasons we will discuss later.)
Task #3 - Find the universe where I win one of the door prizes. (Done.)
Task #4 should have been to ensure that I also landed in a universe where the individuals in question put two and two together and concluded that I did know what I was talking about. Alas, I failed in this task. Probably because I was too preoccupied with winning, and I assumed (fatal error) that recognition would follow.
The moral of this story is that each task involves its own critical observations, and each observation was critical in determining the nature and success of the next task. This is the nature of navigating through Smearland. You need to be aware of what other possible outcomes might arise before they occur. You also need to be aware of how one result will impact your ability to see the next set of outcomes and to generate the forces necessary to select that outcome at will.
What would have happened if I had not been successful in Task #2? Of if I had not thought to find a universe other than the obvious one where the two winners were drawn from a pool of all audience members at random? Certainly, if I had won under those circumstance, my skeptic friends may have remembered our conversation and been a little impressed, but the point is, one way or another, I decided to find a universe where I won, and I did. :) Shhh... don't tell anyone.
Friday, August 03, 2007
Wednesday, August 01, 2007
Journal Club #4
I've talked a bit about expectations. So, what happens when you try to manipulate expectations within the context of a psi experiment?
Pitman, J. A., The effect of manipulating expectations both before and during a test of ESP. Journal of Parapsychology, Spring 2004. (here)
This paper begins with a very general overview of expectations as they may relate to psi performance. The thing to note here is that 'expectations' are never precisely defined in terms of their components - mental imagery, representations of time, degree and nature of emotional content, changes in expectations as testing progresses, etc. It's assumed that everyone has the same definition of 'expectation' and that 'expectation' means the same thing in every study that is discussed. This means that 'expectations' are not being manipulated with a level of precision that will easily lead to future testable hypotheses.
Pitman presents two experiments which have basically the same design. The second experiment corrects subtle flaws in 'random' presentation and increases the number of subjects, as compared to the first experiment. Pitman manipulates expectation in two ways - by artificially inflating (or not) the degree of positive feedback the subjects are given about their ESP 'guessing' trials, and by giving them a placebo (or not) that is presented as a substance that enhances ESP. We are largely left to speculate as to what the actual effects of these manipulations might be upon expectations for success. It is assumed that subjects will have greater expectations of success when given the 'ESP-enhancing' placebo and when given artificially inflated positive ('false') feedback about their success. But again, the nature of these expectations is never elaborated upon.
In both experiments, Pitman found a significant interaction between the placebo manipulation and the feedback manipulation.
"When participants were given false feedback, there was a placebo effect... However, when participants were given true feedback, there was no placebo effect... Further, when participants were given a placebo, there was an effect of feedback... However, when participants were not given a placebo, there was no effect of feedback." (Results - Experiment 1; confirmed in Experiment 2)
In other words, unless your expectations were (theoretically) artificially inflated in two different ways at the same time, your performance was no different from that of everyone else in the study.
Pitman acknowledges that this is a problematic result. "Without a significant effect for the placebo manipulation in increasing predicted ESP performance, explanation of the additive performance for placebo and falsely positive feedback is challenging." (Her next sentence brings to light an additional problem with this study - "One must conclude that all participants started the ESP test with similar levels of expectancy...". Yeah, that's a problem for many reasons.)
I think it's a fascinating result. Due to the imprecise definition of 'expectation' and the vague manipulations thereof in this experiment (and pretty much all similar studies), any conclusion that I would attempt to draw from the data would be highly speculative. But go ahead and twist my arm... okay, okay, I'll speculate.
The first thing that comes to my mind is that, with two sources for expectations of success, participants may have been able to shift their attention between two significantly different frameworks. For example, when I am told that I got the answer wrong, I may be able to shift my attention and draw upon my expectations about the placebo, and it may take several trials worth of false positive feedback for my expectations to again center on the outcome/feedback. With only a single source of expectation for success, my expectations will probably begin to be altered in the subtle ways that lead to a pull towards the opposite outcome. (I tried some of this out on slot machines in Las Vegas. I could only be successful for so long before my expectations, by virtue of simple exposure to success, had qualitatively changed to the point where, without a much greater effort, I was pulled towards an unsuccessful outcome.)
Before I digress too far, I should point out that other studies have shown effects of a single manipulation of expectation. Pitman points out that J.B. Rhine was successful in getting a single main effect of placebo upon ESP performance. Other studies have also attempted to manipulate the participants' beliefs about ESP in an (sometimes successful) attempt to improve their performance. So further speculation about the results of this study would probably only be productive if I were able to include the other studies for comparison purposes.
Subjects in this study were also asked to predict their ESP scores prior to testing, but it appears that these predictions were collected after the placebo manipulation. Of course it's possible that asking the subjects to quantify expectations had an effect in and of itself, but there are not enough details about when and how this data was collected to allow for useful speculation.
The author attempts to make further inferences about 'expectation' as it relates to the subjects' scores on the 'sheep-goat' scale of psi belief, but these are not the expectations we are interested in, and therefore they will not be discussed here. The expectations that occur at the time of testing are, IMHO, the ones most likely to influence the outcome. Their nature most closely matches the representations of the outcome and success or failure, therefore these are the expectations that I think need to be more carefully teased apart and manipulated in future testing.
Kudos to Pitman for a simple, elegant experimental design, and a fascinating finding!
Pitman, J. A., The effect of manipulating expectations both before and during a test of ESP. Journal of Parapsychology, Spring 2004. (here)
This paper begins with a very general overview of expectations as they may relate to psi performance. The thing to note here is that 'expectations' are never precisely defined in terms of their components - mental imagery, representations of time, degree and nature of emotional content, changes in expectations as testing progresses, etc. It's assumed that everyone has the same definition of 'expectation' and that 'expectation' means the same thing in every study that is discussed. This means that 'expectations' are not being manipulated with a level of precision that will easily lead to future testable hypotheses.
Pitman presents two experiments which have basically the same design. The second experiment corrects subtle flaws in 'random' presentation and increases the number of subjects, as compared to the first experiment. Pitman manipulates expectation in two ways - by artificially inflating (or not) the degree of positive feedback the subjects are given about their ESP 'guessing' trials, and by giving them a placebo (or not) that is presented as a substance that enhances ESP. We are largely left to speculate as to what the actual effects of these manipulations might be upon expectations for success. It is assumed that subjects will have greater expectations of success when given the 'ESP-enhancing' placebo and when given artificially inflated positive ('false') feedback about their success. But again, the nature of these expectations is never elaborated upon.
In both experiments, Pitman found a significant interaction between the placebo manipulation and the feedback manipulation.
"When participants were given false feedback, there was a placebo effect... However, when participants were given true feedback, there was no placebo effect... Further, when participants were given a placebo, there was an effect of feedback... However, when participants were not given a placebo, there was no effect of feedback." (Results - Experiment 1; confirmed in Experiment 2)
In other words, unless your expectations were (theoretically) artificially inflated in two different ways at the same time, your performance was no different from that of everyone else in the study.
Pitman acknowledges that this is a problematic result. "Without a significant effect for the placebo manipulation in increasing predicted ESP performance, explanation of the additive performance for placebo and falsely positive feedback is challenging." (Her next sentence brings to light an additional problem with this study - "One must conclude that all participants started the ESP test with similar levels of expectancy...". Yeah, that's a problem for many reasons.)
I think it's a fascinating result. Due to the imprecise definition of 'expectation' and the vague manipulations thereof in this experiment (and pretty much all similar studies), any conclusion that I would attempt to draw from the data would be highly speculative. But go ahead and twist my arm... okay, okay, I'll speculate.
The first thing that comes to my mind is that, with two sources for expectations of success, participants may have been able to shift their attention between two significantly different frameworks. For example, when I am told that I got the answer wrong, I may be able to shift my attention and draw upon my expectations about the placebo, and it may take several trials worth of false positive feedback for my expectations to again center on the outcome/feedback. With only a single source of expectation for success, my expectations will probably begin to be altered in the subtle ways that lead to a pull towards the opposite outcome. (I tried some of this out on slot machines in Las Vegas. I could only be successful for so long before my expectations, by virtue of simple exposure to success, had qualitatively changed to the point where, without a much greater effort, I was pulled towards an unsuccessful outcome.)
Before I digress too far, I should point out that other studies have shown effects of a single manipulation of expectation. Pitman points out that J.B. Rhine was successful in getting a single main effect of placebo upon ESP performance. Other studies have also attempted to manipulate the participants' beliefs about ESP in an (sometimes successful) attempt to improve their performance. So further speculation about the results of this study would probably only be productive if I were able to include the other studies for comparison purposes.
Subjects in this study were also asked to predict their ESP scores prior to testing, but it appears that these predictions were collected after the placebo manipulation. Of course it's possible that asking the subjects to quantify expectations had an effect in and of itself, but there are not enough details about when and how this data was collected to allow for useful speculation.
The author attempts to make further inferences about 'expectation' as it relates to the subjects' scores on the 'sheep-goat' scale of psi belief, but these are not the expectations we are interested in, and therefore they will not be discussed here. The expectations that occur at the time of testing are, IMHO, the ones most likely to influence the outcome. Their nature most closely matches the representations of the outcome and success or failure, therefore these are the expectations that I think need to be more carefully teased apart and manipulated in future testing.
Kudos to Pitman for a simple, elegant experimental design, and a fascinating finding!
Subscribe to:
Posts (Atom)