REPLICAT.ION 

Posts on replication

Unedited posts from archives of CSG-L (see INTROCSG.NET):

Date: Thu, 29 Sep 1994 14:28:49 EDT

Subject: What are you doing?

############# CHUCK TUCKER 940929 #############

WHAT ARE YOU DOING? (Begun 940927 Concluded 940929)

[RE: Ford 940925. 7:50; Greg 940926; Marken 940926.0830]

I spend a great deal of my time downloading, reading and making copies of the posts on the net. I usually make a copy of two posts per day and give copies to several of my colleagues and some of them I distribute to students in my classes.

I rarely comment to the writers on the net since I consider most of my comments of little interest to the other 130+ folks. On some issues like epistemology, research design, the premises of PCT and previous research I have very little to say that is not being said (or I know will be said!) by Bill, Mary, Tom, Rick or Martin (BMTRM). Since much of the discussion of late has been with folks that are coming on the net to learn about PCT I would rather read how BMTRM deal with their questions and issues. I use their exchanges to learn how I can deal with similar questions and issues in my own dealings with the queries of students and colleagues.

Yesterday, for example, a PH.D. student and I were discussing her comprehensive examination wherein she had questions about the epistemological assumptions of PCT. I did not enter that discussion on the net because this is about the 10th time it has appeared on the net and I have resolved the issue many years ago. So what I did was to give her (after some discussion, of course) three posts by Bill [one which I titled "Perception and Reality: The Color Purple" (940914.1445 MDT)]. I also gave her Bill's post [my title "Self-Esteem: Meaning and THE TEST" (940921.0400MDT)] since she had some questions about research design.

This student and I also had a discussion about a "problem child" that a friend of hers has (in fact she said that her friend calls the child "a problem" rather than use the child's name) and that gave me the chance to talk about Ed's Ford's new book on discipline. She was very impressed with the ideas since she had not thought of PCT as relating to children (this student has never taken a class from me!).

I say all of this to point out that the net is valuable to me even though I don't write on it very much. When I think I have something to write that might be valuable to y'all (although I don't expect that any of you will post me back on the net) I will write it. So here goes.

WHAT'S IN A NAME?

I had a thought after the exchange with the person who questioned the name of the list as Control System Group. In a long but wonderful answer to the query Bill mentioned the Center for the Study of Living Control Systems. I thought why not call the list: Living Control Systems-List (LCS-L) and change the name of Control Systems Group to: The Society for the Study of Living Control Systems (SSLCS). This would all fit with the name of our journal which is (as I understand it): Closed Loop: Journal of Living Control Systems. How about that folks?

REPLICATING PCT RESEARCH

I believe that one good way to learn about the ideas that others have used is to attempt to replicate their research. Most research done in the so-called social and behavioral sciences can not be replicated since the instructions (the section called "Methods") do not describe the activities of the researcher (see Rick's discussion Lord's research for some "evidence" on this point).

I have been critical of Rick, Tom and even Bill because their research lacked such descriptions. So I have asked them to provide me with the programs that they used in their research so I can try follow what few instructions they gave me and do the research "like" they did it. (To be fair I must say that one of Rick's articles did have some adequate instructions and the instructions for the research I will replicate that Tom did are to be found in the MA theses of his students which are very well done - editors, as you know don't like to have all that "junk" in the article about how the research was done!) Rick and Tom have provided programs to me but Bill is still working on a program involving a variety of variables to control a la DEMO1.

I began with Rick's study "The Cause of Control Movements in a Tracking Task" (in MR pp. 61-66) and after several discussions about modifications with Rick we settled on a design which would have two types of disturbances and two levels of difficulty. We developed a short questionnaire to use with each participant (I do not call persons who are in my studies "subjects") and borrowed a MAC and asked students in my classes to participate (without any pay or points - I don't do that either) in the exercises (not study or research). I have had about 20 different persons participate in the exercises to date and still would like to have about five more do so.

At this point in time the data from the participants look exactly like that which Rick collected in the original study. I will report the results after I get the few more subjects and after Rick reviews my draft.

Rick sent me a program for his other study which was never published ["Closed-Loop Behavior: Human Performance as Control of Input" MR pp. 67-75] which I have run many times myself but have not had anyone else do it. I have asked Rick to make some modifications of the program so that I could so some studies with it and I expect Rick to have those changes to me when he can gets away from his book and the net. These studies should be interesting since they will involve participants doing a tracking task with and without disturbances in sequence when they don't know when the disturbances are involved in the program. [NB: Yesterday Rick wrote me and said that he would not get to these modifications until next year so I will proceed with what I have and just replicate his research and discuss its implications for Living Control Systems Theory (oops!)]

I have the program for Bourbon's "social tracking" studies done by his students but have yet to obtain a computer with the proper equipment to do the studies. I have several students that are interested in PCT now and I think they will be working with me on these studies.

STUDIES ON SELF

I have worked with Bill to develop a set of questions trying to examine further how "selves" are disturbed. I have used as my design for these questions the study done by Robertson and Goldstein where they got persons to tell them their "self-identity" and then challenged it and found that all but one person became "upset". Bill and I are trying to do this with "significant others" named by the persons and trying to have them tell us "what disturbs you when X says it". We have found that the questionnaire is quite limited and the next step will be to either interview persons OR set up "role-play situations" wherein disturbances which we know of from the person's previous report will be introduced into the interaction and we will observe the results. I will be working with my class next Spring on this project.

DEMO1 AND GATHERING

I have tried to have these programs (using Dag's disk) put on the fileserver at the university so I can give instructions to my students and have them go to several labs around campus and do some exercises with them. They were working last Spring but changes have been made in the system and they do not work now. Next week I have a new person assigned to me to try to get the problems solved. If I can get this working it will open up the use of these programs for the several hundred students who are in my courses each semester. I hope to be able to put Bill's new program on the filserver and use it for exercises in my courses and hopefully for some research.

STUDIES USING THE TEST

I believe there are more studies using THE TEST than we realize. I think the Milgram studies (including his "crowd" studies), the studies by Sherif (including the autokinetic one), a study by McPhail where the students "walked-out" of his class in the middle of a lecture and he continued to lecture to two student sitting in the front of the room, and the Asch study (with derivations) all used THE TEST. The problem is that they did not collect the data on a person by person basis and thus it is impossible to find out the results of THE TEST. I think most conventional experimental studies use THE TEST when they set up "experimental" and "control" groups (sic) but we don't which "treatment" is the disturbance. But if individual data were available in any of these studies my bet is that they could provide some data for PCT.

I am collecting the data from a set of studies encouraged by Harold Garfinkel (who Rick might have known at UCLA) called "breaching" studies wherein disturbances were purposively set up (like Candid Camera) to find out about a person's "taken for granted world". Erving Goffman also mentioned some of these studies in his works and, in fact, he wrote a number of essays on disturbances (one was on bloopers and what they mean to self-control). I hope that these studies can be used for replications or at least a way to design other studies to use THE TEST at higher levels. (BTW Danny Miller mentioned the Goffman studies in his paper at the 1993 CSG meetings)

SOME MORE ON GATHERINGS

Jogged by a discussion on the net about the "side effects" of individual action using the "ring" formation from the Gathering program as an example Clark and I started to reexamine the simulation for the "ring" and its comparison with our observations of "ring" formation in public gathering places. We concluded that the "ring" in Gathering was not an accurate simulation of our observations of "rings" so I started to modify the program so that it more closely approximated observed "rings". One of the major problems is that no "ring" that we have observed (or others have reported observing) involve 14 single on-their-own self-controlling persons so that the nice "circle" form that Rick calls a side-effect never occurs WITHOUT having 14 persons practice making such a formation over and over again. Also such a formation does not require that the 14 persons "follow" another person as in RING (remember this program was derived from a program devised by Bill called GURU). So what I have done is to make the person into singles, symmetrical and asymmetrical pairs (no triads) and see what they do when they follow another to a certain destination. The observed result (a so-called "side- effect") is not a perfect circle ring formation but a formation that is quite a bit less perfect but it is one much closer to that which we identify as a "ring formation" in public gathering places. So now we are planning to work on the Gathering program some more and write up our results.

While I was modifying the Gathering program I had a discussion with Bob Stewart a friend and colleague of mine for many years and one who uses PCT in his own teaching and research. He wondered if I could devise a program wherein the actors would attempt to achieve two DIFFERENT goals AT THE SAME TIME. I selected the goals for each actor to 1) follow another actor (called "symmetrical withs") AND 2) seek a destination at the SAME TIME. What happened is what we expected from the PCT model: the two actors "freeze" in mid-screen. This can be seen and interpreted as E-motion (the stopping of motion). I plan to work on this some more than perhaps Clark, Bill and I can write up a little paper on "Emotion: A PCT View".

A MODEST SUGGESTION

I really don't plan to say that others should "do what I do" but I do think that some of what I have been doing could be helpful for understanding PCT (or LCST) by others. As Bill has pointed out many times (most recently to Jeff) one has to work with the model to understand it. ONE way of doing this is to get the demonstration disk and use it. Along with this I think that those who create these programs should continue to make them available to others to use in their research. Perhaps there should be much more discussion of our own research on the net (as Ed Ford pointed out) rather than discussing others research (obviously I am guilty of NOT doing this!). Research designs and proposals should be discussed on the net more often (and with more understanding) than in the past. Finally, we should find a way (other than my way of silence) to deal with the issue of epistemology that keeps coming up on the net. The central assumption of PCT is that there may be a "real" world outside of a human organism but it is impossible for an organism to know "it"; "it" (as George Herbert Mead stated many years ago) is simply "there". Yes, it is ALL ALL ALL ALL perception.

Regards, Chuck

Date: Fri, 30 Sep 1994 10:36:51 -0600

Subject: Replication;

[From Bill Powers (940930.0655 MDT)]

Chuck Tucker (940929) RE: replication: pulleys, strings,

weights, and springs

That was an inspiring post, Chuck. I hope you will write some more on the advantages of replication. It's not just fulfilling a formal requirement of science; it's a way of getting to understand what the other guy is talking about, by walking a mile in his shoes. There is nothing that can substitute for the direct experience of the control phenomenon, for seeing how the results just keep repeating and repeating -- and how well a simple theory can predict them.

So here is an ESSAY ON REPLICATION, inspired by you.

In introductory physics, students have to run through a whole series of elementary experiments. They replicate Galileo's experiments by rolling balls down inclined planes and timing pendulums of different lengths. They hook up pulleys and weights and measure how far the load moves when the string at the other end is pulled. They hang weights on springs and measure the stretch of the spring. One after the other, they repeat all the basic old experiments which were once at the cutting edge of physics, done by people who would later have their woodcut portraits reproduced in 20th-century textbooks. The point of all this is not just to salute the past, but to show _why_ the basic physical laws, and the equations that describe them, are considered so important: THEY WORK. A student can use simple algebra to make predictions, do the experiment, and see that the equations predict _what actually happens_. For some students like me, these simple demonstrations open up a new world. They redefine what "understanding" means. They tell us that nature is comprehensible.

I don't think that psychologists and sociologists have ever really believed that human nature is comprehensible. Most of them have never experienced a prediction that actually describes, in advance and accurately, what a human being is going to do. It doesn't seem to bother them that a prediction of behavior may be violated by a large number of individuals in a study; they don't really expect to see predictions work accurately or for everyone. All they hope for is to detect a trend, a suggestion of a relationship, a pale ghost of a natural law that can be seen only in averted vision.

And why are they satisfied with this kind of knowledge? I claim that it is simply because _they have never encountered any other kind_. I think that one reason for Skinner's strong following was that he came up with an experiment in which results were highly predictable: the cumulative record marched up the paper, showing scallops, reversals, recoveries, and other phenomena in a highly predictable way. People could set up a Skinner box, replicate published schedules of reinforcement, and watch the same traces develop. People who got used to seeing such regularity in behavior thenceforth wanted nothing to do with the statistical vagueness of standard psychology. Once you have seen what true regularity looks like, there's no going back. Skinner had everything he needed to put psychology on a new track except a model that could explain this behavior. Neither he nor his followers understood modeling; they did curve-fitting instead.

I think the key to what is wrong in the behavioral sciences is the general lack of replication. In the first place, replicating a typical behavioral experiment would be time-consuming and costly, because in the effort to discover small hints of effects, experimenters must use large numbers of subjects; in trying to get the most out of their efforts, they vary many conditions, hoping that at least one factor will stick up out of the noise far enough to be called significant. In the second place, I don't think that experimenters really believe that if they did try to replicate another person's results, they would succeed. There are too many uncontrolled variables, sampling errors, undocumented details of procedure, differences in available equipment, differences in physical setting, differences in experimenter attitude. Anyone who stops to think of all the variations that could take place in attempting a replication would be discouraged from the start. How could I ever actually _reproduce_ another person's experimental procedure even if I could afford the time and expense? Where could I obtain a comparable population of subjects? And anyway, with the results depending on a slim preponderance of one kind of behavior over another, it doesn't seem likely that I would be lucky enough to have the _same_ relationships just barely reach p < 0.05.

I'm just guessing, of course. Others who know experimental psychology and psychologists better may not agree with me. But the fact remains that there is almost no replication of experimental results in the literature of the behavioral sciences. Experiments are done once, and from then on the statistical findings, if interesting enough in their implications, are taken as facts.

This tells me that behavioral scientists, for the most part, have never had the experience of a truly successful replication. And without that experience, it is impossible to get any sense of reality about the phenomenon revealed by the replication. Knowledge takes on an abstract aura, a sense of floating in intellectual space without an anchor. All things become possible, no fact seems thrust upon one as an inescapable aspect of nature. Understanding nature becomes a matter of plausibility, of possibility, of mathematical and logical manipulations with no ties to observation.

What's important about running the PCT demos is that each demo is in fact a replication of an experiment with human nature. The outcome is predicted, and what is predicted happens every time. The human participants are under no constraints to behave as they do; they hold a control stick in their hands, and are free to move it in any way they please. Yet they move it just as the theory predicts they will move it, creating patterns of movement that could be drawn in advance. These replications reveal the phenomenon of control; it is right there in front of you, and you can make it repeat any time you wish. The whole point is for the person doing the replications to begin to get the sense of comprehending a basic phenomenon of behavior, to realize that there is something about human behavior, even if it is a small and unimportant something, that can be understood just as thoroughly as we understand how pulleys, strings, weights, and springs work together.

There was a time in the history of science when the most important things than anyone could understand about nature were at the level of pulleys, strings, weights, and springs. All the vast array of developments from these beginnings had to wait upon mastery of these simple facts of nature. There was no short-cut -- to take a short-cut, one has to know where one is going. At the time of these first developments, all of physics lay uninvented and unsuspected in an unimaginable future. All that could be done was to keep working with what could be known, to build a base on which more knowledge could be constructed.

There were always those who wanted to jump ahead of this process, to see if they could guess how it would all come out and avoid going through the tedious simple steps. They thought, perhaps, that there would be glory in guessing right long before the plodders of science got there. But in doing so, they misconstrued science itself; what matters is not guessing at the right answer, but developing a method for getting to it that can be applied over and over to develop more right answers, right answers in a superabundance that makes a single lucky guess look trivial. In my opinion, psychology has looked for the lucky guess, the impressive flashy result, the insight that cuts through the details and offers explanations out of the blue. It has neglected the pulleys, strings, weights, and springs, and in doing so has built a house on sand.

While attempting to apply PCT to larger aspects of human behavior is interesting and might in some cases even be useful, I think it is a mistake to give too much importance to that side of our work. There's a great temptation to join all the others who think that they can meet human needs (or make a buck) by talking confidently about things they don't really understand. The true state of affairs is that we are still at the level of pulleys, strings, etc. and have nothing of any permanent value to say about higher levels of organization.

I have no objection to trying to apply the principles of PCT in any way that seems interesting or useful. But I do not choose to be part of that, and I hope that growing numbers of PCTers will choose as I do, to continue to look at basic phenomena and keep working on the knowledge engine that will some day utterly replace everything that people think of today as a science of behavior. Nothing we can say about the higher levels today will be of any important whatsoever in the long run. It will all look like mysticism, superstition, superficiality, and magic, cloaked in the kind of language we have heard before from scholastics, seers, alchemists, and the Priests of Isis. We are at the start of a long task, which will end with a science in which all results are replicable, and replication is taken for granted as a natural requirement of science -- replication within the limits of measurement.

I know that this is possible. I have seen it done, and can do it again on demand. Knowing that, how can I possibly settle for anything less?

Best to all, Bill P.

Date: Tue Mar 07, 1995 12:09 pm PST

Subject: hypostasis hypothesis

[From: Bruce Nevin (Thu 950306 10:46:36 EST)]

I have seen the term "complex environmental variable" or CEV used here, with an image of one projecting the structure of the perceptual hierarchy onto the environment and attributing its properties to the world. Call this the hypostasis hypothesis.

In the discussion of solipsism we see that we have no choice between dealing with our perceptual inputs and dealing directly with the world. Any "dealing with" is by perception.

But as to the structure of the world, we are not limited to the structure of the perceptual hierarchy--meaning here the detailed structure that is projected in CEVs, not the gross structure into levels of perception. We learn, we reorganize, and we can construct theories which, as we know, can run strikingly counter to what seems obvious. (Round earth, quantum physics, control of perceptual input.)

We are able to cover gaps in perceptual input by imagination. We perceive a CEV which we take to be something real in the environment sometimes on the slimmest support by perceptual input from the environment.

Two things now. First, concurrence with others is very persuasive in whether we perceive a particular CEV or some other. The awakening of people by some child crying "the emperor is naked" is really not nearly so common as their somnolent concurrence in how splendidly he is clothed. The unconforming, impolite child is perhaps liable to be punished. I think there is considerable experimental evidence of this kind of collusion in hypostasis. We can probably each think of several thriving, respected, well-funded fields of science that exemplify it. Funding, tenure, respect of one's authority facilitate control of a wondrous range of perceptions, and the reduction of error in the control of those perceptions makes them powerful incentives to collusion and creative ignoring (ignorance).

So it seems that that exploration and testing of one's own perceptions in isolation from the judgments of others is essential for sound science, perhaps even deliberately challenging expectations, what "everybody knows" to be true. This aspect of science is essentially asocial in nature, even antisocial. The more famous episodes in the history of science bear this out.

However -- and this is the second thing -- in science, replication is essential. Constructs must be testable, and anyone, in principle at least, should be able to replicate the tests. What's more, publication is essential. You need someone to say "eureka!" to, and they need to say "yes, something really has been found out here". Science is essentially social in nature. Paraphrasing Berkeley, if a test is made, and no one replicates it, is there a science? Lots of good advances in science are known only to historians because they did not become "part of the science" -- that is, included in the knowledge and assumptions shared by practitioners -- until independently discovered by someone else. The first discoverer(s) failed in the social aspect of science. Presumably other advances are lost entirely.

This is the difficulty we find ourselves in with PCT. It is one thing to say to crowd of fellow onlookers, "That guy over there has no clothes on!" It is quite another to say "None of you has any clothes on. Here--try some of mine for size."

Some folks look and try, but apparently are controlling perceptions of their fellows agreeing with them. One way of reducing error is to ignore perceptions that occasion it. Paraphrasing Duhl about family therapy, we might call this the PCT two-step: get insight, experience error in perceptions of relationships with others who have not got that insight and for whom your behavior under that insight occasions error, abandon insight. (Does this look like three steps? The two steps are "with insight" and "without insight".)

Email, and the annual meeting, and a very small number, slowly growing perhaps, of experimental workers and less formal practitioners, gives us a bit of a scientific community, someone we can say "Eureka!" to, who can reply "yes, you've really found something" or "Well, sort of. How about this instead?" Or perhaps "You said he controlled the knot, instead of saying he controlled a perception of the relationship of the knot to the mark. This shows that you don't really understand PCT yet." (if it's Rick answering ;-). (Actually, I should bite my keyboard, Rick, you've been a model of diplomacy lately instead of demanding a conversion experience like your own as a rite of passage. But I do digress.) All this is good, but at least from time to time it is not enough. That is, it seems to me that the people who really know best how wonderful the thing is that we have here--Bill, Mary, Rick, Tom, Ed, Clark, Chuck, I won't pretend an exhaustive list, those with their hands dirty--it seems to me that each of these people controls a perception of recipients of grant money and teachers in colleges and universities and students and administrators of agencies also learning and applying PCT and participating in graduating it from hypothesis/hypostasis to tested, modelled, verified "fact" at deeper and deeper levels. And controlling such a perception of course occasions error. And chronic error means reorganization.

The outcome of reorganization is (by hypothesis) uncontrolled and unpredictable. Given the social nature of science, we need to keep one another abreast of our reorganizations. We can't let Tom get too depressed, we need him :-)

Bruce

Date: Tue Mar 07, 1995 4:34 pm PST

Subject: Hypostasis hypothesis

[From Bill Powers (950307.1625 MST)]

[Bruce Nevin (Thu 950306 10:46:36 EST)]

This is certainly true, but it's not the whole truth. Don't forget that there is a very large non-living world with which we interact, and the way we perceive CEVs must also be consistent with our interactions with that physical world. Even though most people thought and agreed at one time that the Earth is flat, interactions with the physics of a round world eventually displaced that notion -- in the face of what most people believed. So while science is in large part a matter of consensus, it is more than that, too.

In fact, isn't our problem with conventional science that we have observations that seem to contradict what mainstream science believes? As you say:

Yeah, verily, to both paragraphs. However, if you have found out how to test your own perceptions in isolation, which means confronting your own hopes and desires and understanding the temptations they pose, then it becomes much easier to be confident about replication, and less important to have someone to say "eureka" to -- except for the sake of sharing the joy.

I don't agree that the social side is any more important than the facing-nature-in-isolation side. If you can't do both, you can't be a scientist. If your only concern is the approval of your peers, then you will fudge the results if they don't approve. If your only concern is to explore nature face to face all by yourself, then when you die everything you have learned will die, too.

Given the social _side_ of science. When we're not socializing, however, we have to be doing, which is something each person does alone.

I talked with Tom a few days ago. He seems more determined than depressed.

------------------

Oded Maler (950307)--

Yes. Unfortunately, language reflects the beliefs of the current and immediately past generations. We inherited a language in which behaviorism is firmly entrenched, as well as other points of view. So we can say "You're making me angry," and "I felt his grief," and see nothing wrong.

As to the genetic factors, I quite agree -- I think we are genetically inclined to construct perceptions in the classes intensity, sensations, configuration .... system concepts, although not to construct any particular examples of these classes. On the other hand, I don't rule out anything but inherited perceptions that are peculiar to the world a person happens to be born into.

Best to all, Bill P.