Every day, innumeracy affects 8 out of 5 people.

Chapter 5
    Sec 5.2

A study where we actually DO something to people, animals, or objects in order to observe the response is an EXPERIMENT.  The individuals on which the experiment is done are the experimental units.  When the units are human beings, they are called subjects.  A specific experimental condition applied to the units is called a treatment.

The distinction between explanatory and response variables is important!!!  The explanatory variables in an experiment are called factors.  Many experiments study the joint effects of several factors.  Each treatment is formed by combining a specific value , called a level, of each of the factors. 

In principle, experiments can give  GOOD evidence for causation while a census does a good job of describing differences between two study groups but can say nothing about cause and effect.  Experiments also allow us to study the specific factors we are interested in, while controlling the effects of lurking variables.  Using experiments we can also study the combined effects of several factors.

Comparative experiments are usually performed in a laboratory which is a controlled environment to protect against lurking variables.  The general design in a flow chart is:

units ---  treatment  ---  observe response

With comparative experiments we need to guard against the PLACEBO EFFECT.  Especially in medical experiments where there are two groups, one treated with medication and one treated with a placebo (dummy treatment), many subjects respond favorably even to a placebo.

To limit the effect of the placebo response and also the confounding of variables, we can design a study comparing two groups who are treated the same in every way which restricts the differences between the groups to the effect of the treatment (response).  The group of patients who receives a sham treatment is called a control group, because it enables us to control the effects of outside variables on the outcome.  CONTROL is the FIRST BASIC PRINCIPLE OF STATISTICAL DESIGN OF EXPERIMENTS.  Comparison of several treatments in the same environment is the simplest form of control.

Randomization of subjects in comparative experiments is the SECOND RULE used to assign the experimental units to the treatments.  Systematic differences among the groups of experimental units in a comparative experiment cause BIAS.  How can we assign experimental units to treatments in a way that is fair to all of the treatments???  The statistician's remedy is to rely on CHANCE to make an assignment that does not depend on any characteristic of the experimental units and that does not rely on the judgment of the experimenter in any way.  The use of chance can be combined with matching, but the simplest design creates groups by chance alone.  See Ex. 5.12 (page 295) for an example.  Randomization, the use of chance to divide experimental units into groups, is an essential (THIRD RULE) ingredient for a good experimental design.  When treating two groups, the size of the groups should be approximately equal.  To avoid making quick conclusions about cause and effect from a single study, it is necessary to repeat the experiment multiple times so that the possibility that chance has caused the response will average out.  
Repeating, the 3 principles of good experimental design are: 
1)  control the lurking variables, usually by comparing 2 or more treatments
2) randomize the assignments of treatments to experimental units
3) replicate (repeat) the treatment on many units to reduce chance variation in the results.

An observed effect so LARGE that it would rarely occur by chance is called statistically significant by using the laws of probability.  You will often see the "statistically significant" in reports of investigations in many fields of study.  When all experimental units are allocated at random among all treatments, the experimental design is completely randomized.  The randomized comparative experiment, because of its ability to give convincing evidence for causation, is one of the most important ideas in statistics.

Cautions about experimentation...
Good experiments require careful attention to details.  The double-blind method avoids unconscious bias.  In a double-blind experiment, neither the subjects nor the people who have contact with them know which treatment a subject received.  The most serious potential weakness of experiments is lack of realism.  Many behavioral science experiments use subjects that know they are subjects...that is not realistic.   

Sometimes a "matched pairs" design is used to compare subject preference between two objects.  The two objects taken together are called a block.  Arrangement of the treatments within the block must be fair and random.  Tossing a coin, which doesn't seem too scientific, is a great way to accomplish this.  The order that the treatments are given can also influence the subject response, so order must be randomized also, again by a coin toss.  The matched pairs design uses comparison of treatments, randomization, and replication on several experimental units.  A block design is the random assignment of units to treatments within each block.  A block is a group of experimental units or subjects that are known to be similar in some way that is expected to affect the response to the treatments.  Blocks are a form of CONTROL.  They control the effects of some outside variables by bringing those variables into the experiment to form the blocks.  Blocks allow us to draw separate conclusions about each block.  Blocking allows more precise overall conclusions.  The idea of blocking is an important additional principle of statistical design of experiments.  A wise experimenter will form blocks based on the most important unavoidable sources of variability among the experimental units.  Randomization will average out the effects of the remaining variation and allow an unbiased comparison of the treatments.

Index