April 20, 1991

Bill Curtiss

Sierra Club Legal Defense Fund

2044 Fillmore Street

San Francisco, California 94115

Re: MTC's Conformity Assessment Procedures -- The Pseudo-science of Transportation Modelling

Dear Bill:

First my qualifications to make these judgments: I have a B.A. Magna cum Laude from the University of California in mathematics, With Special Distinction in Mathematics. As a Junior at U.C. Berkeley, I ranked 37 1/2th out of 1300 college math students in the nation in the annual mathematics contest sponsored by the Mathematical Association of America. I have an M.A. in mathematics (including study in statistics) from Harvard University. And my Ph.D. from the University of California at Los Angeles is in psychology, concentrating in psychometrics. Psychometrics is the science of the measurement of human behavior and traits, and forms the scientific basis upon which transportation modelling and all other forms of human measurement rest. I taught measurement theory -- specifically, reliability and validity -- at California State University, San Francisco. I have been a computer programmer for 29 years. I taught computer science for U.C. Berkeley Extension. In other words, I am an expert in mathematics, statistics, scientific method, measurement science (including modeling), and computer science.

Modeling is really a very simple process, when the modeler is not trying to make it mysterious. A scientific principle is expressed in the form of a mathematical formula. Then data are substituted for the variables in the formula, allowing a result to be computed (e.g. emissions of CO, from vehicle type, speed, temperature, etc.). When the formula is in dispute, statistics must be used to determine if it does what its users want it to do. The relevant factors are reliability (giving repeatable results) and validity (measuring what it is supposed to be measuring). Both are measured using correlations, and only qualify the model to be utilized in situations similar to those in which it was validated, if at all. For example, an intelligence test that was validated only on white, middle class Americans could be expected to give meaningful results only when used with such subjects.

If every measure must be validated by comparing it with "the real thing", one might ask why measures and models are used at all -- why not just use the "real thing"? The answer is simply practicality: the test is relatively quick and easy to administer, whereas rigorous scientific research is very slow and expensive. A yardstick is available in any hardware store, but a highly accurate scientific instrument is unwieldy and extremely expensive. However, one must never forget that the reliability and validity of the measure or model is strictly limited; a judgment of reliability and validity doesn't confer any magical ability to predict accurately in all situations, nor any special consonance of the formula used with the forces that guide the universe!

Even in physics, the "hardest" of the sciences, reliability and validity are limited. Newtonian physics may be adequate to predict events on Earth, but fails utterly when applied to the behavior of the stars or the nucleus of the atom. There, the more accurate formulas of Einstein's Relativity must be used. When we come to predicting human behavior, both reliability and validity tend to be so low that accurate predictions are impossible. In other words, the probability that your conclusion is correct would be very close to 0.5. It could not be relied upon. And where the stakes are as high as they are with highway construction (air pollution and massive environmental destruction on one hand, loss of millions of dollars of federal and state subsidies on the other), transportation models are far too unreliable a tool.

MTC and its consultants violate all of the rules of measurement science in attempting to predict emissions changes due to highway expansion. The documentation for both MTCFCAST and STEP reveals no evidence that reliability was ever measured. It makes vague, non-quantified claims of validity, but shows no evidence that the concept of validity was even understood. After running the model, the results were compared with one set of data. Then in a process they called "calibration", they modified the coefficients of the formula to make the model conform to that set of data. They imply that this process makes the model valid. Actually, all that it does is to make the formula "predict" one set of data. If it were to be applied to another set of data, or if a different factor were to be "predicted" using it, even if it were applied to a similar set of data, there is absolutely no guarantee that it would continue to predict accurately. In other words, this process does not result in a valid model. It merely conforms the data to the model. As an analogy, it is as if MTC had a ruler made of putty, and stretched or shrank it in different situations to make it register in a way convenient to the situation. If the formula is wrong, changing its coefficients won't help. An entirely different formula may be required!

All of the models use a standard formula that they call a "logit model equation". An example is P(q,i-j) = exp(Uj) / SUM(exp(Uk)) (k=1 to j). Here "exp" means e raised to a certain power, where e is the base of the natural logarithms. Out of the billions of possible formulas that could be used in the model, there is absolutely nothing special about this one, that qualifies it to be used in transportation modeling! The probability that it is the best formula to use is practically zero. The fact that it has been used by others has nothing whatever to do with whether it is valid.

In short, it is extremely unlikely that MTC's models have sufficient validity even to predict in situations similar to ones used in the past. And even if they had some reliability and validity in such situations, the probability that the models would continue to work in new situations (e.g. the collapse of a segment of a freeway, or the expansion of a freeway not studied before) is vanishingly small. Stated more simply, "Garbage In -- Garbage Out" (GIGO). Saying that they are "state of the art" with regard to transportation and air quality modeling merely compares garbage with garbage. Transportation modelers are not known for their impartiality, nor for their sophistication with regard to statistics or measurement science.

So how, then, are we to predict the effects of freeway expansion, if all current models are worthless? We have to fall back on basic scientific research, which is not easy or cheap, but which is the only way we have of reliably answering such questions. Agencies or researchers that receive funding for supporting the building of freeways have little motivation to develop accurate models, when they have models that give them the conclusions that they wish (basically, an extension of the status quo). They have even less interest in funding or conducting honest, unbiased research on this question. It can only be accomplished by scientists who have not been "bought" by highway interests.

On the other hand, why do any research? Isn't it obvious that expanding highways can only encourage prople to drive more, and hence worsen air quality?! And if, as required by the greenhouse effect, we must decrease traffic by 50% below current levels, isn't it obvious that we won't need all that extra pavement?! Out of the mouths of babes...

Sincerely,

Michael J. Vandeman, Ph.D.