Anonymous public comment

Following a modest review effort, I would like to offer these comments: 1. The document fails to acknowledge much prior literature that is germane and of value to environmental modelers. Entire books and volumes of journals that support the art of environmental modeling and simulation have evidently been overlooked. 2. In the Executive Summary, the definition of a model encompasses "physical, biological, economic, or social" systems, but omits chemical systems. Especially where the transport and fate of pollutants is concerned, the modeling of chemical interactions can be of fundamental importance. 3. The Executive Summary makes no reference to computer programming, which can be a substantial part of any computational model development effort and requires great care to do well. The omission of any mention of computer programming under the model development heading here is inconsistent with the recognition of "encoding" (on page 6) explicitly as a model development activity. 4. The Executive Summary claims, "Model quality is an attribute that is meaningful only within the context of a specific model application." To the contrary, a model can be poor apart from a specific model application if the model framework contains errors in logic or coding, or if fundamental laws of science have been neglected where they are necessary to arrive at a sensible output of any sort. 5. Page 3 mentions a "Models Knowledge Base" that "provides convenient access to standardized documentation of the models’ development, scientific basis, user requirements, evaluation studies and application examples." Is that a reference to the "Models Knowledge Base" at http://cfpub.epa.gov/crem/knowledge_base/knowbase.cfm? If so, those web pages do not appear to provide any such documentation. 6. Redundant text appears in a few places: -- A statement that appears on page 10 reads, "A model framework is a formal mathematical specification of the concepts and procedures of the conceptual model, usually translated into computer software." A statement on page 12 reads, "The model framework is a formal mathematical specification of the concepts and procedures of the conceptual model." -- Page 8 says: "Decide on the type of model or models to use: Discuss alternatives and compare different types of models, e.g. empirical vs. mechanistic, static vs. dynamic, simulation vs. optimization, deterministic vs. stochastic, lumped vs. distributed [sic]" Page 10 says: "Some of the choices among model types include: empirical vs. mechanistic, static vs. dynamic, simulation vs. optimization, deterministic vs. stochastic, and lumped vs. distributed models." -- The following lengthy text is duplicated verbatim on page 8 and page 9: "[...] identification of the environmental domain being modeled; specification of transport and transformation processes within that domain that are relevant to the policy/management/research objectives; specification of important time and space scales inherent in transport and transformation processes within that domain in comparison with the time and space scales of the problem objectives; and any peculiar conditions of the domain that will affect model selection or new model construction." 7. The structure of the document and the concepts presented is often inconsistent: -- Page 2 says, "Chapter 2 discusses the role of models in environmental decision making. Chapters 3 and 4 provide guidance on elements of model development and evaluation, respectively." That is not consistent with the actual contents of Chapter 2, in which Box 2 clearly presents the "Steps in the Modeling Development Process". -- Immediately following Box 2 is Figure 1, the right-hand side of which deviates inexplicability from the outline given in Box 2. -- Under Section 3.2, Problem Specification, conceptual model development is mentioned. But in both Box 2 and Figure 1, conceptual model development is presented as distinct from problem specification. -- Earlier in the document, the Executive Summary placed "identify the environmental issue" -- presumably identical to identifying the problem -- under the heading of Model Development. Identifying the problem is best thought of as separate from developing a model to address it as the guidance document itself indicates in subsequent sections. The document appears to be structuring the same fundamental concepts in at least 4 different ways within the space of about 20 pages. That makes it difficult to comprehend precisely what relationships among these concepts the authors intend to convey. 8. Please provide examples of actual statements representing Problem Identification and Problem Specification for a specific type of model application. 9. While "problem specification" has its own section (3.2), it is also mentioned as part of a "first stage" under the "Problem Identification" heading (3.1), so it is not clear exactly how the two concepts are to be distinguished. 10. After the first few "Problem Specification" bullets, a sentence begins, "Once the need for a model has been identified...." That step is out of sequence as presented in the text, since it would not be reasonable to "decide on the type of model or models to use" (the bullet above in the text) until you have determined that one is needed at all. 11. The caption to Figure 3 says, "An optimal level of complexity exists for every model, which is represented in the figure as the “ 'point of minimum uncertainty.' " -- A discussion of the phenomenon of overparameterization -- how it can be recognized and addressed -- would be appropriate here. The issue concerns not only sensitivity of individual parameters to the model output, but the possibility that different combinations of parameter values may result in the same goodness of fit to the data. -- A rational approach to model development would not seek to "optimize" complexity, but rather the value gained from an expenditure of resources. That optimum may actually lie to the left of the "point of minimum uncertainty". 12. Page 1 says, "Models have a long history of helping to explain scientific phenomenon [sic] and of predicting outcomes and behavior in settings where empirical observations are limited or not available." The fact that this language is taken word for word from the NRC report should be clearly indicated, as is standard practice, by using either quotation marks or a separate text block with wider margins. 13. Revisiting Box 2, one "Model Development Step" presented there is "Model Development". That amounts to a circular approach to explaining the term. 14. Page 12 says, "The mathematical framework is usually translated into a form amenable to running on a computer (i.e., algorithm development and model coding)." Perhaps "by" should come after "i.e." here? In any case, having an algorithm and a set of code is not sufficient. After the code is written, it must be compiled to provide an executable that can then be run on a computer. ------------------------ Time constraints do not allow for further detailed comment by this reader. Generally, the content of the document is of uneven quality and the technical discussion is less than robust and complete. Given the level of expertise that EPA has been known to employ in the development of a variety of rather sophisticated environmental models, it is difficult to believe that this document represents the best guidance that the agency has to offer on the subject. The Federal Register notice indicates that these materials "have undergone a rigorous process of internal and external peer review." It appears, rather, that not even serious proofreading was undertaken. A bona fide and genuinely thorough review for the purpose of quality control is urged before final release. Or perhaps the release of a final version should be reconsidered altogether. Given the superior quality of the NRC document and the credentials of its authors, an EPA guidance document that approaches the same concerns less well and with no strength of authority seems superfluous -- even more so in light of EPAs own previously published QA guidelines. It has evidently taken more than 5 years for the provided draft guidance to emerge for public comment. As taxpayer, I wonder whether continuing the drawn-out effort in light of what has been produced so far is a good use of taxpayer dollars.