Roger Caiazza
The New York Draft State Energy Plan prepared by the New York State Energy Research & Development Authority (NYSERDA) is currently out for comments. There is absolutely no indication the New York State is treating the stakeholder comment period as anything but an obligation so I had no plans to invest time and effort developing technical comments that would be ignored. Then I read the Health Impacts Analysis chapter. It is so bad that I had to document this embarrassing scientific travesty for the record.
Alberto Brandolini has stated that: “The amount of energy necessary to refute BS is an order of magnitude bigger than to produce it.” To fully document the problems would require an overly large post so this will reference articles at my blog addressing the different components. Nonetheless I show that the NYSERDA analysis chose its health impact goals and then contrived an analysis to support those claims.
Health Benefits Claims
In a recent article Doreen M. Harris who serves as President and CEO of the New York State Energy Research and Development Authority and Chair of the New York State Energy Planning Board summarized the health benefit message in the Draft Energy Plan. She said: “Additional analysis shows that continued implementation of the State’s energy policies would provide substantial public health benefits throughout the State in all communities, with the greatest benefits realized in disadvantaged community areas.” She made some specific claims: “This includes reduced emissions and cleaner air resulting in avoided hospitalizations, work loss days and emergency room visits due to asthma.”
The relationship between inhalable particulate matter and emergency room visits due to asthma is frequently cited as proof of air quality impacts. In my analysis I only looked at those parameters because of the frequent references and because I found historical data for both parameters.
Before continuing I should note that asthma health impact claims related to air quality is a shaky proposition from the get-go. I used Perplexity AI to generate a summary of the confounding factors affecting asthma related emergency room visits. There are environmental, socio-economic, healthcare access, clinical, comorbidity, behavioral, clinical management and psychosocial confounder factors affecting asthma. Claiming that any one of the factors affecting emergency room visits is agenda-driven science.
Health Impact Relationship
Correlation does not indicate causation. Claiming causation when then is no correlation is tone-deaf agenda driven science. I posted an article that documents there is no relationship.
I compared data from two sources. The New York State Department of Health has developed the New York State Asthma Dashboard that includes asthma emergency department visits data. The New York State Department of Environmental Conservation (DEC) operates an ambient air quality monitoring system across the state and prepares annual reports. The Methodology Appendix in the Health Impact Analysis chapter of the Draft Energy Plan compares the observed inhalable particulate matter (PM2.5) with their model predictions to validate their approach as shown in Table A-3 below. That analysis used data from 19 monitoring sites. I used the same sites except for the near-road monitor because they are not intended to capture average ambient concentrations.
Source: Draft Energy Plan Health Impacts Analysis
In my article on this relationship, I provided plots of the observed data for county-level pollution and emergency room visits. I did not think there would be an obvious relationship, but I was surprised that it was so bad. Only two of the sixteen comparisons suggested that there was a relationship that indicated that inhalable particulate concentrations influenced asthma emergency department visits.
Air Quality Analysis
I have a long and wide-ranging background in air quality modeling. When I read that the health analysis estimated benefits from reduced exposure to inhalable particulate matter concentrations at the community scale, I was taken aback because of the level of effort required. Estimating the impacts of all the sources of air pollution down to the level of 4,911 census tracts in New York State is challenging simply due to numbers. The second challenge is that they considered five pollutants and the Appendix notes that both primary and secondary pollutants were considered. Inhalable particulates (PM2.5) can be emitted directly but most of the observed particles are secondary pollutants formed in chemical reactions from NOx, SO2, VOCs, and NH3. The chemical reactions that create secondary pollutants vary by season, meteorological conditions, and distance/time from the emitting source. When modeling local impacts, it is sufficient to only consider straight line impacts determined by hourly wind directions. However, secondary upwind pollutant reactions occur over multiple hours necessitating more sophisticated transport patterns to track pollution transport.
The solution to this policy impact challenge is to use a simplified average impact analysis. EPA’s CO–Benefits Risk Assessment (COBRA) screening model fits the bill. COBRA uses the well-established and proven Climatological Regional Dispersion Model (CRDM) that categorizes parameters affecting pollutant dispersion and transport. This approach is best suited for local impacts of primary pollutants. When used for secondary pollutants it is less appropriate because there are more factors involved.
The Draft Energy Plan needed an analysis that addressed disadvantaged communities at a finer resolution than COBRA provides. This analysis was conducted using a newly developed air quality and health impacts modeling framework—the NY Community-Scale Health and Air Pollution Policy Analysis (NY-CHAPPA) model. My problem with the NY-CHAPPA model is that it over-simplifies the air quality analysis. The most important air pollution impact parameter is wind direction, because impacts only occur if the wind is blowing from the source to the receptor of concern. CRDM uses 16 wind categories, but NY-CHAPPA only uses four. Given all the sources in the analysis I think using only four wind directions is unacceptable. This gives results that are just too crude to be representative of the actual relationship between sources and receptors.
Given that this is a new modeling approach, I believe it is incumbent upon NYSERDA to verify that their new model is valid. The Appendix to the Health Impact Analysis chapter purports to validate the model for this reason. An air quality model verification analysis uses historical meteorology and emissions input to predict air quality concentrations and compares those results with observed concentrations over the same time period. The process is not complicated. It is necessary to compare model results against observed concentrations. Obviously, the observations need to be for the same time period as the predictions. The NYSERDA analysis does not do that. On page A-13 the draft states: “Because model projections were only available starting with year 2025, these results were compared against multiple years of observational data”.
When I first read that statement, I did a double take and read it again. I could not believe it. It is inconceivable that anyone could claim to evaluate model performance by comparing observed historical concentrations against future predicted concentrations. It is just plain wrong. The verification statistics presented are worthless. The biggest problem describing this situation is finding the right terms to describe the enormity of the error without using profanity.
Context
There is no question that reducing air pollutant emissions will provide health benefits, but the relationship is complex, and in my opinion usually exaggerated. NYSERDA’s claimed public health effects are listed in Table 2 of the Health Impacts Analysis chapter. I addressed whether the avoided emergency room visits due to asthma benefits which range from 1,100 to 3,600 fewer cases per year are meaningful relative to historical rates.
Source: Draft Energy Plan Health Effects Chapter
I compared the emergency room visits due to asthma health effect relative to observed data from the .New York State Asthma Dashboard. Table 1 lists the annual asthma emergency room visits for different age groups. All my analyses used the total asthma emergency department visits. Of particular interest note that the Covid Pandemic changed the identification of asthma. In my opinion, limiting the comparison data from 2009 to 2019 would be more representative of an actual relationship.
Table 1: NYSDOH New York State Asthma Dashboard Asthma Emergency Department Visits
Emergency room asthma reporting changed in 2020 due to Covid. Because this changed the reporting metric, I ran the statistics for the data available from 2009 to 2019. Table 2 lists simple statistics describing the data for that period. The range of emergency room visits over all 10 years of data before Covid is 47,636. The maximum number of avoided emergency room visits is 24% of the standard deviation and 7.6% of the range of observed emergency room visits. The predicted improvement is a small fraction of the observed emergency room visit variation.
Table 2: NYSDOH Asthma Dashboard Asthma Emergency Department Visits Statistics 2009-2019
In my analysis of the context of the predictions I also looked at the inhalable particulate variations. The average predicted concentration reduction for all three modeling scenarios is less than the range of observed annual concentrations. This means that the predicted reductions are within the range of inter-annual variation and that, contrary to the messaging, this suggests that the results will not be observable.
Discussion
My recent posts address shortcomings of the NYSERDA analysis of health benefits of the net-zero transition analyzed in the Draft State Energy Plan. I believe that the air quality analysis used to predict health impacts was overly simplified. NYSERDA used a new procedure to estimate health impacts that needs to be validated but the alleged verification process was fatally flawed. One of the key health concerns is the effect of inhalable particulates on asthma related emergency room visits but there is no observed relationship between annual average PM2.5 and emergency room visits related to asthma for the New York State monitoring stations used in the NYSERDA analysis. I also showed that the predicted impacts on emergency room visits, and inhalable particulate air quality reductions are within the range of observed variations.
Conclusion
My comments should precipitate, at a minimum, a revision to NY-CHAPPA to include 16 wind directions and a valid verification analysis of the modeling. I don’t expect NYSERDA to respond. Instead, I expect that my comments will be ignored like all my previous submittals. It is clear to me that NYSERDA established the public relation slogans for the goals of the program and then perverted the science to get answers to support those claims. When I described this to one of my friends, he remarked that this is proof that science and NYSERDA cannot be used in the same science.
Roger Caiazza blogs on New York energy and environmental issues at Pragmatic Environmentalist of New York. This represents his opinion and not the opinion of any of his previous employers or any other company with which he has been associated.