Sunday, November 29, 2009

The Censored Folder

In the leaked documents, there is a folder full of tree ring data files many with labels of -CENSORED and -FIXED located at  FOIA\documents\mbh98-osborn.zip\mbh98.tar\TREE\ITRDB\NOAMER\. In this folder there is the infamous BACKTO_1400-CENSORED file AND a bunch of others e.g. BACKTO_1400-FIXED, BACKTO_1400, BACKTO_1300-CENSORED, BACKTO_1300-FIXED, covering several time periods, as well as a host of files that appear to compare or generate the various versions of the time series.

Let's take a look at a plot of BACKTO_1400-CENSORED vs. BACKTO_1400-FIXED vs. BACKTO_1400:





The red data and red trend line is the CENSORED version, which shows a downtrend, particularly in the 20th century ("hide the decline"?).
The green data and green trend line is the FIXED version, which shows a slight uptrend, particularly in the 20th century.
The blue data and blue trend line is the "BACKTO_1400"-no other label- file, which also shows a bit higher uptrend over the whole period than the FIXED version, but in the 20th century shows a slight downtrend, less than the CENSORED version.

 It is also clear that the 3 datasets vary markedly in either direction on a yearly basis throughout the entire 1400-1980 period!

UPDATE: Looking at only the 20th century data produces the same trends, only amplified by a factor of 2.4! Also note the near perfect inverse symmetry of the linear regression lines (the regression equation for the CENSORED data is f(x)=0x - 2.39 and the regression for the FIXED data is f(x)=0x + 2.41). I would wager that selecting a slightly different date range will produce the perfect inverse regressions - i.e. "The Flip" noted here, or maybe the date range is correct and its just a rounding error by my spreadsheet program). What are the chances that the "adjustments" between the CENSORED and FIXED datasets are anything other than flipping the data (although it's more complicated than that because eyeball inspection shows the data are not inverse, but by some additional mathematical manipulations one can flip a data set to make the inverse look completely different, yet in the end result in the exact same regression line). With data files labeled CENSORED and FIXED, what are the chances that flipping the data was an innocent mistake, rather than deliberate? Looks like a smoking gun to me.



Next up, regressions on the 1400-1899 data.

3 comments:

  1. I posted the 2nd graph with the following comment at the realclimate.org site (a site controlled & founded by Michael Mann & his supporters) and received the reply from Gavin Schmidt (a Mann supporter and colleage) below:

    In the FOIA documents, there is a folder full of tree ring data files many with labels of -CENSORED and -FIXED located at FOIA\documents\mbh98-osborn.zip\mbh98.tar\TREE\ITRDB\NOAMER\. In this folder there is the well discussed BACKTO_1400-CENSORED file and many others e.g. BACKTO_1400-FIXED, BACKTO_1400, BACKTO_1300-CENSORED, BACKTO_1300-FIXED, covering several time periods, as well as a host of files that appear to compare or generate the various versions of the time series.

    I posted a plot of BACKTO_1400-CENSORED vs. BACKTO_1400-FIXED vs. BACKTO_1400 with linear regressions at:

    http://1.bp.blogspot.com/_nOY5jaKJXHM/SxQebvCIjuI/AAAAAAAAARA/xlK6gks_Bh8/s1600/ScreenShot337.bmp

    The red data and red trend line is the CENSORED version, which shows a downtrend.
    The green data and green trend line is the FIXED version, which shows an uptrend.
    The blue data and blue trend line is the “BACKTO_1400″-no other label- file, which shows very slight downtrend over the period. BTW, these same trends are present as well throughout the entire 1400-1980 data set, but the trends are amplified by a factor of 2.5 during the 20th
    century.
    It is also clear that the 3 datasets vary markedly in either direction on a yearly basis throughout the entire 1400-1980 and 1900-1980 periods.

    Comments?

    [Response: "censored" in this context means removing some of the data and seeing how well things perform (for verification of that, look it up in the emails). In MBH98, the more data you remove the worse the reconstruction, and at some point your reconstruction will not pass the verification step. Removal of all tree ring data - in that reconstruction - leads to the method breaking down before around ~1700 (Newer reconstructions are more robust). However, just plotting the numbers in each individual test doesn't tell you very much, because you seem to be implying that each of these sensitivity tests are somehow equal. They very clearly aren't. - gavin]

    ReplyDelete
  2. Their reply does not make sense. (I have an electiral engineering degree with computer science minor and have been around using mathematical filters for 20+ years for signal processing). What they are doing is basically signal processing, where the objective is to get the "true" signal with the least amount of data points. The better your algorithm is at this the more it can work in a noisy environment. Think how wireless phones have improved over the years....


    It is telling that their "censoring" consistently changed the slope of all the curves from negative to positive to fit their desired result. His reply would have been more plausible if the results were more random in how the signals(charts) degraded as data was removed. I would ask him for a detailed response to how this is plausible that all the "bad" data was negative to the slope..

    Keep up the good work. Design reviews line by line and data point by data point is how the scientific method is supposed to work for research and development. Engineers and Scientists must be able to explain every step.

    ReplyDelete
  3. wow first time I have seen this, and the team has the balls to say removed bad data is censored ? this is so def. of cherry picked.
    some one is going to jail for fraud and conspiracy to commit.

    ReplyDelete