The Endurance of University Data Records - Be Discouraged
Posted by Heading Out on March 5, 2010 - 10:21am
Much has been made of the destruction or loss of data from the files of the Climate Research Unit (CRU) of the University of East Anglia. Dr. Phil Jones, the Director until this all became public, did not do himself much benefit with his remarks before the Commons Select Committee in the UK that is looking into the Climategate matter. Destroying the raw data, or not making it available, so that all that one can use is the modified and gridded data, means that there are no checks that the adjusted data has been properly derived. But the destruction of research data is not only encouraged in some institutions, it is mandated by regulation. It is, however, a point that a lot of folk may have missed. So I thought I would mention this since I suspect that it affects much more than just the data at my own university.
I retired from the University last Friday, and have spent today throwing away about 80% of the material in one of the three offices that, transiently, it has been stored in. (It happened that in recent months the three folk who had worked with me on many of my research programs also retired, and so their records were boxed and collectively stored with mine until they could be sorted.) We worked through file after file, with data going back to the first experiments that I had run some 40-years ago when I came to this place as a very junior Assistant Professor. That data was still on graph paper, with hand-plotted curves. It went into the trash barrels. As did many of the journals that I had paid large chunks of money for over the years, and almost all of the correspondence dealing with the millions of dollars of contracts that I have managed during my term here.
I am working with the University Archivists. They and a couple of students helped me work through many of the files, and did most of the actual disposal. We have done some pretty interesting things over the years (I was incredibly fortunate to be involved in many of the activities that changed my discipline from an academic curiosity into something that impacts, in one way or another, many people's lives every day.) But not much of that is being kept.
A treatment for skin cancer that discriminates between healthy and diseased tissue – save the patent – the rest into the trash. Cleaning the Statue of Freedom atop the Capitol building in Washington, save the proposal, the final report and one paper. The rest – into the trash.
You might think that I am being deliberately destructive, but this is what the regulations require. For those as incredulous as I was, until last week, here is the information:
Notice that the applicable dates are for three years back. In other words three years after I get a contract or grant I am supposed to archive the proposal, report and the sample of data (the paper that I mentioned earlier), and then four years later, I am supposed to destroy all the research data. Hope the sanctions are not too onerous, since, until today I had kept everything. Now much of it lies in grey trash bags stacked down a hallway.
I actually got a bit annoyed about this last week, and it was then that this all came to my attention. As it happened when my pension was calculated (ours is based on length of service) the record did not show that I worked for part of 1997. Now I knew that I had, but if I (or actually the Center staff) had followed University rules, rather than what I had wanted, then there would have been no other record against which to compare the facts relative to the records that are in the Central Personnel Office. Given, however, that we had kept the records, the copy was found, sent up and the matter was straightened out within about an hour. (However, since a large number of boxes have recently left for the incinerator I don’t believe that my replacement as Director has continued my cautionary practice).
In the past the Center has been audited, and had, on another occasion to defend a set of experiments that were investigated by a government agency. In the first case I found a record from a period that I suppose I should have had destroyed that showed that the audit inquiry was misinformed, and in the other I was able to supply all the documentation required (foregoing that it took several full weeks of several individuals time to copy – this being before much of our information was stored digitally). As a result, and based on that information, the inquiry was discontinued.
The amount of space that is needed to store digital records is trivial against the bookcases of material that have just gone into the trash. But storing the material only in digital form has some risks. I have just finished a comprehensive review of one of our programs, requiring data that was stored digitally back in about 1987. I cannot open the files for any of the information. I can’t find readers that will read some of the disc storage that I recorded it on. (And where I made copies of the files and transferred them, the current versions of the software won’t read files from that far back.) It wasn’t in this case too much of a problem since, in violation of policy, I had the paper copies and just scanned them in to get what I needed (and created a digital copy), but those paper copies will be in those grey bags next week.
There are problems with data storage. If I kept the written records, then when I vacate the room, the books will go onto a bookcase in the hall for the students who want them to help themselves (my colleagues already have), a small amount of material will go home, some will go to the Archives, but the majority will burn, or be landfilled. Because the person who follows me into that space has their own research and documentation, which they will put into the bookcases that I am vacating. There is not enough room to store the material. We used to use microfiche to do that – I haven’t seen anyone use one of those readers in years, I stopped when ours broke.
The Federal Government and the National Labs are no different. I was on a National Panel which needed some information on a project from one of the National Labs and we wrote for it. It was about 15-years after the experiments. They no longer held the data, and there was no-one there who we could talk to about the work. (Which was one of those supposedly crazy ideas that folk go out and try, and bless my socks, this one worked, and might have been helpful if we could have found out more.)
So while I continue to think that it is madness to be spending the amount that we are on research into the possible problems of the greenhouse gases without a more robust set of raw data that everyone agrees has integrity and that has been compiled in a way that is logical and transparent, I have to point out that the protocols governing records at Universities are not supportive of my position. Not that this makes me feel any better; rather the reverse. And there are many, many research programs that do not have that level of visibility.
I used to joke in my class that disasters happen in about 20-year cycles, because nobody read anything that was older than that, and thus missed some less-than obvious design features, which became forgotten until their lack led to disaster. But I had not realized that the data was all gone. And in the digital age, if it isn't on the web who knows where to look for it.
Troubling thoughts!
Contact
- Content: editors at theoildrum dot com
- Tech support: support at theoildrum dot com
License
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
Thanks Dave!
This is serious issue. I think businesses (in general, not just Universities) have pushed for short data retention limits, partly to save on storage, and partly to limit their exposure to liability suits. If the records are gone, it is hard to prove anything one way or another about whether an analysis was done correctly. I know as consulting actuaries, the records underlying our analyses were deleted fairly quickly, although I don't remember the time frame being as quickly as three years. The final reports were stored separately from the work papers, so it would be easy to throw away the work papers. The records retention people handled that, without any intervention by consultants.
In the energy field, there are many different kinds of records that would seem to be affected, especially if the research is done under grants. It seems like the calculations underlying EROI analyses might be affected, as might be the details on experiments on how creating biofuels from algae actually work. Unless the final reports contain a whole lot of details, it seems like this method could make it harder for a new researcher working on a later grant to continue where the previous research left off.
Maybe other countries have different laws on this. It is hard to believe that those who made these laws, especially for universities, really thought through the consequences.
I have to recount a story I heard years ago. I remember it because I have been so involved with engineering documentation over the years:
A friend worked for Ford Engine and Foundry Div. in the late sixties. As most corporations do there was a big cost cutting program and drawings were pored over to find ways to cut costs. A group of engineers were studying pistons and found a small hole drilled between one of the ring lands and the center of the piston. No one could determine why the hole was there but it was easy to find out how much it cost to drill the hole. So these guys got credit for saving X dollars.
Then winter came, and a bunch of warranty claims started showing up with pistons seizing in the cylinder walls. Now its time to panic. Instead of trying to figure out what was causing the problem they called the guy out of retirement who had been responsible for the piston design. He took one look at the (new) drawing and said:
"Where is the hole?"
Not many engineering drawings have a section titled "Why its like this"
That is so trenchant!
Management never wants to pay for proper documentation... "it costs too much". As a result, the brains of the guys doing the work are the valuable assets, not the diagrams and databases. Of course, the trolls in accounting feel everyone with the same job description is interchangeable so they fire everyone once an economic downturn occurs.
I have seen this repeatedly in the mining business. Due to the cyclical nature of commodities, every five to eight years companies "retrench" and fire their "brain trust". The supposed savings from eliminating technical personnel are swamped by the cost of rediscovering what the company used to "know".
Of course, I've charged major consulting hours to "rediscover the wheel", so I guess I shouldn't complain, but it offends my sense of efficiency.
There appears to be more than just a question about the loss of the original data from the University. There are big questions about how that data, that can no longer be validated, was used in the computer models here's reference at the Guardian Newspaper. I'm not sure any of the research centres are giving open access to original raw data, source code, assumptions etc for modelling.
I'm a climate sceptic, not denier, what we're told should be open to question. The evidence for global warming is very strong so I find myself not agreeing with HO on this next statement.
There are two problems with this statement. The University in question is only one of three major research centres. One could take out all of the UEA data and there would still be an overwhelming case for AGW. The other is that LTG really highlights to me is the need for systemic thinking. It appears likely as the economic crisis develops it will mask the problems of PO and AGW.
I'm not sure why the original data was not archived.
Good point. And my reply: I did modelling myself and very often look at the reports based on (economical or tecnical) forecast models. According to my experience the authors of these reports virtually NEVER publish the source code of their model, and mostly not even the most important algorithms or mathematical equations.
The only information contained in the reports is the assumptions and the results. The rest is black box.
Maybe this way the authors want to
a) avoid that competitors pirate copy their core know how
b) impress their clients (as some modelling is much simpler than it appears) and
c) avoid that their work is scrutinized and possibly criticized for mistakes
On the other hand of course it is understandable that the credibility of this muddeling is limited - including climate modelling.
I'd also wish that important models like climate models would be published entirely - including the source code. But I fear this will never happen.
Never release model code?
Here you go:
http://data.giss.nasa.gov/gistemp/sources/
http://www.giss.nasa.gov/tools/modelE/
http://www.ccsm.ucar.edu/
http://edgcm.columbia.edu/
http://www.mi.uni-hamburg.de/Projekte.209.0.html?&L=3
http://www.nemo-ocean.eu/
http://www.gfdl.noaa.gov/fms
http://mitgcm.org/
dozens more here
http://www.realclimate.org/index.php/data-sources/
Glad I could help!
Fine, so things in climate research may be different.
But try the same with other topics where money matters more directly - e.g. the energy supply scenarios from the IEA or EIA or similar national studies, or also studies about the future of power generation from nuclear energy or renewables.
I bet you won't find ANY truly transparent study - if you do please let me know!
do any academic types do those studies? isn't it only industry or political players?
good luck! haha
Exactly. But this type of "scientific studies" is what billions of government or industry money are based on.
And nobody cares.
Funny thing is, well actually it's not funny at all, that climate researchers are being blamed for not being as open as other research fields while perhaps being the most open field of all. It says lots about the state of the climate debate...
QFT
The idea that climate data, is in general closed, or that vast pieces of it are lost is an invention of the fear-uncertainty-and-doubt campaign of the side of the media skeptical of it all.
Just some examples of raw data and methods that are available:
Dozens here:
http://www.realclimate.org/index.php/data-sources/
Free climate station data here: (I have this imported into a database on my laptop, and was able in TWO days to pretty much confirm what GISS and HADCru claim for temperature trends)
http://www.ncdc.noaa.gov/oa/mpp/freedata.html
GISS also has all of their data and source code freely available:
http://data.giss.nasa.gov/gistemp/
It really is much ado about nothing. Scientists in the last few years get spammed with FOIA requests, some of which don't even make sense, and the sole purpose of them is to create a story. A recent example of this is detailed here, where a climate researcher was given an FOIA request for data that was already public!:
http://www.realclimate.org/index.php/archives/2010/02/close-encounters-o...
+1 Thank you.
Thank you very much for posting those links. I am just a reader here, but I think you should be allowed to write a key post on the integrity of climate change modelling and data collection.
Of course the corporate media are trying to dissuade us all from acknowledging AGW, and they are resorting to lies in order to accomplish this. But lying as a way of life can backfire. Rupert Murdoch, who has done a lot of lying on behalf of the rich and powerful, has also recently bought Dow Jones and the Wall Street Journal. I don't know that I'd be comfortable relying on the WSJ if I was a small or mid-level investor.
This is not a climate change site, and we do not support any particular view with regard to AGW. We do not run AGW articles, unless we have particular insights, such as there do not seem to be enough fossil fuels to produce the claimed CO2 rise. Of course, some, particularly those aiming for 350 ppm, may still feel limiting fossil fuel use is important.
With peak oil apparently here, and likely to create indirect impacts through a financial unwind, it is not clear to me that even if we wanted to, we could reduce fossil fuel use any more quickly than it will naturally fall off. I expect the fall off in fossil fuel use will be much more rapid than the percentages AGW groups are advocating. So this leaves me scratching my head--is the AGW discussion really all that important? Or is it something that those who are unwilling to mention peak oil have grabbed on to, to support their agenda?
So who's doing it? Aliens? Or is it possible that humans are doing other things which exacerbate CO2 levels besides burning fossil fuels, like burning off the rainforests?
This is not an AGW article, but I'm responding to your comments. Are you a believer in preposterous coincidences?
I think Gail is remarking on Kjell Aleklett's research which suggests that there's not enough fossil fuel in the ground to push atmospheric CO2 above 450ppm or 550ppm, even if we burn it all.
It's an encouraging result, which I'd like to see replicated, but it's not going to save us, even if it's true. If we stabilize at 450ppm, that's close to 2x the highest value in several million years. (Currently, at nearly 390ppm, we're at the highest level in 15 million years, and back then "global temperatures were 5 to 10 degrees Fahrenheit higher than they are today, the sea level was approximately 75 to 120 feet higher, there was no permanent sea ice cap in the Arctic and very little ice on Antarctica and Greenland.") There are a number of good reasons to believe that doubling atmospheric CO2 results in about 3C global temperature increase.
There is no reason to be optimistic about the survival of civilization if sea level rises 75 feet in a century or two. If we want a biosphere that resembles the one in which we evolved, we must figure out how to remove hundreds of billions of tons of carbon from the atmosphere and oceans, in the next few decades. Humans may twist themselves into knots trying to avoid this fact, but ultimately we'll be forced to deal with the harsh reality. The sooner we start acting like adults and stop acting like frightened children, the better.
Good comments Barrett.
I did indeed click on the study she referenced, and saw it when it was originally put up. It's a decent piece of research, but certainly not conclusive on the issue of AGW, particularly with unknowns like gigatons of methane existing in tundra and clathrates with questionable stability.
What about some level of precautionary principle? I think AGW should be taken seriously even if it was at a 2% possibility. Jeez, we're talking about the ongoing diversity and carrying capacity of the planet in the context of a great mass extinction. Meaning no disrespect to Dr. Aleklett's work, AGW is scary as hell to anyone who values the planet and its currently-evolved species and ecosystems.
I agree, Greenish.
The likelyhood of getting injured in an auto accident in a given year are about 1%, of getting killed even lower. Yet laws require seatbelts and other saftey considerations as well as insurance. Why should I have to wear a seatbelt if the the chance of needing it are only one percent? Why should I care about AGW if the chances of it being reality are only 2%? It'll only effect every living thing on the planet. Then again, with peak oil I may not have to worry about a seatbelt either!
Good point. All I can add is that we are talking about two types of probabilities. The seat-belt probabilities fall into the "frequentist" camp based on actual statististics, whereas AGW has at least some Bayesian characteristics brought on by considerations of prior beliefs.
This is a well known debate. One of E.T. Jaynes great insights was to observe that to provide a probability, you have to start with a probability or a set of probabilities. Probabilities don't spontaneously spring up.
If you get this from prior statistics leading to correlated results, you are in the frequentist camp. If you have only a theory not correlated with actual results, somewhere you have to invoke Bayesian reasoning to obtain a probability. Otherwise the best you can say is that your theory is right or wrong. Most likely they are placing uncertainties on their parameters, as conditional probabilities, which means they have a most likely outcome with outliers on both sides.
This would be the same thing as saying that you have a 1% chance of getting killed without wearing a seat-belt but then you add a +/- % to the historical data's uncertainty. Which is quite pointless IMO if you understand how probabilities work in this case. You would have to say that you have a 2% probability of getting killed with a 50% probability or a 2.5% of 25% probability, and so on. Weathermen don't do this but they could if they wanted to.
In this case AGW becomes a belief upon belief system; The first stage of belief is whether the model is correct in that it gives you a falsifiable prediction, and the second level is on the beliefs in the impacts of your parameters.
In a sense there is no point in doing the second set of beliefs unless it is a completely probabilistic or stochastic model. Then the falsibility considerations only apply to how much of an impact the overall effect has on the real world data. That however is a hard one to sell politically because people want black&white results.
I only go the stochastic route myself, because I am not selling anything, just trying to understand the truth.
Paul Krugman has an interesting link to the economist Martin Weitzman who has been trying to understand the probabilities, and he doesn't really succeed:
http://www.economicsclimatechange.com/2009/12/extreme-uncertainty-of-ext...
Like I said, I get around this by using maximum entropy considerations on a completely stochastic model for anything I am uncertain about. This gives me built-in error bars as it gives the most likely probabilistic outcome with all other probabilities built-in to the functional form as maximum entropy uncertainties. That's what makes it a Bayesian belief system, but one based on observations of entropic systems.
That is why I can see through quoting oil reserve estimates as proven, probable, and possible. They have no theory for reserves in the first place and likely only base probabilities on what would occur under similar situations. A very heuristic frequentist approach that gives no insight into what is happening.
Hmmm. I think your mathematical insights are, as usual, deep and interesting. Moreover, having not slept in nearly 48 hours and being more than a bit dingy, I have no business posting anything today. A review of my past comments would show that I'm not one who crusades here pushing an AGW agenda.
But a few comments seem called for, like the ones I've made.
It seems to me that the burden of proof may lie with contentions which run counter to known physical principles; such as the belief that rising CO2 will NOT cause the atmosphere to retain more heat and acidify the oceans. The physical principles themselves are falsifiable in a junior highschool lab. Entertaining a belief in large-scale exceptions to such physical principles, even tentatively, is extraordinary.
Of course, there's also the principle, noted by Nate and others, that humans are often biased to err on the side of rationalizing away stressful ideas as well as having a steep discount rate regarding the future. So any rational planning (were it to exist) should apply a reasonable correction factor for those tendencies.
And I call shenanigans on that whole line of conflating the reasonable extrapolation of known physical laws with belief.
best.
On that last bit, they are giving margins of error for their projections, so the uncertainty has to arise from somewhere. The uncertainty is resolved by a prior belief in what can contribute to this uncertainty.
I suppose one can delve as deeply as one likes into the epistemological basis of certainty and bring it rhetorically to bear on any given model's margins of error, but to what useful purpose?
The utility of the scientific method is in uncovering regularities which have explanatory and predictive value. Such regularities as thermodynamics and quantum mechanics are the basis of much of our entire technological civilization. Not to put too fine a point on it, but in their current form they have never been wrong. Focusing on the niggling uncertainty of particular models rather muddies the central point that there is no scientific controversy whatsoever concerning the existence and probable causes of AGW.
It's a bit embarrassing seeing some TOD editors and regulars declare agnosticism on it, which is exactly reminiscent of the Kansas Board of Education's position on evolution.
Part of the problem is that the uncertainties are muddled with the possibility of metastable trajectories (as in chaotic systems). If some of the uncertainty is in the exact trajectory it will take, then many people will politically gamble that it may take a trajectory that is fairly benign.
I believe this stems from the pop science aspects of chaos. People think that chaos drives the equilibrium or steady-state values of the system whereas we know this is not true. How do we convince people of this distinction?
Thanks Web. I understood the difference when I posted, yet was hoping a "finer mind" would point out why our culture would address the two differently. Just the idea that AGW is slowly creeping into our lives like an angel of death, rather than the sudden, violent, in-your-face reality of an auto accident makes action on it difficult. We're the proverbial "frog in the pot". I think the same will apply to peak (anything) oil, until the crash occurs.
True if AGW is based upon models alone, which it isn't.
First there was the Physics theory of greenhouse gasses which made predictions, then empirical evidence from measuring the IR absorption bands of e.g. watervapor and CO2, then observational evidence from satellites which measure reduction in outgoing IR radiation in said absorption bands and finally GW statistically rising out of the temperature chaos worldwide in the 80's.
Paleontology research also confirms CO2 as a forcing on the history of climate (Iceages caused early researchers like Arrhenius and Fourier to look for explanations in the first place). I probably forgot some here as well.
Basically it comes down to: AGW was predicted by theory, then confirmed by empirical evidence and the earths climate history.
So AGW is not dependent on models, just the future predictions are. The models are based on physics that has been confirmed by multiple paths of evidence. It doesn't mean that all about the physics of climate is known, but (according to the scientists) the vast bulk of it is.
So about 150 years of scientific layer upon layer confirm AGW which requires extraordinary evidence to make extraordinary claims about AGW not happening or saying it is a belief system...
I just caught the end a bit today on the shellfish-or rather what they eat--producing N2O, one very potent greenhouse gas. I guess the critters they eat are nitrogen rich because of the fertilizer runoff and that the environment in the shellfish digestive tract makes the N2O production possible. I didn't hear just how significant this might be though, I'm guessing that is yet unknown.
We are likely facing financial collapse, which is not built into Kjell Aleklett's and others' models. Financial collapse is likely to bring fuel use down even faster than their models suggest. The credit collapse will hit all fossil fuels, not just oil (and will hit uranium, as well, I expect). So their models are high side models. I doubt we really have any ability to bring fossil fuel use any faster than financial collapse will dictate, and it is not at all clear to me that it would be desirable--it seems like if we were able to, it could lead to even worse near-term die off. We really don't understand interrelationships very well a networked system (like the world economy), and pulling on one "string" in a complex system can cause unintended effects other places--even when we think what we are doing can have only beneficial outcomes.
I don't think those who are focusing on AGW have any idea what a predicament the world is in. They have their eyes so firmly focused on BAU, that they assume that is where we are headed, and any discussion needs to be couched in these terms.
I think we need to look at AGW as part of the whole picture. The emphasis on AGW, without putting it in a peak oil context, leads to very misleading views of what is important.
I don't think those who are focusing on AGW have any idea what a predicament the world is in.
Well, some of us do.
I think we need to look at AGW as part of the whole picture.
Indeed.
The emphasis on AGW, without putting it in a peak oil context, leads to very misleading views of what is important.
I'd phrase it more like: "leads to very misleading views of what is possible."
If energy were to remain as cheap and available as today, it's quite possible that we could find some set of solutions for global warming in the next few decades; cornucopians and lukewarmers might be proven right. But in a post-Peak energy environment, we will have continuously less energy available to mitigate / adapt / sequester carbon. Because we're already committed to another century or two of warming, ocean acidification, and sea-level rise, that's a formula for catastrophe.
While I can understand the position of commentators who want to keep the focus of this site on resource depletion and its potential for major disruption of our way of life, I am surprised to read comments such as "I don't think those who are focusing on AGW have any idea what a predicament the world is in."
A few idle thoughts on why AGW is critically important to the discussion on peak oil.
- There is in fact a clear difference between the "AGW-doomers" and the "peak oil/credit doomers" and that is that one is based on a well established physical science while the other is based on assumptions/models about how human societies will react when resource availability reduces. So even though I take the 'predictions' made here and elsewhere quite seriously, I take them with more grains of salt than the 'predictions' made by scientists about the effects of increased CO2.
- Unfortunately, there is more than enough fossil fuel (coal plus unconventional oil) to boost atmospheric CO2 to dangerous levels. Hansen has explicitly incorporated "peak resources" into his predictive models. (search for 'hansen peak oil')
- Indeed, since the concepts of societal collapse consequent on peak oil are not accepted in the mainstream, it is not possible for scientists to publish work that includes such scenarios, even if they believe them personally. I came across a very interesting post on the difficulties Hansen had in publishing even his (non-collapse scenario) analysis...
- The assertion that 'economic collapse' will preclude us burning all the fossil fuel reserves is highly speculative, particularly since in respect of potential bad effects, we would have to avoid burning it for the next thousand years or so. Financial/societal collapse may occur in the US and other countries, but I doubt it will occur everywhere. Furthermore, we cannot be sure what kinds of societal responses will occur in reaction to ongoing collapse. There are entirely feasible scenarios where large scale industrial activity continues in some form in some or all parts of the world.
- Peak oil / collapse scenarios are serious for human-kind but essentially harmless for the planet as a whole (environmentally speaking). AGW however (at least in the more alarming prognostications of scientists such as Hansen and Lovelock) has some very severe possible consequences for the entirety of this planet's ecosystem.
- My opinion is that rather than (or in addition to) saying "we need to look at AGW as part of the whole picture (of peak oil)", we need to say "we need to look at peak oil as part of the whole picture of ecosystem collapse".
AGW is a critically important topic for peak oil people, because AGW is the only theory to challenge business-as-usual that enjoys widespread belief. and its claims are vast - believers are asked to visualize a planet that looks different from space.
by contrast, almost everyone agrees that oil is finite, but almost nobody is capable of believing that oil can run out before something better comes along.
this paradox should be studied.
Agreed,
our atmosphere is being used as a pollution sink as we burn fossil fuels. AGW is totaly relevant to the topic.
If AGW weren't a view that is being used to hide the problems of peak oil, and to give the general population a view that we can "fix" our problems with a few poorly thought out solutions (like "cap and trade"), electric cars, wind turbines, and solar PV, I would have more interest in it. Wind turbines and solar PV do not replace oil, and are not long-term solutions. They are fossil fuel dependent, like the rest. Electric cars are a mirage off in the distance.
Maybe we need to distinguish the AGW science and the AGW movement. It is the movement I find offensive, because it is spreading so much false propaganda, I think we should stay as far away from it as possible.
Gail, I'd respectfully suggest that AGW itself (the science) is neutral with respect to the 'problems of peak oil'. The scientific basis is very strong. However, I'd agree that the response is sometimes problematic, and this is of course an issue with the political process. The implications of AGW is that the world has to drastically reduce its emissions of CO2 from consuming fossil fuels. That is unimaginable to 'business as usual' because the world's economy is so funamentally based on fossil fuel consumption. The only way that the political and business establishment is able to contemplate such a course is by somehow postulate a transition that does not "impact" on our way of life. Obviously (to people here at least) that is a contradiction.
The "AGW movement" is made up of disparate groups and individuals, all with their own additional agendas and biases. So I don't think it is possible to make some blanket accusation of "false propaganda". My viewpoint is that people like Gore are working within the "system" to try and lead some change in energy investment direction. They must make so many compromises to get traction that they are easily seen as "bau apologists". But to be fair, you can imagine what the outcry would be if Gore actually did say anything that suggested that Americans needed to be ready for a lower standard of life.
By contrast, read James Lovelock "the revenge of gaia" for instance - he makes some wild claims but he's pretty much in the doomer camp in his prognostications of systemic collapse. He implicitly accepts many of the assertions of peak oil / societal collapse in recognising that its too late or not possible to build enough "renewable energy" to replace the fossil fuel, and that we may be facing a future of "survival scenarios".
There are many in the "AGW movement" who eschew renewable (wind/solar) and actively push for nuclear (eg Lovelock, Hansen, and see bravenewclimate).
Climate change discussion does not "hide the problems of peak oil."
The solution to the problem of peak oil is to burn less fossil fuels, and eventually burn none.
Part of the solution to the problem of climate change is to burn less fossil fuels, and eventually burn none.
It's like saying that the broke alcoholic, when we're talking about the damage booze is doing to his liver, we're "hiding the problems of his dwindling money supply." If he gives up drinking we'll deal with both problems at once.
To the devoted doomer, everything depends on fossil fuels. Even electric rail, which was perfectly workable with late-19th century technology and emissions levels.
Yet when I post a rebuttal to false propaganda against carbon-free nuclear power, I am censored by TOD staffers. When I demand an explanation for this, I am stonewalled. When I ask for my own words to be given back to me so I can publish them elsewhere, I cannot get them.
If you think I will forget how you (personally) have blocked any front-page rebuttal to Dittmar's falsehoods on TOD, you are dreaming. If you think I will ever forget how you have stonewalled my demands for an explanation of who censored me and why, you are deluded. The work of 4 separate authors proving that Dittmar's series is based on falsehoods has not been published on TOD because you, personally, refuse to allow it. You have one hell of a lot of gall.
Thinking about censoring this? Go ahead. Make my day.
I know this is not my fight, but this statement baffled me--TOD has some sort of hold on your work?
I know your replies to Dittmar tend too much toward personal attacks to be as effective as you would like them to be, the tone of this post, though edgy, is certainly diplomatic enough. If your rebuttal is in the same tone I for one would very much like to see it--the personal attacks are much better left between the lines and far more effective when the reader must connect the dots...in my not so humble opinion that is.
Yes. When a comment is "hidden" (whether by accumulation of user flags or by exercise of editorial power), it disappears even from the author's own listing of comments. Only the editors (and SuperG) can tell it was ever there; for everyone else, it is Down the Memory Hole. I've caught Gail doing this to me a few times, and goodness knows how many times I didn't notice. Well, when you disappear a comment that I worked hard on to finish the analysis before the comments closed, and leave a very nasty sniping job un-rebutted, I notice. I notice the shabby treatment afterward too.
The sharpest thing I said to Dittmar in rebuttal to the comment I linked above was "You again? I'm not your friend." Then I proceeded to show how the material at his link was anything but supportive for his position, and made a better case for the opposite. I suspect that it was censored because the editor responsible (whoever that is; nobody owned up to the act) is a strong anti-nuclear partisan, and the conspiracy of silence afterward is damning. Just be aware that some of the PTB here are anything but honest brokers.
I would very much like to put it on display, at my own blog if necessary. However, since I trusted TOD with the only copy of my work, the refusal of TOD's editors to send it back to me means I cannot even do that. It seems there is something they're afraid of, and that should tell you something.
I'm keeping backup copies of anything of substance now. Fool me once, shame on you. Fool me twice, shame on me.
likely to create indirect impacts through a financial unwind,
What I do not understand is why there are not more people who believe in AGW but are not shouting from the rooftops that "only" 30% of the money spent on a carbon reduction project goes to actual carbon reduction.
http://www.environmentalleader.com/2009/12/08/uk-report-just-30-of-carbo...
Note how 30% of the 70% goes to places like Goldman Sachs - the same group who helped bring you the very unwinding you speak of.
I am pretty sure that a significant amount of AGW talk is motivated by peak oil. After all, it is much easier to under-exaggerate the negative consequences of global warming "So, it'll be 2 degrees warmer all the time? So?" than peak oil. It is also clear to me that most government proposals for CO2 emissions reduction more or less follow the more optimistic models of fossil fuel availability, meaning that the official plan is "Burn every last barrel we can possibly extract".
Gail said:
For shame! This is supposed to be a science-based website, FFS. This attitude is the Achilles heel of TOD, and I've had to comment on it repeatedly. Please educate yourself. I suggest you start by listening to all the podcasts here: 4° and Beyond and here: Interviews with top climate scientists and more interesting interviews here: Radio Ecoshock.
This may help:
THE CARBON CONTROL KNOB (57 min, 13 MB) — Prof Richard B. Alley speech to American Geophysical Union Bjerknes Lecture Dec 2008 San Francisco.
and:
Paul Kingsnorth Deep green British author & broadcaster says governments & climate conferences will never stop fossil fuel use. We make climate change - & won't stop until a crumbling economy & climate chaos force us. Revisioning the new ecology with The Dark Mountain Project. 6 MB podcast, 2009.
Unless they're denialist articles. :)
When members of this site have had trips to foreign countries paid for them by fossil fuel companies, or worked for years for fossil fuel companies, I don't think we need to spend much time wondering why articles denying climate change are considered "particular insights", but those raising alarm about climate change are not.
It's as important as peak fossil fuels, yes. Certainly more important than the prospect of using nuclear weapons for mining, or whether we should bother paying off our debts.
It's quite valid to say that a site is for discussing X, and not Y. But when A, B, C and D are discussed, and when "Y is all nonsense anyway" is discussed, we start to wonder whether the exclusion of discussing Y is really about keeping the site focused, or whether it reflects a bias by the editors. And when we look into it, we notice that if Y is true, it would hurt the financial interests of those particular editors. So we start to wonder...
That we have editors of TOD even questioning the overwhelming consensus and data on AGW, proves the point of how screwed we really are.
This is the event that will push the extinction of many of our fellow beings also, as we are now over 1000 times the background rate.
This is insane behavior.
Peak Oil is an anthropocentric event that will effect the population size and lifestyle of a species.
IMO, it is unfortunate that you chose to post this here. Innumerable other sites cover the climate change issue from all sides, let them remain specific venues for that debate. No stories here have been posted in years about climate change specifics, which is as it should be. Little in the way of constructive debate will arise out of this, only endless salvos of flames for yourself. And what information proprietary ethical issues in government funded institutions have to do with energy I'm at a loss for.
On the other hand Leanan regularly posts stories about climate change issues in Drumbeat; I consider that her own prerogative and never remark about it, as the Herculean work she puts into each issue should make her decision to do so past accountability. As members of the staff I suppose it's not much more of a stretch to include pieces here on Whatevergate, but I still think this is getting away from TOD's purported reason for being, apart from, as I said before, pouring gasoline on those flames, which should be arriving shortly.
I think you are missing the point of the post. I used the tie in to the discussion of climate, since data retention lies at the heart of some of the questions that currently swirl around that topic, and which were, as I drew attention to, one of the issues that the Parliamentary inquiry brought up. But it is merely a topical lead in to the discussion of a much broader concern. The initial data that I described deleting dealt with experiments to establish optimal operating parameters for equipment to be used as a drill for going through rock, there are many other examples that I could have given, most deal with energy production, but all tie in to the underlying theme of the post which is how we store data, and when should we delete it. (And if not, the unanswered question as to how we manage to retain it in a useable form).
Raw data just isn't that important. All data has to be massaged so that it makes sense. Then you verify the paper by replicating the experiment not using the old data.
GW has been verified independently by CRU and NASA and others using different mixes of data.
What makes data valuable is that it obeys the laws of physics. If it doesn't then either the data is wrong or the laws of physics are wrong and the laws of physics are rarely wrong.
Experiments and experimenters are often wrong.
The reason GW theory is right is it makes sense.
The handful of denier scientific hypotheses have been found rather far fetched(cosmic rays make clouds, Lindzen's Iris) and quickly disproven by experiment.
All the deniers have left is Anthony Watt looking for poorly constructed weatherstations and the Medieval Warming Period based on tree rings before the invention of thermometers, which doesn't disprove man-made GW from massive CO2 in the slightest.
Not so fast. You don't replicate historical data, not unless you have a time machine. Oftentimes you don't replicate expensive or disruptive field work either, as the necessary funding or permissions may not be forthcoming a second time. That's the sort of thing proper archives are for, but I suppose proper archives are not the sort of thing that sounds sexy when university deans are giving public speeches about their supreme (self-)importance (and thus cash-deserving status) in the world. But be that as it may:
It just doesn't matter whether "GW makes sense". Let me repeat that: it doesn't matter. It doesn't matter because it has blown up far beyond being solely about laws of physics. It's beyond even caring about physics. Like it or not - and it has become blindingly obvious that the (stereo)typical insular-minded university professor likes it not one little bit - it's now also and supremely about politics.
After all, when you go around blathering casually about draconian shutdowns of major economic activities, there's bound to be political blowback. How could it conceivably be otherwise? On what planet? When you yammer about turning Britain (and by implication the USA, Japan, and the EU) into "a very poor third world country" (Mayer Hillman, quoted by George Monbiot), you can't expect people just to sit back and say, "sure, why not". In other words, it is a supremely political question whether it's worthwhile to forgo hundreds of trillions of dollars (or euros) in economic wealth in a mostly-impoverished world - or instead to take one's chances with Business-As-Usual (BAU), or at least with whatever BAU-like path proves possible under emerging resource constraints.
In order to support the draconian politics that's so facilely bandied about, the science will need to be unimpeachable beyond even the best levels attained in, say, aviation. This is rather unfortunate - dare I day it, "inconvenient" - but there it is. The wasteful, reckless, cavalier institutional sloppiness described in the keypost simply will not do for a question of such magnitude, no matter how much you or anyone else here might scream and yell.
But it gets even worse. The modeled consequences of a BAU-like path are after all hypothetical - no matter how much the models may "make sense", from a political point of view they're still just undocumented ghosts (trust me, I wrote a secret model, and it's all mine although you the taxpayer paid in full for it, and I refuse to say how it works but the pixels it draws on my screen scare the hell out of me) in a computer. Worst of all, agreement to deviate from the BAU-like path has to be virtually unanimous since a characteristic of exponential growth is that if even one player opts out of self-restaint, everyone's efforts will be vitiated in time.
So: whether you, or I, or anyone else here likes it or not, the bar is set extraordinarily, perhaps unprecedentedly, perhaps un"fair"ly high. Now, and I've said this before, it may be a shame, or in some cases even a tragedy, that some retiring tweed-jacketed (or, in the USA, track-suited) university professors who might have been counting on quiet careers in an obscure backwater find themselves in the harshness of the limelight, but like it or not there it is. Deal with it or it will deal with you.
So science that has a political/economic/social implication has a higher bar than regular ol' science(like the size of the universe, something nobody cares about)?
That's just wrong. Science is science. It has its own rules.
Look at evolution. Darwin knew it was going to cause a furor but when Wallace published he had to move ahead. He didn't wait for the invention of radiocarbon dating and more data collection. He knew what he had found made sense and so did his colleagues.
Was the theory of evolution unimpeachable? No, but it has the experts convinced it's true. Same with global warming.
As for the expense of mitigating I don't hold climate scientists responsible for finding cheaper methods. Corporations will take their cues from the government.
If the government tells them to phase out coal emissions like they are phasing out freon or reducing acid rain, they'll do it.
Unfortunately, the government and media is run by corporations. It will take a lot of guts to turn that around.
"Was the theory of evolution unimpeachable? No, but it has the experts convinced it's true. Same with global warming."
I agree. A different example:
http://news.yahoo.com/s/nm/20100304/sc_nm/us_dinosaurs_asteroid
Does anyone here think this issue will now be laid to rest? Notice that these people were able to access data and research going back at least 20 years, likely more.
I doubt, though, that we've heard the last of this one either.
Correct. Democratic centralism, which amounts to a notion that at some arbitrarily chosen point you shut off discussion and reject new evidence, is best left behind to the living hell that was Stalinist Russia; it certainly has absolutely nothing to do with good science. And after all, the cause of the fate of the dinosaurs is an issue with minimal practical impact, not the sort of thing that politically justifies - or has gotten - an all-out no-expense-spared comprehensive global search for evidence. So why would anyone be foolish enough to think the latest declaration absolutely settles the matter for all time??? Or to put it another way, hardly anything in science is ever 100% settled.
The persistence of Flat Earthers would be an example, though one would have to admit that the better and more settled the science is, the loonier the claims of its deniers sound.
In other galaxies, maybe not, who knows? But on planet Earth, YES, of course, absolutely, shout it to the rooftops.
Few people give the proverbial rat's behind about whether the size of the universe is a little bit off - or even an order-of-infinity off, if any of the wilder speculation about multiverses turns out to be even slightly right. But it has absolutely zero consequences for daily life - in popular language it's called an academic issue, which means, "who cares, except for eccentric boffins who we already well know to be barmy?". On the other hand, many people do care very much about being sentenced to eternal poverty on the basis of what can only look to them like more of the same eye-rolling millenarian stuff that's been coming down since long before the Oracle at Delphi. IOW, the one thing has no consequences, the other does. So the public standards of proof are vastly different, irrespective of what may satisfy specialists.
Of course it does. But the very reason we're discussing AGW in public rather than relegating it to some obscure seminar room on the thirteenth floor of some obscure Meteo Hall is mitigation. Mitigation is mainly about politics, not science, despite the fact that choices will need to be informed by science. No amount of yelling that "it's just wrong" will alter that one jot, nor will it alter the fact that politicians and the public will demand a much higher standard of proof with respect to something they care about, such as wealth vs. poverty, as opposed to something utterly devoid of practical import, such as the size of the universe.
Of course not, well you shouldn't; neither should I nor anyone else; it's not climate scientists' job. But that's changing the subject, which was establishing a quality of proof sufficient to move the political mind, which (outside of North Korea) normally requires also at least some moving of the public mind.
Sorry this is all so ... well ... inconvenient and non-idealistic, but once again, there it is, like it or not. Be that as it may, it still mystifies me that science types here and in the broader world seem to find it so difficult to grasp that politics and academic science simply do not play by the same rules, or that the existence or severity of consequences might, at least in practice, affect the strictness with which said rules are applied. After all, we do expect that expensive, time-consuming jury trials may be held in murder cases, but most of us would think it preposterous to impanel a jury and spend hundreds of thousands over a routine parking ticket. Why, oh, why, would AGW be any different vis-a-vis universe size? [George Mobus, can you help???]
Your attitude is inconvenient and non-idealistic. So are many others.
Perhaps a parabel will help.
The CO2 that results from burning fossil fuels has made our modern world possible and science now is getting in the way of all that.
I expect you think that corporate backed climate deniers who have no scientific basis for their
opinions are heroic and also believe that scientists have been paid off by Greenpeace or Al Gore or that atheist scientists have joined a worldwide ecological religion.
Righteous and right on!
That is exactly why I have been working the depletion models as far as possible from a formal first-principles approach. We have to get this right as too much depends on it. Right now we have Michael Lynch and some other lightweights nipping at our heels, but it seems at some point that the big guns will come down on us.
I just haven't been able to articulate it as well as you have.
Indeed, the political points that apply to AGW will often apply similarly to other issues of great social import such as resource depletion or the mitigation of toxic levels of pollution.
Absolutely agree that the AGW "debate" is about politics, not science. Wars have been fought over far less than is at stake in the question of whether we need to curtail CO2 emissions. It's a matter of literally trillions of dollars in the future value of fossil resource and the income to be derived from them.
However, ..
That's the framing that one faction in the political war would like us to buy into. But that's all it is. A carbon tax (or its equivalent) will mean a massive shift in the allocation of economic activity, but it is by no means obvious that a shift to non-carbon energy resources would have negative consequences for the economy as a whole. In fact, a plausible case can be made for the opposite -- that accelerating the shift away from fossil resources is the best hope we have for economic recovery. I'm not sure that's right either, but it's at least a point that can be argued.
The fact remains that the physical world cares nothing for any of this. AGW is real and presents a degree of threat that is subject to genuine scientific inquiry. Those of us who don't have skin in the political game should not forget that we have more than just skin in the geophysical game. We have plenty of reason to care about scientific truth, independent of what's convenient or inconvenient for any particular economic interest group.
Yes, KLR, I think you missed the point of Dave's post.
I've been concerned about the loss of natural history information in old bound texts that are slowly disintegrating. I caught an oblique reference to a rare fossil discovery (Silurian psilophytes) in the area in which I used to live. I found the 1921 journal article and map in fragile condition in the dark basement of a university library. It was way down in tightly squeezed compactus shelves. A year later it was moved to a top floor solarium with a nearby balcony but has most likely been moved again.
Anyway I photocopied the map and after some early morning walks and haggles with farmers I found the long lost spot. Trouble is no qualified researchers gave a damn so I guess I'll carry that secret to the grave. I was even going to pay for electron microscope work on the material. My point being that with the GFC and Peak Oil no-one is going to carry on certain lines of research. Even if some key journal pages have been scanned and can be found online the momentum is not there. Effectively the knowledge has been lost.
It's not just bound texts. Years ago I was working for an organization involved in searching for hydrocarbons in Greenland (this was before any offshore drilling there). One of my colleagues found a reference to a Cretaceous fossil dredged from the bottom of Baffin Bay by the M'Clintock expedition in search of Franklin in the 1850s. We thought another look at the fossil would be a good idea.
The paper describing the fossil noted that it had been lodged with the Royal Dublin Society. It appeared that the fossil collection of the Royal Dublin Society had been transferred to the Natural History Museum in the 1870s, and this museum was still in existence. We found an address and wrote to them. Several weeks later we received a letter from the museum, saying that they could not help. The Republicans had occupied the museum in 1916, and had used all the rocks and fossils to fill sandbags as part of their defences. All the specimens had been recovered, and were still in storage, but the labels had been separated from them . . .
Data will only survive in the long run if it is archived with multiple copies in different formats in locations, and it is of continuing importance to many people. I think we have to live with the expectation that most data simply will not survive. There's too much of it.
I imagine it's not helped by Universities asserting that they own the rights to data, inventions and conclusions. It is logical that those people involved will keep copies but legally it's not theirs to choose.
Alvin Toffler discussed these issues in 1970 in his groundbreaking book, "Future Shock". He described the problem broadly with the term "Information Overload".
As the amount of information created by an cooperative effort (company, government department, university) increases, the problem of retaining and storing the rising flood of information becomes critical. What to save? For how long? How do you deal with security with critical classified information (recall that many nations in the world today would kill to be at the level the Cold War powers were at in the 1960's on nuclear weapons research).
The whole problem of historical illiteracy is seldom touched upon, whether it be technical amnesia, or whether it is occurring in the larger culture.
This cultural amnesia can be almost touching...A friend of mine recently bought an MG Midget and drove it to our workplace. A young woman who works there was almost beside herself with shock as she said "Who would imagine you could ever build a car so TINY?!"
And there can be opportunities...It is given as a statistic that books about the technical ideas of Nikola Tesla are still among the most checked out and examined books in the New York Public Library. Do you know why that is?
Think about it...since most engineers and technicians know that their bosses are technical amnesiacs, all they have to do if they are pressed for an exciting new line of research or a project that someone may provide research funding for, said technician can go to the work of Tesla and whip up something from one of his long forgotten (but still very advanced, even today) projects, pass it off to the bosses as his own newest breakthrough line of research and come across like a genuis.
Don't think it doesn't happen.
RC
A 1950's vintage electrical engineering book made the following suggestion:
Make a breadboard with many flashing lights.
Every few weeks change the color of the wires and their length.
When management comes by and looks at what is going on, they'll see the wire changing, assume you are working and you can then actually DO your job VS plenty of meetings about how your job is coming along.
Here are some tricks I have learned.
Graphed data is as good as good as table data if you have the tools. Rare that you need any kind of significant digits.
Recreate the data trends with models. Plot the data points along with the model. The model will algorithmically compress the data if the fit is good.
If you only have graphed data, use software to digitize the data or else use the transparency mechanism in photoshop or gimp or paint shop to create a layer. Or create a cropped image and place it as the background of a plotting program (excel perhaps).
Publish any data plots that you have on imageshack or one of the other free image hosting sites and then link to it in a blog. That should provide some persistence.
This is usually good enough, at least for what I do.
It seems like for 99% of the data, I wouldn't worry. If it comes to understanding some physical phenomena, the "truth" is still out there waiting to be measured and understood. The fact that someone did not keep archives or threw them away is beyond anyone's control.
These are just my thoughts, FWIW. I am a formal first-principals modeler but an ad hoc data collector. I know other people have the roles reversed, where they are sticklers for the data and end up resorting to heuristic fitting exercises. The latter analysts tend to scream the loudest when they get imperfect data. I just want any and will deal with it.
YMMV
That only works if the unadjusted data is graphed or otherwise presented. I don't think that would be the case.
Like biophiliac commented (http://www.theoildrum.com/node/6271#comment-596242) the data not presented visually usually has lost complete context. Graphed data usually has the context in the accompanying text. The only thing to worry about is the massaging of the data.
Which brings up an interesting point for graphs on oil depletion. Many times I see data that gets run through a moving average causing much of the fluctuations to disappear, which makes it easier to visualize a trend or a fit. Yet, the moving average will by definition add a latency to the data as well. Whether those side-effects are significant depends on the importance one places on accuracy and precision. The AGW skeptics go to town on massaging and post-processing data.
Historical graphical file formats you probably cannot now read and display:
xbm
xpm
ppm
tga
pcx
If you store your information as images, you have to be sure that others will be able to read them, no different from any other file format. The platform you are using now may not exist in 10 years. It looks like gif, png and jpeg are likely to exist for a while, but gif is already being phased out with preference to png, and in 10 years, gif may not be readable either.
It ads a phase shift rather than latency. Fourier filtering can be used to remove high frequency artefacts from data without the phase shifting caused by various other forms of digital signal processing (like moving average). I've always wondered why it's use isn't more common.
e.g.
http://terpconnect.umd.edu/~toh/spectrum/FourierFilter.html
Perhaps just the coding/processng requirements.
Of course it will remain readable. In the future, image reading algorithms will be adaptive and be able to figure out the graphics format algorithm on the fly. I think the statement is naive, as software is not hardware. I can see a problem dependent on the physical media the data is stored on but not in the algorithm.
The MVA is approximately a latency and not a phase shift. By definition a phase shift only applies to a given frequency, and a typical plot is a waveform composed of multiple frequencies. We can argue this but I am simply showing the practical effect.
A fourier transform approach may be of some use, but all the creative filtering doesn't help with the data composed of the most recent data samples due to edge effects on the most recent data. That is one of the problems we see with all the climate data and oil production data. People arguing over whether the last 10 years showing a decrease or increase, for example. The reason you see the moving average used is because it is so simple. A moving average is just a finite impulse response (FIR) filter in DSP terms.
You know, I wrote this with the huge disclaimer that said YMMV - YOUR MILEAGE MAY VARY. I am just stating things the way they appear to be going (all data processed as graphs), with some useful tips for interpretation of archival data, not the way I want them to be (apart from removing the filtering).
It seems to me that scientific data should be archived on the web in digital form and in open formats. I don't know who would pay for this nor who would manage the sites, but it would certainly be a valuable service.
It can be hard work looking after data and old reports. I have worked on low energy housing since the last energy crisis in the 1970s. Today's researchers are unaware of all the good work that was done then because it's all on paper and not on the internet. Also for some things it is necessary to see how things have improved - are today's houses actually more energy efficient than those of thirty years ago? Has people's behaviour changed? You need to look at the old reports.
My office is stuffed with great material from then. Time and again we've given the paper to our university library to archive and a few years later they've come back saying they haven't room and could we stick it in a skip please?
Inspired by Walt Patterson at Chatham House who has put all his old books and papers on-line:
http://www.waltpatterson.org/
I've started turning stuff into pdfs and putting it into the university digital library. Hopefully they won't come back in ten years and say it's full up!
Here's some samples from the heady days of Milton Keynes energy research of the 1980s:
http://oro.open.ac.uk/19946/
http://oro.open.ac.uk/19860/
http://oro.open.ac.uk/19861/
As for the original recorded data, much of that has been transferred from tape cartridges to genuinely floppy disks to not-so-floppy disks to CD and now lives in a very tiny obscure corner of my hard drive.
Thanks for all your hard work in this area! It seems like people forget that there was a lot of work done in the past, that might still be relevant.
I see similar things from my experience as a layman. An enormous amount of interesting stuff was published in the popular press during the energy-crisis years of the 70's, and I gobbled it up. Today, almost nobody knows about it. People are even ignorant of commercial products of the period ("how will I mow my lawn without gasoline?" "well, GE built something called the Elec-Trak about 40 years ago, and we've learned a few things since then... maybe you can find one for sale if you look.").
It occurs to me that a few thousand hours of student work-study time digging through microfilms of magazines from the era to find material for re-investigation could be some money very well-spent.
No doubt a lot of good stuff is fading away forever, interesting proposal on the student time. The shear mass of things now being done digitally is going to the make the archiving and indexing no picnic, and the triage processes will be very uneven. We may have the internet and huge computer memory capacity but we aren't and will never be Star Trek's Borg. That is probably a very good thing, a certain degree of privacy is desirable--that was one lesson I learned well in the village.
But the losses can be great and irretrievable. Entire languages fade yearly these days. The amount of time and energy that went into developing them is probably incalculable. Many, many unique and instructive ways of looking at the world or at least parts of it disappear with each. Life goes on.
From earlier post: I now understand what happened to your reply, it may actually have disappeared and not be anywhere by now. (I may have actually read that reply as the that node went to no comment in the few minutes between when I posted a reply and went back and realized I'd missed a piece of code when explaining how to post pictures that are stored on the web.) I run many of my replies through my mail program to get some sort of spell check and I used to send a few of the more thoughtful ones to myself...haven't done much of that lately, but then I never have too much of a detailed technical angle in what I say.
I do need to thank you. It has taken months but I pretty much finally have habituated myself to the 'control' + 'click' option. I doubt I would ever have thought of it without your suggestion back when I bemoaned losing all my 'new' flags in the thread about the huge marine two stroke diesels ?- )
In my experience in scientific research, most data that is more that a few years old is useless because the context in which it was collected has been lost. Published data usually has enough context and interpretation to be potentially useful indefinitely. The message should be: if the data/results are useful, publish or write some kind of report with the data/results, otherwise it will sit on a hard drive or a shelf until it is thrown away and the effort wasted.
As a corollary, I would say that other peoples raw data is also useless (e.g. the CRU data), regardless of age, unless they are actively working with you to interpret the data. There is never enough context to correctly interpret other peoples data.
I think one thing we had going for us in an insurance consulting setting was the fact that the insurance company was responsible for collecting and keeping its own data. All we as consultants were doing was analyzing it. So even if we didn't have data records going back forever, the client certainly should have (if it didn't go bankrupt).
Also, one of the things actuaries do is look at patterns in past data, and how those are predictive of the future (not assuming peak oil!!). So each time a client sends data, it sends a whole triangle of historical data (because old policies and old claims would have payment amounts and and other values at different points in time). So each year's data would look like the prior triangle, with just one more diagonal added. (If there was a change in history, we would ask why.) So each year, we would automatically get a new copy of the old data, negating the need for the consulting firm to keep the very old data files.
Of course, in the insurance historical data is the real gold. The better your predictive models (which depend critically on your historical data) the better your potential for accurately pricing insurance premiums and thus making a profit.
In some areas of science the experimental data itself is not really important except as far as it supports the ensuing publication. In general, if the theory is important there will be multiple subsequent experiments that replicate and validate the theory.
Exceptions to this are of course the study of historical data (including weather observations, which to my knowledge have been archived and maintained by all the appropriate weather observing institutions) and data of "unique" or "expensive" events such as nuclear tests, explorations, fossils, etc. These are commonly kept in museums, which specialise in that kind of long term preservation.
Medical data often has privacy concerns so mostly ethics committees will insist on destruction after a certain interval. This may be of concern if the data can be "re-analysed" in future with better techniques or if new information comes to light.
Obviously in a fast-changing world there are always going to be occasions when "new information" appears that might impact on previous interpretations of data. Ideally it would be possible to re-visit the old data, but apart from the logistical issues involved in locating and retrieving that data, it is often better to 'start with a clean slate' and collect new data.
I have just registered so I can comment on this topic.
Firstly, I would like to commend this site for the quality of its articles, and perhaps more importantly in view of some other blogs, the quality and tone of the comments. I will endeavour to keep my comments in match with the tone, at least.
Retention of data, information, articles, books and so on is a complex subject. Timescales will vary from months to millennia. The type of media will change over time and with electronic media, there will be a need to maintain the format in a readable form and regular transfer to current equipment. This, in itself, will be an energy intensive task.
As an aside, in trying to educate my former institution on this subject, I used the example of the Domesday Book dating from 987. This can still be read by anyone. The BBC created a new version for the thousandth anniversary, 1987, on Phillips laser disk. That laser disk version lasted barely ten years and has now been transferred to CD and will undoubtedly need to be transferred from CD soon.
Of particular relevance to Energy matters, I would assume ( and others may correct me) that there are records, data etc, held by the fossil fuel industries which are likely to be of great importance to future generations. Specifically, if we are to adopt carbon capture and sequestration on a significant scale and store in, for example, oil wells, then detailed records of the geology etc, of those wells are crucial.
Amen. Although US oil records on reservoir sizes, etc, are supposedly publicly available, we have no guaranty that it will always be in the likely location, such as the Texas railroad commission.
We could learn something from the open source or FSF movements whereby redundant copies float around in a distributed fashion preventing anyone from locking down the source. In this case it would be data and not source. It wouldn't be as strong as FSF because we couldn't place a copyleft on the data, but anything would be better than what we have.
This begs the question: How well are TOD posts and threads being archived? If nothing else, I think years of TOD and the posts/discussions here may be a worthy historical document, useful to some historian (500 years out?) trying to understand "The Greatest Transition". Who knows. Some of the posters here may become "Prophets"!
"This begs the question: How well are TOD posts and threads being archived?"
My method
Wait until there are no more comments.
Copy Post to Word
Date properly in sequence, i.e, 20100306 Article Name
Clean out the Trolls and irrelevancies if any (try the Grauniad and anything to do with Climategate on for size).
Reformat all diagrams to 6.5" width.
File by topic on hard drive.
Since I plan on living for ever or dying in the attempt, the Posts are quite safe ...
But seriously folks, Heading Out has touched on the very important topic of "Erasure of Corporate Memory", deliberately or accidentally. all the old farts are retiring or dying off. These are the people who know why things were done the way they were.
I specialise in Troubleshooting as in the BC TV Series of the same name from the 1970's.
I've had the creditors seize the computers.
I've had engineers running a junior in Calgary buy a very significant shale gas property and then thrown out 20 boxes of geological data "because we don't have the space". Please help. Oh, and we can't afford to maintain the computer mapping system for BC. Well, duh. I remapped the project on mylar in 3 months using microfische logs.
And for bureaucrats, politicians and accountants, information on paper in boxes is just a waste of rental expenses in the basement until the shit hits the fan.
The Worst American Author Presently Writing once wrote in one of his books "If you didn't write it down, it never happened". And it's true.
Corporate Memory or Social Memory is probably more important than written records. In 1963 I moved to Brisbane, Australia. I noticed a peculiarity of development in the western suburbs: most of the houses on the hills were built in the 1920s or 30s; but all the houses in the valleys were built after World War 2.
In January 1974 the reason became obvious: the Brisbane River rose to its highest level since 1893 (when two ships floated into the Botanical Gardens), about two meters higher than any other flood in the twentieth century. Much of the new development in low-lying areas was under water, and many companies realized that putting mainframe computers in the basement was a bad idea.
The Social Memory of the 1893 flood stopped development in low-lying areas for half a century. But after that, the decision makers were people who did not remember the floods. The data were still available, but the personal memory was not.
Ird,30+ years after the 74 flood the clowns have been building on the river flood plain.I don't know about social memory but there appears to be a blindness to the facts of topography.
One would think that local and state governments have a responsibility to zone with worse case floods in mind.But the developer dollar speaks their language.
In the USA, what they often have is Federal flood insurance. They build (and people buy) because somebody else gets stuck with the bill when floods happen.
I think the question of how well Oil Drum posts are archived is an important question. I don't have a good answer.
Many of my posts are republished on Energy Bulletin. When Bart Anderson first approached me about doing this, one of his points was that this would provide some redundancy as to where the articles are located. If something happens to articles at one site, the other site will still have them.
As an IT guy one of my jobs was to perform data backups and archiving. Our weeklies were redundant, kept in two seperate buildings on site, and a weekly was also sent to "The Vault" (protected storage). One set of weeklies was recycled every 5 weeks, one kept for annual archives. Our annuals were recopied twice yearly (to avoid magnetic degradation) and one copy went back to indefinite storage. We dealt with numerous data formats from many countries, some very outdated, so the applicable software was usually stored with it. Our owner was really anal about backups and archiving because he had lost millions on a contract that had been put on hold for three years. The magnetic media (tapes) had degraded and months of still valid data had been lost. Optical media lasts longer but should be "recycled" every few years, and if data formats are outdated, the software should be archived as well. We kept two old PDP 1100s alive just to access and convert old data (mostly from Europe) and move it to more modern platforms. The company made a lot of money just converting and modernizing client's old data.
There are companies that do just this, provide indefinite storage and insure integrity of data. They have vaults that are climate controled, secure, and hardened against nuclear blasts and EMPs. Of course, how good the original data is, and how or whether it ever gets accessed again is another question.
As I said, TOD posts and threads (even the Trolls) are worthy of this sort of preservation, if only for its historical value documenting interesting times.
I've been waiting for the open source community to come up with some code that would allow you to track a filename across various storage locations and have a tool that can checksum the media to suggest when rotation onto new media should happen.
It seems no one has scratched this itch yet.
To me, it looks like we are presently swamped with information and have no means to reliably store it, index it or serve it up on request. (Certainly in any long-term context, eg. decades, generations, centuries). In that circumstance, I think HO's university policy should be considered correct though perhaps the shockingly short timelines in the policy need to be revised somewhat upward, agreed. At the bottom, the useful things from a research project are the results / conclusions, and the context those were drawn in. eg. would the materials analysis process which R. Diesel used to determine that his cylinder walls need to be 30 mm thick be of any use to anyone today? Obviously not, as the alloys available today, the lubricants, fuels and a host of other factors are all so different that his original data is pointless.
Bottom line is, we need to learn to trust the peer review process as it is, or to fix it if it is broken. Relevance and accuracy of specific data sets to a conclusion is an issue for the reviewers. If others wish to later challenge the conclusions, they need to do their own basic research and put that through the peer review process.
Of course the question of establishing a broadly available screened and shared data set for specific questions of broaqd interest such as climate change is one which should be handled on its own merits. The fact that it hasn't been done yet as a generous donation of time to society by some past group of researchers doesn't negate their conclusions.
Such data would be of great interest to anyone investigating the history of materials science and historical engineering practices, and perhaps museum workers maintaining machinery of that vintage.
My own experience as a scientist and historian of science is similar. In the XVIII century records were often better kept than now. Nature as an article on this issue recentely. The discussion was related to scientific fraud. The problem is that scientist have very little support for keeping archives. If you change of university each 3-5 years, you will loose a lot of stuff. Once you get retired, nobody care about your stuff. Often, the scientist bring old papers with him. Once he died family migth dump his old stuff in garbage. As a counter exemple, recentely, Stanley Miller died. In his file, someone found 50 years old sample of a variante of is original experiment. By analysinng this goo with modern techniques, scientists discovered than a slight modification on the original setup brough a new light on the problem of origin of life.
I brought this issue a couple of time to our governement. In astronomy, old data are considered as important now. Many observatory are archiving data automaticaly. There is also effort to keep old astronomical plates in vault.
The Miller example brought new insight, and brings up the point of at least archiving experiments of great historical value.
I guess it also depends on whose values we are talking about. I'm quite sure it wouldn't be very hard to find people alive today with specific agendas who may want to actively suppress information about experiments such as Miller's. After all, it sounds a lot like the work of the "Devil".
I suspect that if humanity survives at all it will go through many cycles of enlightenment followed by long periods of darkness and ignorance. Not to mention that sometimes both enlightenment and ignorance exist side by side in the same historical period separated by geography, culture, social class and levels of education.
This sounds like the premise of Asimov's Foundation books.
I think that one thing we have not paid enough attention to is how the composition of food has changed over the years. There are a lot of different issues:
1. Soil depletion. Trace minerals may in fact now be gone, or present in much lower amounts.
2. Industrialization of farming. Feeding animals grain instead of grass, adding antibiotics and growth hormones.
3. Rise of "manufactured" food, with short shelf lives, and many chemical additives.
4. Changing fashions on what is perceived as good to eat. For example, we used to eat organ meats, but not much any more. (This also reflects chemical issues).
5. Pollution issues--mercury in bodies of water, for example.
Without actual samples of food from many years ago, I don't think we can really tell what we are now missing. We can sort of guess at all of the "stuff" that has been added. Important things, like the Omega-3 fatty acid balance has gotten badly messed up, but there may well be other things, too.
I agree with the "20 year disaster cycle due to failure to revisit fundamental design flaws" and think it applies to many other areas as well. Just a quick glance at some of the "alternative energy" proposals on the table today proves this point: "waste-to-ethanol," nuclear, anaerobic digesters, gasification...
Congratulations on your retirement!
I very much appreciate your post, but I disagree with this statement: So while I continue to think that it is madness to be spending the amount that we are on research into the possible problems of the greenhouse gases without a more robust set of raw data that everyone agrees has integrity and that has been compiled in a way that is logical and transparent...
There are a number of separate data sets at different institutions that completely support those at CRU, so we have plenty of robust data, all of it saying the same thing: Earth is warming. Everyone agrees, meaning the entire scientific enterprise.
It remains to be shown that any of these data were not compiled with integrity, or not in logical and transparent ways; it remains to be shown that any climate scientist has behaved in an unethical way. There is no evidence to indicate that thousands of climate scientists have conspired to "hide the decline."
I continue to think that it's madness to be spending the paltry sums that we are on climate research, mitigation, adaptation, carbon sequestration, etc. If we're to have future generations of humans, the whole world needs to pull together, and soon.
Human caused climate change is a given, and no credible scientist who is not ideologically committed or whose financial interest are at stake, would argue against it
But retaining data?
It is a bit messy.
Thanks for the kind words - in regard to the points raised, I think you might want to check some actual numbers, and perhaps read the "Climategate" e-mails, before making generalized statements. In regard to research dollars, there are, for example, dramatically larger sums being spent on climate research than, for example, on improving the productivity of oil wells, or understanding the decline rates from the gas shale wells.
This site does not take any of the many positions possible on the AGW debate. This article is not intended as a statement on AGW. Oil Drum staff members have a wide range of beliefs on the subject. HO is by no means alone in his beliefs.
I think an AGW agenda has been pushed by those who do not want to talk about peak oil. We can disagree on how important that agenda is. I personally think the direct and indirect impacts of peak oil will trump the impacts of AGW, so AGW (whether true or not) is of vastly less importance than the information in the press would suggest.
Those pushing the AGW agenda would also like to promote the idea that BAU can continue, if we only adopt renewables. I think that is just a bunch of hype, to make politicians look good, and avoid facing up to the real issues.
So I would appreciate people feeling that they are free to attack HO or anyone else because of their beliefs on AGW. Such behavior is not appropriate in an Oil Drum setting, in my view.
Gail,you and Heading Out are entitled to your opinion and to express it but so are others entitled to express theirs.
I,for one,am not"pushing the AGW agenda" but it is a very important part of the bigger picture and it is essential to understand as far as possible that picture.
Renewables and nuclear power are part of the possible energy alternatives.Whatever the political implications it is essential that these are debated and examined.
You may well be right that peak fossil fuel will have a more profound effect than climate change.On the other hand you could well be wrong.Maybe you think that population overshoot is even less important than either.If you try to see the bigger picture you may be less inclined to rail against those who are less than impressed by HO's gratuitous (and off topic) comment on AGW in the final paragraph of his otherwise useful article.
Gail, this statement by you appears utter BS, and I am choosing my words carefully.
I, and presumably many others, are very concerned about AGW regardless of any beliefs abut BAU. Mass extinctions, remember? We're just one species.
Gail,
I see you deleted my post and this response indicates that I think that you were dead wrong to do so. I have been part of this DB for almost 5 years now. I know what is appropriate and what is not. I was not rude or disrespectful. Just blunt. You were mistaken not I.
HO absolutely knew what he was starting and did it deliberately. He has done it many times in the past. He pushes an agenda and he has no evidence. You do us and TOD a disservice when you support it. He has no evidence to back up his AGW belief and neither do you. The facts are well known and more than well documented. The research is solid, the facts and the data have been checked more ways than you can count. It is there for anyone who does not have an agenda to look up, study and learn.
You blatantly fall into the same trap of HO. You think your expert ice gives you license to pontificate on any subject as if you are an expert. When called on it you think you are being attacked. But on the other hand what is the main tenet of this site. If you can't back it up you are talking drivel. And you are. And he was too.
There is no such thing as an anti-peak oil agenda in AGW issues. BOTH are serious concerns. The issue is not your issue (Peak Oil) competing with their issue (AGW). Let's not be immature about these important subjects. Peak Oil may mean the end of our civilization as we know it. AGW may mean the end it period. There is a difference there.
If you really think that Peak Oil trumps AGW you are singing in the dark. It is far more catastrophic in its potential. It is far more important than peak oil. Peak oil just means that our civilization has to adapt, shrink and learn to do with less. If, as that civilization draws down due to exergy issues, we end up in some desperate struggle to maintain BAU we will make the problems of AGW much worse. Not less. What is already baked in due to the current CO2 (and related gases) concentrations is going to take us to a bad place already. We are certain to make it much worse as our energy problems worsen.
Your statement that concentrating on AGW means that one wants to support BAU is just totally baffling.
The biggest weakness of TOD is this very subject. TOD is supposed to be a place where you provide evidence to back up what you say. Except if you want to play AGW denier. Then it is ok. That is BS and you should know it. You state that TOD does not take a "position" on AGW. This is also nonsense. There is no position to take. Back up your beliefs with evidence and we'll have a discussion. Throw out opinions garnered from Fox News and you get what you deserve.
It is amazing to me how many people fall into the trap constantly pointed out in the upper right hand corner of this page. A man has trouble understanding something if his way of life and thinking depend on him not understanding it.
WYO
If you will read my comment above you will find that my motives were not quite as sinister as you imagine. Rather mine is a general concern, that the current interest in the CRU data base allows me to highlight, and it was focused because of my retirement and encounter with "the system."
I actually do look at the evidence and am amusing myself by looking at some of the raw climate data at the moment. I do not post that here, but on Saturdays at my own site, which is Bit Tooth Energy . I even walk you through the process that I am using . I don't have an axe to grind, I am not (and have not been) paid by anyone either to write anything at this site, or that one. I am actually just curious and figure that others would benefit from the evidence that I find. It was the same sort of reasoning that started me writing here.
HO I think your trying to make out that your more neutral on the subject than you are, this is a comment in reference to your experiment on your blog spot
This is what John OSullivan says about Climate Change
John Sullivan
HO and Gail
I think you're both struggling sometimes with your neutrality position. HO article was useful up to a point then right at the end it appears that the whole article was to support a view that nothing should be done until there is certainty about AGW. Well the evidence strongly suggests it is taking place. If we're to take the same view towards AGW as PO as outlined by Robert Hirsch in his mitigation of PO, we need decadal lead in times.
For me, PO and AGW are both signs and symptoms of Overshoot in population. By the time we get to the stage of certainty on that one, I don't think anyone will really give a sh*t about storing raw data. But I also feel it is possible to ameliorate the situation if everyone got out of denial mode, whether its AGW or PO, and went to precautionary principle mode!
This just adds more evidence as to the position Heading Out clings to.
We are having pseudo scientific ideology masked as something else.
I hope you are right. We all have our different backgrounds/experiences/perspectives, and it is certainly annoyingly easy to get sidetracked away from useful discussion. A good hand on the rudder is fine with me, seeing how useless discussions can become.
"I personally think the direct and indirect impacts of peak oil will trump the impacts of AGW"
Herman Daly says otherwise.
Personaly, I think its impossible to call and it will be a close run thing. We have clear evidence that resource depletion and environmental degredation are happening simultaneously and this is the dilemma humanity has to face.
read the "Climategate" e-mails, before making generalized statements...
It must be understood that Whatevergate is a manufactured controversy, a weapon of political warfare deployed by the same fossil fuel industries that oppose the Peak Oil theory. TOD has felt only a taste of what this PR machine is capable of, because no government is seriously undertaking policy measures to deal with oil depletion. The moment that happens, prominent Peak Oil enthusiasts will feel the sting of Marc Morano, Sen, Inhofe, and Heartland Institute. Imagine Matt Simmons or Kenneth Deffeyes being targeted by Sen. Inhofe for criminal prosecution for their research.
I can hardly wait for the moment to happen. Just another challenge in these challenging times.
It will be funny when Inhofe asks to see the oil depletion data, and then we get to point directly at him. ha ha
But then again, it's well known that sarcasm does not work on these people.
Heading Out,
thanks for this enlighting (or rather overshadowing) posting.
When I was a student I once also had to clean up the office library of a deceased professor, and as far as I remember many of his books were to be thrown away. But these were all printed books or journals (of which by German law at least a few samples exist in a Federal Library), but no things like original protocols.
I may understand that documents are deleted a researcher may declare worthless by himself (e.g. because they became totally outdated) or because of physical decay. But this is certainly not the case for many research topics, e.g. related to climate change.
And a standardized erasure of scientific records sheds considerable doubt about the entire scientific credibility of the entire institution. And if such rules are valid on a national (e.g. US) level this would translate into:
DO NOT TRUST US SCIENCE BASED ON INFORMATION OLDER THAN 7 YEARS![embarrassement intended - I am embarrassed]
If this is really the case this calls for political action.
BTW:
Exactly. For example in about 1890 a German economist wrote a textbook on how to avoid a economical crash as we are experiencing right now (based on the experiences of 19th crashes - there were quite a few of them). Much damage could have been avoided if some of our folks would have taken this old book from the shelf and read it. Obviously they didn't.
Ancient (or current) knowledge is of no use to those who care nothing for the longer term, only tomorrows financial gain.
Back when I was writing, receiving and producing final reports on National Science Foundation grants, we had to:
1. Discuss our data archive plan.
2. Place relevant data into an archive with international agreed upon standards.
3. Show NSF that we had done so when the grant was complete (at penalty of not getting another grant).
This was true for genetic, phylogenetic, biological specimen, and weather-related data we collected. I can still go back and find my old data on open, publicly accessible and institutionally managed databases.
Here's the database for specimen records, for example:
http://www.tropicos.org/SpecimenSearch.aspx
But my raw computer files and sequence lab notes are indeed toast by now.
Greetings,
You said that you can't open many of the files you created in the 80's.
I am the author of this seminar on file formats which explains how general and serious the problem you mentioned is, and what are the reasons:
http://mfioretti.com/how-file-formats-can-be-used-favor-or-hamper-innova...
everybody is welcome to download the seminar (it has a CC license), but the main reason I'm mentioning it here is that I would really like to have a few more details about those files you mentioned for the next edition of my seminar.
If you can, please let me know either here in the comments or by direct email at mfioretti, at nexaima, dot net. Thanks!
Best Regards,
M. Fioretti
xbm - X BitMap
xpm - X PixMap
ppm - Portable Pix Map
tga - targa
pcx - Personal Computer eXchange for PC Paintbrush
I involved some very well-known scientists in an analysis of black-market product samples some years back, and the results were highly newsworthy. However, after the analysis was done and published, the scientists strongly pressed for the original samples to be destroyed. Since we wished to (and subsequently did) offer the same samples to international bodies and anyone else who wanted to replicate or disprove the analysis, I refused to either destroy them or give them access to the samples again. I have no reason to doubt that their rigorous analyses were correct, but my personal standards of accountability and chain of evidence apparently rubbed them the wrong way. Once making news with our project, they stopped working with us and did it without our involvement with the money and offers that came their way, which were many.
Just one item in a list of several dozen aspects of the culture of academia that fail to impress me.
I recall reading a long time ago that much of the data from the Landsat program is just sitting on magnetic tape having never been seen by a single human being. There was not enough in the NASA budget to hire enough people to keep up with the high rate of data coming in. Much of this never analyzed data is about the distribution of vegetation types back in the early 80s and would be valuable in showing how much the climate has changed since then and how it is affecting plant life.
This is the normal state for satellite data. There is money to feed the private sector hardware manufacturers but no money for academics to make use of the product. It seems the funding agencies are staffed by politically appointed hacks. The same BS goes for model work. The question asked to the proposal writer is "why do you need so many people to run a model". You could get a monkey to run a model. That is not the point. You need people to analyze the output.
There is no money for people and hardware to transfer data from tapes to DVDs or whatnot. So the tapes sit there an decompose.
As to data retention, I can see that a private company that employs you would have a real problem about someone walking away with saved information/data, and that you might risk prosecution. However, universities are supported to varying degree with tax dollars. The wages of research are partially supported by tax dollars and/or tuition. Surely, there is an argument that such knowledge is owned by the commons.
I would think that a researcher who has spent much time and effort would make copies and retain the means to view such as a tribute to their life and efforts, at the very least. To hell with the rules.
My basement has stacks of boxes containing data and computer printouts from my university research days. Every now and again I wonder why I'm keeping it - nobody is ever going to be interested in it (and indeed there are boxes of tapes and disks that probably can't be read anymore). But I don't throw it out yet - as you say Paulo, it seems to be some remaining artifact of all that work and effort... The university of course has long forgotten about it except as a list of publications somewhere (perhaps...).
The problem with the archival of digital data is well known within the IT industry, has been for decades but is largely ignored by users. There are no good solutions. Typical hardware cycles are 3 years, so within 3 cycles often there is no hardware available to read a particular media. Often the software also does not exist any more.
1. Keep data in the most open, basic, human readable format. This essentially means plain ASCII text, and one of the reasons I like wikis.
2. Keep the data in a live archive which is constantly migrated to the current platform.
3. Print it out on acid free paper if you want to keep it more than 10 years.
And note. A backup is not an archive.
There are professional archival companies who can manage the process for you but as mentioned, most institutions and computer users simply don't care.
Well, there are limits to everything, see a comment from 80,000 AD in Donald Kingsbury's novel Psychohistorical Crisis:
" 'There are a hundred quadrillion people out there all writing their memoirs and taking cubes of their newest baby, and you expect everything to be linked with everything else, and instantaneously, all the way back to the cave paintings of Lascaux? That a student's life should be so easy! The Galaxy is a vast place,' he said tritely."
Documentation. Documentation. Documentation.
Diminishing, vanishing, human kind forgetting its history. Its like civilization itself has reached some sort of limit - too much systems complexity - law of diminishing returns - or just plain shortsightedness: greed and stupidity...
I can only offer my sympathy and join the angst of others here who have voiced their grief about this issue - having to live under the trolls who seem to control everything nowadays...
Here's my story.
Our lab maintains the standards for SI-units for aviation - meaning we're a calibration lab - an accredited one - just below the national level. Actually we have some units which are better what the national level has to offer so we compare those to ones internationally. Anyway - thousands of pieces of equipment pass through our hands every year - for decades they have - and we have kept records of everything: how the SI-Units were copied to us, verified and compared, uncertainties calculated for each unit and process, re-checked, tools, instruments and work processes carefully documented, audited and accredited - all the way down to the actual tool being used outside the lab in the real world.
I recently inherited this mess from my predecessor who had been the main architect for creating much of the system along those decades. I have magnetic tapes and 8 inch floppy disks I need special antique equipment to read, with obscure database and word processing files from the 80's on them. Not to mention bookshelves full of paper files from the age of the 'paperless office'.
Fortunately its not all so bad. For the past decade we've been able to maintain a migration/update cycle so that virtually all the data from the age of the first windows word processing environment and sql-databases is now running safely on a modern database server - all converted, indexed so that you can honestly call it an 'archive'.
However the dark clouds gather. People are getting complacent, standards are being lowered (not a good idea in a 'standards'-laboratory .. poor pun I know sorry) and everyone seems to have suddenly forgotten why we do everything the way we used to: "why is this instrumentation required?", "why do you have to produce this and that documentation for this process", "why do you have to perform so many checks, gather so much data, compare so many units etc." - these are the question the trolls keep asking me - and really they're not interested in my long technical explanations - all they see in our department is a huge money hole that 'doesn't produce anything' (as if that wasn't so ironically true of their own department - of course in their minds they do 'produce' something - something every organization seems to value the most these days: 'savings').
Lets say we design a new system, instrumentation, directly related to flight safety - whose purpose is to fulfill the requirements set by the aircraft manufacturer, and ultimately aviation authority and law. The design, manufacturing, calibration, verification all need to be documented, the documents archived - archived in a way that they they leave a permanent record - a record which at any time can be examined and analyzed with ease. After all, the instrumentation has an expected life span of decades. And the instrumentation produces data - all the time - over years, decades. This needs to be archived - preferably in raw format directly to a database, from where the whole of the data can be analyzed numerically with a computer - a paper record, a print slip won't allow you to do that!
These are the things we need to do from time to time. Yet our department can't have funding for additional equipment or personnel for "documentation" - "because that's not your core-competence", we are told. And we know that getting an outside company or consultant to come over to do the job is pointless since what we do is so specialized that anyone wouldn't do it for less then half our annual budget. So we have meeting after meeting with the trolls - and they continue asking me "why does this instrumentation need to be documented so thoroughly", "after all we're not 'aircraft designers'" and "why do you want to store all this data, if the purpose of the instrumentation is just to calibrate flight instruments - what do you need the data for afterward if the calibration is OK?" - These people have no experience or education in engineering or any kind of real science - yet they are the ones in charge...
And every time one of our engineers retires, his papers are carefully boxed up and sent away - so that the 'replacement' - a new fresh engineer straight from college arriving few months afterward - can sit on a shiny empty desk - and start over - everything - decades of experience, learning from trial-and-error, mistakes, discoveries - all gone - because the trolls at personnel place no value, can't even grasp the concept of 'knowledge' - they care for things they can count: desks, people, beans. Meanwhile the organization that ones created the whole system of keeping track of the most fundamental units of physics in aviation - slowly corrodes, forgets away its basic knowledge - and what is left is a bunch of 'technicians' who know which buttons to press of the old machinery, left behind by the mystical masters - who actually knew how all of it worked and why...
On an above post jjhman's anecdote about the magic hole on a piston describes my position - except I'm surrounded by such holes - and every time the trolls find one, we have to just give up and take it out for them - because either we have no documentation about why its there - or time or personelle to dig out the reason why its there - i'm just waiting for the day that something crashes because of this new mentality.
Intriguing, can you give us some exampels?
Question: What are the chances an infinitesimal (.04%) trace gas (CO2), essential to photosynthesis and life on this planet, is responsible for runaway Global Warming?
Answer: Infinitesimal
The IPCC agrees. See the IPCC Technical Report section entitled Global Warming Potential (GWP).
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket."
— Eric Hoffer (The True Believer: Thoughts on the Nature of Mass Movements)
What's the chance that breathing a mere 450 ppm (.045%) in air of phosgene will kill you?
Thanks for acting in a typical alarmist fashion by comparing an inert gas with a chemical weapon.
At least you didn't resort to lewd language as with most posts of warmists.
In any case the real culprit to global warming is the greenhouse effect, and that is caused by water vapor. If you are uncomfortable with currently warm planet earth you should lobby for cap and trade laws for steam.
No, Helium is an example of an inert gas. Get your facts straight, pipsqueak. Or is that the helium talking?
:) :) :)
In any case the real culprit to global warming is the greenhouse effect, and that is caused by water vapor.
It's true that water vapor contributes a larger part of the radiative flux than CO2 (about 75 W/m2 versus 32 W/m2), but the important difference is residence time in the atmosphere, which is about one to two weeks for water vapor, versus one to two centuries for CO2.
The CO2 merely has to act as a catalyst, big boys like the sun and water will do any heavy lifting. Small amounts of catalytic substances can hurry along big reactions.
Picture an old pan balance with a huge and endless stream of sand breaking over the fulcrum and splitting evenly (on average) into the two pans. Inhabitants of the pans have been shuffling the sand around in them with little effect for a long time but now one creature has begun to whittle at the fulcrum. After a while a little bigger notch gets whittled from one side than the other so more sand flows to that side and begins to wear it even faster. More sand then follows and wears it faster and now significantly more sand falls on one pan than the other. The balance will change, the pan getting more sand won't be able to hold signicantly more than the other since they were both overflowing to start with but it will keep a little more sand a little longer and sink a little farther before the pans reach equilibrium with the new flow distribution. With luck the increased wear on the fulcrum will spread across the whole top, the notch will disappear and the system will balance not far from it original point. But if the whittlers keep up or increase the whittling of the bigger notch the sand will groove it ever more deeply, the flow will substantially change and the balance point will be altered a lot more and a lot more quickly than if the whittlers had desisted or better whittled the opposite side to match the more worn side of the fulcrum. Anthropomorphic carbon release seems to be a lot like whittling one side of the fulcrum.