US Minerals Databrowser

Jonathan Callahan, a PhD chemist who has worked for almost 20 years as a data analysis/data visualization expert for various Federal science agencies (NOAA, NASA, EPA), has created a new data browser, called US Minerals Databrowser.

Many of you are familiar with Jonathan's Energy Export Databrowser, which he created in 2008, showing imports and exports of energy products by various countries around the world, based on BP's Statistical Review.

The new US Minerals Databrowser uses as its base the minerals data set found in US Geological Survey Data Series 140. This data set provides consumption/production, import/export, price and usage data for a large number of minerals used in manufacturing, construction, and agriculture. Oil Drum readers may be interested in this data set, because many of the minerals are ones which have been described as being nearing or past their peak, or ones for which import dependence is an issue.

The data, when it is complete, can go back as far as 1900. The most recent year varies, but is often as recent as 2008 for some data elements. The Databrowser shows up to five graphics for each mineral:

1. US Production / Imports - This graphic also shows US consumption and exports.

2. World Production / Price - This graphic also shows US production.

3. Price Evolution - Displays price vs. world production, the way an economist might look at it.

4. Usage history - (Only for items without an *) - Shows how consumption by product has varied over time.

5. Usage pie - (Only for items without an *) - Shows consumption by product for the most recent year available (often 2003) as a pie chart.

More detail describing these charts and their interpretation can be found on the About page.

Many of us use Jonathan's Energy Export Databrowser very regularly. This new tool promises to be very helpful as well, when questions come up about a mineral, and wether production is likely to hit a peak in the near future. It is also helpful to know what a particular mineral is used for, and how that has changed over time. To give an illustration how this databrowser can be used, I put this post up the same day that a Campfire post on potassium (from potash) and phosphorous (phosphate) is up.

Jonathan tells us he would like constructive feedback on his new databrowser. So give him your thoughts in this thread, or contact him at jonathan.s.callahan [at] gmail [dot] com.

Jonathan, this is a very helpful dataset. Any idea if 'reserves' information about these minerals is available or planned for the MineralDatabrowser?

It's great to be able to pick any mineral (e.g. titanium as one example) and see the price, production, and consumption trends over the years, especially by individual industries.


There are no data on 'reserves' in the USGS DataSeries 140. This dataset forms part of the USGS Commodity Statistics and Information (minerals as part of the economy) and is compiled from industry surveys. Given the data sources, this dataseries is more focused on economic than geologic information. The description of this dataset states:

The USGS collects data on a monthly, quarterly, semiannual, and annual basis from more than 18,000 minerals-related producer and consumer establishments that cooperate with the USGS. These companies voluntarily complete about 40,000 canvass forms that survey production, consumption, recycling, stocks, shipments, and other essential information. Data are also gathered from site visits, memberships on domestic and international minerals-related committees, and coordination with other government organizations and trade associations.

One comment I would like to add about 'reserves' information in general is to remind readers that reserves are modeled rather than measured. In contrast, the data in the USGS DS 140 are all actual measurements (survey responses) that are compiled to get the final numbers we see. Yes, some heuristics may have been involved but these numbers are about as close to actual measurements as we will ever get. I believe the error bars in the measurements to be relatively small -- under ten percent, say.

Any numbers on reserves, on the other hand, are estimates or, to put it unflatteringly -- 'guesses'. Good guesses by experts in the field no doubt but guesses nevertheless. Historically, the error bars on those estimates are difficult to quantify.

Reserve numbers are of course important but I would argue they are not as useful as the information that comes from historical trends. Reserves estimates, even if highly accurate, do not take into account very important human factors like technology, economics and politics that are embedded in the production and consumption data.

Despite my concerns, I would love to have access to a good compilation of reserves numbers from a reliable source. There is certainly much to be learned by examining those numbers and if anyone is aware of such a compilation please let me know.

-- Jon

If you did choose to show reserve projections (if you were satisfied with the obtainable data), would you show a future depletion plot as a straight line, logistics curve, or other with default caveated error bars, say 25%?


You are putting me on the spot. ;-)

The BP Statistical Review contains data on reserves yet the Energy Export Databrowser very pointedly leaves out any analysis using reserves data. As regular readers are not doubt aware, there are serious issues with the reserves estimates for oil and natual gas -- especially in OPEC nations. I have no confidence that any presentation of those reserves numbers would help people make informed decisions about the future. Under 'people' I include everyone from interested individuals to industry executives to investors and politicians.

To answer your question with respect to minerals data: if we did have 'satisfactory' data on mineral reserves I can imagine presenting current global consumption as part way along a curve -- either Hubbert or Gompertz. In general I think it is inappropriate to use these statistical approaches for individual nations where 'above ground' factors can be important. But I believe they are at least another tool in analyzing the problem when working on a global scale. (I have done some error analysis of Hubbert Linearization using data from various individual nations and found the results to be disappointing.)

The US Minerals Databrowser is aimed at an educated but still general audience where I believe a simple display of existing trends is most effective. At present, I have no plans for including statistical analyses as separate products.

However, if anyone were interested in funding a tool for interactive application of statistical models to these data I would be happy to talk to them.

-- Jon

Coal reserves numbers are chock full of uncertainty as well, as Heinberg's work shows. Would be nice to have a roundup of these WAGS for resource size that are bandied about by many as solid fact.

Kudos from me as well. Very slick.

I am suspicious that prices don't follow the pattern that economists would predict--prices don't necessarily rise with shortages. It seems like this data set might be helpful in starting to look at this issue.

This may be at least partly due to the huge array of subsidies - and biased subsidy statistics.

More details in a new report:

I love this database. It's very convenient. I used to google the info. and search for it the hard way when trying to explain my position on thin film solar panels. My concern is that they are made with elements that will soon peak and will cause the price of thin film PV to soar relatively soon.
Telluride, Gallium, Indium, selenide..., etc
Some closer to peaking then others.
It makes no sense to me to get so carried away with thin film solar when they are made with such rare metals.
I prefer traditional PV, though it's diffuse, but it's too expensive.

For that reason, I prefer solar thermal, and wind. But I do not expect miracles. I expect a reduction in the amount of high quality/useful energy.

I know I'll probably get allot of nasty responses over my concern of depletion of rare elements used in thin film solar.

I always do. But, I still think it's an issue.

leduck, not from me. I am also concerned. We should be concerned about the excessive use of any limited material even if it is called a rare earth element but is not rare. Everything on our planet is limited of course and even things we thought couldn't be used up are being used up. And the more we have to turn to those minerals and elements that are plentiful but more dispersed the more energy we have to use to extract them. I hate this rush to replace oil with solar and wind without thoroughly looking at all the issues related to doing that. You cannot compare doing something in a small way to doing it in a large way. You cannot compare the effects on the atmosphere when small amounts of CO2 were being released to the effects when large amounts are being released (and may trigger positive feedback effects). You cannot compare drinking 8 cups of water a day to drinking 8 gallons of water a day. One will make you healthy the other may kill you. You cannot compare the atmospheric effects of a small number of windmills to what might happen if we supplied most of our electricity with them. That experiment hasn't been tried. Same with solar. All that sunlight was doing something else before being captured by solar panels. What happens if we use some solar panels may be very different from using enough to supply most of our electricity. Well I sort of shifted subjects in the midstream. So be it I think this is important.

For that reason, I prefer solar thermal, and wind. But I do not expect miracles. I expect a reduction in the amount of high quality/useful energy.

Why do you exclude nuclear?

I didn't mean to exclude nuclear. But I still expect miracles even with nuclear.

Tellurium is genuinely rare. The rest are junk metals they don't bother to refine because there is no demand.

Actually in one of the threads here on peak minerals - maybe it was the one called "Peak Minerals" - everyone was hemming and hawing over the scarcity of this and that, including the elements needed for CIGS; some commenter came down very harshly on most of the conclusions being drawn, saying that all that would be needed for gallium production to bump up was for Al miners to slap on $50 million bucks of additional kit. The reserves numbers were wholly irrelevant to the available supply, gallium being a simple sideline to aluminum extraction.

You need to keep this in mind at all times, often there's a simple solution to meeting resource needs that people will be wholly unaware of, whether out of ignorance or not.


I'm glad you pointed out that October, 2007 TOD article titled "Peak Minerals". That article, uses the same data (not sure which lead data they are using), applies a logistics model and finds a bunch of 'peaks':

After an exhaustive examination of the USGS database of the world mineral production (Kelly 2006) we found at least 11 cases of minerals that show a global “bell shaped” curve with a clear peak. Peaking was evident by visual examination and it was confirmed by fitting the data using a bell shaped function.

The value of the databrowser is that it allows you to do an "examination of the USGS database" without the exhaustion. Were a similar article published today, any reader would be able to check the facts on their own to see if they agree with the author's conclusions.

In the case of mercury, as mentioned in the "Peak Minerals" article, I would heartily disagree that the bell shaped curve is a sign of running out of mineable resources. The 2009 USGS Minerals Yearbook report on Mercury certainly makes no such claim. On the contrary, they state:

The declining consumption of mercury, except for small-scale gold mining, indicates that these resources are sufficient for another century or more of use.

The mercury curve is defined more by economics than by geoloogy as shown in the "Production Price" plot:

Decreasing production was accompanied by a decreasing price. Not a sign of scarcity in my boook!

If you want to see another near perfect "Hubbert" curve try looking at asbestos. Is it geological constraint? I don't think so.

My argument with modelers is that once they develop a set of theoretical and computational tools they try to use those tools on everything in sight. When no humans are around, many phenomena are subject to statistical modeling. But once we become involved it becomes equally important -- not more just equally -- to examine the human side of the equation, to tell the economic, political and environmental stories that play such a large role in our use of resources.

Best Hopes for careful application of statistical models.

-- Jon

This is a great tool, and thanks for putting in the effort to make it available. I did a quick assessment and found that of the 84 minerals listed, the U.S. is exporting only 19. But then if you look at asbestos, there is an "inverse" Hubbert Peak with the largest imports running from ~1950 to 1980. Now in 2010 internal consumption has dropped and we're actually exporting. No doubt this is due to regulations banning the use of asbestos in many products. So, there's a lot more to be "mined" here and we really do need to consider why a particular mineral is being imported/exported.

I agree. In my opinion, most of the problem has to to with the way people treat their "models". In most cases, these are not models at all but merely heuristic functions that someone may have at one time claimed "fit" some class of data. Through the passage of time, the modelers forgot that these heuristics actually did not explain anything at all.

For "Peak Anything" most Bell-shaped curves fall under this category. In finance and economics, the same thing happened recently when all the quants on Wall Street blindly used normal classic Bell-shaped statistics when it clearly did not apply. All the financial insurance (tranches, CDOs, etc) built up on this basis caused everything to start to collapse like a row of dominoes.

I read this 70 page document for fun yesterday -- "WARNING: Physics Envy May Be Hazardous To Your Wealth!"
This points to the other problem with modeling. You have to figure out the difference between statistical variability and uncertainty. Under real uncertainty, like the example of mercury prices, you might as well give up trying to understand what is happening.

For another site the other day I dug up a paper whose intro went over some of the terrifically wide of the mark calls economic forecasts have made in the past, it was quite the trail of bodies, yet people still look to the auguries. This is some fundamental human weakness, obviously, since we have the auguries, auspices, and oracles of the Classical world, with their bizarre portents.

Mercury was mostly used in thermometers, hence the massive declines; doesn't look like CFLs or amalgam have brought back its popularity. Looks like it was behaving like a good economic model in the 40s, with a price spike followed by boosted production - war effort? Wonder how those classic laws of supply and demand have degenerated over the last century with other resources. If you chart oil supply addition for the world over the course of the BP data (back to '65) you can clearly see how the market just can't be flooded like it was in the good old days, and the trend is manifestly down - yet this simple enough demonstration is hardly known about for some reason.

You have to figure out the difference between statistical variability and uncertainty.

My business partner is a statistician and she has spent a tremendous amount of time teasing these two apart and then explaining to clients exactly what is variability and what is uncertainty. The difference between these two is absolutely key and is poorly understood by most.

Very Cool!

For data buffs these two TED talks will be like lollies.

changing practice of data mining in an internet age

Hans Rosling digs deep


You might be interested in what Bill Gates has to say on zero carbon energy.
new generations of nuclear

With infinite energy from a blue sky technology such as Nuclear, the planet becomes an infinite source of material, with the down side that we choke on our own pollutants.

jonathan, do you have any additional information on the "balancing items" shown on ng production - consumption ?

the doe has been looking at this of late, but afaict it amounts to a catch all term seeming to say "we don't know". also, afaict,there is no correction for the compressibility of ng in transit.

Not sure if you saw this. I posted it in an earlier drumbeat as the apparent source for the Financial Sense post in response to an earlier post of yours.

Mark Papa CEO of EOG Resources

“We look at the 914 data and we try and tie that back to the IHS data [IHS is a consulting firm] and we really can’t tie it back and it looks to us like the 914 data is just consistently overstating particularly in the other states category. The second item we look at in the 914 data is just the balancing item and the balancing item seems to have grown over time. So we have tied our internal models to the IHS data and even though the gas rig count has gone up considerably over the last 4 - 5 months what it tells us is that the production is still going to be down to the tune of about 3 bcf a day relative to December 08 throughout all of 2010
-Mark Papa

EOG Resources, Inc. is one of the largest independent (non-integrated) oil and natural gas companies in the United States with proved reserves in the United States, Canada, Trinidad, the United Kingdom and China. EOG Resources, Inc. is listed on the New York Stock Exchange and is traded under the ticker symbol “EOG.”

At December 31, 2009, EOG’s estimated net proved natural gas reserves were 8,898 Bcf and estimated net proved crude oil, condensate and natural gas liquids reserves were 313 million barrels for a total of 10,776 Bcfe. Approximately 75 percent of EOG’s reserves on a natural gas equivalent basis were located in the United States, 16 percent in Canada, 9 percent in Trinidad and less than 1 percent in the United Kingdom North Sea and China. At year-end 2009, EOG had approximately 2,100 employees

thanks for responding. yes, i saw that post and couldn't deciper what papa was talking about. i can't even find any reference to "914 data" in the monthly ng reports.

papa seems to be making the claim that ihs data is more reliable than doe data, that may be, but i have found cases where ihs data was bogus.

i dont see balancing items growing over time, what i see is the balancing item being negative at the start of heating season and positive during "storage" season. the amplitude of "balancing items" could be growing. this could be explained by the compressibility of gas in the gas transmission system(sometimes refered to as "packing" of the line).

thanks for responding. yes, i saw that post and couldn't deciper what papa was talking about. i can't even find any reference to "914 data" in the monthly ng reports.

Description of Survey
The EIA-914 survey collects natural gas production volume information on a monthly basis from
a sample of well operators. Production volumes are requested specifically for Texas, Louisiana,
Oklahoma, Wyoming, New Mexico, Federal Offshore Gulf of Mexico and all Other States
(except Alaska). Two volumes are requested from respondents:
(1) “gross withdrawals (wet),” which is full-bore well stream gas minus lease condensate,
oil, and water; and
(2) “natural gas lease production,” (sometimes referred to as “sales production” or “gas
available for sales,”) which indicates the net amount of produced gas that leaves the lease
to go to natural gas processing plants or directly to end-users.

Decoding I think what he simply said was "Our own internal models (and those of others) disagree with the EIA reports and say that real production has dropped 3 Bcf/day compared to December 1998. I note he doesn't say whether he considers the absolute EIA reported production in Dec 2008 to be accurate as he just talks deltas.

The only information I have on "balancing items" is the definition of the term in the Explanatory Notes Appendix of the EIA's Natural Gas Monthly:

Note 7. Balancing Item

The balancing item category represents the difference between the sum of the components of natural gas supply and the sum of the components of natural gas disposition. These differences may be due to data reporting problems or to issues in survey coverage. Preliminary monthly data in the balancing item category are calculated by subtracting dry gas production, withdrawals from storage, supplemental gaseous fuels, and imports from total disposition. The balancing item may reflect problems in any of the surveys comprising natural gas supply or disposition. Reporting problems include differences due to the net result of conversions of flow data metered at varying temperatures and pressure bases and converted to a standard temperature and pressure base; the effect of variations in company accounting and billing practices; differences between billing cycles and calendar periods; and imbalances resulting from the merger of data reporting systems, which vary in scope, format, definitions, and type of respondents. Survey coverage problems include incomplete survey frames or problems in sampling design.

Annual data are from the NGA. For an explanation of the methodology used in calculating the annual balancing item, see the NGA.

Collecting data by survey and then bringing all of that data to truly comparable standard units is not easy and we should be thankful that government agencies take their work seriously in this respect.

Being careful takes time and attention and money

-- Jon

Collecting data by survey and then bringing all of that data to truly comparable standard units is not easy and we should be thankful that government agencies take their work seriously in this respect.

Being careful takes time and attention and money

In this case though Jon we have a specific allegation initially made by Matt Simmons but now also by others. The allegation made by Simmons is that beginning under the last administration the EIA was instructed to show a certain ongoing year-on-year rise in US NG production. They were told what the figures had to show and instructed to adjust models accordingly. Later claims say one of the ways this is done is by cherry picking a non representative sample for 914 survey submissions.

I'm just reporting what's been said...


I just got of the phone with a friend of mine in DC who used to work for the EIA and stays in touch with folks there. They were quite frankly flabbergasted by the claim.

They suggest that anyone interested in this topic would do well to actually read some of the EIA documentation like the Form EIA-914 Methods if they want to better understand how this survey is conducted.

They also doubt that Matt Simmons made the allegations as you have stated them. Do you have a quote from Simmons?

Like you, I'm just reporting what's been said.

-- Jon

He makes the claims that the US Natural Gas Glut is not real frequently in his slides and talks and links are usually posted to them in the drumbeat. However the wording I used above is my recollection/condensation of what I believe he is saying and I should have made that clear. There's another interview somewhere where he talks about this specifically (might be the audio that goes along with these slides) but I'll have to try and track it down again.

Here's a couple of his slides

Note that EOG Resources is listed as a dubious "EIA superstar" by Simmons and low and behold now we have the CEO of the very same EOG Resources saying that the EIA is overestimating production. Small world...

I'd imagine that if this could be done it would not require the knowledge of the vast majority of EIA employees working on the data. Although they might well have worries about the accuracy. Just what are the true error bars on the reported figures anyway +/ 1 percent? 2 percent? 5%? Anybody know?

By the way do you know if these folks at the EIA are also shocked that people question their projections for anything. I've posted these before but here's how the EIA thinks things are going to go (from an earlier post of mine)

Here's what the EIA says will happen EIA Annual Energy Outlook 2009 (with projections to 2030)

Here's the three case world liquids supply summary

Click for larger image
And here's US oil production

Note the "Hubbert's Valley" :-)
And Nat Gas

And coal

Total Energy consumption mix

Party on Dudes!
Oh, yes and here's CO2 emissions

Yep they carry on rising. Maybe not party on dudes then...