Why isn't The Oil Drum indexed by Google? UPDATE: We are!
Posted by Super G on September 30, 2005 - 10:20am
Update [2005-9-30 10:20:36 by Super G]: Progress has been made! Now a search of site:theoildrum.com brings us up. We are also indexed by Google Blog search. If someone out there helped us out, thanks!
My website, www.theoildrum.com , has been live since August 20 but it does not come up in Google searches.We have plently of incoming links. According to our logs, the site gets crawled by Googlebot daily. Further, I am participating in the Google Sitemaps program.
I do have a robots.txt file to exclude certain admin pages from being crawled, but I believe it stills permit the crawling of all of the content pages, because those content pages are getting hits from Google's bot.
Here is the contents of my robots.txt file:
User-agent: *
Disallow: /~
Disallow: /my
Disallow: /user
Disallow: /poll
Disallow: /search
Disallow: /newuser
Disallow: /comments
Disallow: /?op=Any ideas?
This was their reply:
Thank you for your note. We understand your concern about your site's not appearing in our index. Your page has been blocked from our index because it does not meet the quality standards necessary to assign accurate PageRank. We cannot comment on the individual reasons your page was removed. However, certain actions such as cloaking, writing text in such a way that it can be seen by search engines but not by users, or setting up pages/links with the sole purpose of fooling search engines may result in permanent removal from our index. Please read our Webmaster Guidelines at http://www.google.com/webmasters/guidelines.html for more information.Regards,
The Google Team
So I came back with:
To whom it may concern:I assure you that our site complies with Google's webmaster guidelines. We do not use any SEO techniques. I invite you to verify this by inspecting our website at http://www.theoildrum.com .
Our site is a weblog that uses the Scoop platform. Several major websites, such as Kuro5hin and DailyKos, use this platform. They have no trouble being indexed by Google.
We lauched our domain in August, and redirected traffic from our old webpage on Blogger at that time. We, like many sites, participate in a blogroll provided by Blogrolling.com. This program provides ~50 inbound links. An acquaintance warned me that an influx of links to a new domain could raise a red flag with Google, but I saw no mention of this policy in the Google documentation. Is this a possibility? If so, what can be done to rectify the situation?
I look forward to your response. Thank you.
Their response this time:
Thank you for your reply. We appreciate your efforts to comply with Google's quality standards. We have sent your request on to our engineering team who will review your site to determine if it is eligible for re-inclusion. If the website is reinstated, it will appear in Google's search results sometime in the next few months. We appreciate your patience during this process.Regards,
The Google Team
So, they say we will appear in Google's search results sometime in the next few months! I think we can do better than that. Are there any Google insiders among our readers that can help us out?
Contact
- Content: editors at theoildrum dot com
- Tech support: support at theoildrum dot com
License
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
PS: Does your site "ping" weblogs.com etc on every new post?
I read that Google Blog Search collects data from weblogs.com. We'll see if their ban on our site extends to the blog search.
a manual review of the site should prove that the site is not a spamblog ... but a source of real content.
check this article out. it was written by matt cutts, google engineer. topic: file a reinclusion request
That reinclusion request link was very enlightening. Thanks!
According to some the comments given on that page, it seems as if our reinclusion process is underway. But it's good to know there's an automated way to initiate it.
(x contains all urls):
->> internal = [u for u in x if not u.startswith('http')]
->> external = [u for u in x if u.startswith('http')]
->> really_external = [u for u in external if 'theoildrum' not in u]
->> len (internal)
83
->> len(external)
54
->> len(really_external)
35
->> len(x)
137
Total URLs on page: 137
Total URL's referencing your own site as http://www.theoildrum/: = 54-35 = 19
Total relative (on site / style urls): 83
Total external URLs: 35
Note this doesn't count the javascript-created links in ads, google may parse them out although I doubt it.
I can't figure out where to email/post on google
tho i saw this for adding URL
http://www.google.com/addurl/?continue=/addurl
- lorax73
Let's hold off on the pile-on for the time being.
It allmost seems they deliberatly block Peak-oil websites:
"We cannot comment on the individual reasons your page was removed." ...........
If they think you are spamming or trying to trick them into getting a higher ranking, then they will refuse to index you.
But still, I got the same problem and am still not able to solve it..
i hate it when Google bans the wrong type of site and then won't let you buy your way back in ;)
It might be the redirect from the old site. Is this a forced redirect, ie, it automatically takes the person from that old site to here? If so, you might want to put a link on the old site saying "click here" for new site.
Blogrolls are fine, Google shouldn't care about those. Every blog has them. They arer more concerned with circular links, i.e. several mostly identical sites linking to each other. You're not doing that either. They consider this to be spamming them.
Other stuff to look for and change
text that is almost the same color as the background
itty-bitty text
Dear Mom, please check out this great website... :-)
I really do wonder if it was a typo that got theoildrum on the list of misbehaving sites because of oildrum. It would be an easy mistake to make.
I may get around to fixing it eventually, but I suggest you use Firefox instead.
Hi there,
Any new site launched is going to take time to rank well in Google these days. I usually tell people it will take 6 - 12 months before you see any action. And as you have changed your site's domain name, you are effectively starting from scratch. I have changed a few sites to new domain names and it is a very painful process.
Looking at your site it does look like there is an issue as there are no pages indexed by Google which is strange. So hopefully they fix this. That being said, It will still take time for you to get the new domain up and ranking like the old site probably was. To rank well in Google, requires two main factors:
1. content / site optimisation
2. lots of incoming links
1. Site optimisation
You need lots of content related to your primary keywords and updated often - which you have in spades! You also need the right keywords within key areas of your pages to help establish your overall theme. Keywords in the page title, domain name, heading tags (h1) are most important. Looking at your page code I have a few small recommendations assuming that "peak oil" is your primary phrase:
Title: move "peak oil" to the start of the title tag
H1 Tag: make sure "peak oil" is in this
Masthead Image: change the alt tag to include "peak oil"
There is some HTML commented out within the head tags of your pages - I would remove that as it could be considered a spam technique by Google
2. Incoming links.
Links are king. Links determine your page rank and page rank (to a degree) determines how well you will rank. In fact, links can alter how a site ranks irrespective of content. E.g. Do a search for "miserable failure" OR "failure" in Google and George W Bush comes up #1!! This is simply because a few people linked to his site many times using these words in the links. As you now have a new domain name, most of your links are to your old site.
Here is your link status and page rank:
www.theoildrum.com
Google - 0
MSN - 3972
Yahoo - 4730
Google Page Rank: 0/10
theoildrum.blogspot.com
Google - 2880
MSN - 5736
Yahoo - 22900
Google Page Rank: 5/10
you can check links at http://www.marketleap.com
Note: Google never reports all links it knows about and only updates its link count every few months.
So, my recommendations here are:
1. try and get as many of the sites linking to your old domain to change this to your new domain. This takes time so you should focus on changing the quality links. These are links from pages that have a good page rank e.g. anything over 3/10 (not on the hope page but the page your link is on).
If you use Firefox, there is a great little extension called "search status" that will show you page rank and alexa rank.
Also, if you can, make sure that the words "peak oil" are in the link text to your site as this will help your ranking for that term. Make sure that your links have some variety. If they are all the same, this will look artificial to Google and you may incur a penalty.
Use this in the search engines to find links to your old site: "link:theoildrum.blogspot.com"
2. If you can set it up, you can try a permanent redirect from your old site to the new one. This is called a "301 redirect" and may help.
Final Note:
There is also a suspected "sandbox" that new sites can be caught in which can last for many months as well. It is believed that if the rate of new links appearing to a new site is large and sudden, this can trigger a penalty in Google. Others think there is an aging filter that applies to new links. Its probably a bit of both.
Hope that helps...
Unless it's a blog. 6 - 12 hours is slow to be indexed on google. Maybe they are sad that you ditched Blogspot? (a google company)
I started getting google hits the day after I started my blog from scratch.
I wanted to discuss sustainability issues (including peak oil), a completely seperate topic to natural foods, but couldn't afford a new hosting, so I bought the facility of cloaking http://www.sustainableliving.info onto the naturalhub site.
I was unaware google considers this 'illegal'!
Since 'cloaking' to this site, and submitting the 'cloaked' name, my sustainableliving pages have stopped being updated. Thats not so bad, but the URL given in the google results always resiles to a very old google cached version from when I first put the site up.
So the link that google gives is as if it is my page, but is in fact googles ancient cached version.
I consider this at least immoral.
I have emailed google and had zero response.
My page now carries the tiny URL address
http://tinyurl.com/72myz
and advises people that the google cahed version is well out of date - in other words, go back to bookmarks, don't rely on google, google may well mis-inform you!
If this trend increases, I can see google shooting oyself in the foot. People will understand it is unreliable, and go back to bookmarks and portal pages.
As for solutions, I don't know. Google has the power, and is misusing it.
http://blogsearch.google.com/blogsearch?hl=en&q=theoildrum&btnG=Search+Blogs&scoring=d
Make sure the redirect from the old site to the new one is a 301 (permanent redirect)
Changing domains can be tricky with search engines, and sometimes, things go wrong for no apparent reason. Usually related to the search engine thinking it's duplicate content and banning one of them.
Linking to wrong people can cause problems, but as your problems began with the domain change, I think it's related to that.
It's fairly common to hear about problems with domain changes (and with new sites). Be patient, it can take a few months. Very possible you didn't do anything wrong.
Keep adding good content, as you are doing now, and visitors and links will come, google will follow.
Something else that could help is contacting webmasters of links to the old domain and ask them to update them to the new one.
When you say that google's bot drops by, does that mean it fetches the robots.txt file and then leaves, or that it the majority of the site and then leaves? If it's the first, then you can contrast your robots.txt file with that of other scoop sites that are well indexed. I doubt it's that.
Google's policy and procedures for building the index are as obscure as the Saudi oil reserves.
Does the their bot sweep the whole site, or just the front page and the robots.txt file?
I'll note that your robots.txt is different from that of most scoop site.
I'd block the "~" files for all clients in the apache config, not in the robots.txt.
http://nicklewis.smartcampaigns.com/blogging/cnn-guerrillas-in-the-midst-a-viral-marketing-campaign- exposed
This is a long report but a brilliant descovery, and I recommend that you read it in its entirity
http://www.google.com/search?num=100&hl=en&lr=&c2coff=1&q=theoildrum.com&btnG=Se arch