SEO and robots.txt - the robots hate me!
I've noticed that NO robots search my Drupal 6.4 sites - none, zip.
Of course this is not my goal!
I've been thru many videos on Drupal SEO (Search Engine Optimization) and have the Meta Tags module installed and supply Keywords and Descriptions.
Nothing seems to help - it's like the robots are driven away by something.
I have 100+ Drupal sites using the one installation of Drupal 6.4 and one mySQL database each site with a different prefix.
I've moved the robots.txt to my top most location on my hosting location. I haven't played with robots.txt - its direct from Drupal 6.4.
When I look at the code for a page I see: (I replaced the actual name of the web site)
My Web Site Name | - learn to work their system...
Everything works just fine but I can't get the robots to look at my sites.
Any suggestions?
Here's the code:<!DOCTYPE
Here's the code:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
<head>
<title>My Web Site Name | - learn to work their system...</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="copyright" content="Copyright My Web Site Name.com" />
<meta name="keywords" content="Description here" />
<meta name="robots" content="index,follow" />
<meta name="DC.title" content="Web site name| - slogan here..." />
<link rel="alternate" type="application/rss+xml" title="xxxxxxxxxxxx Timeshares RSS" href="http://www.MyWebSiteName.net/rss.xml" />
Since you're hiding your
Since you're hiding your website name, how are we supposed to take a look and figure out why this is happening?
Take a look at https://www.google.com/webmasters/tools as well.
Ok, here's the site
I didn't want to be accused of plugging my web site, but here is the raw code:
www.SkiTheHolidays.com is an example of my web sites with this problem.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Park City | SkiTheHolidays.com</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="copyright" content="Copyright SkiTheHolidays.com" />
<meta name="description" content="Park City, UT was the home of the 2002 Winter Olympics" />
<meta name="keywords" content="Park City Utah,ski,snowboard,5-star accomodations" />
<meta name="robots" content="index,follow" />
<meta name="DC.title" content="Park City" />
<link rel="next" href="/content/park-city-reservation" />
This web site has been up and running for 1 month now. Doing a Google search for SkiTheHolidays.com results in NO matches - Google completely bypasses my site.
I went into the Google link and added a few of my web sites - I'll see in a day or so if Google still ignores me. I Verified my 2 sites with Google.
Thanks for the tip.
This is crazy...
When I do a Google search of skitheholidays.com I get the above post but I still don't get my web site!
Apparently the way Drupal handles multi-web sites prevents Google from finding the Drupal web sites!
Yuck - this is something I never planned on. I wanted just one set of code to handle 200+ web sites and Drupal seemed to fit the bill.
This has never occurred to anyone before?
Now what? Have I shot myself in the foot?
Is there anyone out there who is using multi-website Drupal and Google finds them?
Yahoo search can't find multi-web site Drupal sites either!!
I just can't believe that no one involved with Drupal has ever thought of this problem? What do you guys do with your Drupal web sites? (Multi-websites)
Maybe I'm the first person to find this problem - if so I humbly suggest you dump multi-website Drupal capability - why would you ever want that capability if no search engine can ever find you?
You could use the sitemap
You could use the sitemap module to generate sitemaps after which you tell Google where to find the sitemaps.
I'll try this..
Thanks, I'll try this and then submit that to Google and see if it does anything.
Even crazier...
I looked up 20+ of my web sites on Google and just one appears - www.GoodBuyToyota.com
Why would just that one multi-website Drupal site appear and all the rest have NEVER been scanned?
I don't remember doing anything different with that site verses the other 20+ I looked up.
So apparently multi-website Drupal sites are visible to the robots when something happens.
What's that?
P.S.
Both of these two web sites are using the same Drupal 6.4 code and sharing the same mySQL database.
Well the Toyota web site does NOT use Meta-Tags and Google finds it and the Ski web site uses Meta-Tags and Google can't find it.
Go figure...
The Toyota web site was linked to by a number of other web sites that I listed the car. Is this why this one web site appears in Google and the rest don't?
Google needs to know that
Google needs to know that the websites are there. Are you sure there are incoming links (external sites that link to your website)? Otherwise you might want to manually add all your websites to Google via http://www.google.com/addurl/
Thanks again...
Thanks I'll to that today and see if it makes a difference.
your site
Your domain was registered on September 1, 2008. It takes a while for Google to index a site. That's the only reason your site isn't indexed.
You don't need to submit it to Google -- just get some links. Drupal.org links are nofollowed, so you'll need them from somewhere else. If you write good content, people will link to you.
For ideas on how to get links, search Google for link building tutorials...
--
My Drupal Tutorials