Are Non-Drupal Sites More Google-Friendly?

My site is here...

drupalfan commented 5 November 2005 at 04:16

Wi-Fi Report

Google drops by daily. It last appeared to index the site this evening. It has found my robots.txt file and has spidered my nodes and taxonomy terms.

I have redirected my old sites to point to the new site, funneling them all to one URL. All links point to my site.

I have noticed Google cannot find /q=sitemap.

what do you mean redirects?

wellsy commented 5 November 2005 at 04:38

I am not sure what you mean by redirects?
Also I do not see the url to your site.

wellsy

I mean...

drupalfan commented 5 November 2005 at 04:46

By "redirect", I mean I changed where the subdomains were pointing (yes, these were testing subdomains, and they were spidered, indexed and linked quicker and more complete than the main site on a top-level domain.) For instance, mobility.wifireporter.com now points to www.wifireporter.com.

try this technique

wellsy commented 5 November 2005 at 05:01

When you create nodes use the title of the node as a bolded heading at the top of the node content. Like for your http://wifireporter.com/node/42

<b>Survey: Consumers Giving Up on Mobile Dat</b>
Cell phone companies investing billions in call centers and customer assistance services may be throwing money down the drain if a new survey is correct. Just two percent of subscribers to mobile data services will call for help, according to a survey commissioned by Olista, a cell operator service provider.

This technique works well for me. Reuse this everywhere you want to be found well by google.

By the way you are right google just seems to not know your site exists at present.

wellsy

Good idea, I'll adopt it

drupalfan commented 5 November 2005 at 14:15

Great idea; I'll adopt it and see if it makes a difference.

so I see a problem here

wellsy commented 5 November 2005 at 05:17

So......the domain mobility.wifireporter.com is now simply a redirect to the present drupal site...yes...it seems to be....I was just redirected there.

That may be the problem. Google does not have any direct links to your new drupal area from an existing area.

So it cannot follow the link trail to your new drupal site.

Try adding some actual links on the main page of your regular site. This may not be what you want if the area is still a testing area but it is the only way I can see to get google to find the drupal area.

Believe me...once drupal is in the line of your publically available domain then it will be found.

wellsy

search this on google

wellsy commented 5 November 2005 at 05:36

"wi-fi report"

See what is second on the list-a drupal powered site for sure!

wellsy
like someone else here said "always use a test site"
permit me to add "backup your database regularly"
orchidsonline.com.au

Second on the list...

drupalfan commented 5 November 2005 at 14:14

Odd, isn't it. A domain spammer gets top billing, a Drupal.org post with my site gets third billing and the actual independent site is nowhere to be found on its own. (that link from drupal.org is one of the top referrers to my site, according to my logs.)

One positive note: I'm using the gsitemap module to keep Google aware of new content. It sends the sitemap fine, but was telling Google to grab q=sitemap, which doesn't exist. I went to the Google Sitemap site and submitted the real URL, including "/?q=gsitemap" and Google can download my new sitemaps. We'll see if that makes any difference.

Wi-Fi Report

study the spammers techniques and benefit

wellsy commented 5 November 2005 at 14:28

When you see sites which rate higher than you I recommend you study what it is they have added to their site to get top spot.

Then rework your own content to beat them.

It works!

wellsy

Good Advice

Maniac8888 commented 5 November 2005 at 14:56

That is really good advice and exactly what we did a long time ago for one of our sites that is consistantly ranked #1 out of 12.5M results for a specific key word phrase that is relevant to our business.

Many things are easy to do (i.e. include key words in urls, image tags, etc.), other things are a bit more difficult but it can be done with some effort.

Studying spammers

drupalfan commented 6 November 2005 at 04:49

Chief among the reasons I see www.wifireport.com is coming up first for the "wi-fi report" Google search is the domain name is identical to the site name. It also uses frames and is a very simple site: no css, no tables, just one frame statement. The other point: it doesn't use drupal.

Wi-Fi Report

no explanation

wellsy commented 6 November 2005 at 05:20

Well I have spent quite sometime trying to find out why your site does not search at all on google and to be honest I am unable to come up with any explanation at this stage unfortunately. Sorry.

The fact that google has not followed the very good links from this node drupal.org/node/35857 which has been up since nov 1st is probably the key to finding the answer really. I would have expected google to follow that to your site well and truly by now.

Why not? Any ideas here guys?
Must be something we have all missed I believe.

wellsy

http://siteexplorer.search.yahoo.com/

I dont know if this is good advice...

gautam_chandna commented 5 November 2005 at 16:38

I dont know if this is good advice but

The SEO tricks I can share with you are simply to submit your site to _every_single_place_you_can_find_a_submit_url_link_(but only after checking its pagerank)
I dont even remember now but I'm must have submitted to atleast 30 odd places.

One place I would certainly recommend is the site Ping-o-matic and get yourself submitted in all the blog search engines (even if yours isnt exactly a blog - if it has a feed, it'll do!)

I started my site in september from scratch (Ahste.com) and it's Google PageRank is 6.

Till the first "Google Dance" my site was unlisted, but instantly afterwards Ahste.com was ranked really high for many searches. More than 80% of the traffic on my site (though still a mere 50 to 60 people a day) get there from various search engines.

So I would safely say, that Drupal is very Google friendly.

Best of luck!

--
http://www.ahste.com

also

gautam_chandna commented 5 November 2005 at 16:37

using clean url's is a must.

path.module didnt make too much of a difference for me, but be sure to use atleast one of the following:

1. sitemap module
2. indexpage module
3. list (I'm forgetting the exact module name..)

I'm using all three with links on my main page. That way, (almost) every single page on my site is accesible within 3 clicks.

:-)

--
Gautam

Various Google-friendly Modules

drupalfan commented 6 November 2005 at 04:40

I included the sitemap module (plus corrected it so Google actually picked up the sitemap at /?q=gsitemap, rather than sitemap. I also put a text link to my sitemap.

I installed the indexpage module and it works from the administration menu, but gives an error if an anonymous or non-admin person tries it.

I tried the Ping-o-matic suggestion.

I often search for "wi-fi report" on Google and a non-druple domain registration site, a PHP-PUKE site that mentions my site, and a drupal.org mention of my site appear on the first page of results. Even "site:www.wifireporter.com" on Google says it has no knowledge my site even exists.

Wi-Fi Report

give it a little time..

gautam_chandna commented 6 November 2005 at 05:24

just give it a little time..

My best guess is that google will _not_ index your site until the next google dance (The Index Update of the Google Search Engine).

Usually happens middle of the month (the time when drupal starts saying stuff like "There are currently 2 users and 499 guests online." (for small scale sites like mine:)

Be prepared for this by making your site as accessible as possible. enable the indexpage (or nodelist) module for anonymous users and give it a link somewhere on you main page (anywhere.)

changes you make will not instantly reflect in google - and from my experience, your site will get 100% indexed only at the next google dance. because as of right now, your domain name is just a link to pages on your site - its not a domain name in googles search index. So the bots dont attack your site directly, they do so through the pages you've mentioned that come up on top.

why so?

cause they are the ones leading Google bots to your site. Your domain name isn't even in their engine to know you exist.

The last dance was around the 22nd of October (thats when I got indexed), the one before that was the 14th of September.

as long as your site has

clean urls
lots of data accessible anonymously [or atleast the major keywords you wish to establish for your site]
regular (prefer daily) changes/additions
every page accessible with as little clicks as possible (definately use nodelist.module)
(the name I forgot in my last post:-)

you will be indexed.

--
Gautam Chandna
http://www.ahste.com

Google Dance

drupalfan commented 6 November 2005 at 15:12

My URL has been available since the end of September, so it has been up for any "dance" Google or the other search engines may have desired.

I'll check into the nodelist module.

Wi-Fi Report

Duplicate content?

heine commented 5 November 2005 at 17:52

Perhaps you fell foul of the duplicate content filter. See Bruce Clay Techtips.
--
Tips for posting to the forums

No kidding

Maniac8888 commented 5 November 2005 at 19:18

That's for sure. If one has mirror sites and forgets the robots.txt to exclude indexing on one of them, Google will cancel each site out and the site will never get ranked or listed. Bad news.

Only My Main Page is Being Indexed

drupalfan commented 6 November 2005 at 04:59

Interesting. Google says www.wifireporter.com doesn't exist. Yahoo, however, ranks my site #1 when I search for www.wifireporter.com. However, only my main page is listed. Although there are dozens of news stories on the front page, none of those are indexed.

Wi-Fi Report

Yahoo has several of your pages in the index

ergophobe commented 6 November 2005 at 17:53

Use this instead:

Yahoo has a about six pages for you./

Also, the dupe content occurred to me. Make sure any old URLs that you can control are sending out 301 codes not 302. 302 is the default, so you have to watch for this.

Google Cancels Duplicate Content

drupalfan commented 6 November 2005 at 15:31

What is the solution? I've deleted all test sites that had some duplicate content and which did not include robots.txt files in their home directories.

Wi-Fi Report

I sort of answered this

ergophobe commented 6 November 2005 at 17:57

In my last post. Don't "delete" the sites. Do whatever you want with the content, but you need at least a .htaccess file or httpd.conf settings to make sure those addresses are sending 301 return codes.

Simply deleting them will send a 404, telling the bot that the page is dead, gone, not replaced, not in existence, so don't look for that page elsewhere.

Using a default redirect will send a 302, telling the bot that the page is on vacation at another URL, but will be right back, so don't index that new page.

Using a 301 redirect will tell the bot that the page is still alive and healthy, but has moved to a new address, so please update your address book if you want to come calling.

If Google has already been through and picked up the 404 a couple of times, it may be to late, but if it's still hitting those addresses, get them to return 301 ASAP.

Another clue: .htaccess

drupalfan commented 6 November 2005 at 18:34

My .htaccess file doesn't use 301 to redirect. I'm off now to research how that's done. While I'm at it, what's a robots.txt file optimized for Drupal?

google...

pfmj2005 commented 6 November 2005 at 06:21

From what I understand... Google does not pay attention to your sitemap unless you do register into their sitemap service.... They did not even notice my sitemap until I told them too...

--
Paul Malenke
paul.malenke@gmail.com
AfterDeathGraphics.com

I registered with the GoogleMap service...

drupalfan commented 6 November 2005 at 15:16

I registered and submitted a sitemap which was found and verified.

I'll take down my subdomains and test installs, leaving just my main site . When I come to think of it, I have one main robots.txt file while forbid the directories where the test sites were located. However, there were no robots.txt files in each of those directories. Could this be the snag?

Wi-Fi Report

Yes

heine commented 6 November 2005 at 16:43

Suppose you have a testsite in /testsite. The subdomain test.example.com maps to this testsite. When you forbid indexing /testsite via robots.txt of (www.)example.com, test.example.com is still indexed, because search engines can't find the robots.txt for test.example.com and they can't know it maps to the subdirectory testsite.

My apologies if the above isn't entirely coherent...
--
Tips for posting to the forums

Makes perfect sense...

drupalfan commented 6 November 2005 at 18:29

This makes perfect sense and one clue why my main site was not indexed by Google.

Wi-Fi Report

MSN Search Likes Me

drupalfan commented 7 November 2005 at 04:35

I read my logs and saw msnbot indexed my site and lo and behold - the site is ranked #1 there. I also saw googlebot tripped on a 404 from an old, now removed, file from a test site. Requests for that file now send a 301 to my main site. I'll ask Google to reindex and see what happens overnight. The Yahoo siteexplorer page listed four of my links, yet 3 were to old pages that no longer are available. I've changed my .htaccess file to redirect requests for those to my main site.

By the way, I appreciate all the feedback. Drupal has to have one of the most-informed and most-community spirited group of users. I think I made a good decision to stick with Drupal.

Have you filed a bug report for the gsitemap module?

robertdouglass commented 8 November 2005 at 09:23

The module is actively maintained and if this is a bug, it'll get fixed, but you have to report it:
http://drupal.org/project/issues/26133

- Robert Douglass

-----
My sites: HornRoller.com, RobsHouse.net

Bug Report

drupalfan commented 8 November 2005 at 17:52

Just submitted.

Get rid of PHPSESSID

Jaza commented 8 November 2005 at 11:12

I just had a look at your site, and while it looks great and is already well set-up for a good search engine ranking (assuming that Google knows that your site exists!), there are a few things you could do to improve it:

Get rid of the PHPSESSID cruft in your links. A quick search on drupal.org will reveal many solutions to this common Drupal problem. It has been argued many times (although not proven) that the PHPSESSID problem affects SEO.
Give your site a text heading. The banner at the top of your site, although consisting entirely of text, is all displayed as an image. This is not a huge factor in SEO - drupal.org itself has an image as its header, as do many other Drupal sites. But at the least, give your image header some meaningful ALT text - at the moment, your ALT text is the default 'logo' - change it to your site's name.
Use the path module to alias all content URLs on your site. Preferably, make use of the pathauto module to help you do this. Drupal is capable of producing the cleanest and most search-engine friendly URLs of any CMS that I've ever seen. Don't underestimate the importance of beautiful URLs in SEO! I noticed that your site has some pages aliased, but not all.

Have a look at this recent ALA article on SEO. As the article explains, a site that's accessible to all humans is also accessible to all machines, such as the Google spider. You've already got a clean, tableless layout happening, so that's a really good start! Assuming that you have plenty of good content, and plenty of other sites linking to you, your site is guaranteed to rapidly climb the ranks over at Google and Co. No way is Drupal - the world's most search-engine friendly CMS - going to stop you!

Jeremy Epstein - GreenAsh

All great points

drupalfan commented 8 November 2005 at 18:04

All useful points. Being a sometimes lazy editor, I love the autopath module. I wondered about the PHPSESID -- I'll zap that as soon as possible. I'll also add that alt tag to the graphic.

How to Increase Traffic

drupalfan commented 4 December 2005 at 05:12

It appears I need to boost my level of traffic in order to be listed in search engines. (Nearly three months after initially adding the URL to Google, offering an easily-spidered site, maintaining a fresh output of content, the search engine still doesn't know the URL exists.)

Since nearly 90% of readers find sites via a search engine, gaining a listing is mandatory for survival. Since most reputable non-spam news sites do not offer reciprocal linking, what are some ways to increase traffic to a relatively new site? I've heard of feeder pages, but is this a good technique?

Check out the site, give me some reviews on what's needed. Otherwise, without some potential for more traffic and notice by Google, the effort to maintain the site simply isn't worth it. Thanks.

Since the Jagger update to Google

laura s commented 4 December 2005 at 06:55

The helpfulness of Reciprocal linking has been greatly diminished to the point of almost no effect. Are you participating on other relevant sites? Your site will become more visible to search engines as others put links to your content in their content. And you will get more traffic from people who see and follow those links.

I see you have no meta description (which comes from your mission statement, usually). Using pathauto to rename your taxonomy will help, too -- descriptive category names in the path itself help search engines dig into your site (and helps technorati etc. to track your tags). There seems to be some debate on whether renaming node paths with keyword URLs is better or not. Unless your site traffic is going to be very heavy, or your server is a bit underpowered, it might be a good thing to do.

You also should see about getting rid of your session IDs for anonymous users. I know that search engines don't like all that schmutz in the URLs. Try adding this to your .htaccess file in your override PHP settings area:

php_value session.use_trans_sid 0
php_value session.use_only_cookies 1

Other than that, the lifetime of the site has been mentioned before. Google sandboxes sites for 9-12 months now. There's no getting around that unless you're an established player that they will manually pass through. (Sites I launched last year suddenly got Google hits earlier this year. It was like a ship had arrived and suddenly there were all these people.) If your site is older, be sure you have 301 redirects to point crawlers looking for your old content pages to their new locations.

My last note is about the visual style: The really long sentences in your node display are hard to read. You might try narrower text blocks.

I hope these comments are helpful.

Laura
===
pingVision • rare pattern • scattered sunshine

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

I thought reciprocal links were time-wasters...

drupalfan commented 4 December 2005 at 22:11

...Which was why I asked. The other tactic I've read and seen employed is "feeder sites" which funnel visitors to a main site.

Please explain more about "renaming nodepaths with URLs." What's an example of this?

I have the clean URLs set and the pathauto module enabled. I've also added the nodewords module to provide each node with keywords that are more search engine attractive. Despite comments from some that the meta tags are necessary, the Drupal sites I've seen listed in Google nearly all employ the tags.

I understand about the PHP SessionIDs. I added the lines to the .htaccess file awhile back. This seems to have stopped most IDs, except within the admin functions. Are these IDs only visible to remote users?

What is the Google "sandbox"? Is this a type or waiting list for new sites before being "admitted" into Google results? My frustration comes from people saying their Drupal sites were listed after 1-2 months.

I appreciate the design suggestion. After years of writing for the Web, I knew content should be offered in easy-to-digest bits. I'll make sure line-length is also brief.

Some attempts at answers

laura s commented 5 December 2005 at 00:19

Node renaming: http://yourdomain.com/node/32 ("Great Christmas Cookie Recipes") becomes http://yourdomain.com/2005/12/great-christmas-cookie-recipes ... for example.

For the categories, you might set pathauto to rename them to category/[term-name] so that the URL reflects the name given the category term. (There are many naming conventions, but the main thin is getting the term name into the URL.)

The session IDs I saw were when not logged in. If you add that code I gave before to the appropriate Apache conditional in .htaccess, it should eliminate them. It's not a problem, per se, but I'm told some search engine crawlers don't like all the extra jibberish.

A newer version of nodewords generates a description metatag for each page as well. You'll need to update the database table for that to work.

As for Google, they have been reworking their algorithms constantly. The latest change was "Jagger" but that's just one of many changes. They "sandbox" new URLs to weed out spammers. Usually you can google up the domain by googling the actual URL, but you generally won't show up on any preferred keywords for some time. There are exceptions -- I'm quite sure Arrianna Huffington didn't have to wait 9 months before the Huffington Post was listed -- but for most of us, it's just part of the reality we all face with recently registered domains.

Laura
===
pingVision • rare pattern • scattered sunshine

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

Usually you can google up the domain...

drupalfan commented 5 December 2005 at 04:09

Ah, you've hit on another of my gripes with Google. I understand about the keywords taking time to perculate to the top, but even an explicit site search (ala site:example.com) returns with a 'site unknown.' The site was registered the last week of September.

Have you installed and submitted

laura s commented 5 December 2005 at 06:13

Google Sitemap? If you register with G and submit it that way, they will tell you when your site is crawled, and whether there are any problems.

I can't speak for Google, only offer some ideas on how to make it easy for the big Goog to notice a new site.

Laura
===
pingVision • rare pattern • scattered sunshine

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

Sitemap Submitted

drupalfan commented 5 December 2005 at 13:59

I submitted the sitemap weeks ago. The site was crawled with no errors reported. The Google Sitemap service lists the pagerank as "low." The Googlebot regularly visits and Google regularly retrieves updated sitemaps.

Last resort

laura s commented 5 December 2005 at 16:25

If you dig around, there is a way to contact them for tech support. It might take a few days to get a reply, but if you ask them if your site is blacklisted or something, they might check it out for you.

Sorry you're having all these problems. Nothing is more frustrating than dealing with obstacles beyond your control.

Laura
===
pingVision • rare pattern • scattered sunshine

_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet

Howto contact google

heine commented 5 December 2005 at 16:41

The best way to file a reinclusion request by Goolger Matt Cutts.
--
Tips for posting to the forums

Have contacted them...

drupalfan commented 5 December 2005 at 18:12

Weeks ago. No reply. I just did a search on the domain and while the domain itself isn't listed, a series of posts that include the URL (such as these) are listed.

Download the web dev toolbar

ergophobe commented 5 December 2005 at 00:25

Download the web dev toolbar for Firefox and look at your site with cookies disabled. That's how it looks to Google.

In fact, you have session ids in every URL. I recently did a site for someone who had troubles with this. What you need to do in addition to what you have done is add this to your settings.php file

ini_set('url_rewriter.tags', '');

That should get rid of the session ids and maybe finally get you into the SEs

Should the ini_set be 0 or 1?

drupalfan commented 5 December 2005 at 14:01

ini_set('url_rewriter.tags', '');

Should this be 0 or 1? The " causes Drupal to error out at that line. I've set it to 1.

http://us2.php.net/ref.session

Ed, it should not be 0 or 1.

ergophobe commented 6 December 2005 at 00:12

Ed, it should not be 0 or 1. This parameter for this setting is a string. By default it is set to

a=href,area=href,frame=src,input=src,form=fakeentry,fieldset=

In other words, it is set to those attributes that *should* get tags.

Were you getting a parse error? Were you using one double quote instead of two singles? What you want is to set it to an empty string.

What version of PHP are you using?

See

PHP 4.4.1

drupalfan commented 6 December 2005 at 05:54

I was using on double quote, ie ". Should it be ' '?

--
Tips for posting to the forums

Yes

heine commented 6 December 2005 at 14:28

Single / double quotes open a string literal. Because you used a double quote " a string literal opens; it closes later when another double quote (usually starting another string) is encountered, That's why you get an error.

Use either two single quotes '' or two double quotes ""

Done

drupalfan commented 6 December 2005 at 15:03

I added a double-quote. Things appear to be working okay. I also switched to the gorgeous civicspace theme.