Logs filled with Mollom Errors

Thanks for reporting -- forwarded to the technical backend team at Mollom!

This problem should be resolved in the meantime.

Thanks again for reporting it!

I'm seeing similar errors:
Message Error -110 from 'http://67.228.84.11' for 'mollom.verifyKey': 'Connection timed out'
Data:
platform_name = 'Drupal'
platform_version = '6.19'
client_name = 'Mollom'
client_version = '6.x-1.15'

Do you still experience this issue? We had a temporary server hiccup in the past days, but that should be resolved by now.

In addition to "Error 0" and "Error -110", I am seeing entries in watchdog like:

Type	mollom
Date	Wednesday, December 8, 2010 - 7:35am
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com
Message	Invalid UTF-8 in form values 
Data:
@data
Severity	notice

Type	mollom
Date	Wednesday, December 8, 2010 - 8:33am
User	Jasofina
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
Severity	debug
Hostname	91.201.66.23

Type	mollom
Date	Monday, December 6, 2010 - 1:44pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	Unknown session id 101206900a7eb276d3. This is not a bug in Mollom. If this happens too often, check your site for attacks.
Severity	warning
Hostname	74.254.73.90

Regarding this last message, "check your site for attacks" isn't very helpful or informative.

The Unknown session id ... error should technically never happen -- unless a malicious user tries to take over the user/form session of another user. The Mollom module is not able to do a security review of your site, so that's why the log message tries to make site operators aware of that.

But regardless of that, do you still get 0 or -110 timeout/communication errors?

I'm also seeing a message about the server ... redirected to:
Type mollom
Date Thursday, December 16, 2010 - 14:06
User admin
Location http://www.canadaparticipates.ca/admin
Referrer http://www.canadaparticipates.ca/?q=user
Message Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
Severity debug
Hostname 70.29.21.228
Operations

Is this a normal message or something I need to be concerned about?

This is a usability problem with the watchdog messages; it is simply not obvious what is going on. See also #1000690: Revamp and reduce log messages (was: Not clear that fail-over is working).

I am not seeing any more -110 or 0 warnings. I am getting these though:

Type	mollom
Date	Saturday, December 18, 2010 - 12:32pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	Error 1000 from 'http://174.37.205.152' for 'mollom.checkContent': 'XMLRPC parse problem. The SAX parser returned: An invalid XML character (Unicode: 0x0) was found in the element content of the document.' 
Data:
  post_body = 'Thomas Kovca' . "\0" . ''
  author_name = 'Thomas Kovca' . "\0" . ''
  author_mail = 'paydaywef@mail.ru'
  author_ip = '178.77.66.56'
  honeypot = 'http://rss.sexmoney.com/fhg.php?WMID=71075&PID=1&CTRLID=&Popup=0&WMEC=5&LGID=de&thumbnails=200&count=320&enc=ISO&target=paysite'
  session_id = ''
  checks = 'spam'
Severity	error
Hostname	178.77.66.56

and similarly, this:

Type	mollom
Date	Saturday, December 18, 2010 - 12:32pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	All servers unavailable: 'http://174.37.205.152, http://67.228.84.11' 
Last error: 1000 'XMLRPC parse problem. The SAX parser returned: An invalid XML character (Unicode: 0x0) was found in the element content of the document.'
Severity	error
Hostname	178.77.66.56

@ebeyrent: With the latest official module release, such errors should not appear. Did you update to 1.15?

Yes - I am running 1.15. More errors yesterday:

Type	mollom
Date	Tuesday, December 21, 2010 - 6:23pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	All servers unavailable: '--'
Last error: 0 NULL
Severity	error
Hostname	91.201.66.15

Type	mollom
Date	Tuesday, December 21, 2010 - 6:23pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	All servers unreachable or returning errors. The server list was emptied.
Severity	error
Hostname	91.201.66.15

Type	mollom
Date	Tuesday, December 21, 2010 - 6:23pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	Error 0 from 'http://67.228.84.11' for 'mollom.checkContent': NULL
Data:

  post_body = 'Commodorrenite'
  author_name = 'Commodorrenite'
  author_mail = 'coi.llcumhannhetherington@gmail.com'
  author_ip = '91.201.66.15'
  session_id = ''
  checks = 'spam'

Severity	error
Hostname	91.201.66.15

Type	mollom
Date	Tuesday, December 21, 2010 - 6:23pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'
Last error: 0 NULL
Severity	error
Hostname	91.201.66.15

Also seeing this warning a few times:

Type	mollom
Date	Monday, December 20, 2010 - 12:49pm
User	stelinlindsey
Location	http://mysite.com/comment/reply/27317
Referrer	http://mysite.com/comment/reply/27317
Message	Error '404' from 'http://174.37.205.152' for 'mollom.getImageCaptcha': 'Resource_not_found'
Data:

  session_id = '10122000d25afef403'
  author_ip = '122.161.163.119'
  ssl = false

Severity	debug
Hostname	122.161.163.119

Thanks for reporting back, @ebeyrent -- I have forwarded that information to Mollom's development team.

FYI - still getting -110 errors:

Type	mollom
Date	Friday, December 24, 2010 - 7:23am
User	Ronorits
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	Error -110 from 'http://174.37.205.152' for 'mollom.checkContent': 'Connection timed out'
Data:

  post_title = 'delete sorry -550'
  post_body = 'quash pitiable  
eliminate sordid  -747'
  author_name = 'Ronorits'
  author_mail = 'vi2009.09.09@gmail.com'
  author_id = '121906'
  author_ip = '2.93.244.201'
  session_id = ''
  checks = 'spam'

Severity	error
Hostname	2.93.244.201

This was accompanied by:

Type	mollom
Date	Friday, December 24, 2010 - 7:23am
User	Ronorits
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'
Last error: 1200 'Please try the next server in the server list.'
Severity	error
Hostname	2.93.244.201

Type	mollom
Date	Friday, December 24, 2010 - 7:23am
User	Ronorits
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	Server 'http://67.228.84.11' redirected to: false.
Severity	error
Hostname	2.93.244.201

Type	mollom
Date	Friday, December 24, 2010 - 7:23am
User	Ronorits
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	All servers unreachable or returning errors. The server list was emptied.
Severity	error
Hostname	2.93.244.201

Type	mollom
Date	Friday, December 24, 2010 - 7:23am
User	Ronorits
Location	http://mysite.com/node/add/forum/30
Referrer	http://mysite.com/node/add/forum/30
Message	All servers unavailable: '--'
Last error: 1200 'Please try the next server in the server list.'
Severity	error
Hostname	2.93.244.201

I am surprised that I'm getting so many errors.

Greetings,

We started seeing the same types of intermittent error messages last week as well and created support tickets in both Acquia & Mollom for this exact issue. At first they would happen once or twice a day, but now they are happening frequently enough to be alarming. To be honest, when I found this thread reported in the queue, i was somewhat relieved that we aren't the only ones experiencing this problem.

Our Specs:
Drupal Core 6.19
Mollom 6.x-1.14

We are planning an upgrade soon to Mollom 6.x-1.15, but it seems from this thread that even 1.15 is having the same errors!

When these "server unavailable" errors occur we are getting reports from our users that:
1. they cannot post (expected behavior since we default to disallowing all posts when Mollom servers unavailable since we average ~1000 spam posts a day).
2. they are intermittently getting broken CAPTCHA images served by Mollom from this url: xmlrpc2.mollom.com

We also noticed a dramatic drop in overall Spam/Ham numbers in our Mollom Statistics starting 12/17/2010 (attached screenshot). We originally attributed this to the holidays, but since we haven't seen a recovery at all, we think this might be tied to the same issue since we block all posts when the Mollom servers are unavailable.

Our Drupal logs DO show Mollom spam, ham, correct CAPTCHA, & unsure entries when the servers ARE available, so we can assume it's not a broken code issue. And when we go to the the Mollom Settings page, we do receive the message that Mollom server are communicating successfully.

Sampling of Error Messages we repeatedly & intermittently receive in our Drupal Log daily, somtimes hourly (attached also):

All servers unavailable: '--'
Last error: 1200 'Please try the next server in the server list.'

All servers unreachable or returning errors. The server list was emptied.
Server 'http://67.228.84.11' redirected to: false.

All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'
Last error: 1200 'Please try the next server in the server list.'

Error -110 from 'http://67.228.84.11' for 'mollom.checkContent': 'Connection 
timed out'

Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.

Error 1000 from 'http://174.37.205.152' for 'mollom.checkContent': 'XMLRPC parse problem. The SAX parser returned: An invalid XML character (Unicode: 0x6) was found in the element content of the document.' 

After visiting the Mollom Blog to see if there were any updates that coincided with 12/17/2010 when we started noticing these issues, I saw Dries's latest post 12/14/10:
http://mollom.com/blog/mollom-gets-a-new-backend

Since this backend server change could potentially impact server availability,
which are the errors we are receiving, couldn't this be a possible cause of
our error issues AND statistical drop in ham/spam numbers?

Thanks!
Henry

After opening a support ticket with both Acquia & Mollom, we received the following from the folks @ Mollom c/o Acquia (thanks to Michael Haag & Sun for the help):

"I believe that we've identified the root cause of this ticket, and it was, essentially, series of events that stemmed from two of our main servers having incorrectly configured time zones. Because of the time discrepancy between the two units, there was an unusual situation where our failover system behaved badly, thanks in part to a couple of new parameters we had created on the backend that allowed redirects to be sent way too early (when the system was a bit low on memory, but not low enough to be dramatic).

"It's probably not the case that the servers were actually unavailable at these times, but certainly enough spurious traffic was generated on our backend to make the situation be recorded as such in the logs.

"Long story short, we've identified the root cause, and the events that led to the root cause, and have (a) fixed/adjusted our redirect parameters and (b) made sure that time zone information is correctly set on all servers.

"We apologize for the inconvenience, and appreciate the ticket -- it helped us identify an issue with our fail-over system that we've now corrected.

"btw, most of the messages were harmless -- they look unnecessarily scary because they are rendered as errors. That is something that could be improved in the Drupal module. Some messages were real errors though."

Now, we know for a fact that the Mollom servers WERE actually unavailable since our users were complaining that they couldn't post intermittently and would receive the

Spam filter installed on this site is currently unavailable. Per site policy, we are unable to accept new submissions until that problem is resolved. Please try resubmitting the form in a couple of minutes.

message. Again, defaulting our Mollom settings to allow posts if their servers are unavailable isn't an option for us since we receive an average 1000 spam/day, with highs of 8000 in a day.

Both of our error symptoms have seemed to recovered:

1. We haven't received a Mollom Error log entry related to this issue since 1/4/2010 12:59pm.
2. Our Mollom statistics graph of overall Spam/Ham posts for 1/4/2010 has shown a noticeable uptick compared to the flat line between 12/18/2010 and 1/3/2010 that probably reflects the half day recovery. Today (1/5/2010) will be our first full day to test if our Spam/Ham numbers go back to "norm"

We will keep monitoring and will update if there are any further issues, though we're optimistic that this seems to have fixed the problem. We just hope we haven't lost the confidence of our forum users during this 2.5 weeks of intermittent server availability. We were lucky that it happened during the Christmas & New Year holidays so perhaps the damage was minimal.

The backend Mollom upgrade and the latest 1.15 release are great steps forward, and we understand no release comes without bugs. We can only hope that these issues can be fixed in a timely manner.

So Despite the hiccup, keep up the great work Mollom!

No site outages on our end, related or otherwise, sorry. but IMO it doesn't seem likely this issue would bring down your site. From our encounter the past 2 weeks, this issue would just trigger whatever failover option you have configured in your Mollom settings when Mollom Servers are unavailable. In our case users were unable to post.

Since you said "roughly" coincided, i'd check what exactly the last Drupal log error was at the time your site went down and troubleshoot from there. You might also want to check the Apache logs as well.

The only thing I can think of that might cause outages would be if you are not pruning watchdog and have millions of rows in that table. Conceivably, the large number of Mollom entries could have caused mysql to thrash.

I'm not saying that's what happened, but I've seen similar things happen before...

As outlined above, we think that those communication errors should be resolved by now.

Hmm, it seems the Mollom error log messages are still happening, but the issue with our Ham/Spam numbers has been fixed after Mollom fixed the issue with the failover system.

These errors just randomly appear with no pattern, and then they just repeat for a minute or two, then stop. I've attached a snippet of these errors from today.

btw, most of the messages were harmless -- they look unnecessarily scary because they are rendered as errors.

Does this quote cited from my Mollom ticket explain the following errors we are seeing almost daily now? Are they truly harmless status messages and not error messages we should be alarmed about?

All servers unavailable: 'http://174.37.205.152, http://67.228.84.11' 
Last error: -110 'Connection timed out' 

Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.

Error -110 from 'http://xmlrpc3.mollom.com' for 'mollom.getServerList': 'Connection timed out'

All servers unreachable or returning errors. The server list was emptied.

Error 1200 from 'http://xmlrpc2.mollom.com' for 'mollom.getServerList': 'Please try the next server in the server list.'

If these messages are truly harmless status messages and NOT errors, and DO NOT impact our users being able to post messages, then I agree that a separate ticket should be created to improve the handling of these events in the Mollom module so that they don't appear as errors. But I would feel safer once Mollom verifies that these log messages are "normal". Then we can close this issue once and for all.

Thanks!

PS. There seems to be a related (duplicate?) issue out there as well: http://drupal.org/node/356704. Not sure which one should be marked as the duplicate.

I'm also still seeing issues, even though I've been told the problem has been fixed.

Type	mollom
Date	Tuesday, January 11, 2011 - 8:37pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com
Message	Invalid UTF-8 in form values
Data:

@data

Severity	notice
Hostname	61.35.191.250

Type	mollom
Date	Tuesday, January 11, 2011 - 6:56pm
User	Anonymous
Location	http://mysite.com/user/register
Referrer	http://mysite.com/user/register
Message	Error 0 from 'http://174.37.205.152' for 'mollom.checkContent': NULL
Data:

  post_body = 'votka91'
  author_name = 'votka91'
  author_mail = 'votkalounge1215@gmail.com'
  author_ip = '91.121.13.97'
  session_id = ''
  checks = 'spam'

Severity	debug
Hostname	91.121.13.97

However, if the stats are any indicator, Mollom appears to be performing much better, as the amounts of spam and ham have jumped impressively.

I'm getting the same problem since about 3 hours:

Type	mollom
Date	Tuesday, February 8, 2011 - 12:40
Message	Error 1200 from 'http://xmlrpc3.mollom.com' for 'mollom.getServerList': 'Please try the next server in the server list.'
Severity	error
Hostname	192.168.10.106

Type	mollom
Date	Tuesday, February 8, 2011 - 12:40
Message	All servers unreachable or returning errors. The server list was emptied.
Severity	error
Hostname	192.168.10.106

There was an temporary, overnight issue with one of the servers that unfortunately went unnoticed for frame of time early this morning, leaving the other servers to pick up the slack. Everything should be back to normal now. We're enhancing notifications for this sort of thing so that people get alerted _much_ more aggressively.

subscribe thx .. dblog filled with Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.

* All servers unreachable or returning errors. The server list was emptied.
* Error 0 from 'http://67.228.84.11' for 'mollom.getImageCaptcha': NULL
Data:

author_ip = '196.214.159.90'
ssl = false

* Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
* All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'
Last error: 0 NULL

update: for some strange reason the errors have stopped without doing anything on this side :)

We've been getting 100's of errors like the following for about a week on our network of sites. Ample spam getting through that correlates to the same timeframe.

Examples:

MESSAGE	Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
SEVERITY	debug

MESSAGE	All servers unreachable or returning errors. The server list was emptied.
SEVERITY	error

MESSAGE	Error 0 from 'http://xmlrpc3.mollom.com' for 'mollom.getServerList': NULL
SEVERITY	error

MESSAGE	Error 1200 from 'http://xmlrpc2.mollom.com' for 'mollom.getServerList': 'Please try the next server in the server list.'
SEVERITY	error

MESSAGE	Unknown session id 110318dda865c6263e. This is not a bug in Mollom. If this happens too often, check your site for attacks.
SEVERITY	warning

We'll investigate these things. Two notes:

- Not all of these messages are errors -- some are just informational. We might want to tweak the watchdog messages to make that more clear (if that is confusing).

- It would be helpful to get timestamps. Our internal records show that you should not have seen these messages 'for about a week'. It should have been limited to one day. If our internal records are correct, we might have to improve our monitoring or the error might have been in the network (beyond our control).

I still get a bunch of entries in watchdog from Mollom as well, such as:

Type	mollom
Date	Thursday, March 17, 2011 - 6:50am
User	swatikamath
Location	http://mysite.com/comment/reply/19736
Referrer	http://mysite.com/comment/reply/19736
Message	Error 1200 from 'http://xmlrpc1.mollom.com' for 'mollom.getServerList': 'Please try the next server in the server list.'
Severity	debug
Hostname	122.166.88.151
Operations

Type	mollom
Date	Thursday, March 17, 2011 - 6:49am
User	swatikamath
Location	http://mysite.com/comment/reply/19736
Referrer	http://mysite.com/comment/reply/19736
Message	All servers unavailable: '--'
Last error: 1200 'Please try the next server in the server list.'
Severity	error
Hostname	122.166.88.151
Operations	

Type	mollom
Date	Thursday, March 17, 2011 - 6:35am
User	swatikamath
Location	http://mysite.com/comment/reply/19280
Referrer	http://mysite.com/LFE/toddler/discipline/44252.html?lid=44252
Message	Error 0 from 'http://174.37.205.152' for 'mollom.checkContent': NULL
Data:

  post_body = 'Very nice article. I have few more good tips for potty training at http://guidetoparenting.blogspot.com/2011/03/step-by-step-guide-to-potty-training.html'
  author_name = 'swatikamath'
  author_mail = 'swati.s.kamath@gmail.com'
  author_id = '869713'
  author_ip = '122.166.88.151'
  session_id = ''
  checks = 'spam'

Severity	error
Hostname	122.166.88.151
Operations

On 3/17 from about 6:35am to 6:50am, I got 35 of these messages.

We're seeing a lot of these messages as well. It looks like the servers are mostly reachable but very slow to respond.

A small time-frame of unavailability on March 17th can be confirmed. However, it lasted a few minutes/hours only, and you should definitely not see these log messages by now anymore. Are you able to confirm this?

I'm not a paying user so please don't think this is a complaint, just a "me too!" bug report.

Saw lots of errors last night when a user added a comment:

 32617  20/Mar 22:49  error     mollom  All servers unreachable or returning errors. The server list was emptied.
 32616  20/Mar 22:49  error     mollom  Error 0 from 'http://67.228.84.11' for 'mollom.checkContent': NULL         Data:  post_title = 'REMOVED'
                                          post_body = 'REMOVED'
                                          autho
 32615  20/Mar 22:49  error     mollom  Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
 32614  20/Mar 22:49  error     mollom  Refreshed servers: 'http://174.37.205.152, http://67.228.84.11'
 32613  20/Mar 22:49  error     mollom  All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'                       Last error: 0 NULL

 32612  20/Mar 22:48  error     mollom  All servers unreachable or returning errors. The server list was emptied.
 32611  20/Mar 22:48  error     mollom  Server 'http://67.228.84.11' redirected to: false.
 32610  20/Mar 22:48  error     mollom  Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.

Mollom version 6.x-1.15

I can confirm that there are no recent server unavailable messages in our logs, just the following: Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.. Thanks, sun!

I got this today

Type mollom
Date Monday, March 28, 2011 - 16:38
Message Server 'http://174.37.205.152' redirected to: 'http://67.228.84.11'.
Severity debug
Hostname 212.185.233.236

my site has also disappeared in the last week twice and the index page and includes have had to be re-uploaded for the site to work (I'm using drupal 7)

Severity debug

Whenever the log message severity is "debug", "info", or "notice", you can happily and freely ignore them, as these are informational log messages only. Only warnings and errors are of interest.

I'm consistently getting these errors on my staging server on Acquia:

All servers unavailable: 'http://174.37.205.152, http://67.228.84.11'
Last error: NULL NULL

Severity is error; I'm using it for profanity protection and have it set to pass content when the servers are down (given the poor uptime track record on stage over the last two weeks, I can't do otherwise), so it's basically not doing me any good at all.

Debugging @smichel's issue turned out to be caused by something else: #1198198: Fallback mode is triggered on unexpected 'spam' => 0 response

Reverting to fixed.

Type mollom
Date Sunday, September 25, 2011 - 14:35
User Anonymous
Location http://www.sedansandsports.com/WKJeepSuspension
Referrer
Message All servers unavailable: http://174.37.205.152, http://67.228.84.11
Last error: -110 Connection timed out
Severity error
Hostname 66.249.72.198

@attilahooper: If you continue to see that error, please file a support request on http://mollom.zendesk.com/ - thanks!

I'm going to close to comments on this issue now, since this issue is closed for a long time already.