CAPTCHA challenge consistently returns failure

yelvington - August 28, 2009 - 16:11
Project:Mollom
Version:6.x-1.x-dev
Component:Code
Category:bug report
Priority:critical
Assigned:Dave Reid
Status:needs review
Description

We are trying to deploy Mollom on all our newspaper sites going forward, but we have been stopped in our tracks by a problem.

We are trying to protect the user registration form from scripted attacks with the Mollom CAPTCHA. However, the CAPTCHA challenge is consistently returning failure on Drupal 6 installs.

We upgraded to 6.x-1.10 and found this in the syslog:

Mollom ID '090828a0ca3db82d80' has been tampered with: it was generated
for a user with sid 109313e1e913938df27edba0b3cc866a, but is used by a
user with sid ee758cf8201b40a237e7c343847acf87

Our configuration has our Drupal application servers protected from the outside world. Inbound requests hit a load balancer that chooses one of two Squid/Squirm web proxies. A cache miss is forwarded onto one of several Drupal servers running Pressflow's enhanced D6.13 core.

In addition, I believe outbound requests from Drupal to the Web also are routed through a proxy/firewall.

Any advice on where we should look for this problem?

#1

te-brian - August 28, 2009 - 23:44

I am getting some similar problems. I also have enabled CAPTCHA on the user registration form (which has been aliased to '/join'). In my case I can eventually get the CAPTCHA to pass after a few tries but it's very inconsistent.

Similar to the OP, our server configuration is non-standard. We are using memcache for our sessions and we are also in a load balanced environment.

Please let me know if any other information will be helpful.

#2

yelvington - September 1, 2009 - 14:53

Our development guys currently believe the problem is caused by Pressflow using a nonstandard session handler. Brian, is there a chance you're also using Pressflow?

I think we need to ping David Strauss.

#3

flobruit - September 15, 2009 - 08:35

I'm experiencing the same issue on two sites, one of which is running pressflow and one which is not. None of them are in a load-balanced environment.

Mollom was working fine until about two weeks ago, when my client reported that all text analysis were reporting failures, so we disabled the text analysis. Then yesterday all captchas were failing for the authenticated comment form, so we had to disable that as well. The registration form is still protected, but somehow this one still works fine.

#4

Dries - September 16, 2009 - 06:51

That message can mean at least three things:

  1. your environment is setup properly and spammers are trying to do record-replay attacks by reusing Mollom session IDs,
  2. your installation's session handling is not working as expected, or
  3. you are caching pages that shouldn't be cached.

#5

Dries - September 16, 2009 - 06:56

By the way, I see (1) happening on my sites. Typically always from the same IP address. That is, spammers are using record-replay attacks so it is a legitimate message on my site.

It could be a legitimate message on your site as well -- but (probably) not if it happens (almost) all the time.

#6

Dries - September 16, 2009 - 10:20

Talked with DamZ about this, and it is indeed a problem with PressFlow (and Drupal 7). To enable more aggressive caching, PressFlow and Drupal 7 no longer keep a session if there is nothing to save in that session. As a result, session IDs are not maintained across page requests. Mollom uses the session ID to identify record-replay attacks across page requests.

A quick and easy solution would be to have Mollom create a permanent session ID by storing some dummy data in the session, but that would affect cache-ability of those requests.

#7

Damien Tournoud - September 16, 2009 - 10:32
Version:6.x-1.10» 6.x-1.x-dev
Component:Miscellaneous» Code
Category:support request» bug report
Status:active» needs review

Moved the session matching into $_SESSION, that should fix the issue with the new session manager of D7 / D6 Pressflow.

AttachmentSize
562374-store-mollom-sessions.patch 4 KB

#8

Dries - September 16, 2009 - 21:46

Would be great to get some test reports. :)

#9

David Strauss - September 17, 2009 - 20:05

The problem here is that requests for these pages won't hit Drupal/Pressflow in the first place if the page is being served via a CDN or reverse proxy cache. There's no easy solution to this because Mollom is designed to prevent replay attacks, and proper page caching induces essentially identical behavior.

It would be possible to fix this with an uncached edge-side include with Varnish and some CDNs, but that's non-portable to Squid and cheap CDNs. Javascript could do something similar in, again, a not-totally-portable way across JS- and non-JS-supporting browsers.

A really nice solution would be using lack of the has_js cookie to bypass the reverse proxy and using JS on supporting clients. The Mollom ID would be stored in a cookie that the reverse proxy would also filter out. That would work for everyone and allow the vast majority of anonymous users to load Mollom-powered pages from a cache.

The suggested $_SESSION-based fix above is a fix, but it will cause users to bypass the reverse proxy as soon as they've even *viewed* a page where Mollom works its magic. A slightly less bad quick fix would be Mollom setting a cookie that expires quickly (far faster than a typical session); this would allow users to quickly return to loading fully cached pages.

In the mean time, I've filed this as a compatibility bug on the Pressflow side:
https://bugs.launchpad.net/pressflow/+bug/432090

#10

Damien Tournoud - September 17, 2009 - 20:19

The problem here is that requests for these pages won't hit Drupal/Pressflow in the first place if the page is being served via a CDN or reverse proxy cache. There's no easy solution to this because Mollom is designed to prevent replay attacks, and proper page caching induces essentially identical behavior.

This is not an issue. Mollom will specifically prevent a page from being cached when a CAPTCHA is displayed. All the pages that needs to hit Drupal will hit it.

The suggested $_SESSION-based fix above is a fix, but it will cause users to bypass the reverse proxy as soon as they've even *viewed* a page where Mollom works its magic. A slightly less bad quick fix would be Mollom setting a cookie that expires quickly (far faster than a typical session); this would allow users to quickly return to loading fully cached pages.

Actually, there are two mitigating factors:

* first, a session will only be started when displaying a CAPTCHA-protected forms. Merely displaying a form will not start a session.
* we expire the Mollom session when it has been idle for one hour, and as soon as a Mollom session has been used (when a form is successfully submitted)

#11

j0rd - September 17, 2009 - 23:51

subscribing.

Similar, if not same, issue.

I have a load balanced website with 2x web servers and 1x database. I'm not sure what's in front of it to load balance though, as the host deals with this and I don't have access to that info.

Is there anything in Drupal's "Performance" caching if it's set to "Normal" and the Lifetime is set to "NOne" that could cause this.

#12

David Strauss - September 18, 2009 - 11:26

@j0rd: That is not a similar issue. It is a support request. Please take it elsewhere (new support issue, IRC, forums).

#13

David Strauss - September 18, 2009 - 11:32

@Damien Tournoud: That's comforting. Anything I can do to help this along?

@j0rd: Sorry, just realized you were asking about the effect of the cache configuration, not how to configure the cache on your host. To answer your question, a "normal" cache configuration will not cause this problem on Drupal or Pressflow, regardless of downstream proxies and CDNs (assuming they're configured to respect HTTP cache header RFCs).

#14

David Strauss - September 18, 2009 - 11:34

@j0rd: Wait, still investigating further about the "normal" option on Pressflow. I believe sessions are still lazy, and hence would cause this issue. Standard Drupal 6 should be fine.

#15

j0rd - September 18, 2009 - 16:43

I am using Drupal 6, not PressFlow.

I noticed when I turned my cache off, I was unable to re-duplicate the issue. I am leaving my cache off for the weekend to see what it does to my Spam vs. Ham ratio. Should my "spam" (failed because captcha doesn't show up) ratio go down, I'll assume it's an issue with caching and I'll get back to you guys on that.

#16

jeannine - September 18, 2009 - 20:48

I'm using Durpal 6.14 and but no PressFlow. Mollom is reporting failure everytime. This started for me last weekend (approximately) and i've been watching for updates to make it work all week. My caching has been disabled the entire time.

#17

rleigh - September 19, 2009 - 02:22

I also upgraded recently (from a site that had been on v5), and the captcha was failing all the time (blocked even though entered correctly), despite everything looking right in settings. I tried turning caching off, and still failed.

I turned off the module, ran the uninstall routine, then re-installed and and configured, and now it's working fine.

#18

David Strauss - September 20, 2009 - 21:18

I'd also be open to adding a drupal_session_id() function to PF6 and D7 that returns a session ID and starts a session if one isn't already in progress. Would that make this easier?

#19

j0rd - September 22, 2009 - 16:05

My spam rate is still hovering around 90%. I worry that this isn't all spam and that some stuff is not getting past mollum.

Is there a way to log which failed requests are blocked by mollom?

#20

Dries - September 23, 2009 - 11:06

j0rd et al, have you read http://mollom.com/blog/important-note-to-drupal-5-and-drupal-6-users ?

I'm still looking for some more success stories with the patch in #7. I haven't run the Mollom 6 tests yet with this patch applied but I will.

#21

j0rd - September 24, 2009 - 16:24

Thanks for the heads up Dries. I will update Drupal with my next stage roll out and see if that resolves the issue.

#22

Dries - September 25, 2009 - 18:20
Status:needs review» fixed

Marking fixed because these issues should be fixed now.

#23

David Strauss - September 28, 2009 - 16:48

Has this been fixed for users of reverse proxies? We are using the reverse proxy patch and Mollom on g.d.o, and I'd like to know if we should expect any issues.

#24

Dries - October 3, 2009 - 20:38
Status:fixed» needs review

Sorry, this issue isn't fixed yet.

#25

te-brian - October 5, 2009 - 07:20

Sorry, I'm way late to the party on replying to this. Way back when I posted #1 we were doing lots of debugging and determined that our the mollom cache was not persistent enough across the particular page requests that the mollom issue was cropping up on. For most loads of the page, the cache was being either not properly saved to memcache, or memcache was being dumped altogether. Occasionally it would be properly cached and loaded and the mollom logic would pass.

Our solution at the time, (and its still working) was giving the mollom cache it's own personal memcache bin. This allowed those entries to have the appropriate amount of persistence.

I'll have to run this thread by someone more knowledgeable about the current server configuration of the site to give any meaningful feedback. I'll also see if we can try out the patch and report any findings. As for now, things are running smoothly with mollom happily storing to its own bin.

One more note, my setup does not fall into the reverse proxy category that I believe you guys want testing feedback on anyhow.

#26

lintlin - October 9, 2009 - 16:08

I'm having mollom failures on my site, too. If text + captcha are enabled, comment posters get a "we're sorry, looks like spam, fill out the captcha", but there is no captcha to fill out. If I turn off the text and just use captcha, I get the image but it won't accept the input. It just returns the red warning "Captcha is a required field". I'm using drupal 6.14, so the September issue should be updated.

This is happening since this morning.

#27

davexoxide - October 11, 2009 - 18:10

subscribe

#28

toemaz - October 19, 2009 - 14:07

subscribe (using pressflow + mollom doesn't work)

#29

Scott Reynolds - October 20, 2009 - 16:50

As #25 mentioned, that was our solution as well. Give mollom its own bin. O ya subscribe :-D

#30

Dave Reid - October 20, 2009 - 16:56

I can confirm that the CAPTCHA on http://groups.drupal.org/node/add/account-synch works when I'm logged in, but not when I'm logged out. So it must be related to this problem as well.

#31

robertDouglass - October 21, 2009 - 02:23

The patch in #7 doesn't apply to DRUPAL-6--1-10, but it applies to DRUPAL-6--1. The patch allows CAPTCHAs to work on Pressflow, without CDN/Varnish. Not tested with.

#32

Dave Reid - October 22, 2009 - 17:20
Priority:normal» critical

#33

noelbush - October 27, 2009 - 11:01

I am using pressflow (with current bzr update) and the changes in the patch on #7 fix this problem for me. I had to adjust the patch to fit the latest mollom module -- the modified patch is attached.

AttachmentSize
mollom-session-handling.patch 3.42 KB

#34

noelbush - October 28, 2009 - 10:23

Correction: the patch from #7 sometimes causes an error in (patched) line 166 if $_SESSION['mollom_sessions'] is not set. So the attached revision of the patch, again against the latest mollom release, checks this first. Still corrects the problem for me.

AttachmentSize
mollom-session-handling-revised.patch 3.48 KB

#35

Dave Reid - November 19, 2009 - 19:22
Assigned to:Anonymous» Dave Reid

Getting Pressflow set up shortly to test...

 
 

Drupal is a registered trademark of Dries Buytaert.