Posted by colemanw on February 4, 2011 at 4:27am
11 followers
| Project: | Drupal.org CVS applications |
| Component: | new project application |
| Category: | task |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (duplicate) |
Issue Summary
CiviCRM is a powerful open-source contact management system, used by thousands of organizations around the world. Webform is also a very popular Drupal module that is widely in use. What I found myself doing over and over was creating a webform, adding fields like "first name" "last name" "phone number" etc. And then writing a little temporary module specifically to process that form, calling up the CiviCRM APIs, matching webform fields to contact fields, using hook_form_alter to set every field's #default_value for logged-in users, then saving the contact upon submission and creating a civicrm activity. Doing it once wasn't so bad, but after a while it gets pretty old. And that sort of thing is completely out of reach for someone who doesn't know PHP. So I decided to help myself out (and everyone else in that situation) by writing a module capable of doing all that and more for any webform. This module provides the missing link for anyone who uses CiviCRM and wants to collect information about, or track interactions with, their contacts. It allows you to:-Automatically create webform fields that are linked to the CiviCRM database.
-Have forms auto-fill for logged in users.
-Have forms auto-fill for anonymous users if you send them a special hashed link through CiviMail.
-Automatically log activities when users fill out your form.
-It uses your CiviCRM default strict deduping rule to decide whether to update an existing contact or create a new one when the form is submitted by an anonymous user.
-It imposes no restrictions on how you style, rename, nest, or edit your CiviCRM fields, you can do anything with them that you could do with any other webform field.
Comments
#1
Coder likes it, I hope you do to!
#2
When would you recommend using webform with this module, or creating a CiviCRM profile to use as a front-end form?
#3
Fixed a bug for logged-out users.
Here's the new version.
#4
CiviCRM profiles are useful if you are collecting only CRM data, and you don't need to do any additional processing with the form submissions, like creating activities or sending emails. Here's what you can do with this module that you can't do with profiles:
The big question might be, why did I decide to create webform fields to collect CRM data instead of creating a way for Civi profiles to be embedded in webforms? There are pro's and con's to both approaches, and I make no claims to ultimate superiority in this approach, but I do believe there are more pros and less cons to doing it this way.
The advantages to my approach are:
-autofill works for both logged-in users and hashed links to the form
-civicrm fields can be interspersed with non civicrm fields instead of presented all in one block
-civicrm fields can be used for webform token values (i.e. in post-submission emails)
-civicrm option lists (gender, name prefix, country, etc) can be limited to a smaller subset for the form, and the options can be re-named on a per-form basis
The disadvantages are:
-no support for custom fields (although I plan to add this soon)
-since option lists are stored in the webform, they will not auto-update if you change those options in Civi. I have however created a nice ajax button for updating options.
#5
Thanks for the testing, CiviCRM community!
Another bug fixed here.
#6
Coleman - I think this is just awesome - it opens up whole new horizons for managing complex Profile type forms - and all with the UI that Webform provides for setting fields sizes and all that. Most grateful for your work on this.
And yes Custom Fields will be great addition, esp if that lets us push some data direct in to fields in the Activity that is created.
#7
Have tested this - works great with latest Webform module - failed with 6.x-2.10 - works with 6.x-3.6 on a civicrm 3.2.4
#8
Yes, I believe the 2x branch of Webform lacks the hooks necessary for this module to work. People who wish to use this module are advised to upgrade to the latest version of Webform.
#9
Didn't mean to change the status, changing it back.
Another important note is that, as with civicrm & views, you must have proper db prefixing in your settings.php for everything to work.
#10
OK folks, by popular demand, here is a new version with the option for hidden CID fields, and support for almost every type of custom field there is. I've also enhanced the handling of updated submissions to give (IMO) the expected result of sticking with the original CID instead of switching to that of the person doing the editing. And there's more goodies like creating a personalized intro message for known contacts, and the ability to block unknown contacts from using the form. I've also filled out the documentation quite a bit.
Please note that the schema has changed to accommodate these new features, and since this module is not yet "released" I didn't create an auto-update. If you have been testing this module, please disable it, uninstall it by going to admin/build/modules/uninstall, and then enable the new version. All the crm fields in your webforms will still be there, so just re-enable civicrm for each webform and you're good to go.
Enjoy!
#11
Nice work - will test it next - but to save others losing half an hour on the 'how the hell did i c*ck that up' this version appears under CiviCRM when the previous version appeared under Webform in the Modules list (maybe just on my set up but ....) - anyhow be warned, if you only look for it in the place you just disabled it, and cant find it, that might be the reason ;-)
#12
hey well coleman you are obviously enjoying yourself building this - and it is real good - like finding a new flavour of chocolate.
Tested most aspects i think.
Custom data - brilliant, tested several options including Radio buttons and Checkboxes all work fine and the output in the email has the values
Hidden ID - yay (super brilliant would be to have it automatically build a link to the record but hey brilliant is still, well brilliant)
Tokens - yep that worked great too once i went back and actually copied what you had listed rather than using the usual [first.name]
So big thumbs up and will reset to 'reviewed'
Pete
#13
This is a great module, just what I need. However, the Activity Type list doesn't come up with any options other than "no activity".
I'm using Webform 6.x-3.6, CiviCRM 3.3.3, and I've updated the Settings.php file to include the table prefixing settings.
Any idea why it's not picking up my activity options? I've tried granting/removing a few permissions, but no luck so far...
#14
saparker: That is indeed strange. Those are the same versions of software I'm using, and you are the first to report that problem. Since you didn't mention any error messages I assume there were none, so I further assume that drupal found the db tables, but the query returned an empty result. That baffles me a bit, but there must be a reason :).
The options for that list are pulled using the exact same function as every other civicrm option list, so here's a test: try enabling a field on your form that uses options, such as gender. Then on the webform tab, click the edit button on that field. Now look at the Options box: the expected result is that it will have some options in it like:
1|Female
2|Male
But if my hunch is right, then that box will be empty (even when you click the "refresh options" link), and I am still baffled as to why.
Try going into phpMyAdmin and running this query directly on your civicrm database:
SELECT value, label FROM civicrm_option_value WHERE is_active = 1 AND option_group_id = (SELECT id FROM civicrm_option_group WHERE name = 'activity_type')If that doesn't return any results, then there's something odd about your database. You could send me a dump of those two tables and I'd be curious to see what's up with them. If it does return results (and it really should), then there's something odd about drupal's connection to your civi database.
Hope that helps. And I hope we can get a real project page for this module soon so that these issues can be properly filed.
#15
Coleman - This module is fantastic and long overdue. However, while testing I repeatedly get these two errors. Any suggestions?
* user warning: SELECT command denied to user 'myPrefix'@'localhost' for table 'civicrm_custom_field' query: SELECT * FROM myPrefix_civicrm_prod.civicrm_custom_field WHERE is_active <> 0 AND custom_group_id IN (SELECT id FROM myPrefix_civicrm_prod.civicrm_custom_group WHERE extends IN ('Individual','Contact') AND is_active <> 0) ORDER BY custom_group_id, weight in /myDrupalpath/sites/all/modules/webform_civicrm/webform_civicrm_utils.inc on line 356.
* user warning: SELECT command denied to user 'myPrefix'@'localhost' for table 'civicrm_option_value' query: SELECT value, label FROM myPrefix_civicrm_prod.civicrm_option_value WHERE is_active = 1 AND option_group_id = (SELECT id FROM myPrefix_civicrm_prod.civicrm_option_group WHERE name = 'activity_type') ORDER BY weight, name in /myDrupalpath/sites/all/modules/webform_civicrm/webform_civicrm_utils.inc on line 180.
Thanks for the help.
#16
coleman - thanks for the tips. You're right that, for me (see comment #13 above), it was a general access problem for Drupal accessing the civicrm database: in my error logs, it was refusing access to the civicrm tables. It looks like this is the same problem that jicarola is having above.
The solution for me was to grant SELECT access (in CPanel) to the MySQL Drupal user to my civicrm database. In the past, I've had separate MySQL users for my Drupal and civicrm databases, and it's not caused a problem. Not being a MySQL expert, I don't really understand why Drupal was allowed to populate the webform fields with my logged in data, but it wasn't allowed to perform the necessary SELECT queries. Anyway, the fix works and I'm now busy converting my various webforms to integrate them with Civi.
I don't know if it's too early to suggest improvements, but would be possible to have the state-province as the full name drop-down rather than the 3-letter abbreviation? (I live in the UK, and the 3-letter abbreviations are fairly meaningless here...)
Thanks again for a great module.
#17
OK, in order to solve the above two problems, and to stem the potential flood of similar issues, I have rewritten all queries to the CiviCRM database to use CiviCRM functions instead of Drupal functions.
So db prefixing is no longer required by this module!
Here's the new version, no schema changes, so no need to reinstall or run update.php (unless you're upgrading from a version prior to the one posted in comment #10, in which case follow the instructions in that comment).
Enjoy!
#18
Coleman - It looks like the updated DB queries did the job. No more errors. Thanks again.
#19
@jicarola You're welcome. FYI the problem was that your db_prefixing was misconfigured. If you ever want to use CiviCRM+Views, you'll still have to fix it.
But at least now all the people in your situation won't be posting in my issue queue about it :)
#20
This module is outstanding - we're very hopeful it will solve an ongoing problem for us. A couple questions:
Is there a way to accommodate custom fields already defined in CiviCRM?
Also, any plans to populate the state dropdown based on country input as CiviCRM profiles do?
Thanks for a valuable complement to Drupal and CiviCRM!
#21
@colemanw, I've been testing a bit while I wait for a response to the above, and have run into an error (below). I submitted my webform and it created a contact in CiviCRM with an associated Activity (an event registration). When I open the Activity tab and click "View" for the event registration, however, this occurs:
* warning: array_key_exists(): The second argument should be either an array or an object in /srv/www/domain.org/sites/all/modules/civicrm/CRM/Event/BAO/Participant.php on line 646.
* warning: array_key_exists(): The second argument should be either an array or an object in /srv/www/domain.org/sites/all/modules/civicrm/CRM/Event/BAO/Participant.php on line 646.
* warning: array_key_exists(): The second argument should be either an array or an object in /srv/www/domain.org/sites/all/modules/civicrm/CRM/Event/BAO/Participant.php on line 646.
Firstname MI. Lastname
unrecoverable error
Sorry. A non-recoverable error has occurred.
One of parameters (value: ) is not of the type Integer
Return to home page.
#22
Hi m.e.
I'm glad you are finding this module useful. Here's my answers to your questions:
Here is the new version with the above mentioned improvements. Again, no schema changes so you can just replace the code and continue using it.
#23
Hi again @colemanw,
Thanks for such a quick response. The st/prov list is above my pay grade -- I think we'll just let people type it in since we can't assume everyone's in the same country.
On the activities choice, I'll just make another selection. For our purposes that tab isn't very important ... I just happened to pick the worst option.
Turns out I had my custom data disabled .... now that it's enabled I can see the fields just perfectly.
Feature suggestion: have you considered incorporating the CiviCRM option of creating a Drupal account when an anonymous user submits a form? or possibly assigning contacts to a specific group rather than giving them a choice of groups?
BTW, the way your module has helped us the most is with SSL. I've been struggling for a week without much progress to coax the CiviCRM dashboard to sense the SSL cert and force secure URLs. Webform loads in https without a second glance.
#24
It's working fine for me. I just would like to know if I can automatically add them to a group. Currently it's by "select". For me they won't have a choice of groups. It's just going to a "newsletter" group, so I don't require anything visible in the actual form.
Thanks for the great module.
#25
@m.e. & @15medium
I'm glad everything is working for you.
I just went through the option list table again and found the column responsible for determining if an activity is reserved by the system. So try this new version and you should see a much more manageable list of activities on the civicrm tab of your webform.
In response to your feature requests, one seems pretty doable, and the other is probably beyond this module's scope. To automatically add webform users to group(s) without giving them any choice about it, you can select the groups you want (while editing the groups webform component), add their group IDs to the "Values" field, and then set the field to be disabled. A more unobtrusive way would be to have the form element be hidden, I will try to think of a way this module can facilitate doing that more easily, and also make setting default values easier so you don't have to look up group ids in civicrm.
As far as creating user accounts, it seems like that request really belongs in the webform issue queue, since the ability to create users from webform submissions really wouldn't have anything to do with civicrm.
#26
Thanks again, @colemanw. I do see your point about the user accounts and will give the groups workaround a try.
I'm just noticing that once I've selected fields for the webform, I can't deselect them. Is that intentional? (Update - now I see that I would just do that via a delete on the list of webform fields, not in the CiviCRM part. Leaving this info here in case someone else runs into the same question.)
#27
@colemanw, I've run into a snag - possibly a bug? I've added all the remaining fields to our form and changed the activity to "inbound email." When I log out and submit the form, webform records the submission but the contact record is not being created in CiviCRM. I have not yet updated the module (since yesterday) and an anonymous submission did work then. The only significant change I've made other than adding a fair amount of custom data is to put custom text ("submit application") in the submit button. Any ideas?
#28
Yikes, you're right. Wow. That's a doozey.
OK, after a lot of testing and head scratching, I've isolated the problem. Specifically what's happening is that contacts are neither being created nor updated if the form contains custom fields and the user in question does not have the civicrm "access all custom data" permission. Further investigation led me to discover the source of the problem -- a bug in the CiviCRM contact API which not only fails to process custom data without that permission set, it also quietly crashes and burns and fails to do any processing for that contact if the params include custom data and that permission is not set.
Bummer.
I am filing an issue with the CiviCRM API team (oh wait, I'm supposed to be on that team!).
Meanwhile, I've fixed things as well as I can for this module. Processing for new contacts now happens in two stages -- one for the core data, then a second pass for the custom data. This will prevent the crash&burn scenario from having the contact not be created/updated at all.
But in order for the custom fields to be processed, you still need to grant "access all custom data" permission to all users who will be using this form. If this is a problem for you because you are doing complex things with your ACLs, please bug someone on the CiviCRM API team about it (other than me :) ).
This new version also includes the feature request from #24. In addition to giving your users a choice about joining groups, this module can now create a hidden field that will just quietly add them to whatever group(s) you want. And default options for the regular (non-hidden) groups field can now be set by checking boxes rather than requiring you to know the group ids.
Please note that if you are using groups for mailing lists, not giving your users a choice about opting in may violate the anti-spam policy of your web host, or worse.
#29
Thanks much for the quick fix, @colemanw. Re: the permission thing, our form can (and usually will) be submitted by anonymous users. In what way other than this form would they be able to "Access Custom Data"?
#30
I'm still learning the intricacies of CiviCRM ACLs, but as far as I know, giving "access all custom data" to your anonymous users is not a security risk. Since they don't have much (if any) access to CiviCRM anyway, it doesn't really matter if they're allowed to view custom fields or not. I believe the only reason to ever not grant that permission to all your users would be if you needed to restrict access to a sensitive custom field, like "annual income" or "nude photos"
#31
I'm a noob to both Drupal and CiviCRM. I downloaded the latest tarball, and it's not like a usual Drupal module. Looks like straight php code. How exactly does one install this module?
#32
Well, I was pretty much in your shoes this time last year, so there's hope for noobs :)
There is nothing unusual about this module, it is straight php code, same as any other drupal module.
To use it, extract the tarball into the sites/all/modules folder so the files are located in the folder sites/all/modules/webform_civicrm. Then you may enable it by visiting yoursite.com/admin/build/modules
The file readme-instructions.txt will tell you how to use it.
#33
Ok. Problem solved. Your tarball is just built funny. The tarball contains another tarball with a modified extension (the _6 part). I extracted the tarball and then renamed the resulting file to remove the version info and then extracted it's contents. Now it makes sense. Cheers!
#34
@colemanw, I have the latest version installed now and ran a test as an anonymous user. The good news is we have a CiviCRM contact again! Whoot! The bad news may be something on my end, but I can't seem to find the contact by searching for an activity. The contact is there, the activity is there, clicking the contact's activity tab shows the "inbound email" - subject "financial aid application 2011" - but when I do an advanced contact search for an activity with exactly those values, or even just the subject without the activity type, nada. Any ideas on this? I do have the "Access Custom Data" permission assigned to anonymous users.
There is also one custom field whose value is loading into webform results and shows up in an Edit screen for the CiviCRM contact, but does not show up for a View of the contact. It's the only field of type "money" on the form and I'm wondering if that's why ...
Finally, and this is a nit, one of my integer-select fields is showing out of order in the dropdown. The choices are 0 through 10, but the dropdown shows 1, 0, 2, 3, etc. The values are in the correct order in my custom data definition.
Hope these tests are as helpful for you as the module is for us!
#35
@m.e.
I don't think I can be much help with that, since those all seem to be problems you are having with CiviCRM and not this module. This module creates activities simply by handing the data off to the CiviCRM API, so they should behave just like other Activities, and if they don't it's a problem with Civi APIs.
Your money custom field, again sounds like this module is working correctly in populating that field, but perhaps CiviCRM is having trouble displaying it for some reason. That also sounds like a core or API issue, but I will look into it a little.
Update: I looked into it a little, and was able to create a "money" custom field, fill out the webform, and it showed up fine on my contact summary. Let me know if you have any more specifics about this.
And the order of options, I'm a little stumped as to why they should be any different, unless Civi has an alternate sort method I don't know about. When my module pulls options it orders them by "weight," then "name" which ought to be consistent with how Civi does it. The good news is that you can re-order them yourself by editing that webform field.
Cheers.
#36
This is fantastic! Any plans to release as a standard drupal module?
#37
Hi @sonicthoughts
The purpose of this thread is to go through the process of releasing this as a standard Drupal module. So far no Drupal moderators have reviewed it, so the process is in limbo until one of them gets the chance to do so. I imagine they are all busy with the CVS->GIT migration.
#38
Thanks, @colemanw - it isn't always easy to tell where a problem is occurring.
FWIW, replacing my "money" field with another numeric type solved the problem with that data. There are more than the standard number of select items in this field and upon further testing of the money version I noticed that choosing a value within the usual limit (first 10 choices) generated the display problem while choosing one of the extra values (added after the field was saved) tested clean. Obviously something going on in CiviCRM there....
By using your hidden add-to-group field we got around the search problem (where I'd planned to use a smart group).
So all is solved for us - thank you again! We look forward to the "official" version.
m.e.
#39
Here is the latest version of the Webform CiviCRM Integration module. I'm marking this as the first beta release, and I believe it is fit for use on production websites.* I've been using it on my own live site for weeks already.
Important: If you are upgrading from a previous version, please run update.php after copying this new version to your modules folder.
No bug fixes in this release, since there have been no bug reports :). But lots of little improvements, niceties, and a handy new feature.
New feature: User message. This feature exists to help prevent a major CRM headache: If users view your form while logged-in as someone else, or they click to your form by following someone else's personalized link (i.e. from a forwarded email), they will see that person's details on the form. Not given any alternative, they are likely to manually clear those fields and type their own information, which would cause the existing contact to be updated with a different person's details, throwing your contact data into confusion.
When enabled, users will see a configurable message instructing them to "click here" if they are not the intended contact. The link will take the appropriate action (logging them out if they are logged in, or else getting rid of the personalized hash) and bring them to an anonymous version of the form.
*This statement has not been evaluated by Drupal moderators of this forum. You have been warned.
#40
Holy moly - this is sensational! Just installed it and configured my first fully-integrated webform and it looks like it's working beautifully. Seamless. Coleman - thank you so much; this is fantastic work!
#41
Thanks everyone for your support and encouragement.
Here's beta-2 with the following improvements:
And best of all (my personal favorite):
If you are upgrading from a version prior to beta-1 (#39 in this thread), run update.php after replacing the old module.
To make the real names show up on your webform-results, you may need to visit admin/build/theme to trigger a theme refresh.
#42
Whoops, my bad. Don't use the above version unless you like getting SQL errors on every webform submission. (I need to be a little less hasty about posing the latest and greatest)
Use this one instead.
#43
Coleman - latest version checks out A-OK on my end. This module really is the "glue" between Drupal and Civi that makes it all work nicely; no more messing with those awful HTML form snippets! Thank you SO much - the whole process is almost fun now. Almost ;-)
Rock on!
#44
@bcobin Thanks! I hope you get some good use out of it. Are you using it for political campaigns?
#45
@all
If any Civi webmasters reading this feel like this module is making your life easier and would like to help out, I'm getting ready to set up CiviContribute for my org (and migrate a bunch of data from another system). I would love to be able to pick your brain(s) if you have experience with either fundraising db planning or data migration. Give me a shout if you're available to advise. Thanks!
My email is the same as my drupal username, @woolman.org
#46
Thanks again, @colemanw - this is indeed a huge contribution to both Drupal and CiviCRM. I was able to install the beta version without a hitch, FWIW.
I've been following some threads about adding same-page conditional fields to Webforms. There is now a (separate) module that does this (discussion of merging with Webform is ongoing). Is it out of the question that your module would be able to accommodate conditional fields? I haven't yet attempted a test - wondered if you had.
m.e.
#47
@m.e. Conditional fields are just a bit of JS to show/hide DOM elements, shouldn't interfere with this module at all. Give it a try and let me know.
One thing that may be a bit more iffy is multi-page webforms, I think it should work, and if it doesn't it shouldn't take much tweaking to get it working. If someone wants to test it, feel free.
#49
Update: This issue is now obsolete since Drupal has moved to GIT! You can now find the latest code and issue queue for this project at: http://drupal.org/sandbox/colemanw/1073296
#50
Just noting that testing with http://drupal.org/project/webform_conditional and report back that all was good. Built a Condition based on CiviCRM Field Gender, and set some Custom Fields that were gender specific in to be conditional on the above and it all fired through with data being updated in civi - good stuff
#51
Very happy to see this module. Thanks so much for the effort. In trying it out, receiving the following message.
[15-Feb-2012 19:08:36] PHP Fatal error: Call to undefined function civicrm_api() in /sites/all/modules/webform_civicrm/webform_civicrm_utils.inc on line 1647
The function is below. Not sure if overlooking something in the setup or maybe too find in versions.. This is CiviCRM 3.3.3 and Drupal core 6.22.
Thanks again.
<?php/**
* Wrapper for all CiviCRM API calls
* For consistency, future-proofing, and error handling
*/
function webform_civicrm_api($entity, $operation, $params) {
$params += array(
'check_permissions' => FALSE,
'version' => 3
);
$result = civicrm_api($entity, $operation, $params);
if (!empty($result['is_error'])) {
$bt = debug_backtrace();
watchdog('webform_civicrm', 'The CiviCRM "%function" API function returned the error: "%msg" when called by line !line of !file with the following parameters: "!params"', array('%function' => $entity . ' ' . $operation, '%msg' => $result['error_message'], '!line' => $bt[0]['line'], '!file' => array_pop(explode('/', $bt[0]['file'])), '!params' => print_r($params, TRUE)), WATCHDOG_ERROR);
}
// The API doesn't always sort correctly, so we'll do it here:
elseif (!empty($params['options']['sort']) && !empty($result['values'])) {
$values = array();
foreach ($result['values'] as $val) {
if (empty($val['is_primary'])) {
$values[] = $val;
}
else {
array_unshift($values, $val);
}
}
$result['values'] = $values;
}
return $result;
}
?>
#52
@ civijim
post your ticket here - http://drupal.org/project/webform_civicrm
more chance of it being noticed and useful to others too ;-)
#53
done - thanks.