LDAPsync component

Here's a first run at code. Thanks for taking a look. Help is welcome.

Attachment	Size
ldapsync.zip	4.62 KB

Thank you so much Richard for your work. Haven't tried your first code yet - want to clarify one thing first: do you attempt to only make changes in the Drupal database, but leave the data in the LDAP server untouched? If so, that's exactly what I'm looking for. My LDAP server is OpenLDAP and I am afraid of a possible data loss on its side if I try out your module.

That's right. At our institution, the LDAP information is authoritative.

Richard

Anyone tested Richard's code so far? It would be great if Miglius could have a look into it and comment here what he thinks.

Hi

Some comments from my posting at ldapauth (VERY promising module btw):

If the LDAPSync where to provide some configuration options to skip new users (not all of our ldap users are allowed to login) or present a checkable list of new users found before adding them, I could surely use it!

Another useful option would be a button to update a single user (I now force that by filtering on a UserId in the patch I provided).

wvd_vegt, the sync should ideally happen silently in the background, via cron. Thus adding a manual confirmation step before actually adding new users to Drupal won't do it.

wvd_vegt, I would recommend that the LDAP administrator place accounts that should not have Drupal privileges into a different OU, and then list only the OUs that should have Drupal accounts in the ldapauth settings.

+1 for #7. That's also the way we do it and the way supposed to be by other CMSes like Moodle. You should maintain an exclusive OU in LDAP that contains *all* users to be synched to Drupal and the other systems you want to use the same set of users (Moodle in my case). For example, the contexts I am using for both Drupal and Moodle is "ou=moodleusers,dc=mydomain,dc=tld" only.

It may not be practical to maintain an exclusive OU to Drupal users, b/c the sysadmin may also be using OUs to apply group policies, etc. However, if your Drupal user organization mirrors your LDAP organization, then at least all of the members of a particular OU should either get or not get Drupal user accounts. For example, you might have an OU for real people, an OU for system accounts, and an OU for computers. Only the OU for real users should get Drupal accounts.

Hi,

Due to some pollution we need to mop-up some things (ie pull it through ldapdata). A cron solution might do the trick though.

At the moment though we update manually as the view on top & user records only updates after explicitly pressing F5 (refresh) in the browser.

Furthermore do I have no influence over the AD so any changes there are out-of-scope

Hi,

As i said I have no influence over the AD content or procedures. I only pull users belonging to our department (if data was correct they should all have access) and even there is a bit pollution so I have to filter them and convert/adjust some attributes that are not updated on regular basis.

So our users are already in a nice group, but there is some pollution of the data so back to square one.

I agree that the AD data should be correct at the first place but to force it that's out of my reach (and i'm forced to live with it).

Here's another approach. You could allow ldapsync to pull all users from specified OUs into Drupal and then manually disable those users you don't want logging into Drupal. They don't have to know that you've done this, as ldapsync won't notify users of account creation. Ldapsync will post a watchdog message each time it makes changes to Drupal users, so you could keep an eye out for that and take action when ldapsync creates new accounts that you don't want enabled.

Richard, I'm not sure If you have read Miglius's comment about your new module in another ticket (which I can't find anymore now). He mentioned that he prefers your module to be released as its own sub module of the "LDAP" module group. Are you going to pull it at http://drupal.org/project/ldap_sync?

Yes, I imagined that it would be a sub-module of the LDAP module group. It already depends on ldapauth. It could even be a patch to ldapauth if people prefer it that way.

I also think a separated module is the better choice than providing a patch, as long as Miglius prefers this code to be independently maintained.

I have now uploaded your module to my site and reviewed its code through the excellent Coder module (http://drupal.org/project/coder), having the Show warnings at or above the severity warning level option set to minor (most). It gave a lot of (more than 100) warnings. Thus, I would suggest you to also use the Coder module in assisting for cleaning up your code to be drupalish.

Also, adding a line like
version = 6.x-0.1
to ldapsync.info would be helpful.

Easy enough. Done.

Basic functionality seems solid in my environment -- I created 935 users in a few seconds tonight. It would be great to find out how it does in yours.

I have noted in the code what line you can comment out if you want to test the module but don't actually want to create new Drupal users yet.

Next, I need to work on the settings page and think about how to reconcile the profile field needs of this module with the work of the existing ldapdata module.

Anyone else who would like to write code is most welcome to do so.

Attachment	Size
ldapsync.zip	4.8 KB

There are two options. This module can be release as a separate project and be developed there, or when it's ready and well tested I could include it in the ldap_integration bundle.

Excellent - thanks a lot, Richard! Now, the code of LDAP Synchronization 6.x-0.1-dev is reviewed by Coder with the result "No Problems Found". Perfect!

Once it is ready and tested enough, I would love to see it bundled in the ldap_integration package. It is so important for me, that it's a must have. And reducing Drupal's "yet-another-never-ending-must-have-module" pollution would be more user friendly, IMO.

I have now tried to enable the module for the first time, but immediately got a white screen of death afterwards. Inspecting the error log gave that the ldap_integration sub directory is missing in your package. You are referring to 2 files expected in that directory in the ldapsync_init() function.

Now, I can even no longer access any Drupal page at all!!!

Did you install ldapsync in /modules/ldap_integration? If not, please try that.

Yes, ldapsync does require files in /modules/ldap_integration/ldap_integration. However, it requires them in the same manner as ldapauth, ldapgroups, and ldapdata require them. Additionally, I specify ldapauth as a required module for ldapsync to function. So, I believe I have implemented these requirements in an appropriate manner.

Please let me know if this is fine or if you have a suggestion for improvement.

Richard

My ldap_integration directory looks like this. Note the ldap_integration subdirectory.

drwxr-x--- 4 4096 2009-03-19 12:02 .
drwxr-x--- 36 4096 2009-03-17 21:29 ..
-rwxr-x--- 1 1206 2007-08-29 19:05 ChangeLog
-rwxr-x--- 1 9 2007-01-29 21:06 .cvsignore
-rwxr-x--- 1 17434 2008-08-11 04:10 ldapauth.admin.inc
-rwxr-x--- 1 333 2008-08-12 16:15 ldapauth.info
-rwxr-x--- 1 3519 2008-08-11 04:10 ldapauth.install
-rwxr-x--- 1 15747 2008-08-11 04:10 ldapauth.module
-rwxr-x--- 1 10208 2008-08-11 04:10 ldapdata.admin.inc
-rwxr-x--- 1 339 2008-08-12 16:15 ldapdata.info
-rwxr-x--- 1 1822 2008-07-22 06:06 ldapdata.install
-rwxr-x--- 1 21693 2008-08-12 15:21 ldapdata.module
-rwxr-x--- 1 1675 2008-07-22 06:06 ldapdata.theme.inc
-rwxr-x--- 1 7110 2008-08-11 04:10 ldapgroups.admin.inc
-rwxr-x--- 1 363 2008-08-12 16:15 ldapgroups.info
-rwxr-x--- 1 2374 2008-08-11 04:10 ldapgroups.install
-rwxr-x--- 1 10029 2008-08-11 04:10 ldapgroups.module
drwxr-x--- 2 4096 2008-08-12 16:15 ldap_integration
-rwxr-x--- 1 173 2009-03-17 22:03 ldapsync.info
-rwxr-x--- 1 10088 2009-03-19 11:58 ldapsync.module
-rwxr-x--- 1 14940 2006-04-23 02:54 LICENSE.txt
-rwxr-x--- 1 1546 2008-08-08 05:42 README.txt
drwxr-x--- 2 4096 2008-08-12 16:15 translations

Ah, did not know that your module directory must be a sub directory of ldap_integration. Anyway, the LDAP integration module package has changed its file structure as of version 6.x-1.x-dev (2009-Mar-19). There is no longer a ldap_integration sub directory.

Its structure now looks this way:

Hi,

LDAPsync component team, very happy to see the solution for the LDAP sync. I'm also looking for the same component and I would like to start the test with LDAP. I think that I need to use my own function as $faculty in line 66 right. please confirm

Thank you
Sudahar

Here is a new version of ldapsync that is compatible with ldapauth-1.x-dev.

To try ldapsync, install the attached files within modules/ldap_integration. To sync users, go to /admin/settings/ldap/ldapsync. The sync is one-way (from LDAP to Drupal). I strongly suggest that you first try this module on a test site, as it will create a Drupal user for each LDAP user!

ssudahar, I have removed $faculty from the module code. I was building a profile data synchronization feature into the module but have instead decided to use existing functions in ldapgroups and ldapdata to add features to ldapsync. I will add those features next.

Attachment	Size
ldapsync.zip	5.45 KB

I have tried your current dev snapshot, commented out the line
$user = user_save('', $userinfo);
and uncommented the line
$page_content .= "<p>Create user ". $name ." ". $pass ." ". $mail ." ". $init ."<br />". $ldap_users[$name]['dn'] ."</p>";
within ldapsync.module to enable "debug only" mode.

Then pressed the [ Sync now ] button. This gave the message
Completed LDAP sync. New users: 0. Disabled users: . Orphaned users: 1.
but nothing else. Shouldn't that at least state to which users the mentioned actions apply? Also note the result of Disabled users (empty instead of zero). Why would your module not have added any new user? I have more than 100 on the LDAP server that should by synced.

I am trying to debug your code now. First error found is that you have defined $count_updated_users to be 0, but never used that variable anywhere. It seems that you intended that variable to be named $count_disabled_users instead.

Next error found: when starting the sync, a database error occurs. admin/reports/dblog reports "mysqli_fetch_array() expects parameter 1 to be mysqli_result, boolean given".

subscribing.

The current code is far away from working with OpenLDAP. I would suggest to completely remove the code related to $count_disabled_users, since it won't work on LDAP servers other than MS AD.

The $count_orphaned_users related code has to implement a measure of proof before actually blocking users. For me, that code blocked most of my fully valid users. Had to comment out db_query() there.

Roball, I welcome your help. From your report, I wonder whether the script is not finding any LDAP users when it executes the search. May I assume that LDAP users may already log into your site using ldapauth? Do you know whether the filter $name_attr=* works in OpenLDAP as it does with Active Directory? Can you insert watchdog statements periodically to monitor the buildup of the $users and $ldap_users arrays? For example:

I did previously see the mysqli error but solved it for my setup. This is a difficult error to troubleshoot (try Googling it), since the source of the error is not noted in the error statement.

$count_updated_users is fixed in the next module update (below).

$count_orphaned_users will misbehave if querying your LDAP directory isn't working. I'm not sure how to "prove" that a user doesn't exist if the script is not properly reading from LDAP.

Here is a new version that implements cron and groups support. I could use advice on groups support. Right now, it's simple and well integrated with ldapgroups but woefully inefficient, making one query to ldapgroups for each user in your directory. Can you think of a better way to implement groups support without modifying the ldapgroups module?

Please use this module on a test site only while it is in early development.

Attachment	Size
ldapsync.zip	6.52 KB

Some comments to your new version:

1. Replace require_once(drupal_get_path('module', 'ldapsync') .'/includes/LDAPInterface.inc'); by require_once(drupal_get_path('module', 'ldapauth') .'/includes/LDAPInterface.inc'); to prevent the first problem I had (see reply #20). This allows your new dev code to be tested inside an own directory ldapsync and removes the requirement of being inside the ldap_integration dir.
2. $page_content is not used anywhere - thus remove it, including the misleading comment
3. $count_updated_users problem reported in #28 is still there
4. You REALLY have to remove the code related to $count_disabled_users as mentioned at #31 - or at least make it optional by a checkbox setting
5. As mentioned at #31 as well, the code related to $count_orphaned_users is dangerous, at least with OpenLDAP and thus should be removed or made optional for now!
6. $num_matches is not used anywhere, thus remove it for now to avoid code pollution
7. The indentation at foreach ($ldapresult as $entry) is wrong and thus difficult to debug
8. ldapsync_admin_settings_submit is no hook - thus the comment is not correct
9. ldapsync_admin_settings_submit is no hook - thus the comment is not correct
10. variable_set() within hook_install() does not work - thus remove it and set the vars in ldapsync.admin.inc instead.
11. Please do not add new features unless known critical bugs are fixed first - thank you!

Here is a DEBUG ONLY version, with most of the fixes mentioned in #34 above. Also with code cleanup so coder does not complain. It does not change anything, so it is save to experiment with.

Attachment	Size
ldapsync-6.x-0.2-TEST.tgz	4.01 KB

Many thanks for these, Roball. Any luck finding out why the module's LDAP search is not returning results from your OpenLDAP? It seems that finding this out should be a top priority at this point in time.

Here is a new DEBUG_ONLY version, that fixes one important bug: the ldapsync_search() function has been renamed to _ldapsync_search() - otherwise it would be the implementation of hook_search() - so it would be called every time when somebody uses the site wide search box!

Attachment	Size
ldapsync-6.x-0.3-DEBUG_ONLY.tgz	4.04 KB

subscribing

We currently have two active developers/testers for this project. We are currently waiting for someone running OpenLDAP to test this module and troubleshoot the LDAP query that selects all users (currently "$name_attr=*"). Thank you.

Richard

Richard, I will continue to debug against OpenLDAP in the next view days.

Great module. Subscribing...

subscribing

I have updated the DEBUG_ONLY further (to the below attached 0.4 version), which requires ldapdata.module as well because all the sync mapping is defined in that sub module. Currently, when pressing the [ Sync now ] button at Administer » User management » LDAP » Synchronization, it should only display the LDAP attributes Drupal user fields should be synchronized to, like this:

Attachment	Size
ldapsync-6.x-0.4-DEBUG_ONLY.tgz	3.89 KB

Here is the first working version (ldapsync-6.x-0.5). It works for me with OpenLDAP 2.3.43 coming with CentOS 5.3. I have removed the dependency of the ldapdata.module for now, since data mapping can be done with that module itself.

Tested successfully with the LDAP integration module 6.x-1.x-dev (2009-May-06). The sync will only be performed on pressing the [ Sync now ] button.

Attachment	Size
ldapsync-6.x-0.5.tgz	3.82 KB

Many thanks, roball. Here is one small fix to the module. It now confirms that $ldap_users contains users before declaring Drupal users orphaned.

Attachment	Size
ldapsync-6.x-0.5.tgz	4.37 KB

Roball, something interesting happened to ldapsync when I upgraded to the latest dev versions. My search of Active Directory failed until I changed the user attribute to lower case. In other words, the setting "sAMAccountName" worked fine when I was on the previous dev version, but only "samaccountname" worked after I upgraded. I was unable to figure out why. I was still able to authenticate as an LDAP-authentified user, so I don't think anything is wrong in ldapauth. However, the help instructions for ldapauth settings suggest sAMAccountName, so that instruction is now incorrect, at least for my setup.

Richard

Same here.

I just tried this module, which I am very excited about, and wanted to report my findings.

I clicked the Sync Now button and it created 917 new users and 15 orphaned users.

1. In the new accounts there is only the username. If I open the account and save it then the rest of the fields come into the account. Should they not all come in initially?

2. What is an "orphaned user"?

Thanks again for making this module.

BTW, I was using a March 30 version of the -dev and upgraded to the latest, thinking that might fix things, but I got this error:

• warning: ldap_get_option(): supplied argument is not a valid ldap link resource in /srv/www/htdocs/sites/all/modules/ldap_integration/includes/LDAPInterface.inc on line 82.
• warning: ldap_errno(): supplied argument is not a valid ldap link resource in /srv/www/htdocs/sites/all/modules/ldap_integration/includes/LDAPInterface.inc on line 167.
• warning: ldap_error(): supplied argument is not a valid ldap link resource in /srv/www/htdocs/sites/all/modules/ldap_integration/includes/LDAPInterface.inc on line 167.

Any thoughts? Thanks.

So there were a couple problems with the latest version 6.x-0.5.

A. The attributes needed to be drupal_strtolower so the return from LDAPInterface.inc matched the attributes
B. There wasn't a filter to allow ldapsync to only get real users instead of users & computers, etc. (similar to ldapdirectory)

We made the changes to the above problems.

Please test this version and see if there are any problems.

Greg

Attachment	Size
ldapsync-6.x-0.6.tar_.gz	8.96 KB

better tar ball....

Attachment	Size
ldapsync-6.x-0.6.tar_.gz	4.24 KB

I don't see it in any comments in module files, and I don't see clear notation in this thread: are you guys working specifically against the latest development snapshots of LDAP Integration? Or the latest recommended releases? Just want to make sure to grab the right versions.

Worked great, thanks! Is there a way to bulk update the imported accounts with group/role mappings taken from our existing ldap_integration groups module settings?

Yes, are you comfortable looking at the module code? I thought I put that in the current version, but I may have it only in my Drupal installation. Let me know, and I can publish what I did locally.

Richard

Pretty comfortable - I don't see it in the current version. Can you post your most recent please?

I had this working now I'm out of the water again. When I try to sync, it takes forever (a few minutes) then gives me no status update. I remember before it would tell me how many accounts it saw and updates. Now it just goes to a white screen. I tried upgrading the LDAP Authentication module to the beta 1 and got the same thing.

Richard, would love to see this feature. Any idea when you'll have a chance to upload the code?

Here is the latest code from my production site. I suggest taking a look at this code for ideas, not just installing it on your site, as it does not contain the latest improvements from other contributors to this thread. It would be much appreciated if someone were interested in merging the changes into a new version.

Attachment	Size
ldapsync.tar_.gz	3.88 KB

gettysburger, if you turn "display errors" temporarily, you may find out why your site is going to white screen.

In sites/default/settings.php, add:

ini_set('display_errors', 1);

subscribing.

This is a *must* component! I hope that this component is integrated in the module. Thanks for it.

Richard,

EDIT: This post was way too long and had a lot of extraneous stuff in it. So I am shortening it..

Our Drupal/intranet users are authenticated by an enterprise HTTPD ID (via the PubCookie system) and I am able to auto-create Drupal accounts compliments of the Webserver Auth module. Previously, getting LDAP->Drupal group mapping has depended on LDAP authentication which we don't have. I have installed the latest ldap_integration with John Barclay's group mapping enhancements. But still no luck getting group info for users

But then I found this thread. If we can simply use this for cron'd LDAP->Drupal synch of users and groups, it could hugely reduce the number of hits on our LDAP lookups on our server.

Anybody try it against "389" (nee Fedora Directory Services) or will I be the first?

Jim

Thanks. I'll give this a try.

I set up another test Drupal 6.14 test instance and added latest WebServer Auth and LDAP Integration modules. Then I added LDAP Sync from post #58 and made no modifications whatsoever to those 4 files..

- I got the LDAP Sync configuration menu immediately
- I saw the "Sync Now" button (Duh!)
- I ran sync and it successfully synched all users in the defined ou=People container

I found no evidence of any sync of group/role information. EDIT: Wrong. I was looking at some users that had "webserver_auth" in their authmap and, no, their group membership did not update.

FYI, this was run against Fedora Directory Services v1.x (now "389 Server").

Thanks,

Jim

Regarding group sync, I checked and saw that your group section:

was being called as expected. So I followed the trail to the ldap_groups_login function in the updated ldapgroups.module. There I see it is doing a test for ldapauth right at the top of the function:

I bypassed this by simply commenting out the test and the next time I sync'd, the user and user_role tables populated all of the LDAP groups and user group/role membership. Edit: I think this was a red herring. I had a couple of users in the user table already and their authmap shows "webserver" auth type. Group sync for remaining newly-synced users with authmap auth value of "ldapauth" worked. I guess I can just leave all of these authmap values as "ldapauth" (even though they are not).

Great. So now it looks like I should turn my attention to the ldapgroups module and understand it better. Biggest issue is:

- having orphaned users is OK (we actually want to keep them) but group membership must be kept up to date with LDAP being authoritative.

I need to see what the ldapgroups module does to group memberships when a user is removed form a group in LDAP.

Edit: I checked -- I removed a user from an LDAP group and the change was propagated from LDAP to Drupal when I sync'd.

Anyhow, I am using SSO/HTTPD authentication and, am synching LDAP group membership for (Drupal) roles/authorization. Yay!

Thanks!!

Jim

subscribed (to the right one this time :] )

This is incredibly useful, when will we see this rolled into LDAP integration? or moved into it's own module : http://drupal.org/project/ldap_sync ?

subscribe

Subscribe. I am planning on installing and testing the latest patch as indicated in comment #58.

Thank you for installing and testing! This components needs more of that to move to the next step. Please do suggest code changes or attach a new version if you can.

Richard

Subscribing

I've tested the code in #51 and it appears to be working fine. I can sync and it will bring in my users.

I did find one error with PostgreSQL in #51 as well as #58.

Line 205:

Should be changed to this for PostgreSQL to work:

Hi this will be an excellent module, when will this be available as a dev download, i'm just wondering if i should start fiddling with it now or wait for the module to be released?

Here is a new version with the following improvements:
- added new sync time interval of "never (manual sync only)" to admin settings
- added new admin setting to control whether Drupal users missing from LDAP are automatically blocked
- limited LDAP search to enabled accounts (only works with Active Directory)
- confirmed that ldapsync works with LDAP integration 6.x-1.0-beta2
- fixed SQL compatibility issue noted in #71. Thanks.

To do:
- need someone with a non-AD LDAP directory to add code that limits LDAP search to enabled accounts (line 230)

Given previous comments, I'm happy to move this into its own module, once I look into how to do that! Please let me know what additional issues you find. Thanks.

Attachment	Size
ldapsync-6.x-0.7.tgz	5.49 KB

Subscribing

Has established ldapsync-6.x-0.7, two times it was synchronised, then in general has ceased to work. I add new users in AD, then I press SYNC, gives out "Completed LDAP sync. New users: 0. LDAP-authentified users that do not have an enabled LDAP account: 0." In what there can be a problem?

Added. Has found a problem. FCKEditor has added tags in filter LDAP Integration parametres. Therefore synchronisation also did not work.

There was a second question. The module now can block users that do not exist or are disabled in LDAP, but whether can it unblock again enabled users?

Thanks for trying this out. I'm glad you found the issue with FCKeditor and the LDAP server settings.

Thank you for your suggestion to enable blocked Drupal users that are enabled in LDAP. The only problem is that a webmaster might have blocked such users in Drupal manually, so I don't want to take that control out of the webmaster's hands. I'm interested in what others think about this feature suggestion.

LDAPsync seems to be working fine for us running the latest 389 Directory Server!

Great! Thanks for reporting back.

(changing category back to "feature request" and status to "needs review")

We still encourage more testing to confirm the reliability of this module. I have submitted a CVS account request in order to move this into its own project.

The module is working very well! Its running on our servers and the synchronization with our AD server is coming up correctly. However, there *is* one issue which has to do with the cron tasks. On our servers we noticed a reasonably large spike whenever we ran cron (in our case of running every 10 minutes, it wasn't looking good). What I found is that currently, placing a setting for the ldapsync to run cron a max with its minimum time does not currently work. The reason is that while you store the time interval between syncs in hours, the difference is checked in seconds. So for the values you currently have:

1 hour is 1 second, 6 hours is 6 seconds and so on (with the exception of 999999 which you have a special case for). So my recommendation would be (if you are sticking with hours) to multiply the time interval by 3600. Your cron function then becomes:

999999 is a very long time (over 100 years) in either case and you don't need the extra check.

And that should fix the cron issue.

Thank you BTMash for finding that issue. I've fixed it and will include in the next version I post. I actually made the change in the admin settings field value instead of here, so that the stored value in the database is in seconds.

Richard

Here is a new version that includes a fix for #79 as well as changes requested as part of my CVS application review.

Attachment	Size
ldapsync-6.x-0.8.gz	5.31 KB

Subscribing

I think the status of this issue should be kept on "active" as long as this module gets official.

The latest sync works great. They one thing I do notice is that I need to sync twice. 1) gets the the user and adds them 2) gets all their data and group membership). Is this behaving as it should or is this a known bug.