Contact matching framework

Here are some ideas:

* matching could be done within contacts of specified types or within all contacts in the system it will be nice if it could be configurable.

* multiple matching option sets: in some cases we want to sync individuals, in other - organization and we could use different field set for each match.

* weights vs and/or: i think weight system is better and it is more flexible. If we want to use AND logic we can assign the same weight for each field and set threshold the max value, for OR we need to assign threshold to min value or we can create multiple matching options sets.

* Each field could have different matching algorithm, in most cases it is a strict equality of values. But for the Name and Address fields there could be fuzzy matching and slightly different algorithm. E.g. Bob could be matched as Robert, or Vadim - as Vadym. If we are speaking about name field there are many sub-fields, which could also have different weights. What about address field it depends what kind of matching we want to make: proximity or strict matching. In some countries there are old zip code system that could math to the new one. The phone could be matched differently too. And what about area code system? We could go further and start matching images (like Facebook or Google+ do).

SO THE CONCLUSION: each field type could support different types of matching and should implement special hook to define those algorithms. We do not need to implement this hooks, let other modules do this.

* What is a contact? It is an entity. So what about making this framework to work with other entities, cause it is almost the same code.

* Performance. This should work pretty fast. To speed up the process we need to use SQL for searching/matching value in field. An SQL could depend on each field. In most cases we will use LIKE operator, but in other developer may want to use SOUNDEX (see http://en.wikipedia.org/wiki/Soundex) or some weighted staff. This is important thing and we should carefully thing about it.

* When to do match? When adding/editing contact? Before or after? Well it depends on a use case. In some cases we may want to avoid of adding duple in db, in other cases we may want admin to merge contacts. I am still not sure what hook we need to call. Pre save hook does not allow to block transaction, only to modify data (so we can mark this contact as duple). Actually it is possible to trow an exception, but it is not always good. We can call multiple hooks, have dedicated function to be called manually from modules. There could be modules which will want to do it in cron. So this should be flexible and provide different options.

Your ideas?

Another idea: for strict and AND/OR matching we could use a views module, which will work pretty fast and universal, not sure if it is possible to do weighted search there easily: may be search API.

In this case we do not need to work much, all is done in a views module, we just need to add basic UI for that to select which view we are using and to define some other options.

Another idea: for strict and AND/OR matching we could use a views module, which will work pretty fast and universal, not sure if it is possible to do weighted search there easily: may be search API.

In this case we do not need to work much, all is done in a views module, we just need to add basic UI for that to select which view we are using and to define some other options (as which fields are passing as arguments, active/passive, etc).

I think something along the lines of http://dgo.to/unique_field might help. It allows you to prevent the creation of an entity (might be currently fixated on nodes) based on the values/properties of fields in the current bundle.

Lets not to go too far about our instruments. If we are talking about configurable system that can build conditions we already have two of them at least: rules condition component (http://www.trellon.com/content/blog/using-ctools-access-rulesets-module-...) and ctools rulesets (http://www.trellon.com/content/blog/using-ctools-access-rulesets-module-...). Lets simply use one or the other. All fancy conditions should be implemented as plugins to this system. Lets not discuss these components here.

The question for now is how we are going to have this functionality added to our workflow.

I see following usages of this functionality:
1. When new contact is created we have extra step where we execute condition and if "duplicate" contact is found we propose to edit existing contact and not creating new one.

2. When drupal user is created but crm contact already exists (for example we match by email) we should also add new step and let user decide whether he wants to link drupal user to existing contact or create new contact for him.

3. We have imported a lot of contacts and we want to do cleanup of duplicates. We should be able to run cleanup procedure to see what contacts looks as duplicates and merge them. This is very tricky as we need to keep all the relationthips, activities etc. I think we can provide some semiautomatic merging procedure for this case.

Thanks Yuri, we need to look on the Rules approach how we could implement this. One of the main requirement is auser friendly interface.

Regarding cases above we have couple more cases:

1. Contact is created automatically in crm_core_profile module or by custom code

2. User is created automatically.

So... There should be a function that allow to return existing contact or create new one if nothing does not match.

Spoke with Yuriy:

We can use the access rules sets, but this will not be good cause of performance reasons: in case if we have 1 000 000 contacts we need to execute 1 000 000 queries to load them and the check for a duplicate for those million contacts.

So we decided to use Views with arguments to construct a query to select duplicated contacts. How it will work:
1. It should be a view which returns contact id of duplicates.
2. This view will have a displays with specific arguments.
3. There is going to be a special function which will receive following parameters: view, view display, contact object.
4. This function will load view display and go trough the arguments and form array of fields what we need to get from the contact and pass to the view.
5. View will return contact ids that matches the criteria.
6. Function will return false or array of matched contact ids.

There also going to be additional functions that will be used in different use cases. E.g. one function will create new contact or update existing one.

It's a belt! Arguments could not be grouped and do not allow to have AND/OR conditions in the groups or between them.

So another way is to used exposed filters.