Add a way to define custom data types

OK, a slight problem … The method of explicitly specifying the support for types on the server side and automatically replacing the types accordingly when indexing solves only part of the problem, as I only realized now. After all, modules can also check the fields (and their types) of an index directly, and frequently do (facets, views, DB backend), and with this patch they suddenly can't rely on the fixed set of values that could occur there, anymore. Even though this is an API change anyways, I don't feel quite comfortable making such changes that could heavily influence dependent modules …

So, two solutions for this, that I can think of:
1) Let users only select the field types that the current server actually supports. (Also automatically update this when changing an index's server, of course.) Wouldn't solve the problem for all modules, but at least fpr the backends, which is already something. (Also, this might be a good idea in any case, to avoid user confusion … Or maybe it would be more confusing, can't really tell.)
2) Don't use the "type" key but introduce a new one, in $index->options['fields'][$f]. Can't see anything wrong with that, I guess …

OK, 2) looks much better, but still wanted to give you guys the chance for input on this. Any opinions, or even other ideas?

I'm not familiar with the spectrum of search backends out there. Assuming you do go ahead with #2, is there the potential for a back-end to support a "sub-type" (for lack of a better word) that fails to map to one of your core types? Returning, for instance, an object or some such?

No, I guess we'll have to explicitly forbid this. (Will write that when I update README.txt …) People can, e.g., use the object ID as the property value, and provide a load function.

1. Change search_api_field_types() to return $type => array('label' => 'xxx', 'fallback type' => 'yyyy') instead of just the label, add a drupal_alter(), and add the associated documentation

I have this a bit different, but the basic idea is the same. See attached patch.

By the way, can anyone think of anything else you should/could specify for a custom type?

2. Extends SearchApiServiceInterface to add a supportsType($type) method

I'd just do this with supportsFeature("search_api_data_type_$type").

3. Add logic in SearchApiIndex::__construct() to add a 'index_type' key to $index->options['fields'], based on the current index capabilities

Ah, good idea to add that logic there … I'd have thought about just saving it permanently, but I guess your idea is preferable.

4. Open issues to all the known indexers so that they use 'index_type' instead of 'type' during indexing

During indexing, I'd just use the 'type' key as before, so service backends wouldn't have to adapt. In that case we can tell exactly what types will be known, anyways. (Oh, apart from processors, I guess … But I don't think that would be a problem.)

Attached is a version which should work. (README.txt is missing, though.) If a custom data type is selected by the user, that type is stored in a special "real_type" entry while "type" contains the fallback type. During indexing I replace "type" with "real_type" in those cases where the real type is known by the service class (as determined by $server->supportsFeature('search_api_data_type_' . $custom_type)). In all other cases, modules will just have to check the "real_field" key if they want to deal with some custom data types.

@ mollux: For your use case I'd now suggest the following (probably in its own search_api_geofield (or similar) module):
- Implement hook_search_api_data_type_info() to define a "geofield" type.
- Make sure the Entity API integration of Geofield returns a single value encoding all information. (Remember that the type has to be "text", or other primitive types defined by the Entity API, though. You cannot use the new custom type there.)
- Properly define in your module the format of the data for this type*.
- Patch the Solr service class / module to handle the new type.

* Now that I think about that, we don't have a way to ensure this type, in contrast to the default data types. The user can just select the type for all fields, after all.
Idea: A third key for hook_search_api_data_type_info() to specify a conversion function. Given some data, it will either return the data in the valid format, or NULL if the data cannot be parsed. I guess the index would then have to do this conversion for all values with custom types, between field extraction and preprocessing.
Otherwise, we could just let service classes take care of this, like we (come to think of it, quite needlessly) do for the default types.

@drunken monkey, I made a small change to your patch (and attached the modified one), so the real_type is stored in the index, otherwise it has to be implemented again in the server back-end. If the custom_type passes the 'supportsFeature', for the index, it can be stored permanently as the $index->options['fields'][$field]['type'].

No, we cannot do this, as data alterations, hook implementations, etc., still might not recognize the type. On the other hand, the backend doesn't have to use supportsFeature() anyways, as it already knows which types it supports. Additionally, not all backends need type information outside of the indexing process.
I agree, it is a bit of an overhead, but I don't think it can be avoided easily. We could only shift it from backends to all other modules, and I don't think that's a good idea. (Or we could introduce a third type key for indexed fields, but that would be really confusing.)

your last point is a good idea, provide a conversion function and check it in the index. we can even throw an error I think if you use a the type that doesn't have a good value, but I have to think if that is a good idea :)

I guess if it can't be converted we should just return NULL as if the field had no value. No need to prevent the whole item from indexing just because of an inappropriate type.
But OK, if you also think this is a good idea, let's add a conversion function.

+ conversion callback
+ explanation in README

As far as I'm concerned, there are no more TODOs here.

@ mollux: Can you confirm this works for you? Then I can finally commit this.

Re-roll attached.

If no-one objects until Monday, this will be committed. I can't wait forever here.

Excellent, thanks for testing!
Committed.