I'm working on this feature. I'll submit a patch when ready.

I need it for OSM Nominatim (https://operations.osmfoundation.org/policies/nominatim/) but it should be available for any geocoding service provider with two configurable values:
- period in seconds
- number of requests allowed for the period

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

GaëlG created an issue. See original summary.

GaëlG’s picture

Assigned: GaëlG » Unassigned
Status: Needs work » Needs review
FileSize
12.52 KB

Here's a first patch, it seems to work well for my needs.

itamair’s picture

thanks @GaëlG ... giving a look into this, will write my feedbacks to you asap ;-)

itamair’s picture

nice feature @GaëlG ... and your patch implementation makes sense to me.
Considering that the Nominatim geocoding policy states clear limits on the Geocoding requests frequency, it also makes sense that the Nominatim Geocoder server provider setup/annotation definition already bundles throttle presets. (isn't it?).
The new attached patch corrects a couple coding standards related to correct class method and constructor documentation.

Please let us know if everything makes sense and that you think this new version might be committed into dev.

itamair’s picture

sorry @GaëlG ... another question. I am not that expert on the Nominatim geocoding service and provider.
I couldn't never try it out but I notched that, in the Geocoder module the Nominatim provider configuration require the following "required" parameters:

- Root URL
- User agent
- Referer

I couldn't clearly realize what those parameters are for, but what I ask you is:

- should all the above be required? does it makes sense to you? ... or some of them might be set as optional to let the provider work correctly anyway.

GaëlG’s picture

Assigned: Unassigned » GaëlG
Status: Needs review » Needs work

Thank you very much itamair for the review. I'll answer your questions, but I just figured out that the patch I submitted does not contain the last changes I made. I think I messed up at some time!

GaëlG’s picture

Here's the correct patch. It contains two bug fixes I made that were mysteriously not in patch #2. It also contains your coding standards improvements from #4.

To answer your questions:
- Yes, it makes sense to me that the Nominatim Geocoder server provider setup/annotation definition already bundles throttle presets. Their doc says "an absolute maximum of 1 request per second" and "Make sure you stay well below the API usage limits.". So I put 1 request per 2 seconds. But I just figured out that I put it in the generic nominatim provider instead of the OSM one. I'll submit a new patch for this.
- About the parameters, for the OSM free service, Root URL is obviously required, but it should not change. OSM Nominatim asks for User agent or Referer, so it's required to fill at least one of the two. Personally I filled both and was not banned for that.

GaëlG’s picture

Whoops, here's the patch.

GaëlG’s picture

Status: Needs work » Needs review
FileSize
13.27 KB
1009 bytes

Here's the fix for the Generic Nominatim vs OSM Nominatim problem.

itamair’s picture

thanks again @GaëlG ... great work.
I will test and review it again. Do you think that this is ready for commit (if I don't find major issues?) into dev and into new 8.x-3.x release?

I have got another question and doubt, though.
The 8.x-3.0-beta1 might be still updated without worrying too much about possible back compatibilities,
but still updating the 8.x-3.0-beta1 version users (that already installed the module) would face a big error in the module (the GeocoderThrottle won't work at all), as long as they don't run the following composer additional command:

composer require davedevelopment/stiphle

Isn't it? do you agree?
For sure, I will highlight and underline that need (to require the new davedevelopment/stiphle library) in the new version release specification.
But do you have any clue to make this update easier, more aware and more automatic for already installed 8.x-3.x releases?

Due to this, this upgrade would be even harder if (wanted to be) replicated into the 8.x-2.x branch ...

GaëlG’s picture

Hi @itamair, I think it's ready for commit if the review is OK.
About the composer problem: I guess it won't work if people update without using composer. A hook_requirements() implementation may help to warn people? But if they use composer update drupal/geocoder, I guess the dependency will be downloaded automatically by Composer, no? I'm not a composer expert, I'm not able to confirm this without being able to test it.

itamair’s picture

No ... unfortunately the composer update drupal/geocoder won't download the new library (it seems to me).
May you reproduce the real user scenario (applying your patch to an existing 3.x release ... and verify that it would break the site),
and then may you better/rather warning a hook_requirements() or hook_update() warning that would instruct the user on this?.

itamair’s picture

Status: Needs review » Needs work
GaëlG’s picture

Status: Needs work » Needs review

As I could not commit to drupal/geocoder to test, I cloned the official repo as "insite/geocoder" in our own composer repo, and required it in a project. Then I committed the patch to this clone and did:

[...]$ ls vendor/davedevelopment
ls: impossible d'accéder à 'vendor/davedevelopment': Aucun fichier ou dossier de ce type
[...]$ composer update insite/geocoder
Gathering patches for root package.
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 1 update, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Installing davedevelopment/stiphle (0.9.2): Loading from cache
  - Updating insite/geocoder dev-master (7e01579 => 64693c1):  Checking out 64693c19e9
davedevelopment/stiphle suggests installing predis/predis (~1.1)
Writing lock file
Generating autoload files
[...]$ ls vendor/davedevelopment
stiphle
[...]$ 

So, if people use composer to update geocoder to a newer version/commit, they will get the dependency automagically*.

There is a problem if people do not use composer (until #2622420: Site-builders should be able to use projects with composer dependencies is done). So I started writing a hook_requirements() implementation. But I remembered that composer is already a requirement according to the module documentation, and there is for example no check to see if Geocoder-PHP is installed, whereas it's a composer dependency. We might add that kind of check for people trying to use the module without Composer, but that's another issue, no?

*This is not the case if they simply apply the patch using cweagans/composer-patches, as the patch is applied AFTER composer has computed the dependencies.

  • itamair committed fb3d487 on 8.x-3.x authored by GaëlG
    Issue #3081749 by GaëlG, itamair: Throttle feature: do not send too much...
itamair’s picture

Status: Needs review » Fixed

Right you are. Thanks @GaëlG ... great job on this.
Committed into dev, this new feature (and #9 patch) will be part of the next release.

  • itamair committed f6c3dc6 on 8.x-3.x
    #3081749: Better implementation of the Throttle Feature, that won't...

  • itamair committed c0921ed on 8.x-3.x
    #3081749: Further better implementation of the Throttle Feature, that...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.