Provide drush-command to re-geocode all empty values

Have you resaved the node in question? The trans-coding isnt done on view but instead on save. Though testing for its presence and attempting to transcode might not be a horrible idea. Please reopen with mor einfor if you still ahve trouble.

You did reopen the case by setting it to active :). Thank you. Ill look into this myself.

I think really we should have a drush command that can automatically re-geocode all empty values.

Doing a node_load, node_save should now be enough to re-geocode everything.

Hi cesareaugusto,

Yup for sure. That's why I left this ticket open. :-)

This operation can now be easily done by installing Views_bulk Operations and using the "Save content" event on a view you make. You can even "select all" or setup your own filter to only do it to the content with empty geofields or replace the find content view on drupal like i do on all my sites with something that is usable ;).

We can leave this open for drush support if youd like but i think its a good idea to reuse VBO for the admin version and keep this module what it is good at.

Forgot to mention that i tested this myself to make sure it worked and it did.

Okay, so I have this working for nodes. The command is "drush geocoder-backfill"

Marking as "needs work" since it only works for nodes and really should work for all entities.

No user interface for it yet. That still needs to be done

@michaelfavia
"This operation can now be easily done by installing Views_bulk Operations and using the "Save content" event on a view you make. You can even "select all" or setup your own filter to only do it to the content with empty geofields or replace the find content view on drupal like i do on all my sites with something that is usable ;)."

I have created a view which has a Bulk Content field using the Save Content event. When I execute the operation it reports that it saved the checked nodes but the geocoding does not update...when I manually go to a node and edit then save it geocodes....Do I need to add any fields to get it to update??

See http://drupal.org/node/905814 for solution

I found a little bug in the actual implementation of the drush integration.

In fact, all nodes geodata was dropped because of a case where geocoding was not launched when the node has not been edited.

Here is a patch bypassing this problem.

I need this for User account fields. i spend the hole day for installing drush and no I see it works only with nodes..
Even no way with bulk operations, or shopuld i make a rule to have a user saving option?

Unfortunately this seems not ready for other entities but I think using VBO like said in #11 should help you.

Thx, but there is no save content options in views bulk operation in a user based view. Even nothing similar like save user..

Is there ANY Solution to bring the user account fields in a content based view?? I tried relationsship -author, but that would only work if every user has at least one node

I have seen your other issue post later.
I think it would not be very complicated to develop a little module wich take all users into the database and bulk resave them or a module which adds to VBO the ability to save users :)

Thx, but for both options i have not enough experience..

Sorry then. While this module is not getting a new patch to enable batch update on all entities, you will have no other option. :/

#18 worked for me without error.

This is fixed. I've taken #18 and modified it a bit.

There is now a new drush option --force, that you can add that will force a geocode, even if there is already geodata.

Ditto what cesareaugusto said. Not sure why the original Issue title was changed since the replacement doesn't reflect cesareaugusto's original request.

#17 didn't work. Note: link is to 'Geocode' module issue -- maybe that's why it didn't work for 'Geocoder'?

I am changing this back to needs review. I was initially glad to read this issue thread. I had imported several thousand nodes with the migrate module and found that I hit the 2500 google geocoder limit. I tried the drush geocoder-backfill command and it ran for a bit, but did nothing. I tried the VBO "save" for all nodes without geocoding, but that did not work either. Then I noticed the dates of this thread and saw I was using 7.x.1.2 from August 2012 and DL the DEV.

Trouble! It ran for a bit and ALL my geofields became blank. I even checked the DB tables. Nada. Something is not working right with the DRUSH GEOCODER-BACKFILL command.

If I edit a save a node, the geocoding returns for that node.

drush geocoder-backfill will successfully re-geocode all your nodes, but only in the event your geocoding API is configured correctly and doesn't reach a limit. Since API limits (or, in an academic sense, the geocoding APIs themselves) aren't a known value, it's not the command's immediate responsibility to predict limits or create a queue.

I might be out of line here, but adding that functionality to the existing working solution seems like the purview of a separate issue.

Hi I'm having this error:

drush geocoder-backfill
Killed

Thanks.

update: please see the %CPU

mpisano@bralig.com [~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mpisano 22551 12.8 0.5 302692 69340 ? SN+ 12:24 0:00 /usr/bin/php /h

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mpisano 22551 19.8 0.5 303972 70920 ? SN+ 12:24 0:03 /usr/bin/php /home/mpisano/public_html/drush/drush.php --force g

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mpisano 22551 22.1 0.5 305252 71976 ? SN+ 12:24 0:06 /usr/bin/php /home/mpisano/public_html/drush/drush.php --force g

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mpisano 22551 24.1 0.5 306532 73296 ? SN+ 12:24 0:10 /usr/bin/php /home/mpisano/public_html/drush/drush.php --force g

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mpisano 22551 28.5 0.6 311908 78576 ? RN+ 12:24 0:25 /usr/bin/php /home/mpisano/public_html/drush/drush.php --force g

drush geocoder-backfill
Killed

We have a lot of content types and nodes so running a bulk update on all content is impractical. I'm attaching a patch that adds an additional "types" option, this takes a a comma separated list of bundle (content) types and only applies the update on those matching types.

This patch applies against the current dev version of the module.

Just a heads up, doing this command will wipe out all existing geocoded values. Tested with mapquest and google geocoders selected.

drush geocoder-backfill

I'm on the latest version of geocoder 7.x-1.2+9-dev .

Doing drush geocoder-backfill --force fixes this, but this will only work if you have less nodes than the allowed quota.

The thing is having to use the --force handle defeats the purpose of this drush command. It's only of value if it assists with not going over Google or Mapquest transaction limit/quota--by ONLY geocoding location nodes that are not geocoded.

If you have 10,000 locations you need to encode, you basically can't do it with this drush command at it's current state, because you'll go over the limit, and half your nodes will remain un-geocoded forever.

I agree duntuk. A command called "-backfill" should ONLY update nodes that do not have any geocode data so as to have a way to overcome google and other API lookup limits. Of course we will reach the 2500 limit, but we could backfill again the next day until we get all the imported nodes geocoded. In its current state, it is not possible.

Yeah, I ran into the same exact thing as you noted in #31...

BTW, #32 response is missing the issue--this problem of wiping out all pre-existing geocoded data has nothing to do with going over quota. There should be nothing in this function that should even wipe anything. At the very least this drush command should ONLY add data, and not delete it...

It's purpose is to add missing geocoded data, however, it fails at that with the worst case scenario: not only does it NOT add geocoded data, but it also deletes existing data. That's bad.

I agree with @duntuk.

With 7.x-1.x-dev (7.x-1.2+9-dev) and geofield-7.x-1.2:

• node_load() then node_save() does not work, (no auto-geocoding is done.)
• Using views_bulk_operations to save_content does not work.
• Running drush geocoder-backfill not only does not auto-geocode, it removes geofield settings where already set.
• Running drush geocoder-backfill --force does work, but as mentioned in #36, this could interfere with quotas if you have lots of nodes using geocoder.

What --force is doing differently is setting $entity->original = array();.

I'm using geocoder with a geofield attached to a taxonomy_term entity type, and ran into a drupal core bug #1054162: Taxonomy bundles not supported by EntityFieldQuery (followup) - while some of the problem was resolved, there is still an issue with taxonomy using vid as opposed to machine names. See #1551774: Replace taxonomy_term_data.vid with vocabulary_machine_name for d8 work on this issue.

As a stopgap solution, I patched geocoder.drush.inc to handle taxonomy_term entities in a special, special way.

i also needed to use --force to get things to work :(

When I try to use the --force option, I get the error Unknown option: --force. See `drush help geocoder-backfill` for available options. To suppress this error, add the option --strict=0. When I use geocoder-backfill without the --force option, it appears to be processing in the command line, no messages, and nothing seems to happen (e.g., no geocoding of existing nodes :( ).

Has anyone been able to find a solution? I've tried everything I can think of.

Well, my workaround solution to backfill my geofield was to rerun my feeds import. I had already used Feeds to import my content type, so all I did on the node processor settings was to change it to "Skip hash check", which forces the update of the nodes even though the source data had not changed. I have a unique field which I use in my feeds, but it is possible to use the existing Node ID as a unique field to tie the data together. After rerunning the import and cron many times, all of the nodes get geocoded!

Another nice thing about this approach is that I can control the rate at which the nodes get geocoded by setting cron at 1 or 2 hour intervals, so as to not hit API limits.

If anyone would like more information, please let me know, and I will be happy to elaborate.

I know my comment here doesn't directly address the issue at the top, but I thought it would be helpful to put here for other people, who have found this thread (like me), searching for a way to backfill Geo field data.

For those of you who don't want to use drush I used another way to backfill my Geofield which worked for me. I had about 1500 users that I needed to backfill geo-data for. I had added a geofield to users. It was set to geocode from another field. The field it was based on was a text field which was holding the postcode for the user.

To do this I used the flag, views bulk operations and rules modules.

First of all I created a simple on/off flag that applied to users. Then I created a rule which was triggered when the on/off flag was applied. The action for the rule was simply to set the postcode value for the user to the existing postcode value. In other words changing nothing at all but just resetting the value to what it already was. Finally, I created a view of users who had an empty Geofield, and a non-empty postcode. I then added a views bulk operation field to set the on/off flag for all those users. This triggered the rule I had created and successfully back–filled all the Geofield data.

This method should work just as well for nodes as for users. I am sure there may be a simpler way of doing it but the important thing is it did work.

@bmango is your address handled by the address field module? https://drupal.org/project/addressfield

If yes, in Rules, how were you able to select just the zip code?

Or in other words, can you go into details as to what you selected for each:

Events:

Conditions:

Actions:

SOLVED:

This is now fully working with geocoder-drush-2077317-05.patch from mariacha1

Attached is the latest DEV version patched with #5 (geocoder-drush-2077317-05.patch)

Credit goes to mariacha1 for supplying the patch.

Here is how to use it:

drush -d geocoder-backfill --limit=2500

This patch not only fixes the drush command (fixes the bug where all geocoded values get deleted), but also adds the much needed --limit=NUMVALUE which allows you to limit the geocoding to abide by geocoding (Google 2500/day) quota limits.

@duntuk - I was not using the addressfield module. I just had a simple textfield which was being used for holding the postcode. I think this makes configuring the rule I used a lot easier.

For the rule, the event was "A user has been flagged", there was no condition, and the action was "Set a data value", where the data was "flagged-user:field-postalcode", and the value was "flagged-user:field-postalcode". ie, the data and value were the same. This rule was just to trigger the Geofield update.

I don't know how this would be done for the addressfield module.

With the patched module in #47 I get the error: "Undefined variable: force_reload geocoder.drush.inc:52"

Adding at about line 38:
$force_reload = drush_get_option('force');
resolves it.

The line was patched out in Add a --limit,--only-entity, and --only-bundle option to drush command, enable for any entity

Used the method from #45/#48: huge help.

Saved me a lot of time; thanks.

This issue was fixed in #27. At this point, all remaining issues appear to be a duplicate of #2077317: Add a --limit,--only-entity, and --only-bundle option to drush command, enable for any entity. Closing this issue as the linked issue has a working patch that addresses most issues here.

Used the method in #44 successfully - needed to be able to control the amount of updates / day for the initial batch, so this was possible with feeds import.

I appreciate that this issue is closed, but I had a requirement to update several hundred nodes on a site which is located on a shared server with no access to drush, rather than loading and saving all nodes manually I wanted to use this approach. I took the contents of geocoder_drush_backfill() and ran it through the Execute PHP Code screen supplied by Devel:

$force_reload = TRUE;
$all_entity_info = entity_get_info();
foreach ($all_entity_info as $entity_type => $entity_info) {
  if ($entity_type == 'node') { //TODO: FIX THE LOGIC BELOW and implement for all entities
    if ($entity_info['fieldable']) {
      foreach ($entity_info['bundles'] as $bundle_name => $bundle_info) {
        foreach (field_info_instances($entity_type, $bundle_name) as $field_name => $field_instance) {
          $field_info = field_info_field($field_name);
          if ($field_instance['widget']['type'] === 'geocoder') {
            $entity_load = $entity_info['load hook'];

            $query = db_select($entity_info['base table'])
              ->fields($entity_info['base table'], array($entity_info['entity keys']['id']))
              ->condition($entity_info['entity keys']['bundle'], $bundle_name);

            $results = $query->execute();
            while ($id = $results->fetchField()) {
              $entity = $entity_load($id);
              $langcode = field_language($entity_type, $entity, $field_name);
              $items = field_get_items($entity_type, $entity, $field_name, $langcode);

              if ($force_reload) {
                $entity->original = array();
              }

              // Check for values and if there are no values, reload the entity
              if ($field_info['type'] == 'geofield') {
                if ($force_reload || (empty($items['wkt']) && empty($items['geom']))) node_save($entity); //TODO: fix for all entities
              }
              if ($field_info['type'] == 'location') {
                if ($force_reload || empty($items['latitude'])) node_save($entity); //TODO: fix for all entities
              }
              if ($field_info['type'] == 'geolocation') {
                if ($force_reload || empty($items['lat'])) node_save($entity); //TODO: fix for all entities
              }
            }
          }
        }
      }
    }
  }
}

Just hope this might help anyone else who finds themselves in a similar situation.

• For those who don't want to use drush or dev versions with patches,
• and that use addressfield as a source for the geocode:

The solution in #45 & #48 doesn't work with addressfield (maybe not even in others cases): either using Flag & a Rule or directly a rule's Component, the geofield won't be recalculated if you don't change the actual value of the addressfield (source field), or you manually save the entity from UI (explained here).

The hack solution for me was to set a value inside an unused address sub-field, hence actually changing the geocode's source.
You can do it both with a rule or a component.

Example (component):
Create a new component of type Rule with a parameter of type Content (or your entity holding the geo data)
Set conditions of:

1. entity has field for your content addressfield
2. entity has field for your content geofield
3. (optional) data value is empty for your content geofield

Actions:

1. set a data value on your content addressfield:sub-premise of whatever you want, like your site's name

(this is the hack: we set a value on a addressfield's sub-field which is not used nor displaied on your site, like the "Sub premise")

NB: setting a fixed value will work only once (same value) -> you could calculate an integer and store it on a new field on your content type, or even better add a parameter in the Component settings which you will set when manually launching the rule with VBO, or you could simply cancel the hack value before re-doing the geofield bulk calculation.

Finally you can go to your admin view showing the content type and in your Bulk operations field settings check (add) the created component and save, so it will appear under "Operations".

One more option for those who would prefer something in the admin interface rather than drush (which currently seems to need dev version+patch)

• The solution in #11 is to use VBO and an action to save content.
• As pointed out in #18 this doesn't work.
• However if you take a single line (non-drush part) of the patch from #2077317: Add a --limit,--only-entity, and --only-bundle option to drush command, enable for any entity this solution will work, including for Address Field.

As I understand it, this patch says "although we are skipping if nothing has changed, don't skip if the last geocode didn't work".

NB This patch is just one part of the patch in the related issue. If that patch is checked in, this one does not require a separate check in. All credit for this patch goes to the original posters in the related issue.

Attached patch worked for me on stable version without needing dev.

Hi

I installed the location module and i have like 20,000 addresses but the longitude/latitude are not showing. Is there a way i can use Geocode to automatically generate the longitude and latitude for all the 20,000 entries.

Any comment will be appreciated

This feature request was fixed in #27 as described in #51.

@docans I don't think it makes sense to set the status to indicate not fixed just because you would like some support. If you read the comments above they describe several ways, and what potential bugs/limitations there are. If you are using the Google free geocode service you are rate limited to so many requests per day - a lot less than 20,000 I think. If you need more help, then you can ask in a Forum or raise your own support request.

If someone still have problems to geocode give a try to VBO, just select the nodes you want to geocode and publish or unpublish them.

I solved