Hello again!

A client has asked for an upgrade that I think will fit nicely into your module.
We're going to be keeping the:

  • Ip
  • Uid
  • Path
  • Timestamp
  • All the other geolocation data

The plan is to database the data every time the geolocation happens, or if the geolocation is more than a variable amount of hours old. So for instance a fresh record every 24 hours. We'll be keeping that data in the table for a variable amount of type, wiping it out using cron after a variable amount of days. So, say, wiping out after 1 year.

The other piece going along with that would be a statistics table that keeps total hits broken down to the zip code and month level that should be easy to maintain forever.

Want me to write that in for you again?

Comments

lance.gliser’s picture

I noticed another issue talking about exposing to views. I think this data structure would be pretty handy for that too.

lance.gliser’s picture

Issue tags: +location, +views

I was thinking about the reporting side of it, so I have more to add to the plan.

I'd like to have two views, one of the historical, one of the recent. Both sitting at admin/reports/smart-ip. I'm figuring on exposing the data of the tables to views, creating a couple default views with exposed filters, and embedding them.

A nice addition, not slated for version one of this new feature would be to use the proximity filter from location module if it's installed to enable searching in a radius from a given zip for X month.

arpeggio’s picture

Hello,

Yes, the suggested features are welcome to Smart IP module. The features you are suggesting are:

  1. Geolocation historical data records with statistics table
  2. Views integration of the historical and recent geolocation data
  3. Search feature within the radius from a given zip for X month

Those features sounds good. Thanks.

lance.gliser’s picture

Thought you'd like an update on progress. I've been a bit detained, but making decent progress. The schema, and data is being stored on my local copy. Now I need to rig up the views exposure and default views to use for reporting.

arpeggio’s picture

Thank you for the update. Reminder: please make sure that the additional features are modular, so that the user have option to enable/disable them. Thanks!

lance.gliser’s picture

The heavy lifting of the statistics options is already optional, and in a separate include file. I did add one additional setting to the base options though. In order to have meaningful history, I had to accommodate for sessions that might last upwards of weeks, which would look like there was a lack of activity. To counter this, I've introduced an option to force a fresh geocode on a visitor, if the geocode data in their session is more than X hours old.

arpeggio’s picture

Thanks lance.gliser. You're making a good progress.

lance.gliser’s picture

Status: Active » Needs review
StatusFileSize
new66.85 KB

Alright, round 1 of the feature ready for testing. I added a dependency on views for the reporting end. Check it out!

lance.gliser’s picture

An additional piece should be added, but I'm a bit short on time at the moment. There's a framework in place for whoever would have time to do it. We could use the Location module's proximity filter.

Bah. I think I also forgot to remove a drupal_set_message() that warns about each geolocation log. Might check that while you're reviewing the rest.

arpeggio’s picture

That's great! Please give me time, I'm catching a deadline. Thank you.

arpeggio’s picture

I made a quick review of your patch. Here are my few comments:
- Why do we need to remove the variable_set('smart_ip_extract_zip_done', FALSE); in smart_ip.install?

  * Implements hook_schema().
  */
 function smart_ip_schema() {
   $schema['smart_ip'] = smart_ip_schema_definition_array();
+  $schema['smart_ip_detailed_statistics'] = smart_ip_detailed_statistics_schema_array();
+  $schema['smart_ip_summary_statistics'] = smart_ip_summary_statistics_schema_array();
   $schema['cache_smart_ip'] = drupal_get_schema_unprocessed('system', 'cache');
   $schema['cache_smart_ip']['description'] = 'Cache table for storing temporary data during Smart IP database update.';
   return $schema;
@@ -138,7 +361,6 @@
   variable_set('smart_ip_use_ipinfodb_service', FALSE);
   variable_set('smart_ip_correct_ipinfodb_key', FALSE);
   variable_set('smart_ip_get_zip_done', FALSE);
-  variable_set('smart_ip_extract_zip_done', FALSE);
   variable_set('smart_ip_store_location_csv_done', FALSE);
   variable_set('smart_ip_db_update_busy', FALSE);
   variable_set('smart_ip_location_csv_pointer', 0);
@@ -146,6 +368,9 @@
   variable_set('smart_ip_blocks_csv_pointer', 0);
   variable_set('smart_ip_blocks_csv_last_pointer', 0);
   variable_set('smart_ip_roles_to_geolocate', array(DRUPAL_AUTHENTICATED_RID));
+  variable_set('smart_ip_enable_statistics', 0);
+  variable_set('smart_ip_hours_between_fresh_geolocation', 24);
+  variable_set('smart_ip_days_to_keep_detailed_statistics', 365);
 }

- Please avoid escaping single quotes:

Translated strings where one can avoid escaping single quotes by enclosing the string in double quotes. One such string would be "He's a good person." It would be 'He\'s a good person.' with single quotes. Such escaping may not be handled properly by .pot file generators for text translation, and it's also somewhat awkward to read.

Can I ask for the source code? I'm getting error using my patch application.

lance.gliser’s picture

StatusFileSize
new168.59 KB

That's odd. That line you mentioned "variable_set('smart_ip_extract_zip_done', FALSE);" certainly does not need removed. I must have deleted it by accident at some point.

I'll try to keep the translation strings in mind, just have a habit of single quoting strings, thank you.

I'm attaching a zip for you of the entire project with alterations, including commenting out the statistics drupal_set_message() I warned about, and wrapping the view's labels in t().

arpeggio’s picture

I have used your demo application, its cool! However, it seems it captures the history of one user only, the admin (maybe because it's still in development stage). I noticed that this will be implemented as sub module, that's great! we can easily ported this feature to Drupal 7. Looking forward for the completion of this application. Thanks.

lance.gliser’s picture

Now you've slightly confused me.

The package I attached should have the statistics included right into the main module. It could be taken into a sub module, but I didn't develop it that way.

I'm also not having issues tracking other users. I've got hits from anonymous and other users as well. It might be that you're not seeing others, because it's not giving you a fresh geoencode until after the expiration timestamp. If you log in as admin, and it clocks you, it'll be X (24 by default) hours, until it geocodes you again to force a statistic update.

Perhaps an action should be taken at login to force a fresh geocode? It's not a huge hit on statistical accuracy. Whichever you feel more meaningful.

arpeggio’s picture

I'm sorry to confuse you. Yes, I understand that statistics is directly integrated into the main module, what I mean is that I saw a folder under modules directory of Smart IP with template files for creating new module which I assume that the statistics application will be created as sub module of Smart IP (which is the right move).

In my point of view, when our users start to test the statistics application they will expect immediately logs from their dummy visitors and some will think its not working with other users. If you designed that way because you think forcing a fresh geocode at login will affect the site performance, in my opinion its better to leave it that way or take no action at login. It will be unnoticeable in the long run.

lance.gliser’s picture

So in terms of actions you need to commit it to the module the only remaining item is:

Required:
Create statistics as a sub module that can be optionally enabled, and brings dependancy on views.
Documentation on how statistical hits are created.

Anything else required?

arpeggio’s picture

I suggest adding in the documentation the simple installation and usage steps also. After this statistics application is completed future task for this:
- Port to Drupal 7
- Add IP lookup capabilities to the watchdog module (as in Smart IP issue: http://drupal.org/node/1047824). I think it is more appropriate to integrate this to your statistics application, what do you think?

lance.gliser’s picture

It shouldn't be hard to add that ip lookup to the detail statistic view. It's a kind of "field" that views could produce fairly easily.
It will take me at least a few days to get these things handled.

I do not currently have D7 site. Will you be doing the port, or should I plan on doing that as well?

arpeggio’s picture

If you’re available, why not? =) This is an easy task for the one who author the application and I can help in code review/testing… However, I can also do the porting but I will need your assistance because this application is huge compared to your last contribution.

lance.gliser’s picture

StatusFileSize
new70.77 KB
new169.34 KB

Alright, here's round two for you.

A few changes are required in the main module still. I need the ability for people set the decay time on geolocations coming from your module so we're sure it's set how they like. Also moved the geolocation firing into a single function to fire, a bit simpler to call. I made sure every login and out also fires a fresh geolocation to clear up the behavior you saw.

Otherwise the functionality has been moved into a sub module, that brings in the views dependency. I whipped up the custom field handler you needed to do whois lookups in the detailed view was well.

This will not play nicely with the previous version of the code if you have it still installed. You'll need to removed the two statistical database tables from the database manually, as they have changed which schema owns them during transition.

It's still lacking some documentation beyond the field descriptions, I commented the help section out until we decide on some clean language.

I don't think I can afford time for the D7 port right now. I'm already doing this one off client hours because it's become somewhat of my pet project.

Have a go at it!

arpeggio’s picture

Cool! I have tested the latest patch, it now display the latest log without waiting for the timestamp to expire. Now, I am currently reviewing the code...
I have question, why do we need to swap:

function smart_ip_set_location_data($account, $location) {
  global $user;
  if($account->uid == $user->uid){
    smart_ip_set_session_data($location);
  }
  smart_ip_set_user_data($account, $location);
}

to:

function smart_ip_set_location_data($account, $location) {
  global $user;
  if($account->uid == $user->uid){
    smart_ip_set_user_data($account, $location);
  }
  smart_ip_set_session_data($location);
}
lance.gliser’s picture

It's a bug I found in the code while doing all the revisions. It seems they are backwards in your latest dev version. You would only want to set the user data, if the account being passed in is the same as the user being acted against. I thought about it a bit, and I'm not entirely sure how another user would even get passed to the function, but I wanted to cover the possibility.

arpeggio’s picture

StatusFileSize
new87.72 KB
new42.56 KB

I see, I think that would be very rare or may not be possible to get other user's location stored to another user's data.

I noticed that you simplified the:

function smart_ip_set_session_data($location) {
  // Store the location information in the session, merging any old data not overridden
  if (isset($_SESSION['smart_ip']['location'])) {
    $_SESSION['smart_ip']['location'] = array_merge($location, (array)$_SESSION['smart_ip']['location']);
  }
  else {
    $_SESSION['smart_ip']['location'] = $location;
  }
}

to:

function smart_ip_set_session_data($location) {
  // Store the location information in the session, merging any old data not overridden
  $_SESSION['smart_ip']['location'] = array_merge($location, (array)$_SESSION['smart_ip']['location']);
}

Using the simplified version would introduce bug when using the debug mode. Do we have concern if we still use the former?

At http://localhost/admin/reports/smart_ip_statistics/detailed
the filter field labeled with "And:" is confusing to the user... How does that field work?

I made some modification on statistics applcation. These modifications are just corrections, to comply with Drupal coding convetion. Like spaces in control statements, placement of commas. Please review the diff file I created.

* Just a reminder of the documentation.

lance.gliser’s picture

I didn't recall making that change to the set_session_data function, so I downloaded a fresh copy of smart IP just now. V6.x-1.2-alpha1, the current stable release I patched against, includes the bugged function lines you mention. I suppose it would be wise to get that updated while we patch in the rest.

The detailed view's exposed filter you mentioned "And:" is a default way of handling searching date ranges for views. I didn't create that one, just exposed it. We probably could stand to update the default view to say "Time Between:" so the "And:" makes more sense. The label that would need to be altered is in: /modules/smart_ip_statistics/views/smart_ip_statistics.default_views.inc on line 391:
'label' => t('Time'), would become 'label' => t('Time Between'),

arpeggio’s picture

Actually the attachment at #23 is already the latest version. I’ll apply the suggested label for your statistics views.

lance.gliser’s picture

I'm a bit lost arpeggio. Were there more changes, or is this patch considered committed and issue fixed?

arpeggio’s picture

lance.gliser, I've sent you message in your contact form, did you received it?

lance.gliser’s picture

I did not. It seems I had an incorrect email in my communications tab, even though it let me confirm it... Very odd.

arpeggio’s picture

Is it ok if you share your email?

lance.gliser’s picture

My contact information should be correct. I updated it. But I sent along my information to you just in case.

arpeggio’s picture

Ok thanks. Already sent you a message.

arpeggio’s picture

StatusFileSize
new87.72 KB

The attachment is the latest version of Smart IP with statistics application patched. Please have a quick review then commit it to CVS.If you don’t have CVS account please apply now. You may want to use this thread for your CVS application.

lance.gliser’s picture

I spent a bit of time going through the results of the Coder module's review. I'll do a bit more cleanup, and should bet it applied sometime this week.

arpeggio’s picture

Ok. Are you using the source code at comment #32 for your cleanup? For me that source code is ready for your CVS commit and if there are corrections found then it is strongly recommended be applied to that source code.

lance.gliser’s picture

Actually I fell ill last week. I applied for a csv account, and am still waiting to hear back.

arpeggio’s picture

I hope you're fine now. I checked your CVS application and its status is "postponed (maintainer needs more info)". My suggestion is to follow up and mark your application as active (It seems you've already stated the needed info).

lance.gliser’s picture

Thank you for the tip off of the required status change, I've updated my request.

birdsnbees’s picture

Subscribing.

lance.gliser’s picture

For those curious, still waiting on approvals. Went through the process and haven't heard back.

socialnicheguru’s picture

has this been incorporated into the module?

arpeggio’s picture

Assigned: Unassigned » lance.gliser

Not yet, lance.gliser is responsible for this task.

lance.gliser’s picture

arpeggio, they haven't gotten back to me. I welcome you to commit this one yourself. My time has become a bit more invested lately anyway. Please take what you have, and enjoy. It's been a pleasure working with you.

arpeggio’s picture

lance.gliser, I see maybe a follow up can get their attention to review your pending request. Same here. Thanks =)

socialnicheguru’s picture

is this in d7?

arpeggio’s picture

No, this is in D6 only. For D7, you may want consider IP Geolocation Views and maps.

lance.gliser’s picture

Assigned: lance.gliser » Unassigned
Issue summary: View changes
heddn’s picture

Status: Needs review » Closed (outdated)

Drupal 6 is no longer supported