CSV file was completely changed
Tobias Maier - November 12, 2006 - 08:46
| Project: | Browscap |
| Version: | 4.7.x-1.x-dev |
| Component: | Code |
| Category: | bug report |
| Priority: | critical |
| Assigned: | mikeryan |
| Status: | closed |
Description
the changes mentioned at http://drupal.org/node/67237
are not the only changes we need to get browscap back to life.
I want to clarify, that browscap in its current state and also with the patch mentioned above does not work at all!!!
so please update the module asap!
the greatest problem which i saw today is that the columns in the cvs file were shifted!
(and a few new one were added)
so the import into the db does not work at all!
I _started_ to fix this issue
The patch I added lets the update work again but the new values are not available
So someone should take a little bit of time to do the remaining work.
Thanks
| Attachment | Size |
|---|---|
| nothing_worked.patch.txt | 7.35 KB |
#1
+1 for this patch
Tobias I'm not sure what you mean by "the new values are not available"?
#2
the author added a few new columns to the csv-file.
the patch skips them although i know that we need to save them as well in the database.
If you want you could add these new columns to the db-table, add an update-path and change all necessary sql-queries to reflect this change.
#3
General update: I'm the original author of browscap - I have not been doing any Drupal work for the better part of a year, and am just now shaking off the rust and catching up. I am looking into the changes (URL and format changes), but am running into problems testing browscap locally on my Windows system:
1. Although I can go directly to the necessary browsers.garykeith.com URLs in FireFox, for some reason the domain doesn't resolve in the cron job.
2. Explicitly defining the IP address for browsers.garykeith.com in my hosts file gets me past the above error, but now I always get
Couldn't check version: 10060 A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Any thoughts on these networking issues? If so, please email me through my profile (let's not clutter up this issue with those issues).
I'm going to see how far I can get testing on my webhost (obviously a slower process) today - I have very little time for Drupal work right now, so it might very well be a few weeks till I get back to it...
Thanks.
#4
Although I have hacked the browscap module to work with the new data structure from G. Keith's site, and have gotten the cron to download the csv file to Drupal on Windows, I am stuck with the phase 2 pass in the _browscap_import(). The new data file does not need to fill in data from the parent to the children; in fact, some of the data about the children is now very specific to that child browser. There are, however, four fields that are often null: browser, version, majorver, and minorver. The problem is that they are only sometimes null.
I am worried that I will need to perform multiple passes over the data to work through these null values on top of the original pass. Is there another more elegant way?
I have attached the new browscap.mysql file.
Here is the code for the _browscap_import as it stands. I have not put it into a patch as it is imcomplete and I'm still hacking.
function _browscap_import() {
// Politely check the version for updates before fetching the file
$versionpage = drupal_http_request('http://browsers.garykeith.com/version.asp');
if ($versionpage->error) {
watchdog('browscap', "Couldn't check version: ".$versionpage->error);
return;
}
$browscapversion = trim($versionpage->data);
$oldversion = variable_get('browscap_version', 'Never fetched');
if ($browscapversion == $oldversion) {
// No update, nothing to do here
watchdog('browscap', 'No new version of browscap to import');
return;
}
// Fetch the new version, and dump it in the temp directory
$server = $_SERVER['SERVER_NAME'];
$path = variable_get('file_directory_temp', '/tmp');
$browscapfile = "$path/browscap_$server.csv";
// If we can, download the zipped version and extract the file
if (is_callable('zip_open')) {
$browscapzipfile = "$path/csv_browscap_$server.zip";
$browscapzip = drupal_http_request('http://browsers.garykeith.com/stream.asp?CSV_BrowsCapZIP');
if ($browscapzip->error or !trim($browscapzip->data)) {
watchdog('browscap', "Couldn't retrieve updated browscap: ".$browscapzip->error);
return;
}
$browscapzipfp = fopen($browscapzipfile, "w");
fwrite($browscapzipfp, $browscapzip->data);
fclose($browscapzipfp);
$zip = zip_open($browscapzipfile);
if ($zip) {
while ($zip_entry = zip_read($zip)) {
if (zip_entry_name($zip_entry) == 'browscap.csv') {
if (zip_entry_open($zip, $zip_entry, 'rb')) {
$browscapfp = fopen($browscapfile, 'w');
fwrite($browscapfp, zip_entry_read($zip_entry, zip_entry_filesize($zip_entry)));
fclose($browscapfp);
break;
}
}
}
zip_close($zip);
}
unlink($browscapzipfile);
// Can't handle zip, get the unzipped version
} else {
$browscap = drupal_http_request('http://browsers.garykeith.com/stream.asp?BrowsCapCSV');
if ($browscap->error or !trim($browscap->data)) {
watchdog('browscap', "Couldn't retrieve updated browscap: ".$browscap->error);
return;
}
$browscapfp = fopen($browscapfile, "w");
fwrite($browscapfp, $browscap->data);
fclose($browscapfp);
}
// Parse the .csv file, importing each row (except the first two)
// into the {browscap} table
$browscapfp = fopen($browscapfile, 'r');
if ($browscapfp) {
// Ignore the first two rows (column headers & version info)
fgetcsv($browscapfp, 1000); fgetcsv($browscapfp, 1000);
while (($browserinfo = fgetcsv($browscapfp, 1000)) != FALSE) {
$parent = $browserinfo[0];
// Strip brackets
$useragent = substr($browserinfo[1], 1, -1);
// Replace wildcards with SQL equivalents
$useragent = strtr($useragent, '*?', '%_');
$browser = $browserinfo[2];
$version = $browserinfo[3];
$majorver = $browserinfo[4];
$minorver = $browserinfo[5];
$platform = $browserinfo[6];
$alpha = _browscap_boolean($browserinfo[7]);
$beta = _browscap_boolean($browserinfo[8]);
$win16 = _browscap_boolean($browserinfo[9]);
$win32 = _browscap_boolean($browserinfo[10]);
$win64 = _browscap_boolean($browserinfo[11]);
$frames = _browscap_boolean($browserinfo[12]);
$iframes = _browscap_boolean($browserinfo[13]);
$tables = _browscap_boolean($browserinfo[14]);
$cookies = _browscap_boolean($browserinfo[15]);
$backgroundsounds = _browscap_boolean($browserinfo[16]);
$authenticodeupdate = $browserinfo[17];
$cdf = $browserinfo[18];
$vbscript = _browscap_boolean($browserinfo[19]);
$javaapplets = _browscap_boolean($browserinfo[20]);
$javascript = _browscap_boolean($browserinfo[21]);
$activexcontrols = _browscap_boolean($browserinfo[22]);
$stripper = _browscap_boolean($browserinfo[23]);
$isbanned = _browscap_boolean($browserinfo[24]);
$wap = _browscap_boolean($browserinfo[25]);
$ismobiledevice = _browscap_boolean($browserinfo[26]);
$issyndicationreader = _browscap_boolean($browserinfo[27]);
$crawler = _browscap_boolean($browserinfo[28]);
if ($browserinfo[29] == 'default') {
$css = -1;
} else {
$css = $browserinfo[29];
}
if ($browserinfo[30] == 'default') {
$cssversion = -1;
} else {
$cssversion = $browserinfo[30];
}
$supportscss = _browscap_boolean($browserinfo[31]);
$aolversion = $browserinfo[32];
$aol = _browscap_boolean($browserinfo[33]);
$netclr = _browscap_boolean($browserinfo[34]);
$clrversion = $browserinfo[35];
$internalid = $browserinfo[36];
$result = db_query('REPLACE {browscap} ( parent, useragent,browser,version,majorver,minorver,platform,alpha,beta,win16,win32,win64,frames,iframes,tables,cookies,' .
'backgroundsounds,authenticodeupdate,cdf,vbscript,javaapplets,javascript,activexcontrols,stripper,isbanned,wap,ismobiledevice,issyndicationreader,' .
'crawler,css,cssversion,supportscss,aolversion,aol,netclr,clrversion,internalid) '.
"VALUES('%s','%s','%s','%s','%s','%s','%s','%d','%d','%d','%d','%d','%d','%d','%d','%d','%d','%s','%s','%d','%d','%d','%d','%d','%d','%d','%d','%d','%d'," .
"'%d','%d','%d','%s','%d','%d','%s','%d')", $parent,$useragent, $browser,$version,$majorver,$minorver,$platform,$alpha,$beta,$win16,$win32,$win64,$frames,$iframes,$tables,
$cookies,$backgroundsounds,$authenticodeupdate,$cdf,$vbscript,$javaapplets,$javascript,$activexcontrols,$stripper,$isbanned,$wap,$ismobiledevice,
$issyndicationreader,$crawler,$css,$cssversion,$supportscss,$aolversion,$aol,$netclr,$clrversion,$internalid);
if(!$result)
{
watchdog('browscap', "DB error: ".db_error());
}
}
fclose($browscapfp);
unlink($browscapfile);
// Phase 2 - for every entry which isn't a top-level entry, plug
// in the data from its parent
/*
Here's the complicated data problem. The missing information is scoped within the following fields:
browser, version, majorver, and minorver
However, the information that is missing is random. We need to perform passes which finds the null values in these fields
and fills in the information from the parent ONLY IF NULL in the child for these four fields. Yuck.
*/
$query = 'SELECT useragent,parent from {browscap} WHERE useragent <> parent';
$result = db_query($query);
while ($destrow = db_fetch_object($result)) {
$query = "SELECT * FROM {browscap} WHERE useragent='%s'";
$srcrow = db_fetch_object(db_query($query, $destrow->parent));
db_query("UPDATE {browscap} SET browser='%s',version='%s',majorver='%s',minorver='%s' WHERE useragent='%s'",
$scrow->browser, $scrow->version, $scrow->majorver, $scrow->minorver, $destrow->useragent);
// All done updating the browscap info - invalidate cached data
// from the last version, and record the version we're currently
// using
cache_clear_all('browscap:', TRUE);
variable_set('browscap_version', $browscapversion);
watchdog('browscap', "New version of browscap imported: $browscapversion");
}
}
#5
While walking home, I figured out a hack to make the phase 2 pass work. I believe that with the new .mysql file I submitted in the post above, I've fixed browscap for the MySQL platform and 4.7.x (tested on 4.7.4). Someone may want to make light work out of the Postgres definition file. Please test the submitted patch and let me know if there are any bugs that were attracted to the midnight oil I'm burning.
#6
Oops, I did not mean to change the original title of this thread. Now restored.
#7
Thank you for your new patch
this patch does not follow the coding standards http://drupal.org/node/318
especially http://drupal.org/node/539
we need a table update schema for the new table structure, too
http://drupal.org/node/51220
#8
I think this has been fixed along the way. If any portion of this issue is still a problem, please re-open and state the specific piece.
#9
Automatically closed -- issue fixed for 2 weeks with no activity.