Closed (fixed)
Project:
Drupal.org infrastructure
Component:
Servers
Priority:
Major
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
3 Dec 2013 at 15:19 UTC
Updated:
21 Aug 2014 at 21:00 UTC
Jump to comment: Most recent
Localize.drupal.org's project syncing code is down since the d.o upgrade and its high time we fix it. No new releases or project are appearing now. I looked at the temporary dump provided by drumm, but its far from providing the data we need so I looked at making the cross-query solution work again. We used to have access to the term_node table on the d.o db, but don't have that anymore. We'll need access to the taxonomy_index then now to filter out obsolete and unsupported projects from our list. (see http://drupalcode.org/project/l10n_server.git/commitdiff/583b6eae7c8109f... for proposed code changes).
Please give *select only* access to taxonomy_index to the localize_ro user.
Comments
Comment #1
gábor hojtsy#2124931: Update l.d.o syncing for d.o update of project data postponed on this.
Comment #2
drummI thought the plan was to gather the information needed from individual projects' update status XML. The export I made was only to show what had been updated recently.
Comment #3
gábor hojtsyWell, I looked at doing that but:
- our code does a lot of important things, eg. when a project goes unpublished or changes its visible name, that is reflected on localize.drupal.org; we could only do this by reading and parsing ALL the project XML files continually
- we did not find time to rearchitect the whole thing and looks like we still have DB level access with our user as before for most of the tables we need, and its likely we can find time to at least fix the old code to make it work again vs. rewriting the whole thing
Comment #4
gábor hojtsyWe also use project usage data for example to eg. order downloads for projects based on the most used projects, see top downloads on https://localize.drupal.org/translate/languages/ru. Neither the dump, nor the XML provides that data.
Is it possible to get this moving? I already closed #2151225: The project d4os is not listed on localize.drupal.org and #2139775: Commerce economic not listed on l.d.o as duplicates of #2124931: Update l.d.o syncing for d.o update of project data :(
Comment #5
drummI suppose I'm okay with this for now. I can't grant DB access. nnewton can.
Comment #6
gábor hojtsyThanks that should help us move forward. I'll ping to nnewton.
Comment #7
gábor hojtsyOk, we got access to taxonomy_index, so I moved on to the next query. We also need access to file_managed unfortunately. We do have access to 'files' but that only contains files up to the migration, since Drupal 7 has the file_managed table. Sorry that it took so long for me to figure out. This now should let us get the projects and releases and fill in the queue with releases to parse since the migration.
I still have problems with the project usage table, but that is due to missing indexes => ending up in filesort, so we cannot query it in a way we need to to get the data we need. I'll comment those out, those are needed to order projects properly, but we have some (outdated) data to do that for now. The list of new projects and releases missing is a LOT more pressing :/
Thanks nnewton for the help so far.
Comment #8
pomliane commentedHi,
any news on this issue?
Comment #9
basic commentedGábor, you should have access to file_managed now as well.
Comment #10
gábor hojtsyThis was granted but turned out to be far from enough. See https://drupal.org/node/2124931#comment-8398433
Anyway, this issue is fixed, we'll need to look for other (more restricted) avenues to get to at lease some barely minimal amounts of data to use.