JSON interface to project search

It's possible that I'm missing the part that assumes there is a 1:1 relationship between release nodes and files. It's also possible that I just did a bad job of phrasing the intent of the patch.

The first (and most important) file, build_repository.php, uses project_release_nodes tell if a release is present for a specific api version. The outputted XML file should be the same with no regard to the number of files (or their types) for a release. Only whether or not there has been a release for a particular api version will affect that.

The second file, serve_md5sum.php, takes a file path and name as an argument and returns the appropriate md5sum for this file. Different files will have different md5sums, but since this is the way that md5sums work, I believe that would be the expected behaviour. The release doesn't matter, only the filename used to access the file.

The last file, .htaccess, just makes it easier to use serve_md5sum.php.

After reading the above linked issues, I just don't see how either of them relate to the issue at hand. If there is still a problem that I am missing, please try explaining it again so that I can rewrite the patch. I do not believe that any more development can continue on the plugin manager until I have more sufficient feeds available.

J Rogers

Since d.o has moved over to D6 now (along with a very functional project module) would it be safe to rewrite this bit of code or are there other planned db changes in the near future?

Thanks,
JR

I have rewritten the code to take advantage of the new project schema for D6. Also, the purpose of the serve_md5sum (now serve_release_info) has changed. Given a project name and api version it returns every md5sum, filepath and version that goes with it. This supports multiple files per release.

If this meets approval then I should be able to finish plugin_manager 2. If not, then I'm certainly willing to redo it as neccessary.

Thanks,
JR

Sorry about taking so long. I just got back from Tennessee. (Spring Break.) Also, sorry about the .tar.gz. Here it is in the form of a patch.

Would it help any if I were to give a more detailed explanation of what this should help accomplish that cannot be accomplished now?

Check out the prototype: http://joshuarogers.net/admin/build/plugin_manager. The data was scraped off the drupal website. Sometimes the scraper grabbed the wrong data. Sorry. It was just meant to show what could be done.

This is most definitely unfinished, but I would still suggest trying it. Search for "facebook" or "plugin manager" or "chatroom" or any other term you want to. The goal is to eventually give the user a simple (or advanced) way to search for different modules and themes that they can install.

Having the ability to search for new stuff without having to leave your own site seems to give a feeling of safety to those who aren't very familiar with the inner workings of Drupal. For those like me that can do it, we should be able to have more advanced searches eventually. Giving a full text search just gives more potential to give better results.

I know there are several people who are really wanting to get this in to core. That would be really exciting. Honestly though, what I want more than that is to get this module polished. That led to this conversation in the usability group: http://groups.drupal.org/node/17765. I personally feel that this would be one giant step toward usability.

I'm *not* being an obstructionist, and I'm not resistant to change. I'm genuinely concerned that you're walking into a minefield with your eyes closed, and I don't want to see your legs blown off.

The thought never crossed my mind. I honestly believe that we are both wanting to do whatever needs to be done for Drupal's future. Though I definitely disagree with the outcome, I respect and appreciate your concern. :) Back to the fun stuff though...

And it gives users the chance to wait while you do expensive string operations iterating through every byte of a 5 meg (and growing) array of module/theme data... assuming you haven't already run out of RAM just parsing the XML in the first place. ;)

I took this into consideration during the design phase. XML files are being processed with XMLReader. It only loads into memory the section of XML that it needs to parse at any given instant. It is extremely light on resources. I didn't want to use too much memory.

"The only reason your prototype is at all functional (haven't tried it yet) is because you have a subset of the current data and/or your PHP configuration is intentionally very forgiving for RAM and CPU usage."

I'm pretty sure that I'm using most if not all of the data (at least as of a week or two ago.) Also, mysql is doing the text searching. Since it supports full text indices, I believe that could help out greatly in terms of resource cost.

I do understand that there are likely to be more and more people using Drupal. Quite honestly, I'm thrilled by that prospect. I also realize that many people have limited resources. That's great, because I do too. I'm not trying to get this added to core though. Right now I'm just wanting to make a completely optional contrib module.

"But it'd be such a cool feature, and all the usability folks think so, too" won't convince me otherwise. I agree it *would* be cool, but it just reminds me of all the code in project.module itself that worked fine when d.o had O(100) contributed projects which completely melted and fell apart we hit another order of magnitude in the number of projects.

I'm not worried as much about the cool features as the useful features. I also can't disprove that the same problem won't happen here. I can give two options though:

1) We actually run the script to see how large it would be. I'll take the gzipped file, double it and see if the system can still stand.

OR

2) We let d.o handle all of the searching itself. It has already been pointed out that Drupal's infrastructure can handle a heavy load. I can't say that this would be the way I would want to go though. (Or that it would have any shorter wait times.)

J Rogers

Actually, the more I think about it, the more okay I would be with putting the actual search portion on d.o and just returning XML or JSON to the browser or server that requests it. Would that be a more acceptable avenue to take?

Would it be possible to create an interface that can return JSON/XML results to searches through the various projects hosted on Drupal.org? This method provides several advantages:

• This would completely eliminate the need for a local repository.
• This would lower the memory footprint of the plugin manager module to just a few kilobytes.
• Since the alternative would be to use d.o's search, this should be no more stressful than someone searching normally.
• It would be able to scale as far as d.o can.

I believe that would negate most of the earlier issues that were raised without causing any lack of functionality.

@dww: You were right. The other method was a bad idea. I'm hoping that this corrects that.

The attached patch creates two files: .htaccess and plugin-manager-repo.php. It allows three functions: get a list of project categories, get a list of releases with md5sums (and sorted by recommended version) and search for projects in a particular category with a particular API version with a certain term. (This uses the same Solr search that powers the searching on d.o)

All queries return either XML or JSON data. This allows the plugin manager to work without a copy of the repository. (Thus with much lower overhead.)

If this is inadequate, I am more than happy to work on it. I do need this functionality for my project.

Thank you.
J Rogers

Corrected some typos that got by me the first time.

And finally curse the fact that kwrite automatically creates files ending with ~. Yeah. I forgot it was going to create one of those before I submitted the patch.

To be honest, I had never really stopped to check. :P Anyway, I've disabled it now.

With this patch in place, this is what a plugin manager install would look like:
1) Upon loading the plugin manager, the browser would load whatever.drupal.org/json/categories/6.x to get a list of all categories that are available.
2) Once the user is ready to search, the browser would load whatever.drupal.org/json/search/6.x?text=my search string&tid=15&page=1. This would return the results to the browser for display.
3) Once the user is ready to install, the browser would load whatever.drupal.org/json/release/6.x/coder for example. It would get a list of all releases with md5sums and filepaths. The releases are sorted by recommended version, then major, then minor, then patch. Thus, the one at the top of the list should be the newest recommended.
4) The browser would post the md5sum, version of the one to install to the local server.
5) The server would download whatever.drupal.org/xml/release/6.x/coder. It would read the version and the download path.
6) The server would download the file for the appropriate version then compare the generated md5sum to the posted md5sum.
7) The rest of the magic happens here.

@dww: Okay, 'A', 'B' and 'C' were all me being an idiot. ;)

D: That sounds fair enough. I'll replace this section with a section to give the md5sum of a file in json. Using update sounds good enough.

E: Most of this came directly from project_solr_browse_page. At the end of the function it uses a loop to turn $response into themed html. If it had just returned $response then I would not have needed to copy any of the code. I suppose I could make a patch that moves the majority of project_solr_browse_page into a helper function. Would you suggest this?

@dww: Okay, 'A', 'B' and 'C' were all me being an idiot. ;)

D: That sounds fair enough. I'll replace this section with a section to give the md5sum of a file in json. Using update sounds good enough.

E: Most of this came directly from project_solr_browse_page. At the end of the function it uses a loop to turn $response into themed html. If it had just returned $response then I would not have needed to copy any of the code. I suppose I could make a patch that moves the majority of project_solr_browse_page into a helper function. Would you suggest this?

D: The way the system current works (if you can call it that,) the plugin manager displays an iframe of the release page before a module can be installed. The user has to copy the md5sum from the release page to the plugin manager form. This is done for security sake.

If the server downloads the file and md5sum and the server had been affected by DNS poisoning then it would be it would be simple to make malicious packages appear to be legit. Thus, the user needs to post the md5sum to the server. One possibility would be to use ajax to fetch it from the XML release history. Unfortunately security settings prevent most browsers from loading XML from remote sites, so this isn't an option... JSON data can be loaded though. Serving md5sums in a json wrapper would allow the browser to automatically get the md5sum and then post it to the local server. This would successfully hide that nasty step from the user without endangering them.

E: My mistake... again... Dang it. Sorry... again. I'll get to work on that patch now. :)

I think I might have worded it badly. The worry here isn't updates.drupal.org falling prey to DNS poisoning. The worry is that a server hosting a Drupal install might become poisoned. If that were to happen then all attempts to connect to updates.drupal.org and ftp.drupal.org could be redirected to sinister.example.com.

If the local server is poisoned and it grabs the md5sum then the both could also be pulled from sinister.example.com. If the user enters the md5sum and the server downloads the package, however, then both the server and the client PC would have to be subject to DNS poisoning for this attack to be successful. Otherwise the fetched md5sum and the calculated md5sum would most likely be different.

That is correct.

The reason we haven't attempted letting the browser parse the XML is because of the "Same-Origin" policy (or at least what we understand it to mean.) Simply, it will not allow XML from one domain to be read by a script on another. (Once again, that is my personal understanding of it.) I do know that json data can get around that limitation, however.

As far as knowing which file we are refering to, appending ?filename=path/to/the/file.tgz to the end of the url would precisely identify which md5sum should be returned.

29e is beginning to look unlikely. Issue #453718: Provide easier access to raw Solr results is meant to address is. Unfortunately, it has received no love. :(