Posted by Uwe Hermann on October 9, 2004 at 6:16am
| Project: | Project |
| Version: | 6.x-1.x-dev |
| Component: | Releases |
| Category: | feature request |
| Priority: | normal |
| Assigned: | dww |
| Status: | closed (fixed) |
Issue Summary
As seen in http://drupal.org/node/11158, it might be worthwile to provide an additional drupal-x.x.x.zip distribution in addition to the tarball, as many (Windows) people don't know anything about tar and gzip.
It's also not very widely known that winzip can perfectly extract tarballs.
While we're at it, a .tar.bz2 distribution might be nice, too.
Comments
#1
the general feeling about this, is not to do this. Setting status to "postponed" and will be closed in 2 month if no-one is picking this up. feel free to provide a shell script that we would be able to use for repacking including all themes and modules or update ticket in any other way.
#2
Which script is currently used to build the tarballs? Is it available for download or included in CVS somewhere? It should need only minimal changes to build additional zip files.
#3
It's been postponed for nearly two years.
What's the status, now?
#4
while it is true that not all windows apps know about tar and gz, there are enough tools out there for windows platforms that do understand this.
Since adding other versions of drupal.tar.gz only adds extra work and difficulties for the enduser ("which one should i download") and on the drupal learning curve unzipping a tarball is only a very small step, i label this wont fix.
#5
Looking at the Joomla download statistics zip is favored over tar by a wide margin. I'm not a fan of the format but I can't help fearing that a significant percentage of potential Drupal users stop at the file download stage. Are these users that Drupal wants to keep? I'm marking this issue as active for now as this could potentially be significant.
#6
I don't care about what joomla users prefer.
#7
Citing the fact that Joomla! users choose these helps prove the case for providing them instead of (or in addition to) tar.gz. Just today I talked to yet another new Drupal user who struggled with this problem.
I guess this first step is to work on this to provide support for multiple downloads files inside of the schema/code of Project* (project* team please correct this if I've put it in the wrong place).
#8
When my wife first downloaded Drupal she had zero idea what to do with the tar.gz file after it was downloaded. She then went to the handbooks and spent maybe 30 minutes trying to figure out what to do. I think that this particular time the getting started stuff was in pretty bad shape, and that contributed to the problem, but given that downloading Drupal is the first step for a lot of new users, if they can't even figure out what to do with the file then they're in trouble.
Remember also that by default windows XP and Vista can extract .zip archives, but I don't believe they can extract tar.gz archives. So that means that users then have to go and download and install additional software before they can even extract Drupal. That's not great usability.
This would be pretty trivial to do except for one problem--at the moment release nodes can only handle one attachment. So, with the current configuration, we'd have to have two release nodes, one for a .tar.gz file and one with a .zip file, for each tag. That would be pretty counterintuitive and probably would be a mess to code.
So, in order for this feature request to be possible, project release nodes need to be able to accommodate multiple files. At the moment, project_release is using its own custom code to handle attachment of files, instead of using the core upload module. There's already an issue for this specific part at #179471: release file attachments should use drupal upload functionality. I'm going to mark this issue as postponed until that issue is fixed or until someone makes an alternate suggestion that would be effective.
As I mention in the issue I just linked to, one difficulty with doing this is how to structure the download tables if there are multiple files per release node. Please follow up in that issue if you have any suggestions.
#9
Current status: We're 90% of the way there, thanks to the following:
#357920: Numerous errors when previewing/submitting a new release node
#366448: Port packaging script to new {project_release_file} schema
#539282: Denormalize release info a bit and store latest and recommended releases in {project_release_supported_versions}
#539668: Expose files attached to releases to views
#539676: Expose {project_release_supported_versions} data to views
We're basically now just blocked on these two:
#539682: Replace hard-coded release download table with a view
#555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org)
There's a bit more drupal.org UI to figure out, but mostly that'd be handled via #539682. Almost ready to call this active again. ;)
#10
It's great to see this in the pipe. It came up just today on Twitter. (See #687000: Why gzip, why not zip? - sorry about the noise!)
I think (as I pointed out in the duplicate issue) this is an important step towards generally making Drupal more accessible to new (particularly Windows) users. I would be willing to devote some of my time towards this effort, if that's feasible, to expedite it. Ping me if you want to take me up on that. =)
#11
Oh, and how come #8 says this issue is blocked on #179471: release file attachments should use drupal upload functionality but #9 doesn't mention it? *confused*
If #9 is the correct status report then I guess the issue referred to in #8 can be canned completely?
#12
#8 pointed to #179471: release file attachments should use drupal upload functionality due to misunderstanding (which is easy given how many issues are flying around here). ;)
#9 tried to clarify, but didn't succeed. ;) But, #9 is the accurate list. However, #539682: Replace hard-coded release download table with a view is actually done as far as this effort is concerned. The only reason that's still open is because there's a now a large pile of legacy code that can be ripped out. But, it's now very easy to add zip downloads to the download tables on project nodes.
So, really, the main blocker here is: #555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org). It's featured prominently on the Update status module community initiatives page, but my hands have been full with D7 update manager issues, and sadly, I haven't gotten much help there... :(
After that, the actual changes to the packaging script will be pretty easy (which we can do here in this issue). I've already mostly got that code done from work I did through 3281d Consulting over the summer...
#13
Thanks for the update. All encouraging stuff! I'll take a look at #652566: Support multiple download files per release in release history XML then. =)
#14
To me this is a big red flag saying forget Drupal. Pity didnt even get to try it. Not only I dont want to mess about with tar files but the attitude stinks. 6 years and counting ...
I also found the forum doesn't open and the 'download' tab is actually about project types.
Enough!
#15
We are sorry that the tarball was to too high a treshhold for you. note that most third party unzip utilities (like winzip etc) can handle tarballs and that unpacking a tarball is an easy job comparing to configuring Drupal. But if you really want to give it a second look (and you should :-), try a complete install package like microsoft offers (http://www.microsoft.com/web/drupal/) or acquia (http://acquia.com/downloads) that includes a webserver, php, a database and drupal in one simpel install file.
back to the issue at hand, should we close this one?
#16
@bertboerland: no, we shouldn't close it, it's still not done. It's closer, but stalled, since I have too many other things to worry about and no one has stepped up to try to drive this home without me (or offer to hire me to get this done so I can focus on it instead of work that pays the bills).
#17
I know you have been working on it and do appriciate it a lot. bit the question was / is do we still need the zip files? acquia and msft have been offering installers for those who have trouble with tarballs.
#18
Yes, we still need .zip files. Not everyone uses microsoft or acquia.
#19
Agreed. And Microsoft/Acquia don't provide it for all the contribs. Those are an important part of this.
#20
This is a timely bump. I said I'd look at this but also didn't have time (sorry dww) ... is this still the case? I guess so:
http://drupal.org/node/652566#comment-2347352
#21
subscribe
#22
subscribe
#23
This patch is based on work I did last summer for the script that packages releases for FusionDrupalThemes.com from Top Notch Themes (aka "TNT"). Given a lot of prior work in project_release to support multiple files per release, the changes to the packaging script itself to support .zip packages aren't *too* terrible. There's still work needed to properly display these in the release download tables, but we're getting closer. I'm providing two versions of this patch for consideration:
The first *just* creates the .zip files at the appropriate moments and adds them to the release nodes.
The second patch introduces a bigger change: UNIX vs. DOS line ending conversion. The basic idea is that before we create a .tar.gz we ensure that all the text files in the directory tree have UNIX line endings, and before we create a .zip, we ensure all the text files have DOS line endings. This is mostly code I wrote for TNT, but merged into the d.o packaging script (especially all the packaged distribution stuff). I'm not 100% sure we want this. Here are the implications of the change:
A) Platform-specific packages will have platform-appropriate line endings (probably a good thing).
B) Lots more file I/O during packaging runs. I'll try to get nnewton to take a look at this to see how he feels about it. ;)
C) All the text files (.php, .inc, .css, whatever) in the packages will be packaged with modification times from when the packaging script ran, not from the last time those files were committed in CVS. The conversions all happen after we do the logic to find the most recent change in the directory tree when deciding if a -dev tarball needs to be rebuilt, so it won't break that. I'm not sure anyone will notice or care, but maybe they will.
Now that I think about it, it should be a relatively easy change to the convert_file() function in here to stat the target file before it does anything, do the conversation, and then use touch() to set the modification and access times back to how they were before the conversion. Other than adding even more file I/O to the whole process (making B a bit worse), I think this would eliminate any potential trouble from C.
Regardless of which approach we pursue, once this patch is ready and committed, the remaining issues to resolve before this feature is actually fixed are:
#652566: Support multiple download files per release in release history XML
#539682: Replace hard-coded release download table with a view
#555362: Support multiple download links in the available updates report (.tar.gz + .zip packages like on drupal.org)
#24
The last submitted patch, 11416-23.package-release-nodes-zip-line-ending-convert.patch, failed testing.
#25
Subscribing to remind myself to review this.
#26
Subscribing. Don't know if there's anything I can do to help, but would definitely like to see this...
#27
Subscribing.
Running unzip is easier than tar xzf. (Also easier to remember.)
#28
This is now blocked awaiting the outcome of #995042: Figure out what to do with DOS vs. UNIX line endings in the packaging script (which will hopefully not take too long).
#29
Okay, great. [ #995042] exactly served its purpose, and the decision is that we do not want line ending conversion at all (at least not now). Back to needs review on the first patch in #23.
#30
Although it's slightly a hack to keep avoiding a weight column in {project_release_file}, we can continue to just use the fid for ordering stuff if we're a bit careful. Given how we want things to work for packaged distros (the full distro with core, projects, and the profile itself should be the main link on the project download table), we'll just order this stuff by fid DESC and it all works fine. However, if we want to keep .tar.gz at the front of the list (and I think we do), we want to actually packaging and insert the .zip into {files} first, then do .tar.gz. So, here's a minor re-roll of the simple patch from #23 to get the right ordering.
Meanwhile, here are 3 possible patches to the default download table view and corresponding screenshots to consider the best UI on the download table itself. Note that these are from a garland-based local test site, not something on *.d.o running bluecheese, so the screenies are a bit off from how it'd really look. But, it should be enough to get the basic idea and decide on an approach. Other options are possible, but I wanted to get feedback on these before I went any further.
Option A: Bar delimited with just the filename extension and the filesize
Option B: Bar delimited with the word "Download" in front of the filename extension and the filesize
Option C: ul of possible downloads with "Download" in front of the filename extension and the filesize
Thoughts?
Thanks!
-Derek
#31
In IRC, Bojhan requests without the filesize. :( I don't like this option, since I personally find filesize useful when scanning these tables. If I'm trying to solve a small problem, and I find a module, and the download says "1.1 MB" or "534 KB" and I think "screw that, this is 30 lines in my site-specific module", it saves me a fair bit of trouble. Plus, it can be helpful and interesting to see the size of the downloads across different versions of core and different branches. So, I'd rather not do this, but here it is for comparison.
Option D: Bar delimited with the word "Download" in front of the filename extension
#32
p.s. Also note that this test site is slightly busted and the fids aren't ordered properly from the packaging script changes in #30 -- that's why zip is showing up first here, even though it won't when this is all deployed for real.
p.p.s. I'm actively working on this now, so I should assign myself. ;)
#33
In IRC, greggles proposes a hybrid of A and C: a
<ul>that drops the duplicate "Download" verb in an attempt to improve scanability.Option E: Ul with just filename extension and size
#34
One edge case I thought of in how all these views work. See #1006238: Add a weight limit for project_release_handler_field_files for more.
#35
After testing this more thoroughly on beta.d.o, I don't think there's any way we can keep hacking this by trying to order everything simply via fid. See:
#1006346: Add a {project_release_file}.weight column
So, here's a re-roll of the packaging script changes assuming #1006346 goes in as-is...
#36
I like E in #33. It's scannable and think it's important they are on their own line.
#37
I'm going to categorically rule out option D. The file size is critical information on a download link. Also B) because it's way too crampy and wouldn't scale out if we also decided to add .bz2 for some reason down the line.
My preference is to have the link have an action in it. The current one does: "Download". Which means C).
However, when I expressed this to dww, he pointed out that this is pretty heavy vertically (which is true) and ends up repeating the word "download" like 500 times in a typical table like http://drupal.org/project/views (which is also true).
That leaves A and E, and E is more scannable. so I'd go with that. It's also more scalable if we want Project module to be useful outside of Drupal.org. SourceForge offers like 10 different possible types here, for example: http://sourceforge.net/projects/azureus/files/vuze/Vuze_4510/
So yes. E).
#38
Re-rolled #35 based on recent commits...
#39
After testing this in bluecheese, I am going to favor A) in contrary to popular believe the usage of
<ul>'s doesn't always make it more easier to read. In this case the users perception should try to group download links with projects, since we have almost no visual separation in this table, adding a lot more visual spacing will make it harder to visually group these links and the user will shift to group the download links - rather than seeing them part of a project release (this can be partly countered by adding more vertical spacing between projects, but that will make the table very long)We lose the keyword Download, which has serious down-sides on the conversion - we probably want to revisit this later when we have more time, there is definitely something said for using web analytics and multivariate testing to improve this.
#40
Cool, thanks for all the UI help. I think A is best, too, though sure, I'm happy to revisit this later.
Here's a final patch given recent commits, including the changes to the download table view to implement A.
#41
Committed and deployed!
http://drupal.org/project/drupalorg_testing
http://drupal.org/node/1006918
...
#42
.bz2 wasn't part of this. If we want that, let's open a new issue. Now that I've done all this work, bz2 *should* be pretty trivial...
#43
For the historical record, here's the 1-off-script I wrote (a hacked copy of the main packaging script) to go back and generates .zip files from all the existing release tarballs. After checking with nnewton that this wasn't going to be a problem, I ran this last night, and regenerated all the release history XML files afterwards. That's why *everything* has a .zip version now, too.
#44
Yay! :D Awesome, thanks dww!
#45
Awesome. This is very helpful.
#46
FYI this change made 29000 files suddenly appearing in the queue of localize.drupal.org for parsing. The queue is processed with tar, so its not parsing any of the files at least (thankfully), but it tries, so the 29000 files stopped legitimate .tar.gz releases from being handled. Looks like there was no new release parsed on localize.drupal.org since this was deployed on localize.drupal.org (December 24th, 2010). This includes the last RC for Drupal 7, and unless we solve this quick, the Drupal 7.0 release itself.
The issue discussing this is at #1012812: Localize.drupal.org release parsing broken since 2010 Christmas, and I'd really welcome feedback to help solve it. Also, better coordination with services using release information would be good for such major changes in the future :) Thank you!
#47
This change was at least mentioned in the infrastructure queue (#995042: Figure out what to do with DOS vs. UNIX line endings in the packaging script). People using the infrastructure should pay attention to that queue...
#48
@greggles: thanks. This issue was submitted about 6 years ago, and apart from "soon we'll have" mentioned on the other issue you linked, I have not seen indication of the timing of this being rolled out. Looks like I missed a more concrete indication? I'm happy to prioritize to make localize.drupal.org work as smooth as possible. I definitely prioritized this issue over sleep last night and this issue over work this morning, so we can get this fixed ASAP, and the release backlog is being parsed as I stand by the cron job and kick it more often so it completes parsing the whole backlog sooner than it would normally take.
#49
@Gabor: I was trying to make noise in that infra issue to give the effort visibility to the rest of the infra team. I was also pinging folks in #drupal-infra in IRC. I didn't think a separate email to infra@d.o to announce the release date was necessary. And, in thinking of all the moving pieces related to releases, I simply forgot l.d.o was parsing release tarballs, and didn't know it was querying {project_release_file} directly, either. Anyway, sorry you had to lose sleep over this! It's hard to know which changes I'm deploying are going to impact who, and sometimes I mess up the notifications. My apologies about that.
Cheers,
-Derek
#50
Not that big of a deal and it is already fixed. This is where the code is, so you know for later what is affected. Note we also query project usage data for some time now. The code is all at http://drupalcode.org/viewvc/drupal/contributions/modules/l10n_server/co... I admit I did not pay that much attention around Christmas, I was actually on holiday for the end of the year from before Christmas. I'd love to get notified of these changes if I clearly am not involved in the process of changing stuff, I think being one of the biggest users of this data / scheme that would be great. Thank you!
#51
@Gabor: perfect, thanks. Now that I know, I promise not to make big changes in the future without keeping you in the loop! :) Hope you had nice holidays...
#52
Automatically closed -- issue fixed for 2 weeks with no activity.