I haven't done the VCS archeology to figure out when/why it started doing this, but for at least the last 5+ years, project_issue has been trying (in a very hacky way) to move all attachments associated with issues into a subdirectory of files for you. The way upload.module and comment.module work means it's very hard to do this sanely. It's broken in the UI for end users here:

#194449: Incorrect preview permalink when attaching files

Trying to fix that is just piling hack upon hack. I needed chx to walk me through his patch via skype, and even then, neither of us totally understand why it works. :/

Given how crappy all the plumbing in D6 is for this, I'm tempted to just *stop* trying to move the files at all and just let things work. Instead of adding another 50 lines of code to Project* that no one understands, I'd rather just remove 50 lines of code from Project* that no one needs. ;)

"But wait! There have got to be a bazillion files in there! Isn't that going to overwhelm the 'files' directory?!?"

That might make sense if project_issue was trying to do something fancy to shard things into reasonable-sized subdirs. But it's not. It's just putting *everything* in a giant subdir.

Now, for the current stats on d.o:

% cd /var/www/drupal.org/htdocs/files
% ls | wc
   3373    3883   76158
% cd issues
% ls | wc
 285762  319639 7555491

I'm not talking about moving files that already live in files/issues. I just mean for all new files and comments, we just start putting them directly into files.

Once Project* is ported to D7, we can start doing fancy sharding by hashing filenames and stuff, and it won't require insane hacks no one can understand.

So, before anyone spends any more time on #194449 (which is supposedly blocking #528682: Allow inline images to be posted to Drupal.org project pages, docs pages, and comments without any special permissions) I'd like to ask the rest of the infra team (especially nnewton) if there's a problem just removing these broken hacks and start bloating the files dir itself.

Thoughts?

Thanks,
-Derek

Comments

dman’s picture

It's always been a UI WTF, true.
I've got into the habit of either saving then re-editing, or just guessing (based on insider knowledge/experience) where images I put into issue will turn up.
Either way, it does feel like an annoying hack that was never really finished. The objective it is trying to achieve makes (made) sense, but the UI way it did it happens at the wrong point.

No objection to turning it off, but a better solution (using filefield_paths or similar) should remain on the wishlist if/when rebuilding this function.

dww’s picture

Assigned: Unassigned » nnewton

Yes, we'll fix this for real once Project* (or its replacement) exists in D7. D6 just sucks for this sort of thing...

Also, apparently nnewton likes infra issues assigned to him if we're blocked on him answering, and I think he's basically the one that has to decide if this is okay. Once he replies here, it's easy for me to take the actions necessary to actually do this. So, assigning to him (for now) to get an answer. @nnewton: if you're happy, please assign back to me.

Thanks,
-Derek

nnewton’s picture

Assigned: nnewton » dww

Hi Derek,

So in an interesting twist, I'm fine with this change, but not fine with our issues upload directory in general. As you note above, the files dir itself has about 4k files in it. This is absolutely fine with ext3+dir_index. However, our issues directory has over 200k files in it. Ext3+dir_index can handle a lot more than this, but we are over the usual recommendation for keeping directories performing well. Sharding this out a bit would be a really good idea.

This fix likely requires D7. Until then, I'm fine with putting these in the files directory.

-N

jhodgdon’s picture

Monitoring so that we can move forward on #528682: Allow inline images to be posted to Drupal.org project pages, docs pages, and comments without any special permissions... looks like this has a solution that just needs to be implemented now? Thanks!

sun’s picture

As mentioned in IRC, I'm not really a fan of this stop-gap fix idea.

The clear separation of issue attachments into /files/issues makes sense. (even though it could be improved, as already pointed out in the summary)

Belatedly changing it now means that issue attachments

  1. will additionally pile up in /files
  2. the identical issue attachment file name will be able to exist in both directories (/files/screenshot.png and /files/issues/screenshot.png)
  3. and thus, even if we wanted to, we'd no longer be able to "merge" the new files back into /files/issues (though moving files is not on the table and generally considered a bad idea)

I think the issue with previews got a bit over-hyped. The issue is not new and known for years already. It can be resolved later, or not.

Postponing #528682: Allow inline images to be posted to Drupal.org project pages, docs pages, and comments without any special permissions on the preview issue sounds like strive for perfection to me, which isn't necessarily needed to make progress. Especially as it requires, from my perspective, a rather large and debatable infrastructural change, which additionally has an impact on every site that is running project_issue.module.

Lastly, the entire issue is going to be obsolete as soon as we've upgraded drupal.org to D7 - which we're hopefully able to do in the very near future.

jhodgdon’s picture

Regarding the issue postponement, it *is* necessary in my opinion. One of the main reasons we want non-priv users to be able to post issues is so that we can open up more docs pages (with embedded images) for editing by the masses, so people can be encouraged to include graphics in docs without becoming admins. If we have to write complex instructions like:
http://drupal.org/node/528682#comment-4998252
for them to be able to do it, that is *not* OK for this group of users.

jhodgdon’s picture

greggles just pointed out to me in IRC that people editing docs pages do not have this problem, it is only people uploading images to issue comments. So I guess maybe my objection is not as important.

greggles’s picture

The clear separation of issue attachments into /files/issues makes
sense.

Not to me. I tend to scatter my files everywhere, so "sense" is relative.

* will additionally pile up in /files

What is the problem with this? The issue nnewton raised indicates that piling them up in a less full dir could be helpful.

* the identical issue attachment file name will be able to exist in both
directories (/files/screenshot.png and /files/issues/screenshot.png)

I can only think of one problem with this - that searching for an issue based on a patch will be harder. I don't thin that's worth stopping this.

* and thus, even if we wanted to, we'd no longer be able to "merge" the
new files back into /files/issues (though moving files is not on the
table and generally considered a bad idea)

So, this means your last point is moot. Great!

Postponing this issue for sharding sounds like letting perfect be the enemy of good enough ;)

Let's do this, but even if we don't I agree we should go ahead with the other.

sun’s picture

if not striving for perfection?

drupalorg_project_form_alter(&$form) {
  $form['upload']['#after_build'][] = 'drupalorg_project_upload_after_build';
}

drupalorg_project_upload_after_build($element, &$form_state) {
  foreach (element_children($element) as $key) {
    $element[$key]['#description'] = strtr($element[$key]['#description'], array('files/' => 'files/issues/'));
  }
  return $element;
}
jhodgdon’s picture

Can we try deploying this on the test site and see how it works? This seems like a sensible fix (and it should probably not be on this issue, which is nominally about not moving the files).

Steven Jones’s picture

@sun the issue is that the files are not at those paths until the comment/node is saved, so previews would break, and people wouldn't know why, but if they saved that preview with the broken images, the images would magically start working. See #194449: Incorrect preview permalink when attaching files for the discussion about this.

jhodgdon’s picture

We could do sun's fix and add a description/help text saying "This is the path where this file is expected to be saved. Will not work in preview mode, and subject to change. You may need to edit the comment to fix a path if you use this in a link" or something like that? Would be no worse than the current situation (and I think a little bit better).

Steven Jones’s picture

@jhodgdon we basically got sign-off on this change from nnewton above, so we don't need anything hacky here, we just need to make project not save the files into a subdirectory.

Looking at the code in project_issue module it seems that the rewriting is surrounded by an if statement:


if ($issue_dir = variable_get('project_directory_issues', 'issues') ) {

So presumably testing this would be as simple as setting that variable to be an empty string? No clever heroics needed here.

jhodgdon’s picture

RE #13 - nnewton may have signed off, but several others responded saying it was not a great idea after all. So I am not sure whether consensus has been reached.

drumm’s picture

Project: Drupal.org infrastructure » Project issue tracking
Version: » 6.x-1.x-dev
Component: File server » Issues

This is removing a buggy hack, which is less code to think about when upgrading to D7. There is absolutely no reason not to do this.

With infrastructure sign-off, moving to project_issue for implementation.

dww’s picture

Assigned: dww » Unassigned

http://drupal.org/admin/project/project-issue-settings has a setting for this. There's nothing we need to implement in project_issue to let things improve on drupal.org. I'm going to unpostpone the other issues, and as soon as we're ready to deploy and enable the filter, we can just toggle this switch.

Meanwhile, we'll probably rip out this code, but it's not urgent.

drumm’s picture

Status: Active » Closed (outdated)

We have date-based subdirectories on Drupal.org now. The filename deduping was taking too long for interdiff files. And D7’s file field configuration has been good for updating this configuration.