Drupal.org redesign/upgrade infrastructure and implementation questions:

All very good questions. I'm in no position to authoritatively answer any of them in terms of "how should it be for this process". In fact, the apparent lack of clarity about that seems to be an ongoing issue that the whole d.o redesign is going to keep bumping up against. But, I am a "veteran experienced from the d.o upgrade to D5" so I can at least answer some of them from the "how we've done it so far" perspective...

A) What modules may be considered pre-approved for use on drupal.org?

- Everything currently in use: http://drupal.org/node/27367
- views (since it's already been approved, and already in use in places, and will be a project* dependency after the D6 upgrade).
- That's about it, as far as I can tell, for "pre-approved" use.

B) What is the approval process if a non pre-approved module is required? What person(s) will review that request? What information would be expected in such a proposal (reason for module request, module performance benchmarking, module stability, module maintenance plan, etc.)?

Historically, this has been "ask Dries and Killes". The request has always required an audit by at least 1 security team member, a clear indication that the module has a "responsible" maintainer (or ideally, team of maintainers), and some loose performance benchmarking or at least code sanity review. The criteria might get more formalized for the future in terms of benchmarking, unit tests, etc. Personally, I'd say that "usefulness/user base outside of d.o" should be an important criteria here. Ideally we should only add more modules that are in wide use and have a large following of people willing to support/sponsor maintenance, upgrades, enhancements, etc. That's a big part of why project* has historically been in such bad shape -- it was only useful to d.o itself, so no one else had an interest in improving it.

C) What is the status of the subsite login and searching projects?

No idea. I don't even know who's working on any of this.

D) If we continue to use subsites for d.o, can we launch them as each is ready? If so, practically, how will this work if we both reorganize the site and launch subsites independently (meaning that it's possible some content may "disappear" if it's old home relaunches while its new home is still being built).

I think launching the subsites independently is a reasonable approach to phased deployment. I think the only practical way to do that, however, is to also do the global reorganization in a phased way, too. E.g., we can't completely reorganize the "flagship" d.o site and turn it into the Drupal brochure site until all the other subsites are up and all the content has been appropriately migrated. We can rip out all the forums once forums.d.o (or g.d.o or whatever our solution for the forums will be) is up. We can rip out all the project/release browsing stuff once downloads.d.o (or whatever) is up, etc. So, I think the basic approach should be: Build each subsite, deal with the data migration problem, deploy and reorganize that part of the main site, $i++. Only when $i == $num_subsites can we finally do $drupal.org = $brochure_site. ;)

E) How should content migration be handled? Does all existing content have on-going value? How concerned about URL preservation do we need to be?

Carefully, yes, and a lot. For example, there's a *TON* of historical information and wisdom recorded in issues, which are referred to via CVS commit messages, release notes, CHANGELOG.txt files, etc. IMHO, it'd be a huge disaster if you could no longer find those issues at their original locations (or at least be properly redirected from their original location).

In fact, personally, I think it'll be really confusing and weird if the nid space is no longer unique across the split up *.d.o. It's already a shame that the nids on g.d.o aren't unique with d.o itself. It'd be nice if you could just go to *.d.o/node/NNNNNN and get redirected to the right site where that nid actually lives. That way, if you see "#12345" in a commit message, email archive, release note, etc, that you don't have to try N different sites to find the /node/12345 you're looking for. But, that's just my take on the problem, others will probably disagree, and say that a globally unique nid space isn't desirable or even possible.

F) What is the preferred process for staging.drupal.org? What is the protocol for requesting an account? Reasonably, how many accounts should there be (we may end up having 6+ teams working concurrently on various areas of d.o simultaneously)? What is the setup for staging? Are there any standards or guidelines that need to be communicated and reinforced to the project teams?

I think staging.d.o is too broad. I think we need separate test sites for each of the proposed subsites. Each one can get a scrubbed version of the d.o DB as a starting point to test data migration, etc (and we should set it up so it's easy to reset any subsite to a freshly scrubbed version of d.o's DB). Each team should have 1 or 2 people who are responsible for deployment and admining these test subsites, and those people need to be approved by killes/dries to get shell access to the virtual machine where their test site lives. Generally, the standards for deployment of any code on *.d.o has been: "patch is RTBC'ed by someone on the security team." I suspect we'll have to relax those standards during this process -- perhaps we should just say that the final step before live deployment and data migration with real data is the security audit. But, I don't think it'd hurt to keep high standards even during the development and testing phases.

G) How should peer review of code be handled? Does a team currently exist? Do more people need to be added to that team? Typically, what are the submission protocols and scheduling lead time? How has this been handled to date?

See above. So far, it's basically been either "RTBC'ed by killes" or more broadly "RTBC'ed by a sec team member". The sec team is already woefully overworked and understaffed, so this might become problematic as the d.o reimplementation ramps up. Yes, finding more security-conscious (and performance-conscious) people who can be part of the peer-review team should be a high priority in the lead up to actual development/deployment efforts. There are only so many of us actively playing this role for d.o in the community right now, and we're going to be totally swamped as this process takes off unless reinforcements arrive.

H) How should q/a be handled? Are there any extant or otherwise logical folks who are/should be on this team? Ideally what is the profile of someone who should be on this team? Does d.o have any guidelines used or veterans experienced from the d.o upgrade to D5?

There's no such team that I know of. For the most part, d.o q/a has been handled by "dww and crew try to make sure everything works as expected before it goes on the site". Not exactly scalable to this process. ;) And, I've certainly had more than my share of failures where even after lots of testing, as soon as something goes live, it broke in an unexpected way that required a few frantic hours of cleanup.

To the extent possible, we should be encouraging/enabling automated testability during all of this process. But, at the end of the day, we still need some usability mavens and people who like to break things to do their best to flesh out problems during testing/development, before live deployment. No such team officially exists at this time. The ad hoc d.o usability team to date has been a combination of myself, hunmonk, aclight, webchick, greggles, add1sun, nedjo, and whomever else I could find in IRC when I was working on a change. ;) That's an unofficial, incomplete list, and I doubt it's the right set of people going forward (webchick seems to have more than her fair share of responsibilities these days, for example), but that's some of the background on how it's "worked" to date.

Probably not exactly the answers you were looking for, but that's what I can share as someone who's been through all of this many times, both as an outsider (in the early days of my project* contributions) and as someone with root on the servers. ;) Hope it's useful.

Subscribing. No time to respond now, unfortunately. It'd be great to have a few more comments by members of the infrastructure team though. This stuff is really important.

Thanks for the thorough reply, dww.

I will personally make sure that the modules needed to search across sites and login across sites are ready by the time we need them. I'll recruit volunteers to help with that effort.

I think that *some* breakage of URLs and some loss of obsolete content is expected. But there ought to be a good reason when we do this. We can do a lot with redirects so URL breakage in particular is usually avoidable.

I think we can mostly ignore peer review of custom code. We will only let trusted persons write code for drupal.org. The infra team will give custom code a cursory review as needed. The security team can also be called upon as needed

Let's put some context around this discussion. The redesign is now active and underway. The upgrade of Drupal 6 is still progressing. Redesign deliverables will be ongoing but finish up in the November time frame so we've got a few weeks to get this ready.

I've met just now with Killes and we have a plan for recruiting new infrastructure team members to help with the development, and staging phases of the redesign.

We got 17 volunteers to join the infrastructure team and Killes and I will be working to increase the production team and have resources to support the redesign efforts.

The logistics for infrastructure are straightforward. The development and staging environments will be built on the scratch virtual machine we currently have. Redesign infrastructure team members will be given Linux user accounts and access to a sanitized dump of Drupal.org.

I am glad to see Moshe leading the effort to coordinate search. I'll reach out to Jeremy Andrews as well.

I'll keep working through this list to recruit implementation resources and tackle these projects in order of priority.

Cheers,
Kieran

What modules may be considered pre-approved for use on drupal.org? I understand that the goal is to keep these to a minimum given ongoing maintenance considerations, but I think that the teams who will be working on implementation need to have the module list defined and documented.

From a security point of view, pre-approved is Drupal Core and nothing else. Even modules that we currently run on drupal.org in their D5 version should be re-evaluated for D6.

Of course, it is not very realisitic to only create a website with Drupal core, so we might need to cut some corners if we don't get reviews...

What is the approval process if a non pre-approved module is required? What person(s) will review that request? What information would be expected in such a proposal (reason for module request, module performance benchmarking, module stability, module maintenance plan, etc.)?

Ideally, the security team or other experienced coders should look at any module that is deemed desirable. On drupal.org, benchmarking is usually an ongoing process...

What is the status of the subsite login and searching projects? Will either or both of those solutions be ready for Q4 2008 or Q1 2009 (realistically the timeframe for when d.o upgrade will be in heavy development)?

Moshe will take care of this.

If we continue to use subsites for d.o, can we launch them as each is ready? If so, practically, how will this work if we both reorganize the site and launch subsites independently (meaning that it's possible some content may "disappear" if it's old home relaunches while its new home is still being built).

This depends. If we want to unify searching across subdomains, or have common log ins across domains, this might imply that we need to share the databases to some extend and this would make the subsites interdependend. Otherwise we can launch them independendly.

How should content migration be handled? Does all existing content have on-going value? How concerned about URL preservation do we need to be?

Google loves drupal.org and we should get quickly re-indexed if we change anything.

However, I'd prefer to cause as few broken links as possible, google won't fix internal links for us and not links from other sites to d.o.

What is the preferred process for staging.drupal.org? What is the protocol for requesting an account? Reasonably, how many accounts should there be (we may end up having 6+ teams working concurrently on various areas of d.o simultaneously)? What is the setup for staging? Are there any standards or guidelines that need to be communicated and reinforced to the project teams?

We can provide test setups that operate on a sanitized copy of drupal.org. Contact me through the infrastructure list when you need these.

How should peer review of code be handled? Does a team currently exist? Do more people need to be added to that team? Typically, what are the submission protocols and scheduling lead time? How has this been handled to date?

As with much of the other stuff, there isn't a formal process yet. Often there are requests being made towards the security team. However that team is somewhat overworked and always looking for qualified people to join it. So this can take any time.

How should q/a be handled? Are there any extant or otherwise logical folks who are/should be on this team? Ideally what is the profile of someone who should be on this team? Does d.o have any guidelines used or veterans experienced from the d.o upgrade to D5?

Most of these people are still on the infrastructure team, most notable dww and myself.

About redirects...I'm happy to help with this if/when the time comes. We've discussed it a little on the g.d.o/redesign group. I feel that the best solution is to do 301 redirects from drupal.org to the new locations. This should be possible as new sites come live.

Also, I really like dww's idea that we somehow maintain unique nodes, though I'm not really sure of a great way to do that. In some environments you can just say "this server gets odd nids and this one gets even nids" but with an arbitrary number of subsites and an unknown growth rate that seems like a bad system for us to try to implement...

Re: 301 redirects - exactly. Something like path_redirect might be all we need.

Re: unique nodes (and users) -- can't we have a single DB table shared by all sites that doles out the next unique [nu]id, or is that deemed too much of a performance bottleneck?

@farriss Clearance:

The working idea is that separate from the discussions with MBD ...

FYI: Mark Boulton Design
Thanks

Edit: Thanks @add1sum

@wolfflow, MBD is the design firm we have hired. It stands for Mark Boulton Design, but they are referred to by the shorthand MBD.

@wolfflow -- Just because you can, please don't edit your issue followups like this. It makes things very confusing for people reading the flow of the issue, and it's confusing for people following the issue via email, since they don't get another email just because you edited an existing comment.

p.s. #306132: Add a permission that prevents users from editing issue followup comments ;)

@dww thanks for your feedback, i did not now. I will put my hands off to my comments next time and in future. I apology for this inconvenience. :-(

Assigning the development of infrastructure and implementation for the Drupal.org redesign to myself.

Narayan, Gerhard, and with help from Damian have implemented a new private SVN for sharing the Drupal.org theme. New module is detailed in this issue: http://drupal.org/node/303611

Neil is the lead maintainer for Drupal.org bluebeach theme for 5 and for upgrading the remaining parts of the theme to Drupal 6. Earl did an initial upgrade of bluebeach to test the new theme system a few months back.

See the new http://groups.drupal.org/drupalorg-theme-developers group if you want to contribute to maintaining and developing the new Drupal.org themes.

We've also created a new group for Drupal.org redesign infrastructure team. http://groups.drupal.org/drupalorg-redesign-infrastructure-team

Michelle has added a redesign component to the webmasters issue queue as well.

Please reopen if there are any actionable things to do.