This week, twelve of us have gathered in Paris to continue our work on upgrading drupal.org. Sprinters include: Joeri Poesen, Damien Tournoud, Neil Drumm, Mike O'Conner, Gerhard Killesreiter, Klaas Van Waesberghe, Todd Ross Nienkerk, Aaron Stanush, David Stosik, Morten Heide, Gábor Hojtsy and myself. Upgrading and redesigning drupal.org is a big project, and, when implemented, will be an important milestone for our community. We're hopeful that we can push the drupal.org redesign closer to completion this week.
This week in Paris, our group is split into two teams. One group will implement the new theme designed by Mark Boulton and Leisa Reichelt with the involvement of many people in our community. The second group will continue where we left off in Boston and will also start implementing some of the new functionality planned for the new drupal.org (e.g. better search, improved project pages, better landing pages, a jobs page, an events page, and more).
These week-long sprints are non-trivial. Although all of the participants invest their own time free of charge, we'd like to cover their airfare, hotel and some of the food required to keep them running. We've only been able to hold these sprints due to your generous contributions in the past; we still need to raise more funds so it is not too late to donate. Please consider using the Chipin widget to contribute if you can.
A number of organizations, including One Laptop Per Child, AF83, Four Kitchens, DrupalTherapy, OpenBand and Looforyoo, Capgemini France, NowPublic, Tag1consulting, and Acquia have already come forward with donations of money and resources to help make these sprints be successful.
At the end of the sprint, we'll update you on our progress, so please stay tuned for details.
Comments
It seems to me that one of
It seems to me that one of the strengths of Drupal was the EASE of upgrading and implementing new designs. I thought upgrades were a simple matter of uploading the new files and letting the software upgrade the database. And implementing a new skin was an "upload and activate" feature.
If you're finding that "it ain't so easy" on your own site, then this should tell you something about what needs to happen in terms of usability. If it's THAT much work upgrading a site, then something needs to change in the code itself.
Now, I completely understand how much work it takes to CODE the next version of Drupal, as there is a lot of wonderful code to work with, and a lot of compatibility issues to deal with. But that's a different thing. Once the code is ready, the site upgrade should be easy enough for a single person to implement in a single day.
- Alan Tutt
http://www.AlanTutt.com
- Alan Tutt
Exceptional Personal Development for Exceptional People
http://www.PowerKeysPub.com
You really have no idea
You really have no idea about what you're talking are you ?
This is not a simple update, it's an upgrade.
There is major functionality change. Previously hardcoded paths and filters are now replaced using views.
The layout is completely changed
There are a ton of pages that have a completely different layout.
Did i mention the improved search ?
Also moving a site of this size is never easy. You need to test the move. To make sure that all the data got converted correctly.
Etc.
------------------------------------------------------
Think Smart, Be Free, Choose Open Source.
How many people does it take to implement a theme that's already
Update / Upgrade -- maybe you should define your terms. They sound like the same thing to me. Again, as I mentioned before, I understand there's a TON of work developing the next version of Drupal, but this is different from upgrading a site, especially when implementing a theme that's already been developed.
Functionality would be part of the code, not the site design. I would expect Drupal.org to run on stock Drupal code with contributed modules that have already been developed. If the modules need more work before they're ready, then that needs to be done first, not during an upgrade.
Layout should be handled by the templates, not the content. This is one of the core principles of content management systems in general, and Dupal specifically - a separation between content, code, and layout.
Search functions are part of the code again, not the site design.
I don't think they said anything about MOVING the site, just upgrading it. Testing data conversion routines should be part of code development, not site upgrades. A site upgrade should not happen until the code has been verified to work as it should.
Also, please note the part of the post that says:
"One group will implement the new theme designed by Mark Boulton and Leisa Reichelt"
A whole group of people working a week to implement a theme that's already been developed. ('Designed' - past tense - already done.)
Now, I will admit I missed the part where they are redesigning the CONTENT -- meaning they are rewriting many of the pages on the site, which is something that can take more time than anything else. That I can completely understand. Even so, how many people does it take to implement a theme that's already been developed?
To João -- my site is still running 5.15 because some contributed modules I used for that site were never updated for D6, so that site is stuck. I understand that my choices at this point are to either redesign the content so those modules are no longer required, or to hire a programmer to update the required modules. For now, I've chosen to do neither since the site 'works' well enough.
- Alan Tutt
http://www.AlanTutt.com
- Alan Tutt
Exceptional Personal Development for Exceptional People
http://www.PowerKeysPub.com
hire the programmer
The key to understanding what we do on these sprints comes from your last paragraph. The donations "hired programmers", sysadmins and performance experts to update modules which nobody else updated (because (almost) nobody else uses them). We have important modules which are crucial to drupal.org but would otherwise not get updated, since the rest of the community is not using them. The first sprint in Boston was about (1) updating modules not yet updated to Drupal 6, so the site can run (2) instead of updating our existing search solution, we implemented a complete new one based on ApacheSolr, (3) instead of hackish and per-site custom ways to update sites with modules, we implemented a new system to manage modules and deployment on our systems, and also (4) migrated away from some modules to others, implementing content migration paths and making sure they work well.
So there was a lot to do on the upgrade in itself, and it is not done yet, since we have *dozens* of custom modules nobody else is using to do custom things on the site, which all need to be verified, so we called for the community to help test the staging site upgrade. Big sites like ours usually cannot afford updating the live site and see as trivial things break. We will most probably see things break once we do the live upgrade, but hopefully by that time all the "trivial" issues will be worked out.
With the theme implementation, we have images and HTML/CSS files, but the theme needs to be time proof and be usable on possibly ten sites on drupal.org. We already have a huge number of subsites, and we have plans to have even more, which should be able to use the same theme. We have all kinds of neat things like cross-site navigation with the active tabs properly highlighted, and so on. Beyond that, we have a huge amount of features to develop. If you are interested in that, feel free to follow http://drupal.org/project/issues-term/799 and contribute.
For updating stuff which was note there, setting up new infrastructure, implementing new features, a good theme which can work with different sites with different modules enabled, we needed to "hire the themer" and "hire the programmer". That is what these dollars pay for.
oh, and just to make it clear
Oh, and just to make it clear, these programmers, sysadmins, performance experts, themers, etc. are not actually paid to be here. The money goes for the cost of the flights, hotel rooms (which people share in twin rooms and with renting a flat when it costs less to keep the expenses low) and some of the food we consume. On the contrary, people here give up working for their clients/employers for one week. So the above "hire" concept refers to "get them together at the same room and feed them so they can do the work" not at all "pay them money". The lost income from the clients and the nulled productivity for the employers is not payed at all, these are part of the personal donations of people here and donations from the employers, which also help make this happen.
Thanks for the clarification
Thanks for the clarification, Gábor. You've explained that the 'upgrade' is about more than simply updating the layout, but is also part of coding custom modules as well, which is different from how I read the original post.
However, it seems to me that upgrading the modules would happen in the same manner as upgrading the core Drupal code itself, and would be done before attempting a site upgrade. Otherwise, there is a risk that critical functionality cannot be implemented properly, and then time invested in the rest of the process would be wasted. Perhaps someone figured out that getting a group together was a more efficient way of doing this. If this is so, then maybe the core Drupal development should happen the same way?
I also appreciate your comment regarding the "time testing" phase of implementing the new theme. Yes, on a larger site, there is a greater possibility of more things going wrong, especially if there are a large number of customized pages which depended on a previous layout to work properly. I guess I had this idea that those who built the Drupal.org site were beyond such mistakes and implementing a new theme would be as easy as it is on a smaller site.
And yes, I do understand that a major upgrade needs to happen in a staging area and not on the live site. I do this all the time with my shopping cart site, although I've always felt Drupal upgrades were safer.
@Cool_Goose - nothing is EVER free. All software has a cost in terms of time involved to learn how to use it, if nothing else. When things change, making simple upgrades difficult, the cost goes up proportionately. My experience with Drupal is that too much changes from one version to the next (making upgrades difficult), and while I love the core concepts and feel it's the best CMS to use, I still get frustrated from time to time.
And to anyone else who may be wondering -- I am not against anyone asking for donations to make a project happen faster. I think the folks who have worked long and hard producing this system deserve to be taken care of and well rewarded for their efforts. My initial question was more towards accountability than anything else, so we may know the reasons behind the donation request.
- Alan Tutt
http://www.AlanTutt.com
- Alan Tutt
Exceptional Personal Development for Exceptional People
http://www.PowerKeysPub.com
staging, core development, accountability
Good comments! Let me help answer some of these.
Well, we did break down our tasks to two chunks: "the upgrade" and "the redesign". Both the upgrade and the redesign includes new module development, setting up processes which were not there, documenting decisions, etc. We staged the work, so upgraded the site and announced it on the front page (http://d6.drupal.org) so that people can test our upgrade work before we even started on doing any of the redesign. This week we got back some of those issues as well, and did another upgrade on this test site yesterday to make it work with all the latest fixes. Check out instructions in our front page announcement on how to test it and provide bug reports: http://drupal.org/node/366562 Also, you are welcome to compare the project listing, faceted search and issue search features to what we have on the live site.
It is more and more done this way as well. There was a fields in core code sprint recently (also had multiple front page posts), a media code sprint, user testing sprints, etc. There are also virtual code sprints. There is an ongoing i18n virtual code sprint (see announcement on the development list).
You can watch our progress on the #drupal-infrastructure IRC channel live (see http://drupal.org/irc for more information) or the Twitter feed of Joeri: http://twitter.com/jpoesen for a more filtered reporting. You can look back at what we did by reviewing our contributions to the various modules we use: project, project_issue, project_release, project_usage, comment_upload, comment_alter_taxonomy, image, etc. These are all open source modules, so you can look at their issue queues, their commit history, etc. for what we did over the Cambridge sprint. We have two major tags we use for tagging issues for the upgrade and the redesign separately. Check out the progress and work we did by viewing the upgrade issues at http://drupal.org/project/issues-term/346 and the redesign issues at http://drupal.org/project/issues-term/799 The themers did not open any issues, so you can only see the new feature issues here; for the theming stuff, you can review and participate in the groups.drupal.org group: http://groups.drupal.org/drupalorg-redesign-theme-implementers
I hope this helps to fill your information needs.
Let's be Constructive
I wish I could constructively reply to this comment but I do not have the time. Just be aware that you have probably offended most of this community. Drupal.org is a tremendous site and there is no solution out there that would offer a path that you are talking about, with Drupal or without. Please gather more information before making comments like this.
--
zzolo
Chicago Technology Cooperative
AlanPalazzolo.com
--
zzolo
About the ease of upgrading
@Alan Tutt: for a simple site (like yours), it is indeed a simple case of picking up the newer version and just running update.php. The upgrade path from core Drupal 5.15 to Drupal 6.9 includes converting all necessary data to the new version.
However, Drupal.org is definitively not a simple site. The site handles the issues and releases for Drupal and the contrib modules, and that uses a module family (project and company) that doesn't see much usage outside Drupal.org (see here). And that's just one of the functionalities, there's major working ongoing with a complete redesign of the look and feel of the site (a new theme, exclusive to Drupal.org has to be developed), and adapting the site to the new versions of the Views and CCK modules, etc, etc.
@Cool_Goose: the guy was simply asking a question, which you didn't answer: are "upgrades (...) a simple matter of uploading the new files and letting the software upgrade the database"? He just uses it, and he's not familiar with the innards of Drupal.org. So, please, next time ease off on the "you really have no idea about what you're talking are you", and try to be more polite.
@zzolo: Who offended the community? One guy is perplexed that it takes >$15000 to do something that he will want to do soon (the guy is running 5.15 on his personal blog), and the next one comes and calls him clueless. I am still confused who's the insulter and who's the insulted here.
João Ventura
Venturas.org
João Ventura
Venturas.org
My apologies jcnventura. I
My apologies jcnventura. I get pretty annoyed with people complaining about free software. I'll be more polite in the future ;)
------------------------------------------------------
Think Smart, Be Free, Choose Open Source.
great!
best wishes with you all and I am sure new design/upgrade (as seen on Mark's site) will bring in lot of improvements/new features to D.O. community and will help newbies find their way on using drupal.
good luck and looking forward to it.
Looking forward to the new design
Best of luck with this mammoth endeavour. I love the new designs by Mark Boulton, and once it's all up and running, I'll be pretty keen to point more people to the site - especially my senior management.
More questions...
I too was wondering what was so hard about the upgrade. I had no idea that drupal.org uses some of the more obscure modules! But of course, after reading the above, it makes sense.
@Alan.T Thanks for asking your question. You're not the only one who was wondering.
@Gabor Thanks for your answer. When you get a chance, I would really like to know what exactly had to happen for the upgrade. Eg. what does drupal run on now, which modules, which hackish solutions etc. and how they are integrated. Also would really like to know what modules are used in the new drupal, and how. I'm really glad to hear that the search is being upgraded.
Also, for Newbies like myself and Alan, I am sure we would love to have it explained what custom coding was needed for upgrading. I can understand that changing hard coded links in articles is not to easy, and a script or two for that sort of thing would be needed. What else?
And I cant wait for it to be up and running! Drupal Rocks!
Well we're all learning something here...
I think Drupal rocks, I think the community rocks.
@AlanT. Great question, and from the answers we got it shows that the existing Drupal.Org site has been 'stretched' to accommodate lots of different needs. This was obviously done through necessity over time. However the important decision has now been made to upgrade the site and address these needs in a more efficient way. Regular Drupal sites that use maintained modules will not have this problem.
Well done for asking...I certainly learned from the answers.
@Cool_Goose, @zzolo. You're both probably right...most people (me included) don't have a clue about the amount of work that has occurred over time to develop the drupal.org site...but no-one in the community should get offended by someone asking a simple question through lack of knowledge. Its the only way we can learn.
@Gabor. Thanks for spending time answering this question whilst you're also in the middle of this massive upgrade!
I think the ability to upgrade core drupal sites when they use maintained modules is great. I also reckon the move to Apache Solr will be massive. Searching functionality will be much better.
Way to go guys!
Thank you Drupal
I hope the usability will improve on Drupal.org . I'm not sure how many modules they need to convert from the old D5 to the new D6. However, my best wishes to them on their effort.
The Apache Solr search engine is not available
Please contact your site administrator :-)
being reindexed
It is being reindexed from the ground up, since there were lots of changes to the module and how it indexes stuff.
Search service is down
But while the index job runs we should also be able to search... maybe we get no results, ok - but a message that the service is down sounds wrong!?
It was actually a
It was actually a configuration error. It should be fixed now.
Not yet fixed... for users logged out
The error message is not yet gone... (I'm logged out). Ahhh - logged in - seems working!?
I really wished I could get and idea how well this ApacheSolr works and if it's really faster than the old search engine. Aside if one of you would be able to write up an article about the decision process and performance testing results and why ApacheSolr was the final decision - this would be very great. I'm very interested to use this software on other projects if it's much faster and could help me, but it was the first time I have heard about it when I read one of the upgrade print articles. If there are others search engines it would be great to hear why ApacheSolr has become the first choice.
Now it works with logged
Now it works with logged out... something must be wrong... sometimes it works and sometimes not.
I removed all my cookies and now it's broken again. Not sure if this is related.
Google Analytics site search?
Is one of you able to configure the site search feature in Google Analytics module for d6.drupal.org? I would be happy to see if it works without any modifications with ApacheSolr or if I need to work on it. I saw site search is also not yet used in d.o. The most used search keywords maybe of interest...
My Recent Posts
My Recent Posts should be added. That way one can trace his/her recent posts easily. Should actually be there in dp6. Its a nice feature
--
www.whereis.co.ke
Drupal.org/D6 and MySQL database modifications
Is the new Drupal.org upgrade to D6 using a MySQL database split read/write infrastructure as well? If so, have the patches/modules been contributed or is it something of a customized hack job?
-- Michael
I recently joined drupal
I recently joined drupal when i heard about it's serach engine optimization enable than wordpress.