By Chris Johnson on
Executive Summary
I propose to replace my employer's extranet website, which currently uses all custom-coding to generate dynamic web pages from database queries and trading partner electronic transactions, with one newly built upon Drupal.
Is this a good idea?
Why Drupal developers might care to help me answer this question
If I undertake to convert our current web applications to Drupal, I believe there are some benefits to the Drupal community which may accrue. They fall primarily in three categories:
- My employer will pay for me to spend a significant amount of my time working directly on Drupal or Drupal-related code. I expect this may be a thousand hours over the next year minimum. This means I will be paid to work on Drupal issues which affect my employer, which would include bugs or limitations in core or module code. Fixing bugs is good for the Drupal community.
- My employer will need to develop a large amount of custom code to run its applications. Some of that will be very generic code which will be donated back to the Drupal community if they so desire it. For example, almost all of our applications involve date input for elements. I will spend the time necessary to perfecting a variety of form elements such as date which others in Drupal may wish to use. (It is also true that much of the code I develop with be of no interest or use to anyone else, and/or proprietary.)
- My employer currently employees several contract developers full time. I see a big opportunity to hire Drupal developers on contract to help us out with this conversion project, and with developing new applications in the future (we are constantly developing new software to meet our changing and growing business oppotunities).
Technical Background of Current System
The system of web applications I propose to convert to using Drupal as a framework has the following characteristics:
- PHPLIB is used as a framework for session, user data, authetication and authorization.
- User data is gathered via custom-written forms.
- Most web pages or web applications center around entering search criteria to locate records in a variety of different databases, or to send transactions to trading clients to search their databases.
- The code for these applications has been developer over the past 6 years using PHP and MySQL.
- Very little of the code has been rewritten or replaced during its lifespan. As a result, most of the presentation is in non-CSS, non-semantic HTML 3, although new bits use XHTML 4 and CSS 2.
- Performance and reliability are very important. There are online users 24 by 7, and they are donig transactions involving research or tracking of millions of dollars per month in invoices.
Company Reasons for This Project
- After 6 years of new pieces and applications being added on to the central system, the accretion is rather a mess of different code. It needs to be cleaned up, unified, modularized and make use of reuse!
- There is a desire to have consistent look and feel across all web applications. Drupal provides this out of the box.
- Using a framework with a lot of built-in support for common tasks, such as presentation (theme), form handling (input and feedback messaging), database queries, etc. allows the application developers to focus more on the business and to get more work done in a shorter amount of time.
- Ability to leverage off new Drupal development.
Some Issues to Consider
- The nature of these web applications might mean the node module will not even be needed, and would be turned off in the Admin->Modules page. What problems, if any, will this create?
- Likewise, most of the blocks will never be needed. Many of the current applications are very tight on screen real estate to display the large amount of database data used by the users. Blocks can be turned off and a theme can be developed with economical navigation links which preserve most of the screen real estate for data. Are there performance concerns or limitations that might result from turning all blocks off?
- Many of the database searches take significant amounts of time because they are searching large tables with complicated SQL. Will the much more complex page generation of Drupal versus the simple, straight-forward HTML currently used (outside of the database values returned) be a significant performance cost?
- Other issues?
My Questions
- I think this is technically possible, and because I like Drupal, I would enjoy working on it. But is it a smart thing to do from a reliable application point of view?
- Is this a smart thing to do from a business point of view?
- What issues not mentioned here do you see that may affect this decision?
- Anybody interested in getting paid to work on this? :-)
Comments
Some points
I cannot answer this on top of my head. If the node module can be turned off in the admin menus, then Drupal is ready to live without it.
A good way to do so is to test it and see what it does.
I think turning off the blocks is pretty safe. The only one that your users will need is the login block, and that can be turned off and accessed via the /user URL. I do so on two of my sites, and it works fine.
Has to be really measured to be quantified. Since this is a big application, perhaps a web/app server farm architecture is more appropriate, with the database on a separate machine. Perhaps you are already doing this with your current app.
Maybe you could partition the database into two databases: one for Drupal authentication and content, and one for the data you are storing for the app. This means more complexity as well (two connections, ...etc.)
Using an open framework for your application is a smart choice. Drupal was initially about community building on the web. Some creative guy extended Drupal to be able to do e-commerce.
I have always seen Drupal as an application framework that happens to have some base and community modules out of the box.
You seem to have seen the light too, and see it this way.
Why not? You say maintaining a six year old application is a pain, and I feel it.
There are considerations to keep in mind. For example, do not try to modify Drupal beyond recognition. Since upgrades will be a pain, and you will end up with an old version of the application.
Try to isolate your modules that you will develop, and keep track of them, so the next release of drupal can be integated as smoothly as possible.
Most importantly is to have a check list of the features you need for you application, and a check mark on whether it is available for Drupal or not. If not, then will you develop it or what? This way you have an idea of what you will be missing, and what actions you will take to address them (write it, drop the feature, ....etc.)
--
Consulting: 2bits.com
Personal: Baheyeldin.com
--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba
Thanks and more
My current application is distributed over 16 hosts, and when we need more hardware, we buy it. I was more concerned with the 20 to 40 extra database accesses that are required to generate a Drupal page compared to the zero (both excluding getting the data my users are interested in) right now, as well as the extra hundreds(?) of lines of PHP code that get included and some executed.
I think using Drupal for my applications is a very cool and exciting idea. But since this is a multi-million dollar business and my paycheck I'm talking about here, I don't want to screw it up completely! :-)
More reason to do a prototype
I guessed it is already distributed, since it is on a large scale already. However, you only have one database server, or perhaps a replicated database.
You can perhaps run two different databases on two different boxes, one for the Drupal stuff, and one for your app stuff. This way, you know that the current load on the app database will not increase with Drupal, and the extra access will be on a new database box.
This of course adds complexity, as I said.
If you can get management to buy into the idea of making a mini prototype, where you can write a quick proof of concept thing. It should not be too hard. I find that this is sometimes the best way for managers to have confidence in new projects, because they see the potential first hand. They can even ask you to do the pilot for a key customer (where applicable), and if they like it, they give the green light (and the green back too!)
Just an idea to consider.
--
Consulting: 2bits.com
Personal: Baheyeldin.com
--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba
Very interesting
This sounds like a very interesting thing.
Node is the very heart of Drupal. It can run without it, but in fact no-one will ever want it.
Far too many people (including lots of Drupal developers!) think oif nodes as content. But they are not. A node is simply a point where data gets together, to become a "thing". Anything.
So I can only advice you to look at your ideas, as "can i make this data group a node?". Do a search for "flexinode" and CCK.
Most probably not. Drupal offers you very powerfull and flexible cahcing APIs, on top of its default caching. Unless you have these already built in in the current application framework, you will most probably increase performance a lot.
An often heard joke is: "can drupal ...", "off course, there actually is a module for that already". But actually this can be seen serious. Drupal has not yet found any limits. So yes, AFAIK, drupal is by far the best choise for such a project. Because of its flexibility.
A few. Take into consideration that hacking Drupals core increases your time solving update issues a lot. If you build on top of Drupals flexible core, instead of inside it, you will gain a lot from all the improvements made inbetween each relaese, for free.
Also note that Drupal can very well use some (good) developers. So not only will your hours be greatly appreciated, any outsiders you hire and educate to use Drupal, will help Drupal in future.
And yes, I am very much interested in a contract. I am listed amoungst others whom are probably as much interested: Drupal services
[Ber | Drupal Services webschuur.com]
No Intention of Changing Core / Miscellaneous
I just wanted to point out that I have no intention of changing core to do this project. So it is important to know if it can be done without such changes. I currently believe that is true.
All code for my applications would be in module or theme files.
Regarding nodes: yes, if I can find a way to use them, I most certainly would. Currently I think that perhaps flexinode would be the most useful way to go for something like a trading partner table, for example.
Regarding caching: all users of these web applications are authenticated and connect via SSL. Data privacy is very important. My plan was to use Drupal's user authentication for this purpose, so the caching which is mostly done for the anonymous user will be of no benefit in my situation. Perhaps I am not up to speed on the latest in caching in Drupal. Is it the case that data for authenticated users is now cached to a significant degree? (I think I know that part of the menu system is cached for them, but am not aware of anything else. Could very well be my ignorance!)
When it comes to cache, you
When it comes to cache, you have the oppertunity to cache yourself with a few lines of code where you find it appropriate, specifying time to expiration or custom conditions conveniently where you keep the code producing what you want to cache. You can easily cache the complete output of the page in a database.
The core currently requires all modules to be loaded and some processing when a logged in user requests a page, but this shouldn't take long, especially if you have some php optimizer installed on your server.
Note to responders
If you have any questions about what I want to do, please ask. I tried to limit by original posting to a reasonable size and important information, but of course, had to leave out many details.
Thanks to those who have responded so far! It's greatly appreciated.
I'm in a similar situation
My employer is paying for a project, the foundation of which is the new Node Relativity module. The requirements of the project brought about the need for such a tool (as well as the Edit as New module). I am infecting my office with Drupal even more as a result, and future projects are lining up with other developers using Drupal to develop them.
I'm now mostly done with the relativity module and have started on the proprietary business-specific module that sits on top of it. The relativity module represents about 85% of this project's total time budget, but without it, the project would have taken even longer.
I have used PHPLIB for a
I have used PHPLIB for a long time. If you plan on going from that to Drupal then you are going to need some serious hacks to the core to get the same flexibility and usefulness that PHPLIB provides. You should create a list of all the functions that are used by the present system and do a side by side comparison on how they are implemented by Drupal.
The one thing I think you will not like is the way that Drupal deals with SESSIONS. With PHPLIB this is very easy and adjustable, requires no help from the webserver and can be tweaked to do odd things like cross site SESSION without a lot of extras.
Then there is the biggy. .htaccess. You will need to study this deeply. Going form a custom application to one that is totallly dependant on webserver recognition and control of URLs is going to be work.
Module interdependacy will be a factor in working with a system like Drupal. Simple stripping away things is not possible without changing the way the core is mapped. You may find yourself in need of a way to deliver modules differently from the present system. I have just made one slight change to a module to get some flexibility in security that I needed for my site. In the end I was forced to change 4 modules. Because of the way they interact there was no simple way of making the adjustment.
I am saying this because you should realize that in the end you may be creating a framework of Drupal not using it as one.
---------------------------
www.hivemindz.com (running PHP5)
www.fireorb.org (documentation and hacks)
__________________________
Carl McDade
Information Technology Consult
Team Macromedia
Ohh, I forgot to mention a
Ohh, I forgot to mention a weakness that may prove to be a deal breaker when dealing with a largeand wildly spread system base. Drupal has no real database abstraction layer. This is a must in my opinion for any large system usage. The absolutly first thing I would do before going forward into any other project is to lay in a full database abstraction layer like ADOdb or similar.
---------------------------
www.hivemindz.com (running PHP5)
www.fireorb.org (documentation and hacks)
__________________________
Carl McDade
Information Technology Consult
Team Macromedia
Dear Carl,
I have seen you doing this before on the Drupal Forums: Give out false or imncomplete support. I am not saying you are doing that here, but please bear in mind that saying things like
Is on the edge.
Why do you think Drupals DB abstraction does not suffice?
Why would implementing a new my-pro-db.inc not work?
Why would you want a db abstraction layer in the first pla ce?
Is that correlation true?
Please, think about the impact any of your comments can have. Fot I get the feeling your negative look at Drupal infects your objectiveness.
If you continue to *talk* about things you do not like but never *do* something about them, no-one will benefit. You not, because you become frustrated, Drupal not, because people like Chris Johnson get false or at least coloured information, and the community not, because they see a lot of negative or false comments everywhere.
So, Carl, please reconsider your ideas before posting them, or at least consider discussing them somewhere else then in replies to support requests!
Bèr
[Ber | Drupal Services webschuur.com]
Well , we can take a look at
Well , we can take a look at my case and learn from it.
My case scenario is this:
An earlier client has fallen in love with the use of xml-rpc to filter and deliver information to a website. He presently has two websites using a scrape along with the companies own database to deliver a search engine to visitors and clients. No decision has been made but it has been thought to use a CMS system as a frontend rather than redesign and rebuild all the sites, many of which are done in Microsoft Frontpage for use as a CMS system. This way the search engine can be integrated as a module. The client is hot on using Mambo , I have suggested Drupal and the lead developer has expressed a wanting to use ASP.NET and go from scratch.
He now wants to sell this system as a service and share database information in a network of sites via a search. The network is among companies in the same industry but presently do not share information. The largest forseeable problem that he can see is that during a series of meetings the pontentials have stated that they have different databases. Many use Microsoft Access , some have Microsoft SQL Server, one has Oracle 8 and one has embed MySQL (desktop software) and exports web information to XML.
As you can see this all can get very complicated quickly without a full database abstraction layer and possibly database persistance layer. To avoid further miss communication I will refer to this as "database independance". Right now both Mambo and Drupal are not viable solutions but both have pluses on their side but neither has database independance.
The project time will not include the time and money that it takes to develop the software (full and extendable db abstraction) needed for them to work. As this is an adhoc development group and not a house the there is a need for "self supporting" software. Open source where counts and commercial where needed.
Drupal does have an inplace system for xml-rpc that can be modified to suit the purpose of the project. Multiple sight login is also on the plus side. Being able to post and edit information via w.blogger, replacing Frontpage. I worked on this because I had written w.blogger in my project outline. A real plus came when w.blogger could be used on all node types. Alot of big pluses for using Drupal. But the database problem stands and breaks everything.
Mambo is easier for the end user to work with and they have indicated that they will be slowly moving to ADODB. But the inter-site communcation does not seem to be there.
Changing either to implement ADODB is not going to happen because this would take changes to the core system and more time and resources than we have. So it's a waiting game to see if it happens through regular development
We have looked at two other possibilities which would solve the immediate database problem and allow for extended development by selling the solution. Looking at the time it takes the first choice is to build around something like Xaraya, postnuke or phpnuke and add in Drupal functions looks good. The second choice is to wait for Drupal to start using ADODB or something better to get database independance.
Suggestions for other choices are appreciated.
---------------------------
www.hivemindz.com (running PHP5)
www.fireorb.org (documentation and hacks)
__________________________
Carl McDade
Information Technology Consult
Team Macromedia
The point is
Carl,
I think Ber's point, which I do support, is: "at least consider discussing them somewhere else then in replies to support requests!"
Rather than confuse newbies or start a discussion in a discussion in a discussion, let us stick with the topic at hand, and answer it.
If you have an enhancement request, an idea, a complaint, a comment, then each of these have their proper place in the forums and/or the mailing lists.
It is easier on all of us if things are in the correct place.
--
Consulting: 2bits.com
Personal: Baheyeldin.com
--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba
I was just going by:
I was just going by:
I know that many companies db systems and are in a constant state of adding to or changing them. This makes things difficult when wanting to upgrade or create new software to accomodate a database system in flux. On the otherside there are those that stand rock solid with their choice of db and that choice is not always MySQL or PostGreSQL . I see that Drupal has dropped MS SQL Server support also.
In answer to the topic: I don't believe that at present Drupal makes a good choice for a framework because of it's database dependancy.
In answer to the topic as a support question: add database independancy or wait for it to happen.
Of course you never gave any db info so my point is dead anyway.
I also thought this was a open discussion because it was not in the " support" forum. I am increasingly careful about looking for the forum section where I post.
---------------------------
www.hivemindz.com (running PHP5)
www.fireorb.org (documentation and hacks)
__________________________
Carl McDade
Information Technology Consult
Team Macromedia
I think they just want to avoid topic drift
True, this posting was in a general forum, because it is not a specific support request for help with a Drupal install or with a specific Drupal feature. But I think Bèr and kbahey were just asking that you avoid topic drift. That is, keep your reply postings very tightly on the topic of the original message.
If you believe a discussion of the need for database independence is a good idea, then by all mean, start a topic on it.
At least, that is my understanding of what they are saying.
As for me, I am the senior IT person and an IT manager at my company, and I have chosen MySQL and Postgres as our database platforms. I have used Sybase and Oracle extensively in the past, and I cannot see our needing to use either of those in the future with their high licensing costs and large resource requirements.
Also, given my extensive database and software development experience, I can honestly say that "database independence" is a wonderful goal, but nearly impossible to achieve because each database vendor implements slightly different sets of SQL, despite standards.
Database support is not a large concern for my project. Instead, workability -- that is, making the web application development job ultimately easier rather than harder once over the conversion "hump" and once up the learning curve for Drupal -- is important, and performance is important.
Another consideration for me is the rate of change in Drupal APIs. If each release breaks everything such that I have to recode a lot of web application software to take advantage of newer Drupal releases, then the benefit of Drupal is somewhat lessened. I can always stay with an older release, but then I miss out on patches and new features, both of which might be useful to my business.
So, please continue to post your thoughts. Maybe just try to narrow the scope of the topics discussed in them.
The code is going depend on
The code is going depend on who you have working for you I guess. If it is a "Drupal company" then you are set. If it is an in house group then you may find the gap between learning the Drupal methods and the changes made between versions a bit frightening. I had not thought of this because I don't believe in using the "next best" thing when a large group is involved in the development.
If you have your own group then maybe mimicking Civicspace might work for you. They are the only group that I can see that is using Drupal as framework. You can probably learn a lot from whatching how they handle the API changes.
---------------------------
www.hivemindz.com (running PHP5)
www.fireorb.org (documentation and hacks)
__________________________
Carl McDade
Information Technology Consult
Team Macromedia
The rate of change in the
The rate of change in the APIs is indeed not small (I consider this a good thing). However if you look at the instructions for module authors you will find,that theactual work needed isn't that big.
--
If you have troubles with a particular contrib project, please consider filing a support request. Thanks. And, by the way, Drupal 4.5 does not work with PHP 5.
--
Drupal services
My Drupal services
Follow-up -- initial prototype
I had a long, interesting conversation with Robert Douglass about this, for which I thank him for his generosity with his time and knowledge. I came away with a positive attitude about making this project work.
To that end, I spent a few hours the other night hacking up a prototype with just one of our simple sub-applications based on Drupal. It actually turned out to be quite easy and is completely functional.
This gives me even more support for it being a good idea, so I will be working on a full-blown prototype (with 3 or 4 sub-applications) to demonstrate to my bosses. If they like it, I'll be moving the whole universe of web applications to using Drupal as a framework.
Thanks again for all the feedback and advice to those who contributed it.
Which db tables are essential?
I thought of using drupal as a framework for a web application too. In the basic install Drupal consists of ca. 50 db tables. Which of these tables are essential and which can be safely deleted?
Drupal 4.6 comes with
Drupal 4.6 comes with aggregation, poll, book, forum, locale and a few other things you might not need, and whose tables you may delete along with the modules who depend on them (the admin -> modules page might be a good place to sort these things out).
However, most of the tables in the default installation are needed by the user, node, category (taxonomy) systems, access, search, session and comment systems of Drupal. I consider implementation of these or similar concepts to be a vital part of any framework and therefore believe you should keep these.