Distribution configuration through macros. [#79900]

Recently Chx and Eaton have managed to make all forms in Drupal 'pull'-able, meaning that you no longer need to load up the
page the form appears on , to execute or inspect it.

This has allowed me to write the following patch, which uses a small form_alter, which listens to each form submission, and stores
it in the variable table. Using a simple callback, you can then export all the submissions that have been made, and the install system
will then play them back, when you install a new site.

I made this a patch for now, since making it an additional module was a bit tedious, as you would have to make sure the install profile has it enabled.
Also, keep in mind that this is only a proof of concept so far.

Attached is the patch. All form submissions you make will be visible via the patch 'macro/export'. At any time you can take the output of that page, and save it in profiles/$profilename/macros.inc , and when you install a new site with that install profile, all the configurations you made will be automatically played back, including first user creation etc.

I should also mention that some forms just don't work at the moment, and they are actually not programmatically submittable at all.
The biggest one i ran into was creating new content types. I think moshe mentioned the block admin screen too.

Eaton is working on some improvements that will clean up the code a lot, and I would also like to introduce a couple of nice things ..

Comment	File	Size	Author
#24	patch_116	9.16 KB	moshe weitzman
#23	patch_115	8.81 KB	moshe weitzman
#22	macro_module.tar.bz2	4 KB	adrian
#21	macro_replay.patch	1.34 KB	adrian
#21
#15	drupal_macro_engine_0.patch	8.81 KB	nickl
#15
#14	testmacro_profile.tar.gz	1.94 KB	nickl
#13	macro_profile.tgz	1.94 KB	nickl
#12	drupal_macro_engine.patch	1.22 KB	nickl
#12
#11	macro_module.tgz	1.77 KB	nickl
#1	macros.inc.txt	42.73 KB	adrian
	drupal_macro_engine.diff	3.71 KB	adrian

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

adrian CreditAttribution: adrian commented 21 August 2006 at 23:01

File	Size
macros.inc.txt	42.73 KB

here's a quick demo macros.inc i whipped up in very literally a couple of minutes.

The username / password is 'admin/password' , but weirdly enough you just get automagically logged in =)

These files will also become much smaller, since you only need a) the form id , b) the params for the form id, and c) the post request.
Addtionally, these files should be able to be unit testable in and of themselves, since each action can be validated.

Comment #2

m3avrck CreditAttribution: m3avrck commented 21 August 2006 at 23:45

Ooooh that's very cool! Subscribing now...

Comment #3

Chris Johnson CreditAttribution: Chris Johnson commented 22 August 2006 at 00:17

Seems like this would be even more valuable for testing than it would be for creating install profiles.

Comment #4

adrian CreditAttribution: adrian commented 22 August 2006 at 00:36

No, it's just inherently testable =)

Comment #5

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 22 August 2006 at 03:41

sweet. :)

Comment #6

Dave Cohen CreditAttribution: Dave Cohen commented 22 August 2006 at 06:00

This seems like a great idea. How about a module that does record and playback independent of the install system? So I can start recording anytime, and playback into a site that's already installed. So for instance I could have a recording that configures module X, another to configure module Y, and I could choose either or both for my system.

Comment #7

Arto CreditAttribution: Arto commented 22 August 2006 at 09:20

Great concept, and simple enough implementation. Any chance this might make it into 4.8?

Comment #8

adrian CreditAttribution: adrian commented 22 August 2006 at 09:58

at the very least i would like to see the macro replay functionality make it in.
Thanks to this function that eaton is writing (http://www.pastecode.com/3525) , the code to call forms programmatically will be a lot simpler.

I originally had the export / recording functionality as a different module, but it's a bit more tedious to test, as you have to add it to the
install profile, and have the module running from the very moment you start up the site.

Apart from the happy fun security issue of it saving plaintext passwords, I think there could actually be a lot of use in this as an implementation of the command pattern, which allows such interesting things such as undo for all actions etc. http://en.wikipedia.org/wiki/Command_pattern

So i would like to see a more fully developed system exploiting macros / workflows in the future, which is part of what fapi3 is going to be about.

Oh. btw. I don't think file uploads can work programmatically yet =( This would have been an awesome way to implement mass file uploading.

Comment #9

adrian CreditAttribution: adrian commented 22 August 2006 at 10:09

Oh , and regarding configurations and the like.. we now have an interesting situation... in groups or on the forums, people can now
send their configuration of modules around.

Additionally, this is a very useful way for distributing things such as cck node types. Essentially virtual packages that contain nothing
but configuration, and a requirement to have certain other packages installed.

I think we should have all modules have a macros.inc file, but i am scared as to how the modules are going to be about changing each other's settings (irk). Also, upgrades to configuration would need to be versioned as well... so i think we should hold off until we have a full idea of what's
required, and how individual modules might mess with each other.

Comment #10

merlinofchaos CreditAttribution: merlinofchaos commented 22 August 2006 at 15:42

Very interesting.

Comment #11

nickl CreditAttribution: nickl commented 24 August 2006 at 00:29

File	Size
macro_module.tgz	1.77 KB

This is going to be tedious until we can find a better way of handling this queue...
Can't create a patch against head because we need a new directory modules/macro... yes you guessed it the macro is now a module in its own right.
Added settings to enable recording and/or delete the current macro.
Export to display the coded macro in views_ui export fashion.
Already has a meta descriptor and install file which makes it compliant with http://drupal.org/node/76340

Comment #12

nickl CreditAttribution: nickl commented 24 August 2006 at 00:32

File	Size
drupal_macro_engine.patch	1.22 KB

install.inc patch
Executing profile/distro recorded macros, as defined in macros.inc, on installation.
Fixed a bug and user information is now outputed to the screen.
Need to fix the layouts a bit but the patch is fully functional.

Comment #13

nickl CreditAttribution: nickl commented 24 August 2006 at 00:37

File	Size
macro_profile.tgz	1.94 KB

Profile information and macros.inc example profile.
The profile now includes the macro module for installation which will start recording immediately after installation.
The included macros.inc creates the first user account, changes the site_name and site_frontpage.

Comment #14

nickl CreditAttribution: nickl commented 24 August 2006 at 03:46

File	Size
testmacro_profile.tar.gz	1.94 KB

New installation profile to enable the macro module and for testing the on install macro playback.
Changed the name of the macro file from macro.inc to profile_name.macro in line with convention.

Comment #15

nickl CreditAttribution: nickl commented 24 August 2006 at 03:52

File	Size
drupal_macro_engine_0.patch	8.81 KB

Single patch file containing the macro module files and install.inc changes.
Added import functionality.

Please feel free to add your comments and suggestions. There has been a lot of interest in this patch please take some time to review it and suggest improvements.

Subsequent patches can be created against: https://svn.bryght.com/dev/svn/macro_engine/ which caters for the new directory modules/macro.

Comment #16

nickl CreditAttribution: nickl commented 24 August 2006 at 05:02

Status:

Needs work

» Needs review

Status change... please review.

Comment #17

redraven CreditAttribution: redraven commented 31 August 2006 at 12:34

Been playing around with this patch, very, very useful :)

One of the admin tasks that I wish I could automate on any drupal site I have every created is role creation and access control.

This is the first one I played around with ;)

It does *not* seem to want to record role submission, I guess this is one of the forms previously mentioned to not work yet.

However, I successfully export/imported the form submissions related to access control.

Played around with some other easily automated tasks, and everything worked fine.

Basic block administration seems to import/export fine.

Thanks for the contribution, its going to really push the distribution effort forward!

Comment #18

moshe weitzman CreditAttribution: moshe weitzman commented 31 August 2006 at 13:41

just a note that this fine patch adds a .meta file whioch is not yet in core. also, this patch will have to change i think if programmtic execution gets simplified with drupal_execute(). that patch is RTBC.

Comment #19

adrian CreditAttribution: adrian commented 31 August 2006 at 14:25

Yeah, i'm just hoping we can get the small bit of code needed to get these macros automatically playing, into core before the code freeze.

kind of relies on http://drupal.org/node/80470 though (as serialising the whole form is sinfully ugly).

Comment #20

eaton CreditAttribution: eaton commented 31 August 2006 at 14:46

Drumm has given his blessing on that issue, and we're waiting for Dries' word. Feel free to pop over and express your approval for that patch. ;)

Comment #21

adrian CreditAttribution: adrian commented 31 August 2006 at 20:47

File	Size
macro_replay.patch	1.34 KB

here is a re-roll of just the little bit of code in install.inc which replays the macros.

Comment #22

adrian CreditAttribution: adrian commented 31 August 2006 at 20:50

File	Size
macro_module.tar.bz2	4 KB

here;s an updated version of the macro module, including a new macro recorder install profile, which you can drop into to build an install profile.

Comment #23

moshe weitzman CreditAttribution: moshe weitzman commented 31 August 2006 at 23:36

File	Size
patch_115	8.81 KB

proper patch file now
- removed .install since we don't have hook_uninstall yet.
- changed .meta to .info since thats what we went with.
- removed some unsupported statements from .info
- includes the install.inc change

still needs review. is easier to review now :)

Comment #24

moshe weitzman CreditAttribution: moshe weitzman commented 31 August 2006 at 23:53

Status:

Needs review

» Reviewed & tested by the community

File	Size
patch_116	9.16 KB

To be clear, the proposal for core is the patch to install.inc in #21. i verified that this works so i set to RTBC.

the patch i attach here is for minor improvements to the macro module, which is not proposed for core (not macro.profile). the macro settings page has checkboxes now, and the export has a blank line in between form submissions.

Comment #25

eaton CreditAttribution: eaton commented 1 September 2006 at 00:28

I chatted with adrian on irc earlier and expressed concern about the inclusion of the auto-macro-playback stuff in core at present. I don't think it'sa good idea to put that functionality straight in the install system when we should look at making it something better integrated. And if it is ONLY going to be used by install profiles, I don't think that the 20 or so lines of code needed to execute macros would be horrible to simply paste into the install profile itself.

Thoughts? Anyone?

Comment #26

moshe weitzman CreditAttribution: moshe weitzman commented 1 September 2006 at 01:35

@eaton - yes, it could be outside of installer, but why? to save some lines of code? the install.inc is not parsed on very page view, unless i miss something. you articulated a benefit (less duplicated code), but not a drawback. if this code happens to be buggy, it will only affect macro playing, which is good. nice isolation. i see this is a small win and small risk.

Comment #27

drumm

he/him

NY, US

CreditAttribution: drumm commented 1 September 2006 at 02:42

Status:

Reviewed & tested by the community

» Needs work

Why can't this code go in profile_name_install()? If it goes in core it will need that debug code (print_r) taken out and good documentation.

Comment #28

adrian CreditAttribution: adrian commented 1 September 2006 at 11:10

well, one of the biggest wins i see from this in core, is that we can much more easily test things.

I am building a list of all drupal forms, so we can put together a list of valid input data for them, and we can test if one of them breaks more easily.
Also, i'm concerned that not all forms are working with programmatic submission yet, and this would help us weed out those which are.

Perhaps just putting this all in a 'devel' distribution could be the answer, but i think it could be useful to be able to test all distributions this way.

Comment #29

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 7 September 2006 at 18:01

This is kind of an oddball question, but could this functionality be used to export details about a site, such as which modules were installed, what vocabularies were created, what roles/permissions were defined, etc.? If so, this would make a great documentation tool.

Comment #30

moshe weitzman CreditAttribution: moshe weitzman commented 7 September 2006 at 18:26

webchick - this is not the right tool for that. we need a special module for that, that might or might not tie into the new status reporting system.

this thing records all form submissions which is not something one would want to share.

Comment #31

moshe weitzman CreditAttribution: moshe weitzman commented 21 November 2006 at 04:16

Version:	x.y.z	» 5.x-dev
Status:	Needs work	» Postponed

well, our little enhancement never got into core so i moved macro module and drupal_execute_macro() into the devel package in contrib. lets work on it there, and revisit the issue when 6.0 opens.