"Fluid" syntax for hook_schema() (method chaining).

Update:
In my own projects I went to calling this pattern "Injected API". The argument is then always named "$api". I am going to change the issue description accordingly.

This can be seen in action here:
http://drupalcode.org/project/injapi.git/blob/refs/heads/7.x-1.x:/exampl...
(and changing the issue description, again)

Link to a more recent example / draft, with a slightly different syntax.

Updated issue summary to describe the latest proposal, and not distract with details we are going to discard anyway.

Updated issue summary.

Updated issue summary.

An alternative would be to move to yml files, like with routes.
(far away from saying it should be CMI !!)
That would still be more verbose than what is suggested above, but nicer to type than PHP arrays..

Replacing "$h" with "$api"

Updated summary, now we also update the title.

Related:
#432440: Update Schema API to make it more usable

I should mention that this other issue is trying to solve a different problem, and has nothing to do with the syntax within hook_schema().

This issue here, on the other hand, is only about DX syntax candy within hook_schema(), nothing more.
It does not plan to change the internal array structure.

I thought the linked issue was more thought for the internal implementation of "writing the schema stuff to the db", and for hook_update implementations.

How does the alternative look like? Crud?
That's the same controversy again as we have with CMI for routes: persistent state + crud, vs info + cache. Imo, whatever we do with state + crud, it always means that modules need to do a lot of babysitting, which I would prefer them to not have to care about.

This said, I don't mind moving towards events or yml.
I wonder what other systems do, I did not do any research on that.

Though yes, I know doctrine. I just think we are far away from that, and the particular implications of that would not fit for us.

Also, don't we have enough bloodshed in D8 already.. shouldn't we leave some killing for D9?
The hook_schema and "huge array of definitions" (not talking about implementation) as is doesn't make realworld problems afaik, and this particular change is totally harmless and can even be backwards compatible.

And yes, this is all about lipstick. Very nice lipstick.
And then again, it's not. It's giving a new face to the pig. And then at some day replacing the pig behind, if we feel that way.

Thanks for picking this up!

Since the hook, like so often, introduces some weird dependencies at times.

That would be, e.g. a table structure depends on the definition of a content type, a field, etc?
I don't see this as a problem necessarily.
You need to be aware of the dependencies when you design a module, and then design a clearly defined food chain. If you ignore that, you get into trouble. But those dependencies do not exist because of the hook, but because of what you want to achieve.

But maybe you mean sth else?

declaring a table becomes a matter of a module providing such a class

I think Zend does it that way (or did, when I used that ages ago).
Totally doesn't help with dynamic schema definitions - unless you play with code generation.
I always found that boilerplatey and stupid, writing those classes.

And rightfully, Doctrine lets you write yaml files instead, which lets you focus on the actual definition of the table, instead of the boilerplate getters, setters and friends. This yaml feels very much like our hook_schema(), with some additions.
From there you get those classes via code generation, and you subclass them to implement custom behavior.

For Drupal I don't really believe in "one class per file" approach:
- It does not help for dynamic schema definitions (e.g. for content types)
- It does not allow multiple modules to participate in the schema definition. No, I don't count subclassing as a solution, that is too one-dimensional.
- It creates a lot of duplicate and meaningless code.

I'd rather see
- some code to produce table definitions (hook_schema, yaml, etc)
- operations that are used internally to create those tables in the db.
- generic classes for tables and fields/columns, which are instantiated with specific configuration, instead of being subclassed. Yes, this would give you the introspection you want.
- reusable "behaviors" which can be attached to tables or columns.
- a query builder similar to dbtng (or just keep that)

Actually I don't really care about the other points.
The main thing is, as a module developer I want a pleasant way to define my schema, without doing anything redundant or bureaucratic. What the system does with that, is not part of this issue: It can either just remember the schema somewhere, or if you want, generate those classes as class-per-table, like Doctrine does.

If we compare
- Doctrine-style yaml files (you can also have XML, but seriously who wants that :) ) -> static
- Doctrine-style yaml + php (I think I saw that somewhere)
- hook_schema() + alter with nested arrays
- hook_schema() + alter with method chaining, as suggested here
- a hybrid hook_schema() which allows nested array return value as well as method chaining.
- the same, but moving that to schema builder events, schema factories, or the like.

Some details aside, these are all exchangeable. Except
- some work better than others for dynamic schema definitions
- there's this old "NIH" argument.
- Doctrine afaik has some additional keys to attach behaviors. This is pretty cool, but nothing stops us from doing the same.
- hook_schema + alter is better at letting multiple modules participate in the definition process.

we need to, first, provide an introspection system for the DB layer to tell us what the state of a table is. That involves defining some table object that represents the introspected data, which we can then retrieve at any time.

We can easily have our introspection objects today, with a few classes instantiated with existing hook_schema definitions.
I can imagine very little that this object could tell us, where we would absolutely need a "class per table" approach. Neither generated nor hand-written.
And due to our modularity, we are unlikely to hardcode any business logic into those classes.

Btw,

Since the hook, like so often, introduces some weird dependencies at times.

there is this issue #1446808: Let plugin managers announce changed definitions
and this post in gdo suggesting a "dependency matrix": http://groups.drupal.org/node/267163#comment-850373
(yes, you probably know that already)

Actually I was referring to the bootstrap knot of doom that hooks result in. Plus the lazy load questions around specialty hooks.

I don't see that problem, esp with hook_schema(). If you could explain .. :)
The only issue I could see is, you are creating a content type, and instead of doing the schema change directly, you need to wait for a "schema rebuild" to do that for you? Not sure if that is even the case today..

wait, I thought you didn't like YAML for Routes? :-)

yaml on itself is great.
The issue I have is yaml for static + something else for dynamic + yet something else for altering, that is 3 different "languages" to learn. With nested definition arrays, you learn the static definition first, and then the rest is just obvious. You can open a random *.install file and look for hook_schema(), and you know it all.
We might get used to it if this becomes a common pattern.

This being said: yaml + alter + dynamic, original hook_schema, "fluid" hook_schema, they are all in the same boat if compared to what you seem to be proposing .. so I'd like to get over those basic decisions first, before we go back to DX bikeshed (which this issue was originally about)

Even if we offered a YAML file for module devs as well (wait, I thought you didn't like YAML for Routes? :-) ), we'd still need such an in-PHP representation of a table anyway because we are *not* baking a YAML dependency into the database system. It is actually rather analagous to Routes (PHP object for the definition, ability to dump out to or read in from whatever other serialized formats you feel like writing as a DX convenience). That PHP object has to come first, though.

Yes, sure.

// (somewhere in core)
// get schema from yaml, hook_schema() or others
$schema = drupal_collect_schema();
// create your introspection table objects
foreach ($schema as $name => $table_info) {
  $table = $db->loadOrCreateTable($name);
  $table->setConf($table_info);
  $table->save();
}
// TODO: Why not make schema an object as well :)

// (somewhere in a module)
function hook_update_N($db) {
  $table = $db->getTable('comments');
  $table->changeFieldSetting('title', 'length', 2047);
  $table->save();
}
function hook_schema($schema) {
  $table = $schema->table('comments');
  // ouch.. this is now a wrapper thingie, whereas in hook_update_N() you are dealing with a "real" table object. This certainly might feel inconsistent.
  $table->field_id(..)
  // OR
  return array(...)
}

Not sure what else you want to do with the introspection table object.
My point is, the average module should not be writing a MyTable class, but just provide its definition, so core can just instantiate the generic Table class.

The code above is using the table as an "ActiveRecord". I'm not sure yet that is a good idea.
I always had the idea that the representation of an object as-it-is-now should be separate from the object as-i-want-to-modify-it. Similar to that the representation of an entity-in-the-db should be separate from entity as-i-want-to-render-it.

$table = $db->getTable('comments');
$table_to_update = $table->createActiveRecord();
assertEqual($table->getDefinition(), $table_to_update->getDefinition());
$table_to_update->addField('test', ..);
assertNotEqual($table->getDefinition(), $table_to_update->getDefinition());
$table_to_update->save();
assertEqual($table->getDefinition(), $table_to_update->getDefinition());

I think this depends what else we want to use the table object for.

Nice, I think I'm getting more of a bigger picture now :)
I think the solution is to properly slice the responsibilities and dependencies.

Here is how I imagine it could work:

Module schema definitions:
hook_schema(), yaml or similar (+ alter) tell us how modules want the schema to be like.
This has to depend on modules naturally. But we only use it during "rebuild time".

Stored schema definitions:
In addition, we have to remember how the schema is right now, independent of how modules want it to be. This information can be independent of modules, and available at runtime early in the request.
"remember" can mean CMI or any persistent storage we have available. It could be remembered in the database, but only if we manage to "protect this from itself" (so probably better not).
This schema information will have the basic table definitions, plus any meta info that we like.

Active Record:
$table->save() "active record" writes a schema definition both to the schema storage, and runs the queries to actually create or modify the tables in the database.
This can be used any time to quickly change a schema, at the developer's own risk. E.g. after creating a field, we could just directly say $schema->createTable('my_field')->addColumn(...)->save().
And it will be used in core, on "rebuild time", to bring the schema back to the structure intended and described by modules.
Note: This may conflict with hook_schema(), e.g. if hook_schema_alter() adds an additional column, that our module does not know about. See below.

Query engine:
db_change_table('tablename')->addColumn(..)->execute() is just a query builder engine which only talks to the database, but has no idea of schema definitions within Drupal.
We could use that within the implementation of the Table "active record", but probably no other modules will use it.
Whether we actually need this thing, I dunno.

Schema rebuild:
"Rebuild" means that the schema will be reset to the state that modules intend (via yaml, hook_schema + alter, etc). Any previous changes done with direct calls to $table->save() will be overridden by the information from hook_schema(). So if the field module creates a table, and on the next rebuild it doesn't know about that table anymore, the table will be wiped (*).

This gives us a reliable way to clean up a messed-up database structure and turn back to the structure we expect.

(*) We need a way to avoid unintended data loss on those rebuilds. We could say that harmful operations are not executed, and a warning + diff is sent instead - so the module developer has a chance to resolve it in hook_update_N(). (I didn't really check how we do that today, might be we already do sth smart there?)

Syntax on $table and in hook_schema():
We could let hook_schema() and the active record thing use the same table class, or at least methods that behave similarly.
On the other hand, I would like hook_schema to be purely declarative. Any ->save() should not be executed within there.
I think we should postpone that until after we have a plan for the other things.

3rd party code?
A lot of this sounds like we could find it in 3rd party code. Otherwise, we could code that as an independent library and then bring it back to Drupal :)

DISCLAIMER:
I have not really studied the other issue yet, so some of that might change my mind.

Btw, instead of letting modules use $table->save(), we could also allow them to request a partial schema rebuild - e.g. only for the module's own tables. Not sure if this could bring us any trouble, e.g. if a table "changes its owner" from module 1 to module 2.

There seems to be the general controversy about "collect info and rebuild" vs "allow modules to do crud operations on an existing structure". (or do both)

The D7 equivalent is hook_install() / hook_schema() on the one hand, and hook_update_N() on the other hand. (and in this case you need both)

hook_schema() means I describe how I want the table structure to look like right now.
hook_update_N() means I describe the operations to get from the current structure to the structure that I want.

If I understand you correctly, you propose to ditch hook_schema() entirely, so that in hook_install() I would have to do operations very similar to hook_update_N(). (only that I would know that none of those tables exist yet).

As a module developer, I enjoy writing and modifying hook_schema(), but I hate writing hook_update_N().
So, I would like to propose something to turn this around :)

1. You write your first hook_schema() (*).
2. You install the module, everything happens as we know it. Except that the hook_schema() is remembered somewhere.
3. You modify the hook_schema().
4. You use a drush command to generate your hook_update_N() (*) code, based on a comparison of the old and new schema. This could be placed in a dedicated file or class per update, if you want..
5. You can do manual modifications to the generated hook_update_N().
6. (every time you re-run the drush command, your changes are wiped. your bad. (we could go the extra mile and automatically merge instead of wiping)
7. Drupal can run a generic test, to check that your modified hook_update_N() will indeed result in the intended table structure.
8. Drupal might also generate stub test cases, to check if your data will be correctly preserved / converted.
9. You run drush updb, and the schema changes.

This seems to me like an awesome-DX solution.

(*) I say hook_schema() and hook_update_N() all the time, but this could be replaced by something else - event subscriber classes, yaml, whatever.

EDIT:
This does not account for dynamic schema changes outside of module updates, or hook_schema() depending on dynamic data. In this case, the developer needs to do more stuff manually in hook_update_N().

Ok, I think we are not so far apart.
I always understood (and I think I mentioned that) that this issue is about your "phase 2", whereas the other issue is about the phase 1 stuff.
A different way to see it, is that this issue is where we have a private conversation, whereas the crowd is in the other issue :)
I have no objections with either of these viewpoints :)

I agree with your phase 1.
I still like my fluid syntax, but I agree we should do the low-level API first, and anyway it's all D9. Until then I will simply use my personal InjAPI bicycle, because I hate those huge arrays.

The most crucial change is the source of authority. Drupal currently assumes that the module-defined schema is definitive, and if the database itself doesn't match up then oh-well, things explode. That's backwards. Instead, the database itself should be definitive, and parts of Drupal that rely on database structure for their logic should get that information from the database, not from what the module developer claims the database is supposed to be.

Yep.
I just wonder if we should have a 3rd level in between, where we remember the schema information on every change. This might be a simple cache, but it could also remember some meta information that we cannot simply extract from the tables themselves, or simply, a more semantic and db-engine-independent representation of the schema.

Drupal's schema system requires a fully booted system [..] eliminate any and all such dependencies

Yep.
The authoritative description of the current state of the database can and should be available on early bootstrap.
The "how modules claim or want the schema to be like" can wait until much later in the request.

Btw, isn't my post in #17 already quite close to that?

(I am missing some points there, but the main idea of separating current schema from the intended schema is there)

Shipping a table definition object with a module (by whatever mechanism, class or YAML).

The "whatever mechanism" was the original intention of this issue.
I see this has to wait, but I would like to make a point, while we are at it.

I have seen some frameworks where you have a "one class per table" approach. This class contains one method to define the columns, and then some methods for CRUD operations.
I always found this approach kinda stupid. Even with code generation, you end up looking at lots of redundant code. Or, if the table contains only the definition method, then you have plenty of classes that do nothing.

I would agree with tables being objects, but if all they have is the definition of columns, then I'd rather see them as instances of a generic class, with a configuration argument.
Also I don't think that the CRUD operations, if any, need to live on the same object/class as the schema definitions.

----------

The "Shipping a table definition" is most relevant for modules that have a static schema, not depending on any other state information. Here the schema definition (by "whatever mechanism") is the developer's main point of schema control.
For modules with a dynamic schema, we may still have a (dynamic) table definition, but here it would be used more for testing than for anything else.

(re-add tag)

Answer the expected comment about memory.