Clients module seeks to abstract the creation of connections to remote services, with the idea that modules can plug in to do something with what they get from remote services.

In essence, it's the UI that's at /admin/settings/retriever.

Advantages:
- We get a UI on both Distributor and Retriever site for free
- We get the ability to set up multiple connections, meaning we can go beyond a hub and spoke system.

This is something I was seeing perhaps happening for a 6--3 branch -- except that the problems at #704082: Delete distributed node on the retrieving server through Content Distribution jump the gun a little...

For Clients to be ready for us to plug into it:
- Its UI needs fixing.
- It needs to be a model we can work with. Currently you have to enable Services on each connection, which just strikes me as a fiddly step when the services already have both key access and regular Drupal permissions.
- A resource needs to have access to more than one remote service.

CommentFileSizeAuthor
#2 content retriever abstraction.png121.52 KBjoachim

Comments

adub’s picture

> - Its UI needs fixing.
> - It needs to be a model we can work with. Currently you have to enable Services on each connection, which just strikes me as a fiddly step when the services already have both key access and regular Drupal permissions.

These are reasonably trivial and on the todo list but I'm not 100% on the timeline. I'd be more comfortable stabilising the module further before refactoring the UI.

> - A resource needs to have access to more than one remote service.

A resource is a configured service by definition. You can't really just multi-select them as they can be returning entirely different kinds of data and each service will have different input parameters. I think this would be addressed in the client which would select a resource for each function it requires (probably filtering on service type). If you want to be able to capture whole configurations, Feeds provides an example of bundling various plugins into an importer - maybe your client could do something like this. Import/export of configurations for connections and resources is definitely something I've envisaged in the clients module.

joachim’s picture

StatusFileSize
new121.52 KB

> A resource is a configured service by definition.

I don't think that model is going to work -- certainly not for what CD is trying to do. The client (aka retrieving) sites need to call about half a dozen methods on the remote distributor. It's already hard enough to set CD up; having to require the poor user to create a resource for each service is not good. I don't want to make the setup steps more complex!

I'm not sure we're on the same page in terms of the architecture. What I'd like from Clients is an abstraction of the connection settings part of CD and the call to xmlrpc, so that they can be used on both sites, and so multiple connections can be made. I see a resource as being a tool that does something with remote data -- in this case, you'd have a CD resource on a Drupal connection. Similarly, you could have a Deploy resource on a Drupal connection.

Here's the UI from CD that I imagine Clients should take over.

content retriever abstraction.png

Look at content_retriever_xmlrpc() in content_retriever.module for the kind of code I'd be in favour of farming out to Clients.

I kind of think the UI should be fixed to a point where the module is more usable, because with an API module like this you need people to actually get as far as trying it in order to get feedback about what they need it to do.

adub’s picture

The screenshot from CD corresponds to a connection in Clients (minus the reset and persistent login which could be feature requests). First stab at clients had just this and the client then handled all the method calls itself as you describe. However, this turned out to be unworkable when making a views or feeds client due to the quite basic form injection available - so this was pushed back into the core module of Clients and called resources, which was actually a cleaner approach overall and creates thinner, more maintainable clients.

A CD client could certainly address the connection directly but I think we should set some fixed API calls for connections in this case (I think this would be basically setting the existing ones in stone but we should both review that first).

I'm also wondering how the configuration of this client will work as it sounds like it will have hardcoded dependencies on certain services being available. I thought it would be better to push this back into resources and your client could then be simpler, just assigning the resources and calling them through their API. That's why I was suggesting scriptable bundles of resource configs which would allow your client to provide default setups. But YMMV on that part and there is work needed on making Clients config scriptable.

Maybe you could make a v1 which accesses connections directly as it's closer to your existing implementation and we have the basic parts in place, and revisit moving some of it out to resources later? I think the reset could be handled in your client (or using the http_reset module)? That would just leave the persistent login which sounds like it might be something Clients should be able to do.

joachim’s picture

> persistent login which could be feature requests

How is login being handled right now? I thought the point of that blowfish malarkey was to do just that.

adub’s picture

The blowfish encryption is just so that you don't have to store a cleartext p/w and is a bit safer with stuff like pulling dbs from live > dev. A related thing that needs developing further (though shouldn't be a problem for current implementation of CD) is that clients_drupal currently expects login and key auth on the server - we'd also want to support no auth/oauth etc.

joachim’s picture

I've been talking to heyrocker (of Services and Deploy) about oauth -- he says the Drupal modules aren't ready yet, but some planning around that is probably going to happen at Drupalcon SF.

agileware’s picture

This would be extremely useful.

Something like #827524: Add support for using the UUID module to determine whether a node is inserted or updated would also be useful in this case.
Although if the client was also stored in the content_retriever table that would also make it unique.

agileware’s picture

Also, being able to pull nodes from multiple views of a client would be useful in addition to multiple clients.
See - #827610: Allow retrieving nodes from multiple views

joachim’s picture

Is Clients module ready for us yet?

Last time I looked at it, it was rather too constrictive and significantly increased the complexity in setting up a connection. That was months ago (see date of my last comment), so it may have come along since then.

I think we need to make sure we're not making the process of setting up Content Distribution even more complicated.

The other thing is that I now have limited time to spend on this -- I spruced up Content Distribution for a specific project I was working on and that has now shipped. So my involvement here will be mostly to review patches.

agileware’s picture

I'm not sure about the clients module at this point.

It is quite possible a client (or myself for them) will be implementing a setup that will require this, so patches will come at that point.
It'll be good to have someone reviewing them, thanks.

adub’s picture

Having reread my last comment I'd say nothing has changed - you can just ignore resources and access connections directly, doing all your resource configuration in your client. Resources are meant as a convenience for clients not a hindrance. Since you'd be making the first client implementation to do that, we should freeze the connections API. Also need to keep an eye on http://drupal.org/node/721640

However, I think abstracting resource configuration to the centre isn't a bad design for Clients in principle, and simplifies client implementations hugely. If the current Clients way of configuring resources is a usability faff compared to how you'd be doing it in a CD client (quite believable), I'd be very up for improving it on the Clients side. As I said, I'd also like to see Clients configs exportable/importable and allow client implementations to install presets.

I'm in the same boat regarding dev, have moved on to something else as Clients does what it needs to do for the project it was written for. But I've opened up some discussion regarding co-maintainership for the purpose of taking it forward - suspect it's a bit encompassing for one person to do well in any case. Let me know if you want to be part of that.

xtfer’s picture

Is this particular development path dead then?

We're interested in developing additional features for Content Distribution, including potentially UUID integration and multiple Receivers per site, but not if its going to be rearchitected.

joachim’s picture

Not dead, just resting ;)

If you're thinking of changing the UI so a retriever site can have several sources, then I'd say moving this issue forward is a good place to start. There's no point reinventing the wheel when Clients module has already started building it. There's work needed on Clients module though -- the reason this issue is stalled (apart from lack of developer free time...) is that Clients isn't yet generic enough to work for this; as it stands it actually would add UI and API complexity rather than remove it.

One thing you should take a look at though if you're planning development work either on CD or on top of it is #846402: sync to latest Services.

joachim’s picture

I've started a 2.x branch of Clients module, which provides the UI and API that we need here. So this can be considered resurrected :)

BTW:

> We're interested in developing additional features for Content Distribution, including potentially UUID integration and multiple Receivers per site, but not if its going to be rearchitected.

That's going to be doable soon -- the next stage for Content Distribution is to become an application on top of the new Transport module. If that's something you're still interested in, let me know :)

joachim’s picture

avpaderno’s picture

Issue summary: View changes
Status: Active » Closed (outdated)

I am closing this issue, as Drupal 6 is now not supported.