Files from imports in a Load Balanced Environment [#3152538]

We have a load balanced environment with 2 or more servers. 1 server is the Parent, any other server is the child where they are replicated using one way lsync from Parent server to children. ProxyPass is configured for all administration addresses and POST to be done on Parent. The addresses are in PoxyPass because files are uploaded before method=post. If they were not there then files would become inconsistent and results in file not found issues.

What we have experienced over the past number of years using feeds is that any child server could initiate a feed import when cron runs. In this scenario imported files are only loaded on the server that initiated the import. If Parent, okay all is good. But if child server, then files will be missing from the Parent and other servers so the user sees a broken image and file not found in logs.

Right now the only way to resolve is to manually make imported files consistent by checking on a schedule.

How can we ensure that feed imports are always processed by the parent server?

Comments

Comment #1

17 June 2020 at 13:59

crutch created an issue. See original summary.

Comment #2

MegaChriz CreditAttribution: MegaChriz as a volunteer commented 17 June 2020 at 14:56

Files to import - downloaded via HTTP - are saved to a 'feeds/in_progress' directory on the private file system (or the public filesystem if the private file system is not configured). Files are kept for 6 hours at max. I've been working on fixing that in #1029102: Importing Large CSV document (downloaded and processed in the background).

I've no experience with parent and child servers, so I cannot help you in that direction. I'm not sure if it's possible to configure the private file system for all servers equally - so that they all point to the exact same location?

Comment #3

crutch CreditAttribution: crutch commented 6 July 2020 at 17:05

Thank you MC. I think we figured out the solution for this by setting cron frequency to 'Never' at /admin/config/system/cron. Then we set up a cron job outside of drupal using the cron key address to run only on the parent server. This way no child server initiates an import. So the results are that imported files are loaded on the parent server and then replicated to child servers appropriately.