I think alot of people might be like me and have CSV files of 1000 records or more to import and update. There is no way any shared host will be able to cope with that. Mine times out at about 600 records.

Is there anyway to make this batch or have a restart import if it does?

Love the module. A very significant addition to the Drupal toolkit.

Comments

alex_b’s picture

alex_b commented
Title: Import in batches or timed intervals to avoid timeouts? » Support partial processing of a CSV file
Component: Miscellaneous » Code

That's a great feature request.

* The first step has been addressed - as opposed to alpha 1, HEAD is not loading the entire file, but is reading it line by line from the disk.
* Next, parser_csv_parse() has to accept a timeout parameter and it has to return whether or not it has finished the file.
* If a file doesn't finish, Parser CSV needs to tell FeedAPI about it. Otherwise FeedAPI will mark the feed as checked and won't come back to it until it has checked all other feeds in the system.

parrottvision’s picture

parrottvision commented

That sounds great. So if I have interpreted that correctly:

1: File is read in line by line.
2: parser_csv_parse() brings import timeout to an end
3: Parser CSV will then tell or not tell Feed API if the feed was completed.

Will Paser CSV then rerun the same CSV file but pick up from a certain line number that it previously stopped on based on parser_csv_parse()?

alex_b’s picture

alex_b commented

"Will Paser CSV then rerun the same CSV file but pick up from a certain line number that it previously stopped on based on parser_csv_parse()?"

kind of: feedapi will rerun the same CSV file. That's why feedapi needs to know that the feed (CSV file) hasn't finished.

7wonders’s picture

7wonders commented

+1 on this. Much needed especially when mapping in such things as remote_file cck fields.

reikiman’s picture

reikiman commented

The above sounds good. I'm trying with the alpha2 release and on a feed w/ 40 items or so it times out at the 19th item and never gets past it.

parrottvision’s picture

parrottvision commented

That seems quite low for any host to stop at. But I also thought alpha2 was a download and process incrementally. I think alex_b would have to clarify but I thought it would download and then each feedapi run it would pick up where it left off and then process from there.