my file is too large to import, can I upload it first?

I would like to second this, being able to import from the server would mean that we could use WGET to retrieve the file or could compress then unzip as above.

Surely this is easy to do? We just need to change the upload file user interface to browser the server.

the same here. i also hava a 25mb file and i get nothing. even tried to split it up, but 10mb is too much.

Ive seen a lot of people asking for this feature, being able to cut out the upload and just selecting the file from the server would be so much easier.

1. It lets us compress the file to make it small to then decompress on the server to then select when importing.
2. It lets us use the servers WGET completely bypassing us have to upload the file

Surely this is easy? Change the select file interface to select the file off the server instead of selecting a file on the users machine or even having a directory files/csv where the module looks for the CSVs.

subscribing

Create a local host environment and do it there. After that just export the db and upload it to your server :P

No need to upload first or do it locally -- check out this solution!

"Create a local host environment and do it there. After that just export the db and upload it to your server :P"

lol, it's a nice hack but it's hardly idea.

I have a hundreds of MBs of CSVs that I need to keep updating so having to download them to my local machine to then import them there and to then copy the HUGE database file over isn't viable. The database file itself is going to be massive so I would no doubt be shifting the problem on since there is probably a limit to what size of database can be imported (as well as time consuming uploads).

I need to be importing between 200MB - 500MB of CSVs per week, having this module being able to take it from the server file system would allow me to use the servers wget and cron to schedule these CSV downloads meaning that I can completely automate it and cut out the middle man. (aka my local desktop pc and broadband connection)

I am really surprised a CMS as packed as Drupal doesn't already have this.

Just been told about the Filebrowser module, it lets admin browse the server file system so all we need now is to link it to the Node Import module.

We could either have an extra option in Filebrowser, say an icon next to CSV files that says 'Click here to import' or we can integrate the browser into the node import 'select your file' page.

This is well out side my skills (as im sure you can see) but I will see what I can hack together.

I'm thinking that this is now a case of taking the code from one module and firing it into the other which should be easy compare to the other stuff Drupal offers. Anyone brave enough?

http://drupal.org/project/filebrowser

This may not be a Drupal limit.

Check this out
http://drupal.org/node/259580

Also, you can tweak around php.ini
Set max execution time to 0 in php.ini, and increase the memory limit as well

"Is it possible to import this file directly from my server instead of trying to upload it? That way I can compress it and then upload it."

Has no one been able to take the FileBrowser module user interface and integrate it in the Node Import module to allow us to do this?

subscribing to this thread....

In any case, one way I was able to solve this was:
- first import a file. and do not delete from server once completed.
- this then creates the file on the webserver.
- once this was done, I moved my huge file (already ftp'ed to the server) in place of the dummy file. So node_import now thinks that is what it is loading, next time it runs...
- reimported using same file name, as opposed to selecting a new file...

A hack, but it temporarily solved my issue....

zeezhao, this is a nice trick, but such process will still time out on large files (after max time PHP is allowed to run on your server). This solution is even more reliable: http://drupal.org/node/191197#comment-863661

Thanks, I already use set_time_limit() to avoid php timeout, and other changes in php.ini for memory and file size limits.

I guess the issue here is that for huge files, its quicker to ftp the file to the webserver especially if you are using an ISP, than to load it to the webserver via the node_load interface. On the webserver you can then break it into smaller files using head or tail or other unix commands.

In my example I was loading a file with about 2m records and 350MB in size. So I broke it into four, with 500,000 records per file for loading. See a seperate thread on some of the other node_load performance improvements - http://drupal.org/node/309563

Being able to limit how many rows get done each time would be nice. I keep getting time outs that there is nothing I can do about. I am not the owner of the server.