One of my biggest concern about FF is the bitcache repository that works smartly and nicely but:
- your uploaded files are totally useless without FF.
They are all piled up in a unique /files/bitcache/file folder, with inexpressive hashcode names like "000444bff4acb1342d92c18a946bf6a8e8c5d6bd" and no extensions.

I think it would be important to have the possibility to dump the repository as ordinary files, Suggested feature could be
- select the files to be dumped in Views
- add the correct mime extension (FF does not care)
- match destination folders with taxonomy and groups, gives options about duplicates

VBO currently allows to dump standard files, but it ignores FF files and does not handle any other criteria (because they are traditional files in folders)

Comments

johanneshahn’s picture

johanneshahn commented
Status: Active » Needs review

hi jvieille,
i've scripted a export/import module for ff. its called File batch
checkout latest http://drupalcode.org/project/fileframework.git/commit/b6cce20 git
or dev (until tomorrow).

can u try some exports?

next to do is:

  • implement 'gzip', 'bzip2', 'rar' exports
  • export to directories inside zip from taxonomy if available
  • some watch dog error handling if something fails
  • send zip to browser (currently stores it in export dir)
  • documentation
jvieille’s picture

jvieille commented

Thanks, great work!

It looks sane, however,
- the option " Export all files" downloads converted files using the same extension as the source file, so they cannot be opened
- the default file name could be set by option (the first part before the timestamp)
- I tried to save a bunch of 775 files / 2G - it took more than 45 minutes ( 8*3 GHz / 24 G ram)
- Import does not seem to do anything. It is not asked to select a file. What is supposed to happen?

- what is the "Send file" option?

johanneshahn’s picture

johanneshahn commented

hi,
- the option " Export all files" downloads converted files using the same extension as the source file, so they cannot be opened
is fixed

- the default file name could be set by option (the first part before the timestamp)
thats the filename of your zip archive or directory. u can override it. on every export this name is set to a default file_batch-[timestamp]

- I tried to save a bunch of 775 files / 2G - it took more than 45 minutes ( 8*3 GHz / 24 G ram)
i've set the limit from 20 to 200 files/nodes per cycle. may it goes faster?

- Import does not seem to do anything. It is not asked to select a file. What is supposed to happen?
i've scripted a first batch import now. currently u have to select a vocabulary to import nodes to. if u have folders inside your
import dir this folders will create a new taxonomy term in the selected vocabulary.

the import batch is just a starting point.
the problem is to know which file to import to what.
there are some possible situations

  • normal file node types with and without taxonomy
  • file node attachments with and without taxonomy
  • file node ckk fields
  • other node types with file field

thanks for review and testing, jvieille!

jvieille’s picture

jvieille commented

thats the filename of your zip archive or directory. u can override it. on every export this name is set to a default file_batch-[timestamp]

What I mean is that we should be able to set a filename that is kept from one session to the other. The current free text box allows to change the filename on a per session basis, so it must be entered every time. I see 2 ways for solving this
- save the chosen default file name in the database instead of hard code it
- do not allow to change the file name, but mention in the directory setting that this directory shall be specific for the site in order to avoid a mess if multiple sites save their files in the same directory.
I think the second option is not even en option...

i've set the limit from 20 to 200 files/nodes per cycle. may it goes faster?

That's fine. Maybe a configurable setting would be appreciated.

there are some possible situations

  • normal file node types with and without taxonomy
  • file node attachments with and without taxonomy
  • file node ckk fields
  • other node types with file field

On important case is missing:

  • File node in the context of on Organic Group

As the same file can be concerned by all these cases, a multiple times, it is not that simple.

Maybe
- Save all files in the same directory
- Save all the node attributes in an accompanying text db file
- create directories structures mocking the differents cases : node attachements, cck, taxonomies, organic groups
- fill-up these directories with symlinks to the files

Then the export / import feature should be able to rebuild the file nodes and their respective attachements / relationships and attributes...

johanneshahn’s picture

johanneshahn commented

hi,

save the chosen default file name in the database instead of hard code it

ok we can do this

That's fine. Maybe a configurable setting would be appreciated.

a configurable setting no prob

there are some possible situations

  • normal file node types with and without taxonomy
  • file node attachments with and without taxonomy
  • file node ckk fields
  • other node types with file field
  • File node in the context of on Organic Group

what u think about doing it like i18n String translation [#1114010]?
we take something like that for the import dir structure.

[node:type:mycustomtype:attachment]/file3.jpg -> will attach file3.jpg to new node of type mycustomtype
[node:type:mycustomtype:attachement:100]/file4.jpg -> will attach file4.jpg to node with id 100

[node:type:file]/file1.jpg -> will create/update node of type file
[node:type:mycustomtype]/file2.jpg -> will create/update node of type mycustomtype

[taxonomy:term:12]/file1.jpg -> create/update node and assign to taxonomy term 12
[taxonomy:vocab:2]/file2.jpg -> create/update node and assign to taxonomy vocabulary 2

[og:group:12]/file1.jpg -> create/update node and assign to og group 12
[og:group:user:1]/file1.jpg -> create/update node and assign to og group user 1

just scratching my head now :D

jvieille’s picture

jvieille commented

May be this can be combined with this issue?
http://drupal.org/node/281298

The handling by directory structure seems handy, but how can this look exact!y

[taxonomy:vocab:2]/file2.jpg -> create/update node and assign to taxonomy vocabulary 2

How can a node be assign to a vocab, not a term? I would guess that "term" or "taxonomy" would be enough to just tag with a particular term.

[og:group:user:1]/file1.jpg -> create/update node and assign to og group user 1

I don't understand this example - an OG user has no significance, og and user are separate concepts

I understand that everything can be combined, like in the following
node:type:mycustomtype:attachement:100:user:1:og:group:12:taxonomy:term:26]/file4.jpg

But how the same file could be assigned in several OG, terms, nodes?

johanneshahn’s picture

johanneshahn commented

hi jvieille,
:D LOL your combination

node:type:mycustomtype:attachement:100:user:1:og:group:12:taxonomy:term:26

bring me to think about that whole think.

greetz johannes

jvieille’s picture

jvieille commented

Don(t worry, no hurry

Come to think about it: you have planned to break the bitcache and RDF dependencies (not supported projects), so you might just consider this in the upgrade path to this.
(bulk upload still needed from the other issue))

johanneshahn’s picture

johanneshahn commented

yea,
thats true.
but i think about not to break bitcache before first stable 7.x of ff.
i am a maintainer of bitcache now and can bring it to 7.x too.
the problem is that bitcache is more essential needed by ff as rdf.

...

johanneshahn’s picture

johanneshahn commented

hi jvieille,
how does your current setup look like for displaying/handling files?

mine is:
i only have nodes of content type "file".
they are assigned to 1 term in one specified vocabulary.

may it can help me to find the best solution for needed import/export structures.

jvieille’s picture

jvieille commented

My setup is
- only one type "file" (wondering if anyone had the need for something else..)
- one vocabulary, multiple terms
- most in OG groups
- some in multiple groups
- sometime using CCK (cannot use existing files currently / missing feature)
- using core attachment - not always.

gobinathm’s picture

gobinathm
he/him
Primary language English
Location Milton, Ontario
commented
Issue summary: View changes
Status: Needs review » Closed (outdated)

Closing the issue. It was inactive for a long time & it's related to a Drupal Version which is not supported anymore