C-Command Software Forum

Maximum number of files in a library

EagleFiler
#1

I would like to know if EagleFiler can swallow an immense number of files, like over 450 000 files and counting. Files are nested in folders, sometimes five or six levels deep.

There are all sorts of files (various text files (yaml, markdown, html, javascript, python), pdf, jpg, psd, BBEdit projects, symbolic links (not aliases), etc.). A first try to import the whole thing about a year ago ended with EF hanging and giving up after about ten hours. Also, many of the folders that were imported had a Yellow label added to them for no apparent reason. This is not good as the folders already have many labels including Yellow.

#2

It’s definitely possible—I have a library with 656K files. I wouldn’t expect the import to hang, but if that happens again you could record a sample to help track down what’s causing that. The best way to do a big import is to move or copy the files into your library’s Files folder. That will avoid EagleFiler having to copy them (which can be slow due to Finder comments), and if the import gets interrupted for some reason it will just resume the next time you open the library. Please let me know if you see any other problems with labels. I know there was a bug that could cause that, but I thought it was fixed years ago.

#3

I would like to add that the trial was done using the instructions in the manual about using aliases and symlinks. To tell the truth, the instructions are a bit confusing. Here’s what they say :

Aliases and Symlinks

  1. Close the library in EagleFiler.
  2. Move the library folder to where you want to store EagleFiler’s files.
  3. Move the Files folder to where you want to store the raw files. Make sure that the alias or symlink is called Files.
  4. Make an alias or symlink from the Files folder to its former location next to the .eflibrary file.

Paragraph 3 says to " Make sure that the alias or symlink is called Files .", although, if I understand correctly, the alias or symlink has not been created at that point since its creation appears only in paragraph 4.

Anyway, what was done is this:

The folder containing all the raw files (here its name is QD) is on partition 2. The EF library was created on partition 1 and the Files folder was replaced with an alias (or symlink, I don’t remember), called Files which points to the folder QD on partition 2.

So, EF didn’t need to duplicate any files.

I’ll let you know if it works this time (in a day or two).

#4

I’ve fixed this in the instructions. Thanks!

Is this a hard drive or SSD?

Sounds good. I recommend updating to the public beta version because it includes better error logging. Please also note what’s happening in EagleFiler’s Activity window.

#5

The Mac is a Mac Pro 2013. Its SSD has two partitions, one for the system (Mojave at this time), the other for QD. At the time of the first test, it was on Sierra.

#6

Thanks for sending the sample log. I think the importing hang that you encountered is fixed in the public beta of EagleFiler 1.9.3b3. I’ve not been able to reproduce the label problem, so please let me know if you find a reliable way to trigger that.

#7

I believe the label problem is fixed in EagleFiler 1.9.3b4.