Hi,
I have imported (again) a few thousand mail-messages into EagleFiler. After a while (~5 hours), finally indexing and all other activity stopped.
I renamed one mailbox from “Archive” into “Archiv” (German word for archive). The moment I did this, indexing started again with an estimate of 97 minutes.
Does renaming a mailbox really need to restart indexing? (I did not add messages, nor did I merge mailboxes, nor did I do anything else then renaming.)
Did it actually take 97 minutes?
When you rename a mailbox, EagleFiler only needs to reindex the name of that mailbox. It does not reindex the messages contained in the mailbox. You should be able to see this in the Activity window because it will say “Indexing records” rather than “Indexing messages in ‘mailbox’.”
Mailboxes in EagleFiler are immutable, so once indexing has completed it never needs to reindex them.
Hi Michael, thank you for the quick reply.
Yes, it is still indexing messages in “Archiv”. 3 hours, 18 Minutes remaining.
My guess is that indexing the messages did not actually finish the first time. Renaming the record simply reminded EagleFiler to check that the messages were indexed, and it found that they were not.
The point is that if, as you write, EF did not finish indexing the first time, indexing of a mailbox with 47.510 elements took EF around 10 hours altogether. Because I remembered this thread, in which I/we thought indexing did not finish because of lack of disk space, this is not what is happening in the actual case.
On the other hand, an indexing time of around 10 hours for 47.510 messages is really a lot. And no, the computer did not do anything else as it was not in use. Summing up the whole experience of backing up emails - my primary reason for EF - is rather not that nice:
Importing mail from “Sent” into EF: 5 minutes for about 14.000 e-mails.
Indexing mails in “Sent”: ~4 hours.
Importing mails from “Archive” into EF: 12 minutes for about 47.500 e-mails.
Indexing mails in “Archive”: ~10 hours (and guess what? I opened EF in the background and it started indexing the same mailbox again.)
Not yet have I replaced duplicates (which starts indexing again on both mboxes).
Not yet have I created specific smart folders, etc.
This all on a MBP with a 2,7GHz i7.
Are these really the times, EF needs? Or is there - again - something going wrong?
Added ~4 hours later:
Finally, EagleFiler stopped indexing messages while I was away. (Yes! Hooray!) I closed the library and - just to make sure, EF did not “forget” to index messages - I reopened it:
Currently reindexing of folder “Archiv” is displayed with an estimated time of 74 minutes (message 2000 of 47500).
Estimated indexing time raised to 2 hours, 33 minutes at message 3778 of 47500.
It seems like more than 10x slower than on my MacBook Pro, so there could be something wrong. That said, the indexing time does depends a lot on your specific situation:
What do you see in Activity Monitor while it’s indexing? Is the CPU utilization high? Are there other “ef” processes doing work? Is there a lot of disk I/O? You can record a sample report to see what it’s doing.
Do you mean the same “Archive” mailbox?
You can check for errors in the Console. EagleFiler will start a new index file if it finds that the old one is damaged, and this will be recorded in the log.
Also, you could check out the file YourLibrary.eflibrary/Indexes/IndexingState.plist. It stores a list of keys for the mailboxes that have been successfully indexed. (You can find the number for your mailbox by hovering the cursor over the icon for it in EagleFiler’s Info window.) So you can see from this file which mailboxes EagleFiler thinks are “done.” If, for some reason, EagleFiler can’t write to this file, it will have to keep scanning the mailbox.
One by one:
efindextool between 0-100%.
EagleFiler between 0-130%.
…in the time I used up for writing this entry.
Yes, I do.
Interesting. I found a lot of lines like this one:
04.10.13 12:35:53,870 efindextool[8959]: Couldn't open index at: file://localhost/PATHTODIR/E-Mails/E-Mails.eflibrary/Indexes/45.efindex
…but interestingly these lines stop at a specific time and afterwards they do not reappear.
Mostly I see lines like this one:
04.10.13 16:49:23,632 mdimport[22001]: Imported '/Volumes/Daten/E-Mails/Temporary Items.nobackup/EagleFiler-2013-10-04-0enFl-vTT6mhE0YQzF8UpQ/3850/NAMEREMOVED.flv' of type 'com.adobe.flash.video' with no plugIn.
Okay, it is Key 45 in my case. No “Key 45” in IndexState.plist, but there is a folder “Damaged” with a 45.efindex and 45-1.efindex (both around 2,15 GB) in there.
If this is getting too much into detail and no longer adequate for the forum, please feel free to continue by email!
And: thanks for your help again!
There’s more information about that in this thread, although that doesn’t seem to be the primary problem you’re seeing.
That would seem to indicate that EagleFiler isn’t running into a problem with a particular message’s content. For example, sometimes it gets stuck indexing a damaged PDF attachment, and in that case you would see an eftexttool process that hangs around.
If the OS reports that an index file cannot be opened, EagleFiler moves it aside and starts a new one. So that could explain why the indexing was taking so long. Unfortunately, the OS doesn’t report why the index was unopenable. This could be due to file corruption or an ownership/permissions problem. If you e-mail me the log, I can take a closer look and try to see what’s happening here.
That’s normal. It simply means that EagleFiler encountered an attached file that it doesn’t natively know how to read, so it tried to analyze it using your Spotlight importers.
Mail sent.
To follow up, it looks like Michael’s index files for very large mailboxes are becoming corrupt. (The message data itself is unaffected.) We’re not sure what’s causing this, but he was able to work around the problem by keeping separate mailboxes for each year, rather than merging all the messages together into a giant mailbox.