Reducing CPU usage for indexing of huge mailboxes

EagleFiler
, , , ,
1

This is actually a thread with two questions:

First, is there a way to reduce EF’s CPU-usage when indexing big mailboxes (>1.000 mails)? I am working with huge mailboxes (~20.000 mails) and indexing makes the Mac practically unusable for the time EF does its work. Is there a way to let EF only index such mailboxes when the Mac is idle?

Second, somehow related to the first question: When selecting conversations in mail.app, this leads to several mailboxes being imported (say: Archive, Inbox, Sent, and others). Now each of these mailboxes has around 3.000-10.000 mails in it. I would like to halt indexing, merge those mailboxes, delete duplicates (with the script you provide) and then give EF the Go! for indexing. Is this possible?

Thank you!

2

Normally, high CPU usage won’t make a Mac unusable because it will share the CPU such that each process gets some time. What sort of CPU usage are you seeing in Activity Monitor? Is there a lot of disk I/O?

No.

You can halt indexing by clicking the x button in EagleFiler’s Activity window. It won’t start indexing again while it is merging.

3

Loosely related to the topic: I created said EF library and let EF index everything. EF-Library A was reay.

Then I created a sparsebundle with the according size - EF-Library B - (I have never managed to match the size of what I expected to be imported into the EF-library with what EF adds) and copied the contents of EF-Library A (files, xxx.eflibrary, notes, smart folders, temporary items, to import (xxx)) into the new sparsebundle.

I doubleclicked the eflibrary-file. 2 minutes later, EF started indexing mails as if it had not done so before.

Why?

4

Did it say “Indexing records” or “Indexing mailboxes”?

Which app did you use to copy the files?

5

“Indexing Messages in FOLDERNAME” is a rough translation to the German version I see. I copied the files with Finder.

Follow up to what happened: I let EF index a second time. After it was finished (2 hours later) I closed the library, but did not eject the sparsebundle. I reopened the file by double-clicking the .eflibrary-file and guess what? Indexing Messages in FOLDERNAME. Again. With again an estimated time of 2 hours.

6

Does this also happen for the original “EF-Library A”?

Do you see any errors in the Console? Perhaps something is going wrong such that the indexing never completes. (Normally, however, EagleFiler will be able to quickly determine which messages are already in the index and resume indexing where it left off.)

7

I went back to the non-encrypted library, opened it, and was confronted with indexing for the fourth time.

8

Thanks for following up via e-mail. To summarize our conversation:

  1. I’m glad to hear that your non-encrypted 12 GB of mail did finish indexing.
  2. The problems with the encrypted library seem to be due to running out of disk space. There wasn’t enough free space for EagleFiler to add additional messages to the index. I will try to make a future version of EagleFiler better at detecting this situation so that it isn’t a mystery what’s happening.
  3. To avoid this problem, you can choose a larger maximum library size when creating the encrypted library. A library containing primarily text, and using phrase indexing, may need nearly twice the amount of disk space that you think, in order to allow space for the content index and other metadata.