Search Limitations and Performance with Large .eml Email Library

Hi,

I’ve recently imported my email archive into EagleFiler — it’s around 100GB in size, with approximately 220,000 individual email files.

Since then, I’ve noticed that searches are very slow. Often when I initiate a search, the app becomes unresponsive (shown as “Not Responding” in Activity Monitor). However, I can see ongoing disk activity, and sometimes EagleFiler recovers after reading through 60GB or more of data.

Is this archive size too large for a single library? Would it be better to split the library into smaller ones? If so, what would be the recommended maximum size — in both GB and number of messages — for email archiving in EagleFiler?

Another challenge I’ve encountered is with search results. EagleFiler seems to search only email headers — such as subject, sender, and dates — but not the body content or attachments. Is there a way to enable full-text search within email bodies and attachments? Or is this a limitation due to the file type or format of the imported messages?

Thanks in advance for any advice or best practices on optimizing performance and search capabilities with large email libraries.

Best,
Luís Trevisan

It’s much more efficient for EagleFiler to store large amounts of mail using mailbox files rather than individual message files. A search of that amount of mail using mailboxes should read much less data and be almost instant. (In brief, with .eml files EagleFiler has to read the message data from each individually and also store more data in its database and in memory, whereas with mailboxes it can read a single, small table of contents file for each whole mailbox and the information held in memory is highly optimized.)

It’s fastest if you can import the mail directly as mailbox files, but if you already have the .eml files imported you can use the Merge Message Files command to combine them into one or more mailboxes.

If you do an Anywhere search, that should include the message bodies and attachment contents. This will be the same with mailboxes or .eml files. If the search isn’t working as you expect, you could create a small test library to send us and report what you are searching for and expecting it to find so we can investigate.

Hello, thank you for the quick reply.

I’ve converted the messages to mbox format, but the search behavior remains the same. I can search by sender, subject, and date, but the body content of the emails still doesn’t appear in the results.

I’ve prepared a small sample library for you to analyze. How can I send it to you?

Thank you very much for your help — I really want to get this issue resolved because I’m enjoying the app a lot.

PS: I’ve been using SpamSieve for a while — it’s absolutely fantastic, a true work of art.

You can send it to eaglefiler@c-command.com.

I’ve just sent the email with the sample library as discussed.
Thank you again for your support.

Thanks. It looks like you closed the test library before EagleFiler had indexed the mailbox. You can see the progress in the Activity window. When I opened the library, it built the index and searches seemed to work as expected, finding various words in the message bodies. The e-mail attachments didn’t contain much searchable text because they were PNG files and PDF files that were images, not text. And there’s something weird about the PDF files—macOS’s built-in OCR does not seem to be able to extract any text from them in Preview, either.

After that, I was able to solve the issue by following these steps:

  1. Removed an app
  2. Reinstalled it
  3. Rebuilt the libraries from scratch

Now everything is working perfectly. The indexing is accurate, and I can search emails by sender, recipient, and message body without any issues.

Thanks for your help! I love EagleFiler and I’m excited about what’s coming in the Apple Silicon version.

P.S.: I still had 11 days left on the trial, but I was so satisfied that I purchased the license today!

1 Like