C-Command Software Forum

Problems importing large imap folder of emails

I have an imap folder of about 1000 messages, many of which have attachments. I select them all and hit F1, but I get the Errors window warning of some attachments not being included. I have run the Download Apple Mail Messages - EagleFiler AppleScripts script but I still get the issue. I’ve opened the messages listed in the Errors window but I still get the issue. I’ve done all this multiple times but I can’t seem to get an import that doesn’t complain. Is there anything else I can do?

When you open the messages in Mail, does it look like they are loading all the attachments?

You could also try the third item under Missing Messages and Attachments where it says:

For very large numbers of messages, the script can take a while. A faster option is to Option-drag and drop the entire mailbox to the On My Mac section of Mail’s mailbox list. This will create a local copy of the mailbox with all the messages fully downloaded.

It looks to me like all the attachments are there, yes.

When I try copying the mailbox to On My Mac, the results are much worse. Importing that mailbox’s messages into EagleFiler says that 60 of the messages themselves are missing.

OK, you are probably fine with the initial import then. My guess is that something is slightly wrong with Mail’s data store. If you’d like me to look into this (in case it turns out to be an EagleFiler bug) you could copy a few of the problem messages to a new server mailbox. If you get the same import error from that mailbox, find the mailbox’s folder in ~/Library/Mail/V8 and send me a ZIP archive of it.

Or, you could delete the server mailbox’s folder from V8 (or remove and re-add the account in Mail’s preferences) and this will force Mail to download fresh copies of all the messages. Of course, only do this if you know that the messages are stored correctly on the server.

After making the local copy of the mailbox, I tried importing the messages from the IMAP mailbox once again, and this time there were no error complaints. The number of messages that EagleFiler claims are in the imported mailbox is different from the number of messages that Mail says there are in the mailbox, so I don’t quite know what to think.

I also tried doing a rebuild of the IMAP mailbox, but that made things worse; now when I import into EagleFiler, there are those 60 messages again, this time with missing attachments.

The problematic messages are listed as partial.emlx messages in their mailbox folder on disk. That is what I thought the AppleScript script was supposed to fix, but for some messages it apparently doesn’t.

I can’t send you the whole mailbox but I might be able to send along one of these stubborn partial.emlx messages if you think it might be useful.

This can be because there are duplicates hidden in Mail but shown in EagleFiler or because EagleFiler is removing duplicates.

Deleting the folder on disk is better than rebuilding, in my experience, at making Mail redownload the attachments.

IMAP messages will always be in a .partial.emlx file. The difference is that the script should get Mail to save some additional files in the Attachments folder next to the Messages folder containing the .emlx file.

It might, though it would be more useful if you could (e.g. by moving the message in Webmail) make a test mailbox containing the just that message and then send the whole mailbox folder.

I think you’re absolutely right about the duplicates having something to do with the numeric discrepancy. In that case I think the export has been successful, though it sure was a lot harder than I expected.

I’m not clear on what you mean about .partial.emlx. In this or any of my Imap mailboxes, some messages are .partial, most are not. All of the problematic ones were .partial. I cannot see a pattern to when they are .partial and when not; it isn’t, for instance, whether there is an attachment.

Your explanation about the attachments being separate is also a perfect explanation of why we’d need to make a whole different mailbox in a different venue in order to capture whatever weirdness is going on.

We may never know what the weirdness is/was. Having successfully (I believe) copied the messages into EagleFiler, I think I’m going to stop. Other (smaller) mailboxes on the same server exported just fine, but in general they were not so attachment-filled. The upshot is that we should probably call it a day on this, but it wasn’t wasted; I learned a lot.

One final thought: I think from an instructional point of view, if you were saying that copying the Imap mailbox into On My Mac would cause the messages to be completely downloaded and copied into the On My Mac copy, it might be good to stop giving that advice, as it did not work that way at all; exporting the On My Mac copy into EagleFiler worked much worse than exporting the Imap original, because the latter caused attachment complaints about roughly five messages, whereas the former cause a missing message complaint about 60 messages.

I’ll pass along the diagnostic report, as this includes the logs with the import error messages. The only stuff to look at is the stuff from today, October 8; it all happened today.

I don’t think the pattern is as clear with Big Sur as with some other recent versions of Mail. Some IMAP messages will be .partial and others won’t (even with attachments). What I meant to say is that just because a message is .partial doesn’t mean the attachments aren’t downloaded. It’s quite common for Mail to keep the .partial file and store the attachments as separate files. EagleFiler should be able to find them and stitch it all back together.

Thanks for the feedback. So far, I have not heard from anyone else that it doesn’t work. I just tested dragging a mailbox to On My Mac, with a mix of cached, missing, and partially cached messages, and Mail did the right thing, writing complete messages to the local mailbox.

Thanks! I can confirm that it really does look like some .emlx files were missing, or perhaps had different ID numbers than Mail reported to EagleFiler. A stubborn file (e.g., 46239.partial.emlx) might shed some light on what’s happening with the attachments. Or, I’m happy to call it a day.

Unfortunately 46239.partial.emlx no longer exists; I guess rebuilding the mail folder regenerates the identifiers. I think we should let this rest and just hang on to the data point.