I’ve just started using SpamSieve and it works great for filtering mail messages as they come in. But, I have many years of old mailboxes with tens of thousands of messages that I’d like to filter.
I’m finding that if I select about 500 messages at a time, SpamSieve runs through them at about 50 messages per minute, which seems pretty reasonable. For me, that’s about 2 weeks worth of incoming email.
But if I ask it to do 2000 messages, it slows down to maybe 5 messages a minute. And if I ask for much more than that, it’s one message a minute or less.
Is this normal? Or is there something wrong with my setup?
I would not expect the filtering throughput to really change based on the batch size.
Which version of macOS are you using? Which mail client? Which command are you using to tell it to filter the message?
You could record a sample log during the period when it’s slow so that we can see what’s slowing it down. It may also be helpful to sample the mail client. For example, maybe later in the batch are older messages that have not been fully downloaded from the server yet. Or maybe it is bogged down tagging/moving the messages that SpamSieve filtered earlier in the batch.
Also, if you’re going to filter a large batch of messages, it’s best to temporarily turn off the Auto-train as needed option. This will help both the filtering accuracy and the speed.
I’m running Apple Mail 16.0, Mac OS Sonoma 14.4.1 on a Mac Mini M1. To filter the messages, I put them in an ‘on my mac’ mailbox, select them, and use cmd-ctrl-L. SpamSieve and Mail both have full disc access. I’ve tried Auto-train both on and off, it doesn’t make much difference.
I took a sample log of SpamSieve and sent it to you by email. SpamSieve is consuming about 100% of a core while this is going on, while Mail is at 0.1%, so I didn’t take a sample of Mail.
The sample log seems show that macOS is bogged down formatting an error message. You may be able to see the error in SpamSieve’s Log window, or I can take a look if you use the Save Diagnostic Report command in the Help menu and send me the report file, as described here.
Thanks for the error log. The missing file errors:
CoreData: error: Failed to clone external data reference from
are not related to this. And the files are not actually missing. The errors are not actually errors and are being mistakenly reported due to a macOS bug.
I don’t see any errors from today that are related to the Filter Messages command. There are a few places where SpamSieve ignores errors, so I will look into whether I can add logging for those cases to see whether that might be related the slowness you’re seeing.
One possibility is that the error is related to SpamSieve finding where the message is when it’s stored in an On My Mac mailbox. Do you see the same slowness if you Filter Messages before moving them to On My Mac?
Yes, I saw the same slow-down when filtering a lot of messages in my inbox. So I decided to get organized, move the bulk of the messages into ‘on my mac’ folders by year, and filter within those mailboxes. It didn’t seem to make any difference.
So it seems that among all your users, I’m probably the only one seeing this? So it’s something peculiar to me?
I will explain a little more about how I’ve gotten to this point. Perhaps I’ve done something odd, to cause the problem.
Until recently, I was running Monterey on an iMac 27" Retina 2015 with 1 terabyte fusion drive. I had accumulated over 500 gigabytes of data on that drive, including 100 gigabytes in my ~Library/Mail folder. I have about 290,000 messages accumulated since 1993. During all that time, I did a little spam filtering by hand, but mostly I just let it all flow in.
Then I got a good deal on this M1 Mac with 256 gig SSD, so I needed to migrate most of my old data to an external drive. I wanted to keep my user folder on the internal drive for best performance, while archiving stuff that never changes.
For some reason, moving my “on my mac” folders from ~Library/Mail/… onto the external SSD, and using aliases within ~Library/Mail/, did not work. Mac Mail did not recognized the external folders. So I wound up re-importing the entire directory from a Time Machine backup, and Mail repeated the process of upgrading the folders from V9 to V10.
Then I got myself a copy of Mail Archiver X, made an archive of the 230,000 oldest messages, and deleted the older mailbox folders from my internal SSD. Which all seems to be working OK.
No one else has reported it, but I doubt that many people are manually feeding large numbers of old messages to SpamSieve. Usually it just filters the new messages as they come in.
I’ll try to test what you’re on my Mac and see whether the same thing happens.
It’s also possible that the sample that you recorded was not typical and that if you recorded more from Mail and SpamSieve they would show some other area as the bottleneck.
It should work with a symlink instead of an alias.
Thanks for the tip! But, I couldn’t get this to work either. Finder sees the symbolic link as if it were an alias, and double-clicking on the alias opens the folder on the archive drive as expected. So, I think I’m setting up the symlink correctly. Mail app still doesn’t see it.
I’m curious about your result? Does your Mac do the same thing, os is it some magic unique to my system?
I’m still investigating this. But so far I’ve found that I can run an AppleScript that doesn’t even involve SpamSieve, just Mail, and make it get very slow in the same way (same type of sample log). Script Editor will even hang and not let me stop running the script. So it seems there’s some sort of Mail/AppleScript issue here, and I’m going to try to figure out a way to avoid it.
Very good question! I am expecting the linked mailboxes to appear in the sidebar, in the list of “On My Mac” mailboxes. But maybe there’s something else I need to do, to make them visible?
The mailboxes shown are determined by Mail’s database. You have to use the File ‣ Import Mailboxes… command if you want to add mailboxes, as this is what creates the database entries. You can’t just put mailbox folders (symlink or not) in Mail’s folder in Finder.