Trying to understand the logs and filter failures

I have upgraded to SS 3.01 on my MacBook with Catalina 10.15.7, using the plug-in (I think!). My MacBook is older and I can’t upgrade the OS further.
here are a couple of questions:

  1. What do the various arrows in the log file “type” column represent?
  2. I uncovered spam messages from an address and they were predicted spam, predicted good, then trained good(auto) and in the Junk folder colored grey and yellow. If SS called them spam, why were they auto trained as good? (3 different messages, a dozen log entries today).
  3. I have some messages (these are mailing list messages) that are all blocklisted by address. Most have an address allowlist entry that is disabled. The same messages appear in both the inbox and Junk. Since they are blocklisted, how do I get them to not show up in the inbox?

That’s explained here.

This means that SpamSieve was asked to process the same message twice, and it changed its opinion. If you look in the Log window it should show (in Info ‣ Origin) why SpamSieve processed the message each time and (in Info ‣ Summary) why it thought the message was spam or good.

It auto-trained them as good after predicting them to be good.

If you train a message as good and it matches any of the blocklist rules, SpamSieve will disable those rules to prevent other good messages from being classified as spam. If you want to continue blocking those addresses, anyway, you can re-enable the blocklist rules and (optionally) lock them so that SpamSieve will leave them enabled even if they contradict the training that you do.

I’m not sure what you mean by “same” here. A single message can’t be in two places at once. You can see in the log why SpamSieve did what it did with any given message.

Please see Example 1 here.

I think I get your answers to the rest of them, Thank you.
This one still stumps me, though:

These 5 messages do appear to be in two places at once, and they each have 2 Predicted spam rules. In both folders they are black letters on a blue background. That indicated that SS properly evaluated them against the blocklist rule, correct? But then why is it also left in the Inbox? Look at these entries for Pendleton Woolen Mills in the screen shot of the Log.

This log entry seems to indicate that the SpamSieve Mail plug-in is not up-to-date. You can update it by clicking Install Plug-In in SpamSieve’s Settings ‣ Apple Mail ‣ Setup window. Then there will be more information in the Origin of each log entry. It will then be interesting to see whether the “duplicate” log entries have different origins.

The two reasons I’ve seen that happen:

  • The rule in Apple Mail is incorrectly set to copy the message instead of moving it.
  • Mail and your mail server encountered an error moving the message. This is more common if the Junk mailbox is not correctly set to use a server mailbox.

I think I figured out the issue. My mail server won’t allow me to turn off the spam filter so some years ago I told the spam filter to use the Inbox as it’s spam output (I guess I was thinking that SS would then correctly deal with the spam). I told the server to prepend a tag to the subject so that I could spot messages that were flagged by the server. This has mostly worked. After reading some of the manual (thanks for the great documentation!) I decided to try having it send the server flagged spam to the same folder that SS uses. Now I see that the server has a lot of false positives for spam. I read that you have created an AppleMail script to recover those back to the Inbox. I think I will give that a try and see how it works.

Sounds like a good plan.

It appears that the script to recover server false positives back to the inbox is partially successful.
When the server puts them in Junk, they appear as brown text on a white background. After the script processes them, they appear (still in Junk) as black text on a white background. If I manually move them to the Inbox and process the rules, SS will correctly classify, move, and color them. But the script doesn’t seem to recover them on it’s own.
I don’t find any logs of those (black on white) that are in the junk folder.
I do see that it takes a cycle or two for the script to change the text color from brown to black, but after several check mail cycles they still remain in Junk.

It sounds like SpamSieve is recognizing that they’re good but that Mail is encountering an error when SpamSieve asks it to move them back to the inbox. It might help to rebuild Mail’s database, but in general there’s not much we can do when Mail runs into a problem like this. macOS 10.15 introduced lots of bugs with Mail’s moving of messages, most of which were fixed in later versions.

They should say Predicted: Good or Predicted: Spam in SpamSieve’s Log window, just like the other messages SpamSieve processes.

I’ll consider rebuilding the database. We have a huge volume of messages in the Inbox (IMAP) folder (37,600) and it scares me to have to download those all again.

That’s the strange thing. I have just looked for a log entry for each of 6 messages that just came in and were placed in the Junk folder (brown text) after I clicked get mail again and they turned to black text. To search for them, I copied the email address from the message and pasted it into the search bar of the log file. Then I looked at the most recent entry for that address. On none of them was the most recent entry today.

Well, it sounds like the rescue script didn’t process them. Maybe the text changed color for some other reason. Please note that Get Mail will only apply the rule if there are new messages in the inbox to download.

You are right. I cleared the checkbox for the rule in Apple Mail settings and then I ran the script from Apple Script Editor. It processed the Junk folder correctly. Almost everything in it became colored and the false positives were moved to the Inbox.
Do you have any suggestions as to how to troubleshoot running the script from the Apple Mail rule?

You can test that the rule works by selecting a message in the inbox and choosing Message ‣ Apply Rules. That should get it to process the Junk mailbox.

It didn’t process the Junk mailbox using that test. I’m baffled.

Please make sure that the rule is Active and that there are unread and uncolored messages in the Junk mailbox. You can also make sure no other rules are interfering by dragging the rule to the top of the list.

If still nothing happens, please use the Save Diagnostic Report command in the Help menu and send me the report file, as described here.

Thanks for sending the report. Everything looks correct in your setup, and I don’t see that Mail is reporting any errors when trying to run the script. If the script that you’re testing in Script Editor is the same one that you have configured for the rule, I don’t know why it would work in one place but not the other.

A few more troubleshooting ideas:

  • Sometimes it helps to a delete a Mail rule and recreate it.

  • The rule runs the script using a different process. Maybe it doesn’t have Automation access in System Preferences ‣ Security & Privacy.

  • You can save the script as a standalone app that runs in the background. Then you would not be reliant on Mail to trigger it.

Didn’t change anything.

I get nervous making changes in system operations. I just want to confirm… in the screenshot below, do you mean that the first box should also be checked?

Yes, otherwise the script (when run by Mail) won’t be able to communicate with SpamSieve.

Success! :tada: :tada:
Thank you!

1 Like

I thought I was done since it seems to be processing messages correctly. However, now when I am looking at the log file (any of the SS windows, actually) I see that it is almost continuously searching. Then, I notice that there are several new log entries classifying messages as spam that were received 4 years ago and are still in the Inbox. These are false positives. When I marked one as good, it took a full minute for the new rule to update.
I noticed as I type this, that the system is “bonking” (this is the sound that I have associated with Mail receiving good messages) several times / minute.
My mail server has been having login issues this morning, is it possible that IMAP is attempting to redownload my entire Inbox?
Sorry about the rambling, but it seems that lots is going on since I made the changes in this thread.

Do you mean that you have entered something in the Search field of all those windows? Or do you see the spinning progress indicator because something just happened and it’s reloading the window?

What does the log entry show as the Origin for those messages that say Predicted: Spam?

What does “rule” refer to here?

Is this this sound that you’ve set in Mail’s preferences or in SpamSieve’s preferences?

Yes, it’s possible that the server is telling Mail that these are new messages that need to be filtered.