Bounce Server not Processing Bounce Email Sent by Amazon SES

pradeep sharma

pradeep sharma

Active Member
Hi Team,
I am facing an issue with processing bounces, I have a mailbox that receives bounces by Amazon SES.
It deletes but does not process or mark bounce in the campaign or list.


here is the verbose of Bounce Handler (I have run the bounce processor without deleting email option) but when we enable delete option is deletes all emails

Code: 
[2024-04-21 22:30:06] - Done scanning message ID: 5011
[2024-04-21 22:30:06] - Scanning message ID: 5012
[2024-04-21 22:30:06] - Done scanning message ID: 5012
[2024-04-21 22:30:06] - Scanning message ID: 5013
[2024-04-21 22:30:06] - Done scanning message ID: 5013
[2024-04-21 22:30:06] - Scanning message ID: 5014
[2024-04-21 22:30:06] - Done scanning message ID: 5014
[2024-04-21 22:30:06] - Scanning message ID: 5015
[2024-04-21 22:30:06] - Done scanning message ID: 5015
[2024-04-21 22:30:06] - Scanning message ID: 5016
[2024-04-21 22:30:06] - Done scanning message ID: 5016
[2024-04-21 22:30:06] - Scanning message ID: 5017
[2024-04-21 22:30:06] - Done scanning message ID: 5017
[2024-04-21 22:30:06] - Scanning message ID: 5018
[2024-04-21 22:30:06] - Done scanning message ID: 5018
[2024-04-21 22:30:06] - Scanning message ID: 5019
[2024-04-21 22:30:06] - Done scanning message ID: 5019
[2024-04-21 22:30:06] - Scanning message ID: 5020
[2024-04-21 22:30:06] - Done scanning message ID: 5020
[2024-04-21 22:30:06] - Scanning message ID: 5021
[2024-04-21 22:30:06] - Done scanning message ID: 5021
[2024-04-21 22:30:06] - Scanning message ID: 5022
[2024-04-21 22:30:06] - Done scanning message ID: 5022
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 1.
[2024-04-21 22:30:06] - Acquiring lock for server ID 1.
[2024-04-21 22:30:06] - Lock for server ID 1 has been acquired.
[2024-04-21 22:30:06] - Searching for results in the "Junk" mailbox...
[2024-04-21 22:30:06] - Opening the IMAP connection.
[2024-04-21 22:30:06] - Connection opened successfully!
[2024-04-21 22:30:06] - Searching using following search string: UNDELETED SINCE "21-Feb-2024"
[2024-04-21 22:30:06] - Found 0 search results.
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 1.
[2024-04-21 22:30:06] - Acquiring lock for server ID 1.
[2024-04-21 22:30:06] - Lock for server ID 1 has been acquired.
[2024-04-21 22:30:06] - Searching for results in the "Spam" mailbox...
[2024-04-21 22:30:06] - Opening the IMAP connection.
[2024-04-21 22:30:06] - Connection opened successfully!
[2024-04-21 22:30:06] - Searching using following search string: UNDELETED SINCE "21-Feb-2024"
[2024-04-21 22:30:06] - Found 0 search results.
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 1.
[2024-04-21 22:30:06] - Finished processing server ID 1.
[2024-04-21 22:30:06] - Started processing server ID 2.
[2024-04-21 22:30:06] - Acquiring lock for server ID 2.
[2024-04-21 22:30:06] - Lock for server ID 2 has been acquired.
[2024-04-21 22:30:06] - Searching for results in the "INBOX" mailbox...
[2024-04-21 22:30:06] - Opening the IMAP connection.
[2024-04-21 22:30:06] - Connection opened successfully!
[2024-04-21 22:30:06] - Searching using following search string: UNDELETED SINCE "21-Feb-2024"
[2024-04-21 22:30:06] - Found 0 search results.
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 2.
[2024-04-21 22:30:06] - Acquiring lock for server ID 2.
[2024-04-21 22:30:06] - Lock for server ID 2 has been acquired.
[2024-04-21 22:30:06] - Searching for results in the "Junk" mailbox...
[2024-04-21 22:30:06] - Opening the IMAP connection.
[2024-04-21 22:30:06] - Connection opened successfully!
[2024-04-21 22:30:06] - Searching using following search string: UNDELETED SINCE "21-Feb-2024"
[2024-04-21 22:30:06] - Found 0 search results.
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 2.
[2024-04-21 22:30:06] - Acquiring lock for server ID 2.
[2024-04-21 22:30:06] - Lock for server ID 2 has been acquired.
[2024-04-21 22:30:06] - Searching for results in the "Spam" mailbox...
[2024-04-21 22:30:06] - Opening the IMAP connection.
[2024-04-21 22:30:06] - Connection opened successfully!
[2024-04-21 22:30:06] - Searching using following search string: UNDELETED SINCE "21-Feb-2024"
[2024-04-21 22:30:06] - Found 0 search results.
[2024-04-21 22:30:06] - Closing the IMAP connection.
[2024-04-21 22:30:06] - IMAP connection closed.
[2024-04-21 22:30:06] - Found 0 results.
[2024-04-21 22:30:06] - No results!
[2024-04-21 22:30:06] - Releasing lock for server ID 2.
[2024-04-21 22:30:06] - Finished processing server ID 2.
[2024-04-21 22:30:06] - Processing finished.


Please help!!
 
twisted1919 said:
For Amazon SES, for best results, you should use the web api implementation not the smtp one, see https://www.mailwizz.com/kb/adding-a-new-amazon-ses-delivery-server-web-api/
Click to expand...
Currently, I am not using the Web API for Amazon SES as recommended in your guide (https://www.mailwizz.com/kb/adding-a-new-amazon-ses-delivery-server-web-api/), due to specific requirements of our setup.

Instead, I am utilizing a local PMTA server that routes emails to various SMTPs globally. This approach has significantly improved the processing speed of our Mailwizz system, boosting it by 100 times. It manages queues and connection limits effectively, addressing the challenges posed by Mailwizz’s single-threaded nature and latency issues between servers.

Typically, Mailwizz would take approximately 1.5 seconds to process each email from Europe to various Amazon SES regions. However, I have developed a workaround that increases the throughput to an impressive rate, tested on an 8-core server.

Additionally, I have extensively customized Mailwizz version 1.3 to handle billions of emails per hour, and we are also exploring the latest versions to enhance our capabilities further.

While I await the release of your high-speed extension, I hope you can assist me in resolving the current issues with our bounce server to maintain compliance with Amazon's policies.

Thank you for your attention to this matter.

Best regards,

Pradeep Sharma
 
Can you look inside the received bounce messages, for the message id (might be multiple) and check those ids against you campaign_delivery_log table in the email_message_id column? Strip the < and > if it exists.
As an example, if you get header like:
Code: 
Message-ID: <somerandomtstring@somedomain.com>
Run this query against the database:
Code: 
SELECT * FROM `mw_campaign_delivery_log` WHERE email_message_id='somerandomtstring@somedomain.com';
Does it return any results? If it does, what's the date added in the database compared with the date the bounce handler is extracting messages?
 
Hi @twisted1919
I further investigated the issue and discovered that the message ID generated and stored in the mw_campaign_delivery_log is not present in the DSN Notification sent by SES. I will send a copy of the DNS & Email received by SES at the bounce mailbox via a private message.

We need to find a better approach to handle this situation.

Using Message ID for Bounce Processing is effective as long as the Message ID is returned in the DNS. This method works well with native email systems like Exim/Postfix or PMTA. However, many SMTP providers like SES modify this Message ID and use their own from the return-path domain. This alteration helps them receive the bounce notifications first, which are then forwarded to us, the customers or the email senders.

In response, we might need to introduce additional headers or adapt our bounce processing methods. For instance, identifying a subscriber by their email address, as we currently do with our mailbox monitor, could be a viable alternative.

I also noticed that we previously included several headers such as subscriber_uid and campaign_uid, but these headers are no longer added by default. Reintroducing these headers could assist in better subscriber identification.
 
That's why we recommended using the web api for Amazon, because it does bounce processing properly with the info amazon sends.
If the original message-id is not part of the returned message, there's nothing much we can do, we're not going to change how bounce processing works, you'll have to implement bounce processing on your own in this case. In the past, indeed, we used to have multiple headers to identify a subscriber, but this caused a lot of issues, and generally, any additional header is a potential issue, and in this case I don't think it wouldn't help since no original header is included in the returned message.
pradeep sharma said:
In response, we might need to introduce additional headers or adapt our bounce processing methods. For instance, identifying a subscriber by their email address, as we currently do with our mailbox monitor, could be a viable alternative.
Click to expand...
You can add custom headers to your delivery server if that's what you need. And then try to see if those headers are preserved when the message is returned. And then fetch the right subscriber based on it.

pradeep sharma said:
identifying a subscriber by their email address
Click to expand...
This is not really a solution, knowing the email is not enough to identify the right subscriber, since a subscriber can be part of many campaigns running at same time.
 
@twisted1919
We can consider implementing a feature where we utilize a specific tag for the email message ID that retrieves the value from the zm_serv_app2_campaign_delivery_log table, similar to using [EMAIL-MESSAGE-ID]. This could be incorporated as follows:


Code: 
x-mw-message-id: [EMAIL-MESSAGE-ID]

This setup would output the actual message ID, for example:


Code: 
x-mw-message-id: f1f8b1e829792c5feca45403bdf305350edad552@bounce-domain.com

We could then direct the bounce processor to extract information from this custom header instead of the default message-ID header. Specifically, we could offer the option to use a custom header for bounce processing while maintaining the default value as 'Message-Id'. This adjustment would not disrupt the original flow of how the bounce processor operates and would enhance Mailwizz's capability to handle bounces based on custom header values.

Furthermore, these custom headers can be configured within the delivery servers, allowing users the flexibility to add and utilize these headers for bounce processing as needed.

Please Please Please Consider this
 
FYI all other x header included in email messages are present in copy of email sent as DNS, so adding another x-header with message-id header will do the trick
 
Yeah, but are you sure that header is part of the returned message? Try with a custom header with whatever random value first and let me know.
 
OK, I ll DO the Following.

1. Will add a Custome Header to Mail Sent by SES
2. will check bounce email manually by login into Mailbox

But to be double sure, I have given you access of my bounce mailbox once done i ll ask you to kindly verify the headers by querying the mailbox over IMAP so that you can get and process the headers.




Regards

Pradeep
 
Hi @twisted1919
I have done the testing and i have added a static message-id header i. e

Code: 
x-message-id: messageid-present-in-delivery-server-log

and i found that the same was present in a copy of the bounce email received via SES DNS.

Code: 
x-message-id: messageid-present-in-delivery-server-log

I have sent you a personal message with credentials of my bounce server. kindly look and verify by imap

Regards

Pradeep
 
@pradeep sharma - thanks for confirming. I'll take a look at all this on Monday since that's when @ghimes returns from his days off and we can discuss some potential adjustments related to what you described above.
 
@twisted1919, did your team get time to look into the adjustment in bounce processing? We can achieve this bounce server can process bounces from any type of bounce server based on DSN.

Pleased consider my request!!
 
I am also facing the same issue but I am using Web API.

I thought the hosting server was blocking, so I moved the app to the AWS EC2 Server and disabled the UFW firewall on Ubuntu. Still, the bounce is always 0.

I thought it was the SNS Issue, so I deleted everything and recreated the Delivery Servers. There is still no update on the bounces.

Please need your help on this.
 
You must log in or register to reply here.
Back
Top