Amazon SES Web Api - SNS stuck in pending confirmation

M

mwizz98

Member
Hello, I migrated to a new server and everything's working perfectly except Amazon SES Web Api. The bounces do not show up in Mailwizz.

I've created a new delivery server, and it is creating a new topic and subscription as expected, but the subscription says "Pending Confirmation" and never gets confirmed. I've tried many times but I can't get Mailwizz to confirm the subscription.

I've tried to delete all Amazon delivery servers, all Amazon users, and re-create everything, but it still doesn't work.

The Amazon user has full SES and full SNS access and power user access. I've tried with many users.

Mailwizz is clearly able to reach Amazon since it is creating the SNS topics. Amazon is reaching mailwizz just fine as well, I see it in the logs:

172.71.150.67 - - [17/Mar/2024:04:04:39 +0000] "POST /m/index.php/dswh/29 HTTP/1.1" 200 - "-" "Amazon Simple Notification Service Agent"

When I test the dswh endpoint in a browser, it says error 500, which is the expected behavior. it is reachable from anywhere. Amazon has no problem reaching the endpoint, but Mailwizz is not confirming. I tested Mailgun and it works perfectly, it's just Amazon that Mailwizz is having trouble setting up.

How can I fix this? Please provide some troubleshooting steps. Thank you.

@twisted1919
 
Last edited:
Thank you for your response. Yes I followed the instructions very precisely and have used Mailwizz + SES for years with no problems. I know how to set up the delivery servers correctly, no worries there.

I cannot provide access to this server, but I can run any commands or tests etc. Could you provide me with some troubleshooting steps I can try? In the logs there is the incoming POST request from Amazon but Mailwizz doesn't seem to be putting together a response URL.
 
For a start, you could delete the application logs from backend > misc > application log, then try to validate the server so you get the POST request, then go back to the logs page and reload the page and see if anything has been logged in the app and if it did, add here the log.
 
Here's what I did:

I installed a second instance of Mailwizz on the same server just to test, clean install just in a different folder, and went through all the set up steps correctly (packages and permissions are all correct), then added an Amazon SES delivery server, but the problem remains. The subscriptions still say "Pending confirmation" and nothing shows up in the application logs in install #1 or #2

Other things I've tried:

Open all ports and protocols on the server to all traffic from the internet
Turn off SElinux

Still doesn't work.

The POST request from Amazon shows up in the httpd logs so it is reaching Mailwizz, but Mailwizz isn't processing it for some reason.

The dswh endpoint is alive, when I test it in a web browser it says:

Error 500!​

SNS message type header not provided.
Click to expand...

I would love to figure this out, thank you for your help.
 
@mwizz98 - let us run some tests on our end and see how it works for us and we'll get back to you.
@laurentiu - can you please test this today.tomorrow and write here the steps you took and the results?
 
Sure thank you. And just to add more info, the problem is not just confirming a topic subscription. When I first migrated to the new server, it had SES already set up and topics already subscribed, it could send like usual but but it couldn't get any bounces like on the old server. So apparently the problem is that the server can communicate with SNS the way it's programmed to, like creating topics, but it cannot process information it receives from SNS.

Also the server is in Google Cloud. I don't see how that makes any difference since SNS requests are hitting the logs but I thought I'd mention it.

If we can do any println debugging to the application log or anything like that just give me the code and let me know where to add it.
 
Quick update on my end:

I added log statements to DswhController.php in order to see what's going on

PHP: 
public function processAmazonSes($server)
    {
        $message   = Aws\Sns\Message::fromRawPostData();
        $validator = new Aws\Sns\MessageValidator([$this, '_amazonFetchRemote']);

        try {
            $validator->validate($message);
        } catch (Exception $e) {
            Yii::log($e->getMessage(), CLogger::LEVEL_ERROR);
            app()->end();
            return;
        }

        if ($message['Type'] === 'SubscriptionConfirmation') {
            try {
                $types  = DeliveryServer::getTypesMapping();
                $type   = $types[$server->type];
                $server = DeliveryServer::model($type)->findByPk((int)$server->server_id);
                $result = $server->getSnsClient()->confirmSubscription([
                    'TopicArn'  => $message['TopicArn'],
                    'Token'     => $message['Token'],
                ]);

// Log: Convert the result to an array and then to JSON string for logging
        $resultArray = $result->toArray();
        $resultJson = json_encode($resultArray, JSON_PRETTY_PRINT);
        // Prepare the log message
        $logMessage = "Subscription confirmation result: " . PHP_EOL . $resultJson . PHP_EOL;
        // Append the result log to the file
        file_put_contents('/tmp/sns.log', $logMessage, FILE_APPEND);


                if (stripos($result->get('SubscriptionArn'), 'pending') === false) {
                    $server->subscription_arn = $result->get('SubscriptionArn');
                    $server->save(false);
                }
                app()->end();
                return;
            } catch (Exception $e) {
            }



// Log SubscribeURL to /tmp/file.log
    $subscribeURL = (string)$message['SubscribeURL'];
    $logMessage = "SubscribeURL: $subscribeURL\n"; // Preparing the log message
    file_put_contents('/tmp/sns.log', $logMessage, FILE_APPEND); // Appending to the log file




            $client = new GuzzleHttp\Client();
            $client->get((string)$message['SubscribeURL']);
            app()->end();
            return;
        }
    
    //rest of the code here...
        
public function _amazonFetchRemote($url)
{
    // Specify the path to sns.log - adjust the path as necessary for your environment
    $logFile = '/tmp/sns.log';

    // Log the incoming request URL
    $logMessage = '[' . date('Y-m-d H:i:s') . '] Received request to fetch URL: ' . $url . "\n";
    file_put_contents($logFile, $logMessage, FILE_APPEND);
    
    try {
        $content = (string)(new GuzzleHttp\Client())->get($url)->getBody();

        // Log the successful fetch of the URL content
        $logMessage = '[' . date('Y-m-d H:i:s') . '] Successfully fetched URL content: ' . $url . "\n";
        file_put_contents($logFile, $logMessage, FILE_APPEND);

    } catch (Exception $e) {

        // Log the exception if the fetch fails
        $logMessage = '[' . date('Y-m-d H:i:s') . '] Exception fetching URL ' . $url . ': ' . $e->getMessage() . "\n";
        file_put_contents($logFile, $logMessage, FILE_APPEND);

        $content = '';
    }
    
    return $content;
}

Here's what gets logged when I add new Amazon SES delivery servers:

HTTP: 
2024-03-20 15:02:40] Received request to fetch URL: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem
[2024-03-20 15:02:41] Successfully fetched URL content: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem
[2024-03-20 15:15:00] Received request to fetch URL: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem
[2024-03-20 15:15:00] Successfully fetched URL content: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem

I don't know if I'm poking around in the right place, but it looks like it's just getting a pem file for SSL. It's not printing anything for $result or SubscribeURL.

I hope this is helpful and that you can make sense of it. The SNS subscriptions still always say "Pending confirmation." Let me know what you think.
 
mwizz98 said:
It's not printing anything for $result
Click to expand...
Might be because its throwing an exception which we silently ignore.
In this block:
PHP: 
try {
                $types  = DeliveryServer::getTypesMapping();
                $type   = $types[$server->type];
                $server = DeliveryServer::model($type)->findByPk((int)$server->server_id);
                $result = $server->getSnsClient()->confirmSubscription([
                    'TopicArn'  => $message['TopicArn'],
                    'Token'     => $message['Token'],
                ]);

                if (stripos($result->get('SubscriptionArn'), 'pending') === false) {
                    $server->subscription_arn = $result->get('SubscriptionArn');
                    $server->save(false);
                }
                app()->end();
                return;
            } catch (Exception $e) {
            }
Maybe log the exception in the catch() block, and see if that logs anything meaningful.

Afaik, @laurentiu checked this in the morning by following the tutorial, and everything worked just fine, so whatever it is, it might be specific to your environment.
 
I added the log statement, it logs nothing:

PHP: 
catch (Exception $e) {
        // It might be useful to also log the exception here
        $errorMessage = "Error during subscription confirmation: " . $e->getMessage() . PHP_EOL;
        file_put_contents('/tmp/sns.log', $errorMessage, FILE_APPEND);
    }

Still got the same thing:

HTTP: 
[2024-03-21 12:52:09] Received request to fetch URL: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem
[2024-03-21 12:52:09] Successfully fetched URL content: https://sns.us-west-2.amazonaws.com/SimpleNotificationService-<redacted>.pem

I wanted to see if the $message was even valid and reaching the application, so I logged it:

PHP: 
public function processAmazonSes($server)
    {
        $message   = Aws\Sns\Message::fromRawPostData();
 
    // Convert the $message to a JSON string for logging
$messageJson = json_encode($message->toArray(), JSON_PRETTY_PRINT);
// Prepare the log message
$logMessage = "Received SNS Message: " . PHP_EOL . $messageJson . PHP_EOL;
// Append the message log to the file
file_put_contents('/tmp/sns.log', $logMessage, FILE_APPEND);
 
 
        $validator = new Aws\Sns\MessageValidator([$this, '_amazonFetchRemote']);

And now we finally have something in the logs:

Code: 
"TopicArn": "<redacted>",
    "Message": "You have chosen to subscribe to the topic <redacted>.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
    "SubscribeURL": "https:\/\/sns.us-west-2.amazonaws.com\/?Action=ConfirmSubscription&TopicArn=<redacted>&Token=<redacted>",
    "Timestamp": "2024-03-21T13:07:38.227Z",
    "SignatureVersion": "1",
    "Signature": "<redacted>",
    "SigningCertURL": "https:\/\/sns.us-west-2.amazonaws.com\/SimpleNotificationService-<redacted>.pem"
}

The ARN and Token and SubscribeURL are all valid. The SNS POST request looks as it should. When I go to the SubscribeURL in a web browser, the subscription changes to Confirmed, as it should. (But of course that doesn't really help because when I use the delivery server to send a campaign, it gets no bounces, I tested.)

Why isn't Mailwizz processing it as intended? It looks like it doesn't even go into
PHP: 
if ($message['Type'] === 'SubscriptionConfirmation')
because the log statements there are never printed. It could be an issue with the validator, maybe a missing package or something, just guessing.

Please let me know your thoughts, I think we are very close to figuring this out.
 
UPDATE #2:

I logged the validator exception here

PHP: 
try {
    $validator->validate($message);
} catch (Exception $e) {
    // Log the error message to Yii's logging system
    Yii::log($e->getMessage(), CLogger::LEVEL_ERROR);

    // Prepare a more detailed log message including the exception details
    $errorLogMessage = "Error during message validation: " . $e->getMessage() . PHP_EOL;
    $errorLogMessage .= "Stack trace: " . PHP_EOL . $e->getTraceAsString() . PHP_EOL;
    // Append the detailed error log to the /tmp/file.log
    file_put_contents('/tmp/sns.log', $errorLogMessage, FILE_APPEND);

    app()->end();
    return;
}

and it looks like that's where things are failing:

Code: 
Error during message validation: The message signature is invalid.
Stack trace:
#0 /var/www/app/apps/frontend/controllers/DswhController.php(607): Aws\Sns\MessageValidator->validate()
#1 /var/www/app/apps/frontend/controllers/DswhController.php(69): DswhController->processAmazonSes()
#2 [internal function]: DswhController->actionIndex()
#3 /var/www/app/vendor/yiisoft/yii/framework/web/actions/CAction.php(114): ReflectionMethod->invokeArgs()
#4 /var/www/app/vendor/yiisoft/yii/framework/web/actions/CInlineAction.php(47): CAction->runWithParamsInternal()
#5 /var/www/app/vendor/yiisoft/yii/framework/web/CController.php(308): CInlineAction->runWithParams()
#6 /var/www/app/vendor/yiisoft/yii/framework/web/CController.php(286): CController->runAction()
#7 /var/www/app/vendor/yiisoft/yii/framework/web/CController.php(265): CController->runActionWithFilters()
#8 /var/www/app/vendor/yiisoft/yii/framework/web/CWebApplication.php(282): CController->run()
#9 /var/www/app/vendor/yiisoft/yii/framework/web/CWebApplication.php(141): CWebApplication->runController()
#10 /var/www/app/vendor/yiisoft/yii/framework/base/CApplication.php(185): CWebApplication->processRequest()
#11 /var/www/app/apps/init.php(221): CApplication->run()
#12 /var/www/app/index.php(18): require_once('...')
#13 {main}

The message signature looks like this and appears to be ok:

Code: 
"Signature": "SZxCsKe6mxLu+y7hpMYHJP6rCcULqHOfGocL6eFf2lycUszmnUIJ7kJ5cCyKjp4Y5winiLB3RHKWEcFPLYCfX0TUPfzoeNjN68IQs+0tdXLU1VLVkckaREqpOIEGgtpdO8UfxGVcLVnrRO8DZ3GzplINZe3aUtOEOx2umVo4WKPY0pxZssSLUufo\/dMFDPBSGCp788SV1oVm1z+UNuNxfOWY57nbASzSos6m2bGr5F2FeCyyrb98lQAFhBEL2hHWmx6fnViLvPlEBLAHDUYntEfsmUGvDEaABu6jcvh65KT++EX1Ktt6RNWZpeeKiI6Rou6vww1zqYHcDTFWaS\/A7Q=="

Why is it saying that the signature is invalid and how can that be fixed? Thank you.
 
Last edited:
Thank you for your response. I just checked and it seems to be fine, already using chrony, it matches very precisely the timestamp of the SNS request, I don't see any discrepancy

Bash: 
~]$ sudo systemctl status chronyd
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-03-15 03:44:08 UTC; 6 days ago
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 656 (chronyd)
      Tasks: 1 (limit: 48836)
     Memory: 2.6M
        CPU: 370ms
     CGroup: /system.slice/chronyd.service
             └─656 /usr/sbin/chronyd -F 2

Bash: 
~]$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* metadata.google.internal      2  10   377   917  -8470ns[-9876ns] +/-  137us

Bash: 
~]$ chronyc tracking
Reference ID    : A9FEA9FE (metadata.google.internal)
Stratum         : 3
Ref time (UTC)  : Thu Mar 21 17:02:15 2024
System time     : 0.000002308 seconds slow of NTP time
Last offset     : -0.000001085 seconds
RMS offset      : 0.000005785 seconds
Frequency       : 68.598 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.001 ppm
Root delay      : 0.000105715 seconds
Root dispersion : 0.000140394 seconds
Update interval : 1041.3 seconds
Leap status     : Normal

I changed the timezone to America/New York, which is what I selected when I installed Mailwizz but it still doesn't work, still invalid signature.

Bash: 
~]$ timedatectl
               Local time: Thu 2024-03-21 13:25:21 EDT
           Universal time: Thu 2024-03-21 17:25:21 UTC
                 RTC time: Thu 2024-03-21 17:25:21
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

I feel like we've very close, what else could it be?
 
UPDATE:

I poked around the MessageValidator.php code and dumped some info to logs to see what's going on, everything looks perfect, I expected that but I still was very puzzled. I logged the $algo and it says Algorithm: SHA1

I am running Rocky 9, so I did some digging and found this:

forums.rockylinux.org

Php 8.0.13 openssl_verify() fails on Rocky 9

So, Rocky 8 has been solid, but I just received a bug report at cacti.net that the function is failing on Rocky 9. So, I was able to reproduce, and so switched from the built in PHP to the REMI based PHP 8.1.11, and the openssl_verify() fails there too. I took the same PHP 8.1.11 to CentOS7...
forums.rockylinux.org forums.rockylinux.org

www.redhat.com

Enhancing RHEL Security: Understanding SHA-1 deprecation on RHEL 9

In this article, I’ll go over some typical problems users may face with Fedora SHA-1 status (including some possible workarounds), and how you can update your infrastructure to use a more secure SHA-256.
www.redhat.com www.redhat.com

SHA1 is basically deprecated across Redhat and related Linux OS variants.

Is this something you could update in the application so it uses SHA256 instead? I think every day you might encounter a customer who tries to install Mailwizz on a newer Linux OS and they'll go crazy trying to figure out why Amazon SES is not working.

Thank you!
 
Looking for that error message and in your error stack trace, things point to /vendor/aws/aws-php-sns-message-validator/src/MessageValidator.php:
PHP: 
public function validate(Message $message)
    {
        if (self::isLambdaStyle($message)) {
            $message = self::convertLambdaMessage($message);
        }

        // Get the certificate.
        $this->validateUrl($message['SigningCertURL']);
        $certificate = call_user_func($this->certClient, $message['SigningCertURL']);
        if ($certificate === false) {
            throw new InvalidSnsMessageException(
                "Cannot get the certificate from \"{$message['SigningCertURL']}\"."
            );
        }

        // Extract the public key.
        $key = openssl_get_publickey($certificate);
        if (!$key) {
            throw new InvalidSnsMessageException(
                'Cannot get the public key from the certificate.'
            );
        }

        // Verify the signature of the message.
        $content = $this->getStringToSign($message);
        $signature = base64_decode($message['Signature']);
        $algo = ($message['SignatureVersion'] === self::SIGNATURE_VERSION_1 ? OPENSSL_ALGO_SHA1 : OPENSSL_ALGO_SHA256);
        if (openssl_verify($content, $signature, $key, $algo) !== 1) {
            throw new InvalidSnsMessageException(
                'The message signature is invalid.'
            );
        }
    }
Specifically to the
PHP: 
if (openssl_verify($content, $signature, $key, $algo) !== 1) {
part.

Looking in the docs for the openssl_verify function at https://www.php.net/openssl_verify it says
Returns 1 if the signature is correct, 0 if it is incorrect, and -1 or false on error.
Click to expand...
So it's 1 if correct, 0 if incorrect. But also -1 or false if it fails for whatever reason, which I think it is what happens, but you might want to check for that as well.
 
Ah, I see your reply now.

mwizz98 said:
Is this something you could update in the application so it uses SHA256 instead?
Click to expand...
We're not setting sha1 either, that's something set in the php-sdk from amazon as far as I can tell, but let me check a few things and get back to you.
 
Let's try this, please open /apps/common/models/DeliveryServerAmazonSesWebApi.php and look for
PHP: 
public function getSesClient(): Aws\Ses\SesClient
    {
        static $clients = [];
        $id = (int)$this->server_id;
        if (!empty($clients[$id])) {
            return $clients[$id];
        }

        return $clients[$id] =  new Aws\Ses\SesClient([
            'region'  => $this->getRegionFromHostname(),
            'version' => '2010-12-01',
            'credentials' => [
                'key'     => trim((string)$this->username),
                'secret'  => trim((string)$this->password),
            ],
        ]);
    }
and add a custom signature version:
PHP: 
public function getSesClient(): Aws\Ses\SesClient
    {
        static $clients = [];
        $id = (int)$this->server_id;
        if (!empty($clients[$id])) {
            return $clients[$id];
        }

        return $clients[$id] =  new Aws\Ses\SesClient([
            'region'  => $this->getRegionFromHostname(),
            'version' => '2010-12-01',
            'credentials' => [
                'key'     => trim((string)$this->username),
                'secret'  => trim((string)$this->password),
            ],
            'signature_version' => 'v4', // we've added this
        ]);
    }

Save the file and try again, does it work like this?
My understanding is/was that the sdk already uses v4 for signing....
 
Also, reading the docs seems that you also need to generate new access credentials, so maybe that's the problem after all, the use of older credentials?
 
I have been creating new credentials with every new test, no worries. I've created dozens of them since the start of this process, to make sure they're not the issue.

I just added the new code, but no change, it still says Algorithm: SHA1

And of course I created new credentials. I even created a new user and tested (and gave the user proper permissions)
 
You must log in or register to reply here.
Back
Top