Submissions per connection/session

VVT

VVT

Active Member
Hey @twisted1919 ,

One question, for the SMTP type of Delivery Server, how many emails are transferred/submitted per connection/session ? Is it like one per connection ? Because I could see in logs that the app is making connection each time it submits an email. Correct me if I'm wrong.
 
Yeah, for each email a new connection is spawned, i believe it;s like so because, while php mailer is able to keep alive the session, swiftmailer isn't, and we need both to be stable.
You can at anytime replace any mailer from the app with yours and the custom settings if you need to keep the connection alive, it's just i haven't find a way to do so with swiftmailer.
 
VVT said:
Hey @twisted1919 ,
One question, for the SMTP type of Delivery Server, how many emails are transferred/submitted per connection/session ? Is it like one per connection ? Because I could see in logs that the app is making connection each time it submits an email. Correct me if I'm wrong.
Click to expand...

very good question





twisted1919 said:
Yeah, for each email a new connection is spawned, i believe it;s like so because, while php mailer is able to keep alive the session, swiftmailer isn't, and we need both to be stable.
You can at anytime replace any mailer from the app with yours and the custom settings if you need to keep the connection alive, it's just i haven't find a way to do so with swiftmailer.
Click to expand...


assumed it would recycle always

this is key documentation info (could not find it so far...)

the #1 reason to keep php mailer !speed!
(all the more odd that it was almost removed...)



if u ever used certain stand alone mailers which have their own mini smtp w/ connection recycle, u will see the difference is huge, so this is info one might wish in the tooltip of the mailer choice box

will phpmailer dkim-sign as consistently as swiftmailer?

again, great that mwz has both options!
:)
 
  • Like
Reactions: VVT
@twisted1919 Yeah, could see that "SMTPKeepAlive" is set to false on apps/common/components/mailer/MailerPHPMailer.php and apps/common/vendors/PHPMailer-5x/class.phpmailer.php ,so it initiates new session on every single submission.

Can you tell us how to turn this ON ? Guess we need to call close function after all the emails are submitted. So, it may be tricky to implement I think. If it can be turned on, let's test how fast it can perform over swift. Because, a considerable amount of time is lost between handshakes when we send a huge volume. Again, if there's auth enabled, auth negotiation will also slow down things to a great extend (disabling SMTP auth at mail server doubled the submission speed in my case).

May be this is the reason why I get almost the same speed if I use PHPMailer or Swift - because we haven't explored its potential yet (?).

@frm.mwz dkim signing works well for me (so far). Haven't observed any load spikes, though, there's a small trade off re delivery speed as expected.
 
@VVT - Nope, that should do it just fine. However, i think you;'re doing a confusion here. With a keepAlive connection, you will still see many connections in the smtp log. The only diff is that on mailwizz, the connection is kept open for all emails.
 
VVT said:
@twisted1919 Yeah, could see that "SMTPKeepAlive" is set to false on apps/common/components/mailer/MailerPHPMailer.php and apps/common/vendors/PHPMailer-5x/class.phpmailer.php ,so it initiates new session on every single submission.

Can you tell us how to turn this ON ? Guess we need to call close function after all the emails are submitted. So, it may be tricky to implement I think. If it can be turned on, let's test how fast it can perform over swift. Because, a considerable amount of time is lost between handshakes when we send a huge volume. Again, if there's auth enabled, auth negotiation will also slow down things to a great extend (disabling SMTP auth at mail server doubled the submission speed in my case).

May be this is the reason why I get almost the same speed if I use PHPmailer or Swift - because we haven't explored its potential yet (?).

@frm.mwz dkim signing works well for me (so far). Haven't observed any load spikes, though, there's a small trade off re delivery speed as expected.
Click to expand...

maybe the solution for how to close the connection is here:
http://stackoverflow.com/questions/2333930/persistent-smtp-connection-in-phpmailer

and swiftmailer has plugins to (limit the number of emails per connection, which incidentally do until that number of emails has been transferred) keep alive the connection...and they are mentioned in the documentation here
\apps\common\vendors\SwiftMailer-5x\doc\plugins.rst
though am not sure if they are used already
what do u think @twisted1919
can u adjust the # of emails per connection to max 100 and set the keep alive so it will be rfc compliant and max the speed (and tell where in the code that would be done)?
 
Last edited:
twisted1919 said:
@VVT - Nope, that should do it just fine. However, i think you;'re doing a confusion here. With a keepAlive connection, you will still see many connections in the smtp log. The only diff is that on mailwizz, the connection is kept open for all emails.
Click to expand...
Hmm.. Though I'm not a dev, I read and tried to understand the code. So, what I understood is, if SMTPKeepAlive is set true, it will send a message and "RSET" the connection after successful submission, then if it's end of the list, it will close the connection. But looks like this is not happening. See the below screenshots -

1. SMTPKeepAlive -> true in above files :
upload_2016-6-17_23-42-27.png
2. Manual submission of 2 emails using telnet under same session
upload_2016-6-17_23-43-15.png

#1 You can see connections are opened for each submission and then disconnected. I observed the same trend when SMTPKeepAlive was set to false as well. Also, I enabled auth at postfix and repeated the test, but it was again authenticating each time a message was submitted. I actually didn't understand your statement "The only diff is that on mailwizz, the connection is kept open for all emails." Because if the system knows connection is alive, why there's another "connect from" on the logs ?

#2 I wasn't sure if the RSET command according to the PHPMailer logic would actually result in a "connect from" log entry. So I tried manual submission using telnet. But it never left a sign of rset command on log. And the "disconnect from" appeared when I exited the session. Guess I reproduced the actual workflow of SMTPKeepAlive -> true here.

So, this dragged me in to the conclusion that either I didn't enable it in the right way or it's not working as it's supposed to be.

Now also, I'm not 100% sure if this is worth putting efforts on, but I wanted to give a try because I know postfix can do way better. Now MW can submit ~60 emails a 'minute' to a remote postfix instance, but it can submit 6-9 emails per 'second' via EE Web API (w/o pcntl). I scaled up the size of postfix instance and still it couldn't perform better. Only PCNTL helped. I am not very much bothered about this now because this unanticipated delay gives me better deliverability.

Below are 2 other hints that give some hope about the result if it works out -

https://support.smtp2go.com/entries/27183734-Using-Postfix-to-Speed-Up-Email-Delivery (Understand that postfix is a feeder MTA, but the concept is same. See #3)
https://sourceforge.net/p/phpmailer/mailman/message/3100786/ (May be this email was one of the reasons why SMTPKeepAlive was introduced way back in 2003 to achieve what I've been talking about, but looks like that's not how it works now - or simply that's not how I enabled it)

@frm.mwz I later realized that PHPmailer is used as is in MW and proper handlers are inherited from parent classes. Thx for the link anyway.
 
Last edited:
VVT said:
Hmm.. Though I'm not a dev, I read and tried to understand the code. So, what I understood is, if SMTPKeepAlive is set true, it will send a message and "RSET" the connection after successful submission, then if it's end of the list, it will close the connection. But looks like this is not happening. See the below screenshots -

1. SMTPKeepAlive -> true in above files :
View attachment 2057
2. Manual submission of 2 emails using telnet under same session
View attachment 2058

#1 You can see connections are opened for each submission and then disconnected. I observed the same trend when SMTPKeepAlive was set to false as well. Also, I enabled auth at postfix and repeated the test, but it was again authenticating each time a message was submitted. I actually didn't understand your statement "The only diff is that on mailwizz, the connection is kept open for all emails." Because if the system knows connection is alive, why there's another "connect from" on the logs ?

#2 I wasn't sure if the RSET command according to the PHPMailer logic would actually result in a "connect from" log entry. So I tried manual submission using telnet. But it never left a sign of rset command on log. And the "disconnect from" appeared when I exited the session. Guess I reproduced the actual workflow of SMTPKeepAlive -> true here.

So, this dragged me in to the conclusion that either I didn't enable it in the right way or it's not working as it's supposed to be.

Now also, I'm not 100% sure if this is worth putting efforts on, but I wanted to give a try because I know postfix can do way better. Now MW can submit ~60 emails a 'minute' to a remote postfix instance, but it can submit 6-9 emails per 'second' via EE Web API (w/o pcntl). I scaled up the size of postfix instance and still it couldn't perform better. Only PCNTL helped. I am not very much bothered about this now because this unanticipated delay gives me better deliverability.

Below are 2 other hints that give some hope about the result if it works out -

https://support.smtp2go.com/entries/27183734-Using-Postfix-to-Speed-Up-Email-Delivery (Understand that postfix is a feeder MTA, but the concept is same. See #3)
https://sourceforge.net/p/phpmailer/mailman/message/3100786/ (May be this email was one of the reasons why SMTPKeepAlive was introduced way back in 2003 to achieve what I've been talking about, but looks like that's not how it works now - or simply that's not how I enabled it)

@frm.mwz I later realized that PHPmailer is used as is in MW and proper handlers are inherited from parent classes. Thx for the link anyway.
Click to expand...


It might be necessary to distinguish between bulk sending one and the same email (no personalization, no unique links, etc) and individualized messages as mwz does.

The KeepAlive for PHPmailer is (partially?) described in the above link of my previous post, just as is a way to make SwiftMailer recycle the connection with the respective plugins (have not yet looked into the code where to apply them, as @twisted1919 might soon have the answer and the fix).

This link
https://en.wikipedia.org/wiki/Extended_SMTP
speaks more about pipelining and mentions also other smtp servers capable of doing so.

Re send-out speed and deliverability, if an smtp was used which would be able to manage the queue (as pmta, mailerQ, rabbitMQ, etc) then all we would care for was that mwz gets all messages out asap, and that is what probably all users would want.

If anyone wants to test their mail servers:
http://billauer.co.il/blog/2013/01/perl-sendmail-exim-postfix-test/
 
Last edited:
@frm.mwz Yeah, I agree that MW should push out emails as fast as it can regardless of deliverability and queuing.

@twisted1919 Here's an interesting thing. I just emailed the present maintainer of PHPmailer (GitHub) and he reverted to me immediately :) , such a nice guy, I never expected he would reply. So, here's our conversation -

=====
>
> Hey Marcus,
>
> Hope you're doing great !
>
> I came to know that you're the current maintainer of PHPMailer - congratulations and thanks for that.
>
>
> I'm one of the millions of users who use PHP Mailer in their business applications.
>
> I don't know if this is the way I should reach out to you with my question/doubt :) .
>
> Does PHP Mailer support SMTP pipelining ? is there a class/plugin available so that it can queue up multiple emails per connection without needing to instantiate each connection for each submission ?
>
>
> I'm aware of the setting - SMTPKeepAlive. I set that to true but still I can see PHPMailer is making new connections every time it submits an email to the MTA.
>
>
> Thanks in advance :)

==
Here's the reply
==
It certainly should not reconnect if keepalive is enabled. I just ran the keepalive test from the PHPMailer test suite (see the function called testSmtpKeepAlive) and it generates this output, where you can see keepalive working:

2016-06-20 21:10:42 Connection: opening to localhost:2500, timeout=300, options=array (
)
2016-06-20 21:10:42 Connection: opened
2016-06-20 21:10:42 SERVER -> CLIENT: 220 192.168.1.10 ESMTP SubEthaSMTP null
2016-06-20 21:10:42 CLIENT -> SERVER: EHLO localhost.localdomain
2016-06-20 21:10:42 SERVER -> CLIENT: 250-192.168.1.10
250-8BITMIME
250-AUTH LOGIN
250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: MAIL FROM:<unit_test@phpmailer.example.com>
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: RCPT TO:<somebody@example.com>
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: DATA
2016-06-20 21:10:42 SERVER -> CLIENT: 354 End data with <CR><LF>.<CR><LF>
2016-06-20 21:10:42 CLIENT -> SERVER: Date: Mon, 20 Jun 2016 21:10:42 +0000
2016-06-20 21:10:42 CLIENT -> SERVER: To: Test User <somebody@example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: From: Unit Tester <phpunit@example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: Reply-To: Reply Guy <no_reply@phpmailer.example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: Subject: Unit Test: SMTP keep-alive 1
2016-06-20 21:10:42 CLIENT -> SERVER: Message-ID: <00cce84f42d4d6472d1ea4d52d562c34@Oc.local>
2016-06-20 21:10:42 CLIENT -> SERVER: X-Priority: 3
2016-06-20 21:10:42 CLIENT -> SERVER: X-Mailer: PHPMailer 5.2.16 (https://github.com/PHPMailer/PHPMailer)
2016-06-20 21:10:42 CLIENT -> SERVER: MIME-Version: 1.0
2016-06-20 21:10:42 CLIENT -> SERVER: Content-Type: text/plain; charset=iso-8859-1
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER: SMTP keep-alive test.
2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
2016-06-20 21:10:42 CLIENT -> SERVER: Unit Test Information
2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
2016-06-20 21:10:42 CLIENT -> SERVER: phpmailer version: 5.2.16
2016-06-20 21:10:42 CLIENT -> SERVER: Content Type: text/plain
2016-06-20 21:10:42 CLIENT -> SERVER: CharSet: iso-8859-1
2016-06-20 21:10:42 CLIENT -> SERVER: Host: localhost
2016-06-20 21:10:42 CLIENT -> SERVER: Changes
2016-06-20 21:10:42 CLIENT -> SERVER: -------
2016-06-20 21:10:42 CLIENT -> SERVER: - Sender was changed to [unit_test@phpmailer.example.com]
2016-06-20 21:10:42 CLIENT -> SERVER: - Mailer was changed to [smtp]
2016-06-20 21:10:42 CLIENT -> SERVER: - Port was changed to [2500]
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER: .
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: RSET
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: MAIL FROM:<unit_test@phpmailer.example.com>
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: RCPT TO:<somebody@example.com>
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: DATA
2016-06-20 21:10:42 SERVER -> CLIENT: 354 End data with <CR><LF>.<CR><LF>
2016-06-20 21:10:42 CLIENT -> SERVER: Date: Mon, 20 Jun 2016 21:10:42 +0000
2016-06-20 21:10:42 CLIENT -> SERVER: To: Test User <somebody@example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: From: Unit Tester <phpunit@example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: Reply-To: Reply Guy <no_reply@phpmailer.example.com>
2016-06-20 21:10:42 CLIENT -> SERVER: Subject: Unit Test: SMTP keep-alive 2
2016-06-20 21:10:42 CLIENT -> SERVER: Message-ID: <8767d9db2003b4722e96d1d0dc3409d6@Oc.local>
2016-06-20 21:10:42 CLIENT -> SERVER: X-Priority: 3
2016-06-20 21:10:42 CLIENT -> SERVER: X-Mailer: PHPMailer 5.2.16 (https://github.com/PHPMailer/PHPMailer)
2016-06-20 21:10:42 CLIENT -> SERVER: MIME-Version: 1.0
2016-06-20 21:10:42 CLIENT -> SERVER: Content-Type: text/plain; charset=iso-8859-1
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER: SMTP keep-alive test.
2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
2016-06-20 21:10:42 CLIENT -> SERVER: Unit Test Information
2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
2016-06-20 21:10:42 CLIENT -> SERVER: phpmailer version: 5.2.16
2016-06-20 21:10:42 CLIENT -> SERVER: Content Type: text/plain
2016-06-20 21:10:42 CLIENT -> SERVER: CharSet: iso-8859-1
2016-06-20 21:10:42 CLIENT -> SERVER: Host: localhost
2016-06-20 21:10:42 CLIENT -> SERVER: Changes
2016-06-20 21:10:42 CLIENT -> SERVER: -------
2016-06-20 21:10:42 CLIENT -> SERVER: - Sender was changed to [unit_test@phpmailer.example.com]
2016-06-20 21:10:42 CLIENT -> SERVER: - Mailer was changed to [smtp]
2016-06-20 21:10:42 CLIENT -> SERVER: - Port was changed to [2500]
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER:
2016-06-20 21:10:42 CLIENT -> SERVER: .
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: RSET
2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
2016-06-20 21:10:42 CLIENT -> SERVER: QUIT
2016-06-20 21:10:42 SERVER -> CLIENT: 221 Bye
2016-06-20 21:10:42 Connection: closed

Are you seeing some other behaviour?

Marcus
==

Looking at the above test info, there's only one EHLO signal for 2 submissions. I sent him my log screenshot. Is there a way to see this connection details somewhere in MW logs ? I used to see that for failed emails. Or is there a way to test the connection like he did, like enabling debugging or verbose ?

Here's another hopeful article : https://luxsci.com/blog/maximize-your-outbound-email-throughput-how-to-send-more-email-faster.html

I know I'm being a bug to you n the PHPMailer dev, lol, but I would like to see how it performs once the SMTPKeepAlive works as expected before I give up. :P :D
 
This is his reply after seeing the above log behavior -


You're not creating a new instance of PHPMailer for each message are you? To use keepalive you need to use the same instance again, otherwise it can't track the state of the SMTP connection. Look at the function in the test suite I mentioned to see how to do that.
 
@VVT - I did some investigation and it seems we reset() the connection/mailer after each email, and the reason is because we swap servers according to settings (change server at) and if we don't reset the connection, then the new selected server will use the old connection which would result in some weird behavior.

However, good news is that i fixed that now, so the reset will only occur when the delivery server changes, so we should have persistent connections from now on.

If you unzip attached and put into apps/common/components to replace the mailer folder you'll get the new behavior.
By default we allow 100 emails per connections. This is adjustable by setting something like:
Code: 
'mailer.max_connection_messages' => 1000,
In apps/common/config/main(-custom).php in the params array.

Give it a try and let me know how it goes.
 

Attachments

@twisted1919 Awesomeeeeeee buddy, this really helps :D. Speed got doubled in my case ! Let the data speak ;)

1000 Subscribers before patching ->
upload_2016-6-21_11-41-14.png
1000 Subscribers and same settings after patching ->
upload_2016-6-21_11-40-36.png

So, the delivery speed got doubled in my case, so looks like it was worth your efforts :). I haven't tested if it affects any kind of quota - group/delivery server/hour minute etc, will test it as I get some time !

Once again thanks for your time !
 
twisted1919 said:
@VVT - I did some investigation and it seems we reset() the connection/mailer after each email, and the reason is because we swap servers according to settings (change server at) and if we don't reset the connection, then the new selected server will use the old connection which would result in some weird behavior.

However, good news is that i fixed that now, so the reset will only occur when the delivery server changes, so we should have persistent connections from now on.

If you unzip attached and put into apps/common/components to replace the mailer folder you'll get the new behavior.
By default we allow 100 emails per connections. This is adjustable by setting something like:
Code: 
'mailer.max_connection_messages' => 1000,
In apps/common/config/main(-custom).php in the params array.

Give it a try and let me know how it goes.
Click to expand...

100 is the rfc limit, it might be wise to leave it at that unless you always use your own special solution

can you map the flow of the whole program logic? it would help for testing, understanding, bug fixes, new feature suggestions, etc

so is it better to have an apps/common/config/main-custom.php file then?

have you looked into the swiftmailer plugins to achieve the keepalive?

great this is fixed for phpmailer
:)
 
@twisted1919 Setting 100 msgs/conn and 1000 msgs/conn yielded same results. I can confirm that customer group quota and delivery server quota works as expected. :)
 
VVT said:
@twisted1919 Setting 100 msgs/conn and 1000 msgs/conn yielded same results. I can confirm that customer group quota and delivery server quota works as expected. :)
Click to expand...

receiving servers may just not process more than rfc 100, which might be why there is no more speed gain afterward
(except if you have your own sending solution)
 
frm.mwz said:
receiving servers may just not process more than rfc 100, which might be why there is no more speed gain afterward
(except if you have your own sending solution)
Click to expand...
Don't think this is true. I've sent 1000 emails and could see only one pair of connect/disconnect on postfix logs. I switched the above variable to 500, then could see only 2. Guess "smtpd_recipient_limit" decides this limit and is 1000 by default (in Postfix).
 
VVT said:
Don't think this is true. I've sent 1000 emails and could see only one pair of connect/disconnect on postfix logs. I switched the above variable to 500, then could see only 2. Guess "smtpd_recipient_limit" decides this limit and is 1000 by default (in Postfix).
Click to expand...

the rfc is 100 (gmail e.g. will accept only 100 recipients max per message),
but what the delivery server admins (e.g. gmail, sendgrid, cluster) set is up to them (as usual).

as stated before, if it is your own solution (own delivery server, cluster, etc), that may differ,
but if it cannot be set in mwz for each delivery server (which will only now come up),
then it is wise to leave it at the rfc recommendation (100), as this avoids problems.
(and this is what @twisted1919 did, see above)

it's not personal, just trying to provide facts, to help avoid trouble-making settings,
if e.g. someone reads this thread and thinks, hey, let's slap 10k in there ;)
 
Last edited:
You must log in or register to reply here.
Back
Top