Submissions per connection/session

Discussion in 'General discussions' started by VVT, Jun 15, 2016.

  1. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    Hey @twisted1919 ,

    One question, for the SMTP type of Delivery Server, how many emails are transferred/submitted per connection/session ? Is it like one per connection ? Because I could see in logs that the app is making connection each time it submits an email. Correct me if I'm wrong.
     
  2. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,289
    Likes Received:
    2,391
    Yeah, for each email a new connection is spawned, i believe it;s like so because, while php mailer is able to keep alive the session, swiftmailer isn't, and we need both to be stable.
    You can at anytime replace any mailer from the app with yours and the custom settings if you need to keep the connection alive, it's just i haven't find a way to do so with swiftmailer.
     
  3. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    very good question






    assumed it would recycle always

    this is key documentation info (could not find it so far...)

    the #1 reason to keep php mailer !speed!
    (all the more odd that it was almost removed...)



    if u ever used certain stand alone mailers which have their own mini smtp w/ connection recycle, u will see the difference is huge, so this is info one might wish in the tooltip of the mailer choice box

    will phpmailer dkim-sign as consistently as swiftmailer?

    again, great that mwz has both options!
    :)
     
    VVT likes this.
  4. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    @twisted1919 Yeah, could see that "SMTPKeepAlive" is set to false on apps/common/components/mailer/MailerPHPMailer.php and apps/common/vendors/PHPMailer-5x/class.phpmailer.php ,so it initiates new session on every single submission.

    Can you tell us how to turn this ON ? Guess we need to call close function after all the emails are submitted. So, it may be tricky to implement I think. If it can be turned on, let's test how fast it can perform over swift. Because, a considerable amount of time is lost between handshakes when we send a huge volume. Again, if there's auth enabled, auth negotiation will also slow down things to a great extend (disabling SMTP auth at mail server doubled the submission speed in my case).

    May be this is the reason why I get almost the same speed if I use PHPMailer or Swift - because we haven't explored its potential yet (?).

    @frm.mwz dkim signing works well for me (so far). Haven't observed any load spikes, though, there's a small trade off re delivery speed as expected.
     
  5. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,289
    Likes Received:
    2,391
    @VVT - For now you can try to hardcode KeepAlive to true and see how it performs ;)
     
  6. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    Hmm.. Set the value to "true" on both the files. Still the same. Do I have to do something more for it to take effect ?
     
  7. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,289
    Likes Received:
    2,391
    @VVT - Nope, that should do it just fine. However, i think you;'re doing a confusion here. With a keepAlive connection, you will still see many connections in the smtp log. The only diff is that on mailwizz, the connection is kept open for all emails.
     
  8. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    maybe the solution for how to close the connection is here:
    http://stackoverflow.com/questions/2333930/persistent-smtp-connection-in-phpmailer

    and swiftmailer has plugins to (limit the number of emails per connection, which incidentally do until that number of emails has been transferred) keep alive the connection...and they are mentioned in the documentation here
    \apps\common\vendors\SwiftMailer-5x\doc\plugins.rst
    though am not sure if they are used already
    what do u think @twisted1919
    can u adjust the # of emails per connection to max 100 and set the keep alive so it will be rfc compliant and max the speed (and tell where in the code that would be done)?
     
    Last edited: Jun 20, 2016
  9. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    Hmm.. Though I'm not a dev, I read and tried to understand the code. So, what I understood is, if SMTPKeepAlive is set true, it will send a message and "RSET" the connection after successful submission, then if it's end of the list, it will close the connection. But looks like this is not happening. See the below screenshots -

    1. SMTPKeepAlive -> true in above files :
    upload_2016-6-17_23-42-27.png
    2. Manual submission of 2 emails using telnet under same session
    upload_2016-6-17_23-43-15.png

    #1 You can see connections are opened for each submission and then disconnected. I observed the same trend when SMTPKeepAlive was set to false as well. Also, I enabled auth at postfix and repeated the test, but it was again authenticating each time a message was submitted. I actually didn't understand your statement "The only diff is that on mailwizz, the connection is kept open for all emails." Because if the system knows connection is alive, why there's another "connect from" on the logs ?

    #2 I wasn't sure if the RSET command according to the PHPMailer logic would actually result in a "connect from" log entry. So I tried manual submission using telnet. But it never left a sign of rset command on log. And the "disconnect from" appeared when I exited the session. Guess I reproduced the actual workflow of SMTPKeepAlive -> true here.

    So, this dragged me in to the conclusion that either I didn't enable it in the right way or it's not working as it's supposed to be.

    Now also, I'm not 100% sure if this is worth putting efforts on, but I wanted to give a try because I know postfix can do way better. Now MW can submit ~60 emails a 'minute' to a remote postfix instance, but it can submit 6-9 emails per 'second' via EE Web API (w/o pcntl). I scaled up the size of postfix instance and still it couldn't perform better. Only PCNTL helped. I am not very much bothered about this now because this unanticipated delay gives me better deliverability.

    Below are 2 other hints that give some hope about the result if it works out -

    https://support.smtp2go.com/entries/27183734-Using-Postfix-to-Speed-Up-Email-Delivery (Understand that postfix is a feeder MTA, but the concept is same. See #3)
    https://sourceforge.net/p/phpmailer/mailman/message/3100786/ (May be this email was one of the reasons why SMTPKeepAlive was introduced way back in 2003 to achieve what I've been talking about, but looks like that's not how it works now - or simply that's not how I enabled it)

    @frm.mwz I later realized that PHPmailer is used as is in MW and proper handlers are inherited from parent classes. Thx for the link anyway.
     
    Last edited: Jun 18, 2016
  10. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7

    It might be necessary to distinguish between bulk sending one and the same email (no personalization, no unique links, etc) and individualized messages as mwz does.

    The KeepAlive for PHPmailer is (partially?) described in the above link of my previous post, just as is a way to make SwiftMailer recycle the connection with the respective plugins (have not yet looked into the code where to apply them, as @twisted1919 might soon have the answer and the fix).

    This link
    https://en.wikipedia.org/wiki/Extended_SMTP
    speaks more about pipelining and mentions also other smtp servers capable of doing so.

    Re send-out speed and deliverability, if an smtp was used which would be able to manage the queue (as pmta, mailerQ, rabbitMQ, etc) then all we would care for was that mwz gets all messages out asap, and that is what probably all users would want.

    If anyone wants to test their mail servers:
    http://billauer.co.il/blog/2013/01/perl-sendmail-exim-postfix-test/
     
    Last edited: Jun 23, 2016
  11. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    @frm.mwz Yeah, I agree that MW should push out emails as fast as it can regardless of deliverability and queuing.

    @twisted1919 Here's an interesting thing. I just emailed the present maintainer of PHPmailer (GitHub) and he reverted to me immediately :) , such a nice guy, I never expected he would reply. So, here's our conversation -

    =====
    >
    > Hey Marcus,
    >
    > Hope you're doing great !
    >
    > I came to know that you're the current maintainer of PHPMailer - congratulations and thanks for that.
    >
    >
    > I'm one of the millions of users who use PHP Mailer in their business applications.
    >
    > I don't know if this is the way I should reach out to you with my question/doubt :) .
    >
    > Does PHP Mailer support SMTP pipelining ? is there a class/plugin available so that it can queue up multiple emails per connection without needing to instantiate each connection for each submission ?
    >
    >
    > I'm aware of the setting - SMTPKeepAlive. I set that to true but still I can see PHPMailer is making new connections every time it submits an email to the MTA.
    >
    >
    > Thanks in advance :)

    ==
    Here's the reply
    ==
    It certainly should not reconnect if keepalive is enabled. I just ran the keepalive test from the PHPMailer test suite (see the function called testSmtpKeepAlive) and it generates this output, where you can see keepalive working:

    2016-06-20 21:10:42 Connection: opening to localhost:2500, timeout=300, options=array (
    )
    2016-06-20 21:10:42 Connection: opened
    2016-06-20 21:10:42 SERVER -> CLIENT: 220 192.168.1.10 ESMTP SubEthaSMTP null
    2016-06-20 21:10:42 CLIENT -> SERVER: EHLO localhost.localdomain
    2016-06-20 21:10:42 SERVER -> CLIENT: 250-192.168.1.10
    250-8BITMIME
    250-AUTH LOGIN
    250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: MAIL FROM:<unit_test@phpmailer.example.com>
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: RCPT TO:<somebody@example.com>
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: DATA
    2016-06-20 21:10:42 SERVER -> CLIENT: 354 End data with <CR><LF>.<CR><LF>
    2016-06-20 21:10:42 CLIENT -> SERVER: Date: Mon, 20 Jun 2016 21:10:42 +0000
    2016-06-20 21:10:42 CLIENT -> SERVER: To: Test User <somebody@example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: From: Unit Tester <phpunit@example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: Reply-To: Reply Guy <no_reply@phpmailer.example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: Subject: Unit Test: SMTP keep-alive 1
    2016-06-20 21:10:42 CLIENT -> SERVER: Message-ID: <00cce84f42d4d6472d1ea4d52d562c34@Oc.local>
    2016-06-20 21:10:42 CLIENT -> SERVER: X-Priority: 3
    2016-06-20 21:10:42 CLIENT -> SERVER: X-Mailer: PHPMailer 5.2.16 (https://github.com/PHPMailer/PHPMailer)
    2016-06-20 21:10:42 CLIENT -> SERVER: MIME-Version: 1.0
    2016-06-20 21:10:42 CLIENT -> SERVER: Content-Type: text/plain; charset=iso-8859-1
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER: SMTP keep-alive test.
    2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
    2016-06-20 21:10:42 CLIENT -> SERVER: Unit Test Information
    2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
    2016-06-20 21:10:42 CLIENT -> SERVER: phpmailer version: 5.2.16
    2016-06-20 21:10:42 CLIENT -> SERVER: Content Type: text/plain
    2016-06-20 21:10:42 CLIENT -> SERVER: CharSet: iso-8859-1
    2016-06-20 21:10:42 CLIENT -> SERVER: Host: localhost
    2016-06-20 21:10:42 CLIENT -> SERVER: Changes
    2016-06-20 21:10:42 CLIENT -> SERVER: -------
    2016-06-20 21:10:42 CLIENT -> SERVER: - Sender was changed to [unit_test@phpmailer.example.com]
    2016-06-20 21:10:42 CLIENT -> SERVER: - Mailer was changed to [smtp]
    2016-06-20 21:10:42 CLIENT -> SERVER: - Port was changed to [2500]
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER: .
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: RSET
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: MAIL FROM:<unit_test@phpmailer.example.com>
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: RCPT TO:<somebody@example.com>
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: DATA
    2016-06-20 21:10:42 SERVER -> CLIENT: 354 End data with <CR><LF>.<CR><LF>
    2016-06-20 21:10:42 CLIENT -> SERVER: Date: Mon, 20 Jun 2016 21:10:42 +0000
    2016-06-20 21:10:42 CLIENT -> SERVER: To: Test User <somebody@example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: From: Unit Tester <phpunit@example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: Reply-To: Reply Guy <no_reply@phpmailer.example.com>
    2016-06-20 21:10:42 CLIENT -> SERVER: Subject: Unit Test: SMTP keep-alive 2
    2016-06-20 21:10:42 CLIENT -> SERVER: Message-ID: <8767d9db2003b4722e96d1d0dc3409d6@Oc.local>
    2016-06-20 21:10:42 CLIENT -> SERVER: X-Priority: 3
    2016-06-20 21:10:42 CLIENT -> SERVER: X-Mailer: PHPMailer 5.2.16 (https://github.com/PHPMailer/PHPMailer)
    2016-06-20 21:10:42 CLIENT -> SERVER: MIME-Version: 1.0
    2016-06-20 21:10:42 CLIENT -> SERVER: Content-Type: text/plain; charset=iso-8859-1
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER: SMTP keep-alive test.
    2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
    2016-06-20 21:10:42 CLIENT -> SERVER: Unit Test Information
    2016-06-20 21:10:42 CLIENT -> SERVER: ---------------------
    2016-06-20 21:10:42 CLIENT -> SERVER: phpmailer version: 5.2.16
    2016-06-20 21:10:42 CLIENT -> SERVER: Content Type: text/plain
    2016-06-20 21:10:42 CLIENT -> SERVER: CharSet: iso-8859-1
    2016-06-20 21:10:42 CLIENT -> SERVER: Host: localhost
    2016-06-20 21:10:42 CLIENT -> SERVER: Changes
    2016-06-20 21:10:42 CLIENT -> SERVER: -------
    2016-06-20 21:10:42 CLIENT -> SERVER: - Sender was changed to [unit_test@phpmailer.example.com]
    2016-06-20 21:10:42 CLIENT -> SERVER: - Mailer was changed to [smtp]
    2016-06-20 21:10:42 CLIENT -> SERVER: - Port was changed to [2500]
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER:
    2016-06-20 21:10:42 CLIENT -> SERVER: .
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: RSET
    2016-06-20 21:10:42 SERVER -> CLIENT: 250 Ok
    2016-06-20 21:10:42 CLIENT -> SERVER: QUIT
    2016-06-20 21:10:42 SERVER -> CLIENT: 221 Bye
    2016-06-20 21:10:42 Connection: closed

    Are you seeing some other behaviour?

    Marcus
    ==

    Looking at the above test info, there's only one EHLO signal for 2 submissions. I sent him my log screenshot. Is there a way to see this connection details somewhere in MW logs ? I used to see that for failed emails. Or is there a way to test the connection like he did, like enabling debugging or verbose ?

    Here's another hopeful article : https://luxsci.com/blog/maximize-your-outbound-email-throughput-how-to-send-more-email-faster.html

    I know I'm being a bug to you n the PHPMailer dev, lol, but I would like to see how it performs once the SMTPKeepAlive works as expected before I give up. :p :D
     
    frm.mwz likes this.
  12. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    This is his reply after seeing the above log behavior -


    You're not creating a new instance of PHPMailer for each message are you? To use keepalive you need to use the same instance again, otherwise it can't track the state of the SMTP connection. Look at the function in the test suite I mentioned to see how to do that.
     
  13. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,289
    Likes Received:
    2,391
    @VVT - I did some investigation and it seems we reset() the connection/mailer after each email, and the reason is because we swap servers according to settings (change server at) and if we don't reset the connection, then the new selected server will use the old connection which would result in some weird behavior.

    However, good news is that i fixed that now, so the reset will only occur when the delivery server changes, so we should have persistent connections from now on.

    If you unzip attached and put into apps/common/components to replace the mailer folder you'll get the new behavior.
    By default we allow 100 emails per connections. This is adjustable by setting something like:
    Code:
    'mailer.max_connection_messages' => 1000,
    
    In apps/common/config/main(-custom).php in the params array.

    Give it a try and let me know how it goes.
     

    Attached Files:

    Vpul Shah, frm.mwz and VVT like this.
  14. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,289
    Likes Received:
    2,391
    L.E: I am working to add the max connection number for each delivery server as part of settings.
     
    pradeep sharma and VVT like this.
  15. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    @twisted1919 Awesomeeeeeee buddy, this really helps :D. Speed got doubled in my case ! Let the data speak ;)

    1000 Subscribers before patching ->
    upload_2016-6-21_11-41-14.png
    1000 Subscribers and same settings after patching ->
    upload_2016-6-21_11-40-36.png

    So, the delivery speed got doubled in my case, so looks like it was worth your efforts :). I haven't tested if it affects any kind of quota - group/delivery server/hour minute etc, will test it as I get some time !

    Once again thanks for your time !
     
    pradeep sharma likes this.
  16. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    100 is the rfc limit, it might be wise to leave it at that unless you always use your own special solution

    can you map the flow of the whole program logic? it would help for testing, understanding, bug fixes, new feature suggestions, etc

    so is it better to have an apps/common/config/main-custom.php file then?

    have you looked into the swiftmailer plugins to achieve the keepalive?

    great this is fixed for phpmailer
    :)
     
  17. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    @twisted1919 Setting 100 msgs/conn and 1000 msgs/conn yielded same results. I can confirm that customer group quota and delivery server quota works as expected. :)
     
  18. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    receiving servers may just not process more than rfc 100, which might be why there is no more speed gain afterward
    (except if you have your own sending solution)
     
  19. VVT

    VVT Active Member

    Joined:
    May 22, 2015
    Messages:
    499
    Likes Received:
    138
    S.E:
    Expired
    L.T:
    Extended
    L.C:
    2
    Don't think this is true. I've sent 1000 emails and could see only one pair of connect/disconnect on postfix logs. I switched the above variable to 500, then could see only 2. Guess "smtpd_recipient_limit" decides this limit and is 1000 by default (in Postfix).
     
  20. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    679
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    the rfc is 100 (gmail e.g. will accept only 100 recipients max per message),
    but what the delivery server admins (e.g. gmail, sendgrid, cluster) set is up to them (as usual).

    as stated before, if it is your own solution (own delivery server, cluster, etc), that may differ,
    but if it cannot be set in mwz for each delivery server (which will only now come up),
    then it is wise to leave it at the rfc recommendation (100), as this avoids problems.
    (and this is what @twisted1919 did, see above)

    it's not personal, just trying to provide facts, to help avoid trouble-making settings,
    if e.g. someone reads this thread and thinks, hey, let's slap 10k in there ;)
     
    Last edited: Jun 22, 2016

Share This Page