How increase speed for email/minute

[Ashish Shukla]

Hi,
I am using mailwizz since last 2 years but now I am facing slow delivery speed.
I have 1 customer account only with 55 Delivery server and want to send 5M emails in just 4-5 hours. I don't want to rotate delivery server as I developed reputation for my IP and domains. Usually it takes more than 12 hours. I know many of you already solved this problem so I am seeking help from your side.
I have sufficient server configuration with 96GB RAM and 1 Gbps Bandwidth.
How may I send 5M emails in just 4-5 hours from MW to PMTA.
Here is current Backend cron setting

 

[majid1f]

Check this topic:
https://forum.mailwizz.com/threads/...ge-but-i-want-more-any-ideas.5838/#post-37568

 

[laurentiu]

@Ashish Shukla - https://kb.mailwizz.com/articles/performance-improvements/

 

[Ashish Shukla]

Hi @laurentiu
I implemented all above points but still same issue

 

[twisted1919]

@Ashish Shukla - did you ever debug the send-campaigns command: https://kb.mailwizz.com/articles/debug-send-campaigns-command/ ?
That will tell you exactly where mailwizz spends the most time.

 

[Ashish Shukla]

Hi @twisted1919
Yeah currently I am running manual cron via debug. It's not showing any error or utilization. See below stats

[2019-04-03 10:21:47] - XXXXX@yahoo.com - 40/50
[2019-04-03 10:21:47] - Checking if we can send to domain of XXXXXX@yahoo.com...
[2019-04-03 10:21:47] - Checking if XXXXX@yahoo.com is still confirmed...
[2019-04-03 10:21:47] - Checking if XXXXX@yahoo.com is blacklisted...
[2019-04-03 10:21:47] - Checking if XXXXX@yahoo.com is listed in a campaign suppression list...
[2019-04-03 10:21:47] - Checking if the server is over quota...
[2019-04-03 10:21:47] - Preparing the entire email...
[2019-04-03 10:21:47] - Using delivery server: ######### (ID: 3).
[2019-04-03 10:21:47] - Communication with the delivery server took: 0.8781
[2019-04-03 10:21:47] - Sending response is: OK
[2019-04-03 10:21:47] - Sending OK.
[2019-04-03 10:21:47] - Done for XXXXX@yahoo.com, logging delivery...
[2019-04-03 10:21:47] - Logging delivery took: 0.00229


if I am running 50 Subscriber with 20 batches and 60 campaigns at a time. Then it finished within 643 seconds

[2019-04-03 10:24:00] - This cycle completed in 643.2 seconds and used 2 MB of memory!
[2019-04-03 10:24:00] - CPU usage in last minute: 13.77, in last 5 minutes: 14.33, in last 15 minutes: 12.96!

 

[Dev1@OptiBiz1]

@Ashish Shukla, I am the dev of https://forum.mailwizz.com/threads/...ge-but-i-want-more-any-ideas.5838/#post-38555

- Make sure you're not embedding images in the email
- Make sure you're not attaching files in the email
- Make sure to use PHPMailer
- Make sure you are using Redis Cache and Mutex
- Make sure you enabled campaign queue table
- Make sure MySQL is configured properly for the type of server you have, example: it should have enough connections, specially innodb variables
- Make sure your MW server is using at least SSD, much better to use NVME. This affects MySQL performance
- If your MySQL is hosted in a different server, then latency will be your PROBLEM, specially if it is hosted in a different location in the world.
- Make sure your MW server network limit is at least 1Gbps, if it is less than 1Gbps, then your sending will be limited.
- If your MW server network limit is 1Gbps, Your PMTA server network limit should also be 1Gbps. Because if MW server(or PMTA) is lower, then you won't be able to send as much because of network limitation. To check the bandwidth usage while sending email, use iftop command of linux.
- Make sure your PMTA server is configured to handle many SMTP connections. 1000 is usually ok.

Now, based on your send-campaigns verbose log, Communication with the delivery server took: 0.8781. That is too much for sending 1 email. What is the email size that you are sending? Is it 100Kb? 200Kb?

If the email size is just around 20Kb, but delivery server still took around 0.8, then you need to find a way to lower the delivery server communication. This usually happens when MW server and PMTA server is hosted on different part of the world.

Try checking the latency of your MW server to the PMTA server, is should be close to 1ms as much as possible(for maximum performance). If the latency is around 500ms, then that's a problem. From your MW server, execute mtr -rw [PMTA ip addr]

 

[Ashish Shukla]

Hi @Dev1@OptiBiz1

Thanks for brief guidance, I solved Delivery speed with Redis. Now facing another issue, once I increased sending limit.

Process Status = GIVEUP
Expected response code 220 but got code "421", with message "421 server.domainname.com out of connection slots " Log data: ++ Starting Swift_SmtpTransport << 421 server.domainname.com out of connection slots !! Expected response code 220 but got code "421", with message "421 server.domainname.com out of connection slots " (code: 421)

SMTP server hosted with different location with another hosting provider. I have updated Max. connection messages = 1 at Delivery server setting.

 

[twisted1919]

421 server.domainname.com out of connection slots

Isn't this pretty clear ?

You're sending faster than your smtp server can handle.

 

[Dev1@OptiBiz1]

421 server.domainname.com out of connection slots " (code: 421)

3 things to make sure this is resolved.

- As I have added above:

- Make sure your PMTA server is configured to handle many SMTP connections. 1000 is usually ok.

- delivery server config Max. connection messages should not be 1, if using PMTA, setting it to 1000+ is fine.
If it is set to 1, then it will create an SMTP connection to PMTA for every message which is inefficient. PMTA can handle thousands of messages per connection.

- In Settings > Cron > Send at once, should either be 0 or 1000+, this can also be equal to Max. connection messages.

If using PHPMailer, Max. connection messages affects how many SMTP connection slots.

If using SwfitMailer then Max. connection messages and Send at once are related.They will affect how many connection slots was used during sending. Let's say, Max. connection messages is set to 1000, Send at once = 100. Then it It will create SMTP connection for every 100 messages, then once the Max. connection messages is reached, it will create another SMTP connection.

 

[frm.mwz]

3 things to make sure this is resolved.

- As I have added above:


- delivery server config Max. connection messages should not be 1, if using PMTA, setting it to 1000+ is fine.
If it is set to 1, then it will create an SMTP connection to PMTA for every message which is inefficient. PMTA can handle thousands of messages per connection.

- In Settings > Cron > Send at once, should either be 0 or 1000+, this can also be equal to Max. connection messages.

If using PHPMailer, Max. connection messages affects how many SMTP connection slots.

If using SwfitMailer then Max. connection messages and Send at once are related.They will affect how many connection slots was used during sending. Let's say, Max. connection messages is set to 1000, Send at once = 100. Then it It will create SMTP connection for every 100 messages, then once the Max. connection messages is reached, it will create another SMTP connection.

It is actually PCNTL that will determine to a large part how many concurrent connections are being made. Of course, more are needed (or less is sent) if you have "connection messages" (correctly that is really 'messages per connection') set to a lower number (e.g. 1), but in most use cases that is somewhere between fifty and a few thousand.
Keep in mind that PHPmailer is actually the RFC compliant one (not SwiftMailer), and that it is 2.6 times faster than SwiftMailer (as it does the pipelining better).

 

[Dev1@OptiBiz1]

It is actually PCNTL that will determine to a large part how many concurrent connections are being made. Of course, more are needed (or less is sent) if you have "connection messages" (correctly that is really 'messages per connection') set to a lower number (e.g. 1), but in most use cases that is somewhere between fifty and a few thousand.
Keep in mind that PHPmailer is actually the RFC compliant one (not SwiftMailer), and that it is 2.6 times faster than SwiftMailer (as it does the pipelining better).

Right, pcntl plays a huge role on the number of smtp conns. Number of campaigns in parallel, subscriber batches in parallel

 

[frm.mwz]

Right, pcntl plays a huge role on the number of smtp conns. Number of campaigns in parallel, subscriber batches in parallel

It was about the number of connections (the rest we know).
Try this: disable PCNTL and see how many concurrent connections you get?

 

[Ashish Shukla]

Hi @frm.mwz and @Dev1@OptiBiz1

Thanks for all information, it is really fruitful.

I am using SwiftMailer with PCNTL enabled.

When I reduced volume it was normal. Now we are sending 500K email daily from 1 domain, reputation is good. Now again since a week facing more than 70% of Giveup errors with connection slots " Log data: ++ Starting Swift_SmtpTransport.

Currently I updated max connection message 2000 in delivery server. Send at once is 0 in cron setting.

Also made changes in PMTA config:
<source-group localSources>
max-smtp-in 1000
reserved-smtp-in 100
</source-group>

But still same issue. Now condition is really bad, not able to test email with 10 internal emails

I really appreciate your help.

 

[Dev1@OptiBiz1]

@Ashish Shukla

You said you updated max conn. message to 2000 in delivery server, but your pmta max-smtp-in config is 1000,
you either need to increase your max-smtp-in or decrease you max conn message to less than max-smtp-in.

Refer to this:

- Make sure your PMTA server is configured to handle many SMTP connections. 1000 is usually ok.

 

[Ashish Shukla]

Hi @Dev1@OptiBiz1

It's 2000 messages in one connection and updated 1000 connections in PMTA. Now i changed max-smtp-in 10000. But still same issue

 

[Dev1@OptiBiz1]

@Ashish Shukla make sure PMTA status Connections Max is reflected from the config you changed. I encountered a problem before where changing max-smtp-in doesn't change PMTA status Connections Max

 

[Ashish Shukla]

Thanks we solved this issue, with help of @frm.mwz . I can see max smtp connection.

Thank you all for your responses and help

 

[Dev1@OptiBiz1]

@Ashish Shukla so the issue is with max smtp conn?

 

[Ashish Shukla]

@Dev1@OptiBiz1 It was due to total max smtp connection