Process campaigns using Redis Queue.

Discussion in 'Tips and tricks' started by twisted1919, Feb 27, 2015.

  1. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,125
    Likes Received:
    2,352
    NOTE: Please do not rely on this functionality anymore. Starting with mailwizz 1.3.5.9 the send-campaigns command has been improved so much that there is no need for Redis or the like anymore so in the future we will remove redis from mailwizz entirely.


    Since version 1.3.5, MailWizz adds experimental support for using queues with Redis.
    Using regular cron job sending, on a linode with 2 cores and 2GB of ram(~512 available), sending a campaign with 10k subscribers at once took as shown below:
    Code:
    [root@work-01 ~]# time /usr/bin/php -q /home/domain/public_html/apps/console/console.php send-campaigns
    
    real    17m40.299s
    user    8m1.417s
    sys     2m53.203s
    
    
    When redis queue has been enabled, sending same amount of emails, 10k, took as shown below:
    Code:
    [root@work-01 ~]# time /usr/bin/php -q /home/domain/public_html/apps/console/console.php send-campaigns
    
    real    4m6.028s
    user    2m26.167s
    sys     0m53.307s
    
    
    As we can see, sending with redis queue is considerably faster, ~4 times faster on a considerably small machine.

    In order to use this sending method, your server needs to meet the following requirements:
    1. Redis server
    2. PHP's pcntl's extension installed and enabled with no function disabled.
    3. If possible, install php's native client for redis.
    4. command line/ssh access so that you can run various commands.

    See https://forum.mailwizz.com/threads/redis-install.205/ for a guide on how to install #1 and #3.

    In order to enable sending via this method, firstly we need to enable and configure the Redis Queue from mailwizz backend, so go to Backend -> Settings -> Queue and do the needed adjustments and enable it, then save the changes.

    Once the redis queue is ready for use, we need to decide which delivery servers are going to use the queue. If you try to edit a delivery server now after the queue feature has been enabled, you will notice a new field, called "Use queue". You need to set this field to YES for the servers that you want to use this sending method. Do that and then save your delivery server.
    (P.S: you can have servers using queue and servers doing regular delivery without queue, both work just fine in the same time)

    Now the system is ready to queue all your emails in the redis server when campaigns are being sent via the delivery servers you have choosen to use the queue on.
    If you send a test campaign, you will notice that sending it is going to be blazing fast, like really fast that you'll think something is wrong.
    Well, there's nothing wrong, it's so fast because we just queue the emails in the redis server, we are not sending them at all.
    In order to do the actual sending, we need some kind of daemon that will run permanently and read the redis queue, pick the campaign emails, send them and then empty the queue.
    For this exact reason, a new mailwizz command line command is available.
    If from command line you run:
    Code:
    /usr/bin/php -q /absolute/path/to/apps/console/console.php
    
    you will see something like:
    Code:
    [root@work-01 ~]# /usr/bin/php -q /home/domain/public_html/apps/console/console.php
    Yii command runner (based on Yii v1.1.16)
    Usage: /home/domain/public_html/apps/console/console.php <command-name> [parameter                                                                    s...]
    
    The following commands are available:
    - archive-campaigns-delivery-logs
    - archivecampaignsdeliverylogs
    - bounce-handler
    - bouncehandler
    - daily
    - feedback-loop-handler
    - feedbackloophandler
    - hello
    - migrate
    - option
    - process-delivery-and-bounce-log
    - process-subscribers
    - processdeliveryandbouncelog
    - processsubscribers
    - queue
    - redisqueue
    - send-campaigns
    - send-transactional-emails
    - sendcampaigns
    - sendtransactionalemails
    - update
    
    To see individual command help, use the following:
       /home/domain/public_html/apps/console/console.php help <command-name>
    
    
    So the command that interests us is the queue command.
    Please note that this is not a command that should be placed in the cron jobs to be called once at x minutes. This is a daemon that once started it will run permanently and you should not have more than one instance of it running in the same time.

    It can be called from command line like:
    Code:
    /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0
    
    The --workers param means how many separate processes will be started to check your queue. Please note that all these processes will run in parallel and might eact a considerable amount of ram and cpu if the number of workers is too high. This is PHP, not the best tool for this kind of stuff.
    The --interval param, is the number of seconds between the queue checks. In the above case, the queue will be checked once at 5 seconds by each worker.
    The --verbose param is a debug flag, if it's set to 1, you can see how and what is processing.

    Once you call this command (you should call it with --verbose=1) you will see how it starts processing all the emails from the redis queue.
    Once you are sure this works as expected you can kill the queue process so that you can later start it again in background.
    To do this, simply run:
    Code:
    ps aux | grep queue
    
    Which will return something like:
    Code:
    [root@work-01 ~]# ps aux | grep queue
    root     21641  0.0  0.5 550952 11116 ?        Ss   Feb04   2:18 /usr/bin/php...
    
    
    Where 21641 is the process PID and we can kill it with:
    Code:
    kill -9 21641
    
    Then we can start the process again and send it in background with following command:
    Code:
    nohup /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0 >/dev/null 2>&1 &
    
    After you do this, the process will run till you kill it, till the server reboots or till it dies because of some fatal error, like insufficient memory or the like.

    To recap, when using redis queue, the cron job that runs once a minute, the one that calls send-campaigns will load your campaigns from database and instead of sending the emails directly, it will store those emails in the redis queue from where they will be later picked up by the queue daemon that you have started and it runs in background and this daemon is the one that does the actual delivery. Each worker will process 100 emails at once, so when you have 10 workers, you'll process 1k emails at once. Keep this in mind so that you don't use all your server memory because of this.

    That's it.
    If any questions related to this, please ask but keep in mind, this is still experimental stuff.
    For me it worked fine while i tested it but I recommend you to test this enough before you rely on it.

    Thanks.
     
    Last edited: Dec 8, 2015
    Man, justme, Rob and 2 others like this.
  2. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
  3. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Hi

    I've been playing around with this and I have redis installed on a separate server, the daemon obviously won't find the queue in this instance, how I can place the daemon on the redis server? Or is that not going to be possible?

    Cheers

    Rob
     
    Urano likes this.
  4. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,125
    Likes Received:
    2,352
    Hey,

    I think it doesn't matter where redis is installed, why would it?
    I mean, as long as you give the right details into the configuration, it should connect(remotely) and put/get data from redis without issues.

    Did you try it? Do you get any specific issue?
     
  5. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Hi

    OK, probably I'm not understanding where the "queue daemon" is firing instructions to and where the email is actually sitting, if I can get my head around the workflow/process then it shouldn't be too difficult.

    I setup a delivery server and set it to use redis (another server), set the daemon running using verbose=1 and it fires as it should. Campaigns are queued but emails aren't sent.

    Using "redis-cli monitor" on the redis server it shows that there is an "email:queue" so redis is receiving the data.

    A little more investigation is needed it would seem :)

    Cheers

    Rob
     
  6. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,125
    Likes Received:
    2,352
    Hey,

    Okay, that's good, if emails are sent into redis you are half way there.
    Now you need to process them from redis and empty the queue, and you do that by running:
    Code:
    /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=1
    
    Once you run the above, mailwizz will connect to Redis and get all queued emails and process them.
    Please note that the above command is a daemon that once started runs forever, it's not something that should be added as a cron job, but started as described above, with nohup when you are done testing.

    Thanks.
     
  7. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Hi

    Got this sorted in the end. I noticed in another post you use PHP 5.6 for dev so I upgraded to 5.6.7 and it started working fine. I was on 5.5.9.

    Now to run some simulations.

    Cheers

    Rob
     
  8. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    29
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Is it possible to automate the following command, so the daemon starts automatically upon reboot or if it dies:

    nohup /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0 >/dev/null 2>&1 &​
     
  9. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Not to my knowledge, funnily enough I've been working on a monitoring check today and some way to reinstate it if it stops working. I'll let you know what I work out.

    Cheers

    Rob
     
  10. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    29
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Couldn't a simple scipr
    Couldn't a simple script that search for "emails-queue" process and starts a new process if it isn't found work?
     
  11. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    29
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Question. Does this Redis queue adhere to the limits in settings -> cron? So if I set the limit to 5,000 emails per minute, we wouldn't go over that?

    I'm trying to find the best settings for the Redis queue process so that we hit our limit yet dont continually eat CPU and RAM when there are no emails to send.
     
  12. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    29
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Is there a way to track the rate at which Redis queue sends the email? I presume the "Estimated completion rate" for each "Campaign overview" is the rate at which emails are being queued into Redis, not the actual rate of emails being sent. I'd be interested to know how fast Redis is sending emails.
     
  13. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    29
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Question. Does Redis need to be configured (i.e. change any Redif settings in redis.conf) to make it properly work for MailWizz or is default fine?

    Furthermore, how does MailWizz make use of Redis? Does it clear out Redis when it is done with all queues? Are there specific keys created? I ask because I'm thinking of writing a script that checks Redis for specific keys related to MailWizz; if it finds some, it checks to see if the Redis queue agent is running. If not running, it runs it. If it finds no keys, it kills all Redis queue agents that are running. Throw this script into a cron and that way the Redis queue agent is run / killed on an as needed basis.
     
  14. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
  15. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,125
    Likes Received:
    2,352
    A few answers for you guys:
    That could work as well, something like:
    Code:
    #!/bin/bash
    ps cax | grep  queue >/dev/null
    if[ $? -eq 0];then
         echo "Process is running."
    else
         echo "Process is not running, start it..."
    fi
    
    ref: http://stackoverflow.com/questions/9117507/linux-unix-command-to-determine-if-process-is-running

    Actually if you run:
    Code:
    php -q /the/path/to/console.php queue --show_size=1
    
    You'll get back the number of emails in queue.

    Yes it does.

    Mailwizz uses resque behind the scenes, so have a look at https://github.com/resque/resque for techincal stuff.
     
  16. Michael Wilding

    Michael Wilding Active Member

    Joined:
    Apr 28, 2015
    Messages:
    260
    Likes Received:
    28
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Does Redis need to be installed on the server where the mailing software is or the SMTP server?
     
  17. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,125
    Likes Received:
    2,352
    Doesn't matter, but best is to be installed in same machine where mailwizz is.
     
  18. Michael Wilding

    Michael Wilding Active Member

    Joined:
    Apr 28, 2015
    Messages:
    260
    Likes Received:
    28
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
  19. Michael Wilding

    Michael Wilding Active Member

    Joined:
    Apr 28, 2015
    Messages:
    260
    Likes Received:
    28
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    I'm getting this when I try to start the daemon:

    Could not fork worker 0

    Any ideas?
     
  20. Rob

    Rob Active Member

    Joined:
    Feb 17, 2015
    Messages:
    404
    Likes Received:
    99
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    1
    Have you got php-redis installed?
     

Share This Page