We're thinking to remove the Queue Tables feature

Status
Not open for further replies.

twisted1919

Administrator
Staff member
Hello,

When we introduced queue tables in the past, it was because we wanted faster sending when calling the send-campaigns command, and the queue tables feature, offered that. While it offered a slower start because it had to populate the data in the queue table, once that was done, the sending was super fast.

A few days ago, one of our customers had an issue with Autoresponders and Segmentation when using queue tables.
While we were investigating the issue, we end up in a place where we had to compare the speed of the sending process while using queue tables vs the regular sending process.
We tested both sending options with a segment of 100k subscribers from a list of 105k subscribers, and the timing we got is:
With queue tables:
Code:
./send.sh 1.19s user 0.75s system 0% cpu **22:52.81** total
./send.sh 1.20s user 0.67s system 0% cpu **22:38.38** total
./send.sh 1.17s user 0.63s system 0% cpu **22:02.75** total
./send.sh 1.17s user 0.64s system 0% cpu **22:36.07** total
./send.sh 1.18s user 0.72s system 0% cpu **22:16.99** total

Without queue tables:
Code:
./send.sh 1.19s user 0.70s system 0% cpu **20:40.77** total
./send.sh 1.17s user 0.66s system 0% cpu **21:13.36** total
./send.sh 1.18s user 0.71s system 0% cpu **21:58.16** total
./send.sh 1.17s user 0.75s system 0% cpu **21:21.06** total
./send.sh 1.19s user 0.71s system 0% cpu **21:48.04** total

While not much faster, sending without queue tables is actually faster now than the queue tables sending.
This is no surprise as we have improved the send campaigns command a lot in the last years.

Now, going back to the issue that lead us to this benchmark.
When using Autoresponders + Segments with queue tables, when new subscribers are added in the system, be it by import or subscription, we need to check and see if that subscriber matches a Segment for which we have a campaign going out already, and if it does, we have to add that subscriber in the queue table for sending. The work we do in order to keep the things in sync is just crazy and at this point I don't think it worth it since the regular sending is now faster than using queue tables.
Another problem that queue tables have, and we found out recently, is when using autoresponders + segments, and your send your campaign, then wait for new subscribers, if you then decide to modify your segment criteria to match more subscribers, then the newly matched subscribers won't be added to the queue table for sending, resulting in losing sending to some subscribers. While this scenario us very unlikely, I know it might happen.

That being said, we're now thinking to simply remove this feature as there is no benefit from using it anymore.
We can't remove it just like that, so our plan is as follows:
1. The next release, in the following days, will contain code that will make sure queue tables are not used for Autoresponders with Segments anymore.
2. When September starts, we remove the queue tables for Autoresponders for good, we simply don't use them anymore for ARs.
3. When October starts, we remove the queue tables feature for Regular campaigns as well, so we remove this feature for good from the application.

What does this means to you?
*) If you are not using queue tables, then you don't have to worry about a thing.
*) If you are using queue tables, when we switch off this feature, the sending will continue as it does now, nothing at all will break, so again, nothing to worry about.
*) When installing the next release, if you are using queue tables + autoresponders + segments, in case some subscribers were missed by the system to send them the autoresponders, they'll receive them now, so depending on your setup, this might mean one email or 10, you should check your system.

As always, we're open to discussion about our actions, so if you think the queue tables should stay, please provide a valid benchmark between the sending options and we can discuss it.

Thank you.
 
Hi @twisted1919 - this sounds good, if there's no benefit to something, there's no reason to have it in the software. They key is removing it so it doesn't affect the users of MW who might be using it - which it sounds like you are taking great care to do. Awesome!

My question: the ONLY way to have queue tables enabled is by putting THIS line in the main-custom.php file, correct?

'send.campaigns.command.useTempQueueTables' => true,

If so, I don't have that line in my main-custom.php - so that means we are not using queue tables in our installation. So we won't be affected. Am I right?

Thank you!
 
My question: the ONLY way to have queue tables enabled is by putting THIS line in the main-custom.php file, correct?

'send.campaigns.command.useTempQueueTables' => true,

If so, I don't have that line in my main-custom.php - so that means we are not using queue tables in our installation. So we won't be affected. Am I right?
That's totally correct, yes. You need to explicitly opt-in for this feature, and if you haven't already, then all good :)
 
Hi @twisted1919 - this sounds good, if there's no benefit to something, there's no reason to have it in the software. They key is removing it so it doesn't affect the users of MW who might be using it - which it sounds like you are taking great care to do. Awesome!

My question: the ONLY way to have queue tables enabled is by putting THIS line in the main-custom.php file, correct?

'send.campaigns.command.useTempQueueTables' => true,

If so, I don't have that line in my main-custom.php - so that means we are not using queue tables in our installation. So we won't be affected. Am I right?

Thank you!
Does this mean we can comment out that line to bypass the queue tables and thus fix the issue that led to this in the first place?
 
Does this mean we can comment out that line to bypass the queue tables and thus fix the issue that led to this in the first place?
Yes, if you remove that line from main-custom.php, in case you have added it, then you'll stop using queue tables.
 
Update, we're holding off from the last two points for now:
2. When September starts, we remove the queue tables for Autoresponders for good, we simply don't use them anymore for ARs.
3. When October starts, we remove the queue tables feature for Regular campaigns as well, so we remove this feature for good from the application.
 
Status
Not open for further replies.
Back
Top