Improving ElasticEmail API

Discussion in 'General discussions' started by Lakjin, Jun 7, 2017.

  1. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    in /apps/common/models/DeliveryServerElasticemailWebApi.php change this...
    Code:
                if ($response['status'] != 'success' || strpos($response['message'], '-') !== false) {
                    throw new Exception(Yii::app()->ioFilter->stripClean($response['message']));
                }
    
    To this...
    Code:
                if ($response['status'] != 'success' || substr_count($response['message'], '-') != 4 || strlen(trim($response['message'])) != 36 || stripos($response['message'], 'error') !== false) {
                    throw new Exception(Yii::app()->ioFilter->stripClean($response['message']));
                }
    
    This helps better identify that the response is the job ID (the unique ID ElasticEmail returns upon successful send) and identifies if ElasticEmail returns an error (I've had an issue with their API returning 403 error but MailWizz not properly detecting that).
     
    Last edited: Jun 7, 2017
  2. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    Why does the $response['message'] must be 36 chars in length ?
     
  3. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Because in their v1 API (which MailWizz uses; you may want to consider upgrading to v2) returns a GUID on successful delivery, which is 36 characters in length with 4 dashes.
     
  4. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    What happens when delivery fails -- where is the failed delivery recorded? Does MailWizz attempt redelivery? After adding the above code, I no longer see failed deliveries (the one that threw 403 errors) in delivery_log but they also have not been delivered.
     
  5. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    @Lakjin - We don't retry sending failed emails. We used to, but dozen version ago we removed this because of the overhead.
    Most likely it needs more attention, i'm looking into it today.
     
  6. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    677
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    Definitely a good idea to bring this back, with a blast/improvement (and allow those who don't want it to switch it off).
    Especially now, that there is @AHK's extension!
    Thx for looking into it :)
     
  7. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Would you consider adding it back? It seems like a critical feature for deliverability reliability, as delivery can fail for numerous reasons. What about a simple approach of simply keeping a delivery_failed database and reprocessing the emails in that database at the end of a campaign?

    EDIT: Another approach would be simply attempting redelivery right after failure.
     
    Last edited: Jun 8, 2017
  8. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    @Lakjin - Sure i am considering to bring it back, but not in the way it was till now.
    I was thinking to do a check right before the campaign must be put to sent status, and remove all the emails with giveup status, do this 3 times, and then mark the campaign as sent. This way we know we tried.
    I just need to make sure and weight the impact this might have on performance though and more than this, how this will work with the temporary tables feature, which will be pretty tricky since it implies repopulating those tables.
     
  9. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    677
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    Do you mean remove them so that one cannot even find that htey were GiveUps? This would make @AHK's extension unusuable, and it is so good to have it!

    If it is optional, then folks decide if they want to focus on delivery or speed.

    Can you remind if the temp tables work with pcntl, and if they are enabled by default now (so far it was in beta and not enabled, correct)?

    Maybe combine options:
    # temp tables and not much error handling (and no pcntl)
    # no temp tables, full error handling, pcntl
    ?
     
  10. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    His extension does exactly this, it removes giveups to send to them again ;)

    Yeah, most likely we will put an option in admin, makes sense this way.

    They do work just fine with pcntl, but they are not enabled by default just yet.
     
    frm.mwz likes this.
  11. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    677
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    Ah, thanks for clarifying what "remove giveups" means. I like this kinda 'remove', and hence pledge to remove all errors (ie take care of them)!
    ;)

    Maybe a notification email with all errors (ie all stati except 'success') could be sent (optionally) after each camp/cron? This way one would have a record of (hopefully) decreasing delivery errors (if any), and be able to handle manually if need be?

    Thanks for working into this direction, it is one the great strengths of MailWizz to take care of things far more thoroughly than other mailers!
    :cool:
     
  12. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    How about an easy solution something along the lines of:

    -Create a new "delivery_failed" table
    -When delivery fails of an email, instead of putting that email into "delivery_log", put it into "delivery_failed"
    -Every time send-campaigns cron runs, check "delivery_failed" table for emails that need to be resent. Emails that are resent successfully, or have been retried 3 times (or whatever limit you set), can be put in "delivery_log" table

    This way, no impact on performance and it will work just fine with temporary queue tables.

    I think this feature is critical for normal uses. In my situation, it is even more so critical due to issues I'm having with 403 delivery errors. Apparently ElasticEmail's API limits 30 concurrent connections per IP, so about 10-15% of my emails are returning with 403 errors due to having too many connections to their API and MailWizz is not resending the emails later on. I've added temporary code to attempt retry, but my temporary code retries immediately after failure, which isn't a solution given the connections limitation.
     
  13. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Were you able to figure out why failed emails aren't in delivery_log?
     
  14. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    677
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    Have you tried with no more than 30 pcntl?
     
  15. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    That would work, of course, but that means our sending is slower than I want. ElasticEmail has helped me find a temporary solution to the problem, as the problem is at their end of 30 concurrent connections limitation, while they work on a more permanent solution.
     
  16. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    An update here, i added back the ability to resend to failed emails, you have to enable it from Backend > Settings > Cron:
    Screenshot 2017-06-14 14.15.25.png

    The way it works is just i explained above, right before closing a campaign and marking it as sent, we do a lookup for all the failed emails and we mark them for sending once again. This works just fine for regular campaigns and autoresponders, and also when using queue tables.
     
    frm.mwz likes this.
  17. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Awesome, thank you. When will the version with this be released?
     
  18. twisted1919

    twisted1919 Administrator Staff Member

    Joined:
    Dec 27, 2014
    Messages:
    10,287
    Likes Received:
    2,389
    I don't have an ETA right now, maybe we can make a release over the weekend.
     
  19. Lakjin

    Lakjin Active Member

    Joined:
    Mar 1, 2015
    Messages:
    391
    Likes Received:
    30
    S.E:
    Expired
    L.T:
    Regular
    L.C:
    6
    Got it, thanks!
     
  20. frm.mwz

    frm.mwz Well-Known Member

    Joined:
    Mar 8, 2016
    Messages:
    3,708
    Likes Received:
    677
    S.E:
    2019-11-27 02:17:39
    L.T:
    Regular
    L.C:
    7
    That's great.
    Would this mean that, once the 3x retry is done, any failed ones (are they still/again marked/findable?) can still be attempted with AHK's extension (e.g. one would have time to prepare other DS)?
     

Share This Page