Email validation before sending

[holgroup]

I just took a look in the code to make sure I don't speak from memories.
The code responsible for this looks like:

$url = sprintf('%s/?key=%s&email=%s', $apiUrl, $apiKey, $email);
        try {
            $response = (string)(new GuzzleHttp\Client())->get($url)->getBody();
        } catch (Exception $e) {
            $response = '';
        }

        if (empty($response)) {
            return $emails->itemAt($email);
        }

So the HTTP request is made regardless, BUT if it fails for some reason, we assume the email is valid and move on, so the explanation here would be that there are failed requests, but the number is just too high, and I think you should open a ticket with them and have them offer some insights into this, and we can continue from there.
I'll have to ping my colleague @ghimes to take a look at this since i'll be away for a few days. If anything, open a support ticket and discuss with him.

Thanks for this, much appreciated for the detail.

Unfortunately, the situation just goes from bad to worse, and it turns out that the CLI solution that was supposed to be one of the valid fixes isnt really worth the time. The email verification process asks a question to the third party service, and then goes ahead and adds them as confirmed anyway without any kind of flag that the email hasnt been checked (yet) having been added to the list. Now the 20k subscribers I have in my list, I dont have a clue whats valid or not. So far across 2 different third party solutions I have now burned my way through about 20,000 credits unnecessarily.

If the service hasnt received a positive confirmation, then they should rightly be added to the list as unconfirmed so the customer has a clear understanding of how using said mailing list with so many high probabilities of invalid addresses will obliterate and ruin their sender reputation. I have opened a ticket with yet another third party service but there is a very serious flaw in the logic of the validation process within the mailwizz script.

@ghimes please advise tomorrow when you can?

 

[holgroup]

And in fact given the service response from bulkemailchecker there are many other options for the verification process to enrich the subscriber list with the results to give the user / customer more granularity:

Status="passed"- A passed response is an address that has passed all tests.
--Event="mailbox_exists"- The address provided passed all tests.
Status="failed"- A failed response is an address that has failed 1 or more tests.
--Event="mailbox_does_not_exist"- The address provided does not exist.
--Event="mailbox_full"- The mailbox is full.
--Event="invalid_syntax"- The address provided is not in a valid format.
--Event="domain_does_not_exist"- The address provided does not have a valid dns.
--Event="mxserver_does_not_exist"- The address provided does not have a mx server.
--Event="blacklisted_external"- The address provided was found in your (Mailchimp, Postmark, Sendgrid, etc.) blacklist.
--Event="blacklisted_failed"- The address provided was found in your failed blacklist containing previously failed addresses.
--Event="blacklisted_domain"- The domain provided was found in your domain blacklist.
Status="unknown"- A unknown response is an address that can not be accurately tested.
--Event="is_catchall"- Is a Catchall mx server config.
--Event="is_greylisting"- Greylisting is active on this server.
--Event="inconclusive"- Transient error, please try again later.

Additional Responses:​

"email"- The address that was validated.
"emailSuggested"- The address contained a typo and we have suggested the most likely repaired version.
"mailbox"- The mailbox that was validated.
"domain"- The domain that was validated.
"mxIp"- The ip address of the mail server that was validated.
"mxLocation"- The location of the mail server that was validated.
"isComplainer"- The address is a frequent complainer.
"isDisposable"- The address is a disposable email address.
"isFreeService"- The address is a free email account.
"isOffensive"- The address contains bad words.
"isRoleAccount"- The address is a role account.
"validationsRemaining"- The amount of validations remaining in your account balance.
"hourlyQuotaRemaining"- The amount of api requests remaining in your account for this hourly segment.
"execution"- The amount of time taken to validate this address in ms.

 

[holgroup]

The people at bulkemailchecker are saying:

"Unfortunately no, we only monitor successfully authenticated and processed requests which are the same statistics located inside your account dashboard. The hourly limit and concurrent connection limit failures (the only systems that would block a request from our end) are not tracked per account at this time.
Your account is permitted 1,500 hourly requests and 1 concurrent connection to the RESTful API v3 endpoint. Are you exceeding these rates?
Are any plain-text or HTTP response code errors being returned when this issue occurs?"

 

[twisted1919]

but there is a very serious flaw in the logic of the validation process within the mailwizz script.

When you import in MailWizz, we assume your emails are all correct, we trust you know what you are doing. This is why they import as confirmed.

Also, this is why, if validation fails in the sens we don't get an answer, we just let the email pass, otherwise, you'd remove an email which might as well be valid.

We only block an email after validation runs successfully, when the returned response we're sure it is a bad email address. The validation service can return many statuses, we don't care about all of them, only about the status that says for sure the email is bad, otherwise, it is going to pass.
We're not going to block emails where we had no response or where the status of the validation is not clear.
If you need it working any different way, you can have a different validation extension to do what you need.

And in fact given the service response from bulkemailchecker there are many other options for the verification process to enrich the subscriber list with the results to give the user / customer more granularity:

I think I covered this above.

The people at bulkemailchecker are saying:

"Unfortunately no, we only monitor successfully authenticated and processed requests which are the same statistics located inside your account dashboard. The hourly limit and concurrent connection limit failures (the only systems that would block a request from our end) are not tracked per account at this time.
Your account is permitted 1,500 hourly requests and 1 concurrent connection to the RESTful API v3 endpoint. Are you exceeding these rates?
Are any plain-text or HTTP response code errors being returned when this issue occurs?"

I think you're exceeding those limits given our discussion, so this might be an issue as well.

But regardless, I also think you have a point and we can improve on this area, for a start, we can at least check the response from the provider and see if there's something we can do against it. We won't be able to wait until the provider says you're good to send again, but maybe we can take an action when this happens, whatever that means, in this case, aborting the import most likely, which again, is not something I am keen to do for reasons I expressed above.
So yeah, check in with @ghimes and we can continue from there.

 

[holgroup]

When you import in MailWizz, we assume your emails are all correct, we trust you know what you are doing. This is why they import as confirmed.

Also, this is why, if validation fails in the sens we don't get an answer, we just let the email pass, otherwise, you'd remove an email which might as well be valid.

We only block an email after validation runs successfully, when the returned response we're sure it is a bad email address. The validation service can return many statuses, we don't care about all of them, only about the status that says for sure the email is bad, otherwise, it is going to pass.
We're not going to block emails where we had no response or where the status of the validation is not clear.
If you need it working any different way, you can have a different validation extension to do what you need.


I think I covered this above.


I think you're exceeding those limits given our discussion, so this might be an issue as well.

But regardless, I also think you have a point and we can improve on this area, for a start, we can at least check the response from the provider and see if there's something we can do against it. We won't be able to wait until the provider says you're good to send again, but maybe we can take an action when this happens, whatever that means, in this case, aborting the import most likely, which again, is not something I am keen to do for reasons I expressed above.
So yeah, check in with @ghimes and we can continue from there.

Thats great, thanks for taking the time to listen. I think certainly there are many improvements that can be made, I (for one) would be prepared to pay more for something more concise on the validation side.

I think also (as I have seen with other software) you can allow the user to specify maximum sending threshold / quota as you do with the delivery server, which would then throttle the requests to the verification service to be sure of not breaching limits as it would appear I may have done. In the case of B.E.C. as well as enriching the list with various data items received in the response, and this should be fairly easy to add as you are already receiving the response anyway it just needs handling, you can also check quota:
"validationsRemaining"- The amount of validations remaining in your account balance.
"hourlyQuotaRemaining"- The amount of api requests remaining in your account for this hourly segment.
"execution"- The amount of time taken to validate this address in ms.

 

[holgroup]

@ghimes the way I see it these improvements for the email validation and hence the overall experience to be more useful, and certainly for this to be something that would provide much more useful feature set for agency users, such that the following should be considered:

1. Right now, you have quotas for the delivery servers, these supply limits based on hourly, daily, monthly usage of the delivery service there also needs to be quotas applied to the e-mail verification services.
2. Because the API quotas are based on the number of API calls permitted are both dependent on the customer and more specifically the API credentials, These quotas need to be applied at the customer level where you would input the API key not at the back end administration level.
3. For CLI imports there is no status page showing how many have been imported so far as you get when you're doing a live file upload.
4. Neither the live file upload or the CLI file upload have a look for showing what has passed and what has failed, I can see from the rest calls made using Ajax in the browser that you do log responses coming back but these are not written anywhere in the programme this needs to change.
5. Right now, you are aborting the import process for live files if it takes too long to get a response, given that this is an Ajax call you should be able to simply re-queue another request after a predetermined amount of time. This should be part of the configuration at the same level as the API key being provided for the customer level within the system.
6. Right now, your colleague has proven that your code at e-mail addresses to the subscriber list regardless of whether it receives a response or not, the only time something is ever done is where a negative response is received. This is a ridiculous way of handling things and means that subscriber lists are dirty because they have not fully been checked on import. You should only mark a subscriber as confirmed if you've received positive response back from the third party, if it's failed you should mark that accordingly. No response back from the third-party verification service should be marked as an unconfirmed or such other status in the database.
7. There is a lot of information available from all of the API responses for the integrations that you're offering for e-mail verification, these can and should be used to enrich the subscriber list within the database.
8. You need the ability to be able to copy custom fields between lists, other software providers hold custom list separately within the table structure, these are then applied to each list in turn. Particularly for lists that have dropdowns with a large number of entities available for choice it should be possible to copy these detailed custom fields between one or more lists.
9. To be able to offer external third party validation as part of a solution provider model / agency basis, you should also permit API credentials to be added at the backend admin level. And also possibly allow for credits to be added (per monetised purchases) for agency model users.