topic Re: Webhook suddenly stopped working in Integration and Testing

Webhook suddenly stopped working

AnandBhagwat — Wed, 17 Jul 2019 13:43:14 GMT

Hi,

I have been using Webhook and receiving notifications from the sandbox account for the last couple of months and everything was working fine. However today suddenly the webhook notifications have stopped firing.

I made the webhook inactive and tried ping and it is sucessfull. However when I submit a transaction via the Virtual Terminal or through my code no notification is triggered ( I have subscribed to Payment- AuthCapture transactions)

I even switched the endpoint to use http//webhookinbox , but the same thing happens there, get a message for ping, but when I submit a transaction via the Virtual terminal there is no message

Can anyone help me find out what is going on ?

Also the notification history shows the following - so there was one notification delivered today and then it stopped.

{
"_links": {
"self": {
"href": "/rest/v1/notifications?offset=0&limit=100&from_date=7/16/2019 12:00:00 AM&to_date=7/17/2019 12:58:22 PM"
}
},
"notifications": [
{
"_links": {
"self": {
"href": "/rest/v1/notifications/9da23d44-bb2c-473c-81eb-215b6bea112e"
}
},
"notificationId": "9da23d44-bb2c-473c-81eb-215b6bea112e",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-16T04:44:59.243",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
},
{
"_links": {
"self": {
"href": "/rest/v1/notifications/4733a36e-4794-4be2-a844-274a9a2c6d70"
}
},
"notificationId": "4733a36e-4794-4be2-a844-274a9a2c6d70",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-16T10:42:40.757",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
},
{
"_links": {
"self": {
"href": "/rest/v1/notifications/09f36c3d-df98-491a-8862-e049be4d006b"
}
},
"notificationId": "09f36c3d-df98-491a-8862-e049be4d006b",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-17T05:41:58.717",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
}
]
}

Re: Webhook suddenly stopped working

kreynen — Wed, 17 Jul 2019 14:30:18 GMT

I don't know if this related, but trying to use a sandbox to write something to monitor the webhook notifications for retry attempts. We've had issues in prodcution where webhook integrations are disabled, but no one can figure out why.

The way I was approaching this was to enable the webhook. Once I confirmed it was working I would change the server configuration so those requests would fail with a 401 response.

After ~48 hours, Authorize disables the webhook as expected, but when I look at https://apitest.authorize.net/rest/v1/notifications?deliveryStatus=Failed it returns an empty [].

I was expecting to see at least 1 failed notification since I thought that was what was triggering the webhook integration to be disabled.

If I check the details of the notificaiton for the last 2 days, they are all deliveryStatus=Delivered with only 1 showing any retry attempts.

I am very confused. Is there an issue that is preventing the API from returning failed notifications on a sandbox?

Re: Webhook suddenly stopped working

Renaissance — Wed, 17 Jul 2019 19:41:53 GMT

@AnandBhagwat

Strangely, I had the exact same issue yesterday. Pings in test mode hit but repeated test transactions with it in active mode did nothing.

@kreynen

Webhooks are only marked failed when all retry attempts have been exhausted. Just because one notification for a particular webhook doesn’t come through it will not mark the webhook as failed. Sounds like you’re already on the same page but just FYI.

Re: Webhook suddenly stopped working

kreynen — Wed, 17 Jul 2019 19:48:58 GMT

@Renaissance Just so I'm clear, shouldn't there be at least 1 notification where deliveryStatus = Failed if the webhook integration was moved from active -> inactive?

I've been approaching this assuming I'm going to be able to figure why the webhook was deactiveated by monitoring the retries and failures. If retires get's > 5, someone should probably look at what is ahppening. I thought the webhook would remain active while it attempted retries 10x over 48 hours and only get disabled if the issue causing the retries wasn't resolved and the notificaiton deliveryStatus = Failed. Is that not the case?

Re: Webhook suddenly stopped working

Renaissance — Wed, 17 Jul 2019 21:08:37 GMT

@kreynen

You have a few things:

1- is the webhooks endpoint active or inactive in the merchant interface (production or sandbox, both work the same. Production test mode is useless however, for testing)

2- has the webhook been delivered

3- if 2 is a no, has the webhook had the maximum number of retries

#1 has nothing to do with how many failed or successful webhooks have been attempted. You can also create these programmatically, but the point is your webhooks will not be disabled completely unless you move them to inactive either in the interface or programmatically.

#2 is an obvious deliveredStatus webhook

If #3 answer is yes, then you have a a failed webhook. You have to distinguish between failed webhooks and unsuccessful notifications. Each webhook will have an Id, each notification will had an Id. So for instance, if you have a webhook for transaction Id 123456789, there will be exactly 1 webhook Id for that transaction. Each notification attempt for that transaction will have the exact same webhook Id. The notifications tied to that webhook will have a unique notification Id tied to the notification. A failed notification != a failed webhook. A failed webhook only happens after the maximum number of notification attempts have been made.

I am making a post because I am inexplicably having this issue again today. Webhooks are delayed for no reason I can pin down.

Re: Webhook suddenly stopped working

kreynen — Wed, 17 Jul 2019 21:56:48 GMT

So if the action hasn't either sucessfully connected with the end point (deliveryStatus=Delivered) or failed 10x (deliveryStatus=Failed), is it not returned in https://apitest.authorize.net/rest/v1/notifications ?

The webhook is active and working when I start. I'm causing it to stop working with a sandbox and a test server so I can write something to watch for retries and catch the problem before it gets to the 10th attempt.

When our test server is configured to accept the webhook POSTs from the sandbox, it works and I see something in the NGINX logs like...

35.192.89.4 - - [16/Jul/2019:14:08:12 +0000] "POST /authnet_accept/webhook HTTP/1.1" 200 31 "-" "Go-http-client/1.1" 0.648 "198.241.206.38, 198.241.206.38"

Then I change the server's configuration so POSTs to /authnet_accept/webhook result in a 401 response from the server and I'll see...

35.222.236.208 - - [16/Jul/2019:14:07:38 +0000] "POST /authnet_accept/webhook HTTP/1.1" 401 3097 "-" "Go-http-client/1.1" 0.000 "198.241.206.38, 198.241.206.38"

That is all what I'd expect. Some of the payloads from the Webhook posts trigger functionality. Others are ignored, but the server always responds with a 200 leading Authorize.net to believe everything was a success.

What I can't figure out is why I'm not about to find a notication in https://apitest.authorize.net/rest/v1/notifications?deliveryStatus=Failed where I see all 10 retries.

Based on this thread, I'm beinging to think that the APIs in the sandbox are missing data, but I haven't worked with these Authorize APIs enough to be confident I'm not missing something simple.

Re: Webhook suddenly stopped working

Renaissance — Wed, 17 Jul 2019 22:06:01 GMT

@kreynen

If this pertains to the past few days, auth.net appears to be having issues. I have ran transactions on very well tested urls to confirm, and we are getting nothing.

Re: Webhook suddenly stopped working

Renaissance — Wed, 17 Jul 2019 22:07:31 GMT

And I get an empty array too, although I am not sure I have had genuine failures yet. It hasn’t been 48 hours on mine since I confirmed this issue .

Re: Webhook suddenly stopped working

kreynen — Wed, 17 Jul 2019 22:11:46 GMT

So everyone else is just ignoring all the green on https://status.authorize.net/ and posting here when they see issues?

Re: Webhook suddenly stopped working

Renaissance — Wed, 17 Jul 2019 22:24:44 GMT

I suppose. Do you have a server side log of the IP addresses that hit your App? That is the place to go.

Re: Webhook suddenly stopped working

Renaissance — Thu, 18 Jul 2019 11:14:58 GMT

@kreynen

To follow up, your webhooks will not be disabled completely, I do not think. I think the one webhook
expires and then the rest are delivered business as usual. For the triggering functionality, that would be something related to your script, wouldn’t it? I think auth.nets work is done once they’ve delivered the payload.

Re: Webhook suddenly stopped working

AnandBhagwat — Thu, 18 Jul 2019 11:58:42 GMT

Well, the webhook has started working again today. Not clear why or how . I have not made any changes to code or configuration. Perhaps someone from Authorize.NET can comment on this , was there a downtime yesterday for webhooks ?

Re: Webhook suddenly stopped working

Renaissance — Thu, 18 Jul 2019 18:39:58 GMT

@AnandBhagwat

Same exact thing here. For sure there was something going on with the system that was delaying or impeding webhooks from hitting endpoints. It is extremely rare. There has hardly been a week go by for the past 9 months that I don’t run at least one test transaction, and in my development phase I will often run dozens in one day to test new functionality. I run all my test on a remote server that is configured to receive webhooks. I cannot count how many times I have deleted all the records of my customers, orders, items, etc tables due to the list being too massive to deal with. I’m at over 1,500 webhooks received, and 99.5% come back within 5 seconds of hitting the pay button.

I was working with a different and newer client when this issue first came up. So after a while I switched to my heavily vetted preexisting test site url. I ran two transactions and it took over an hour for them to come back. That was at about 8 pm yesterday. After those 2 came back, things started being business as usual immediately, with instant responses.

The day before they were “down” intermittently. On and off again you would get an instant response. Like I say, first issue like this ever in going on 9 or 10 months of continuous testing.