Webhook suddenly stopped working

[AnandBhagwat]

Hi,

I have been using Webhook and receiving notifications from the sandbox account for the last couple of months and everything was working fine. However today suddenly the webhook notifications have stopped firing.

I made the webhook inactive and tried ping and it is sucessfull. However when I submit a transaction via the Virtual Terminal or through my code no notification is triggered ( I have subscribed to Payment- AuthCapture transactions)

I even switched the endpoint to use http//webhookinbox , but the same thing happens there, get a message for ping, but when I submit a transaction via the Virtual terminal there is no message

Can anyone help me find out what is going on ?

Also the notification history shows the following - so there was one notification delivered today and then it stopped.

{
"_links": {
"self": {
"href": "/rest/v1/notifications?offset=0&limit=100&from_date=7/16/2019 12:00:00 AM&to_date=7/17/2019 12:58:22 PM"
}
},
"notifications": [
{
"_links": {
"self": {
"href": "/rest/v1/notifications/9da23d44-bb2c-473c-81eb-215b6bea112e"
}
},
"notificationId": "9da23d44-bb2c-473c-81eb-215b6bea112e",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-16T04:44:59.243",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
},
{
"_links": {
"self": {
"href": "/rest/v1/notifications/4733a36e-4794-4be2-a844-274a9a2c6d70"
}
},
"notificationId": "4733a36e-4794-4be2-a844-274a9a2c6d70",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-16T10:42:40.757",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
},
{
"_links": {
"self": {
"href": "/rest/v1/notifications/09f36c3d-df98-491a-8862-e049be4d006b"
}
},
"notificationId": "09f36c3d-df98-491a-8862-e049be4d006b",
"deliveryStatus": "Delivered",
"eventType": "net.authorize.payment.authcapture.created",
"eventDate": "2019-07-17T05:41:58.717",
"webhookId": "52b926e4-c911-4e50-b06d-ee2d2ba73bea"
}
]
}

[kreynen]

I don't know if this related, but trying to use a sandbox to write something to monitor the webhook notifications for retry attempts.  We've had issues in prodcution where webhook integrations are disabled, but no one can figure out why.

The way I was approaching this was to enable the webhook.  Once I confirmed it was working I would change the server configuration so those requests would fail with a 401 response.

After ~48 hours, Authorize disables the webhook as expected, but when I look at https://apitest.authorize.net/rest/v1/notifications?deliveryStatus=Failed it returns an empty [].

I was expecting to see at least 1 failed notification since I thought that was what was triggering the webhook integration to be disabled.

If I check the details of the notificaiton for the last 2 days, they are all deliveryStatus=Delivered with only 1 showing any retry attempts.

I am very confused.  Is there an issue that is preventing the API from returning failed notifications on a sandbox?

[Renaissance]

@AnandBhagwat

Strangely, I had the exact same issue yesterday. Pings in test mode hit but repeated test transactions with it in active mode did nothing.

@kreynen

Webhooks are only marked failed when all retry attempts have been exhausted. Just because one notification for a particular webhook doesn’t come through it will not mark the webhook as failed. Sounds like you’re already on the same page but just FYI.

[kreynen]

@Renaissance Just so I'm clear, shouldn't there be at least 1 notification where deliveryStatus = Failed if the webhook integration was moved from active -> inactive?

I've been approaching this assuming I'm going to be able to figure why the webhook was deactiveated by monitoring the retries and failures.  If retires get's > 5, someone should probably look at what is ahppening.  I thought the webhook would remain active while it attempted retries 10x over 48 hours and only get disabled if the issue causing the retries wasn't resolved and the notificaiton deliveryStatus = Failed.  Is that not the case?

[Renaissance]

@kreynen 

You have a few things:

1- is the webhooks endpoint active or inactive in the merchant interface (production or sandbox, both work the same. Production test mode is useless however, for testing)

2- has the webhook been delivered

3- if 2 is a no, has the webhook had the maximum number of retries  

#1 has nothing to do with how many failed or successful webhooks have been attempted.  You can also create these programmatically, but the point is your webhooks will not be disabled completely unless you move them to inactive either in the interface or programmatically.

#2 is an obvious deliveredStatus webhook

If #3 answer is yes, then you have a a failed webhook. You have to distinguish between failed webhooks and unsuccessful notifications.  Each webhook will have an Id, each notification will had an Id.  So for instance, if  you have a webhook for transaction Id 123456789, there will be exactly 1 webhook Id for that transaction. Each notification attempt for that transaction will have the exact same webhook Id. The notifications tied to that webhook will have a unique notification Id tied to the notification.  A failed notification != a failed webhook.  A failed webhook only happens after the maximum number of notification attempts have been made. 

I am making a post because I am inexplicably having this issue again today. Webhooks are delayed for no reason I can pin down. 

[kreynen]

So if the action hasn't either sucessfully connected with the end point (deliveryStatus=Delivered) or failed 10x (deliveryStatus=Failed), is it not returned in https://apitest.authorize.net/rest/v1/notifications ?

The webhook is active and working when I start.  I'm causing it to stop working with a sandbox and a test server so I can write something to watch for retries and catch the problem before it gets to the 10th attempt.

When our test server is configured to accept the webhook POSTs from the sandbox, it works and I see something in the NGINX logs like...

35.192.89.4 - - [16/Jul/2019:14:08:12 +0000] "POST /authnet_accept/webhook HTTP/1.1" 200 31 "-" "Go-http-client/1.1" 0.648 "198.241.206.38, 198.241.206.38"

Then I change the server's configuration so POSTs to /authnet_accept/webhook result in a 401 response from the server and I'll see...

35.222.236.208 - - [16/Jul/2019:14:07:38 +0000] "POST /authnet_accept/webhook HTTP/1.1" 401 3097 "-" "Go-http-client/1.1" 0.000 "198.241.206.38, 198.241.206.38"

That is all what I'd expect.  Some of the payloads from the Webhook posts trigger functionality.  Others are ignored, but the server always responds with a 200 leading Authorize.net to believe everything was a success.

What I can't figure out is why I'm not about to find a notication in https://apitest.authorize.net/rest/v1/notifications?deliveryStatus=Failed where I see all 10 retries.

Based on this thread, I'm beinging to think that the APIs in the sandbox are missing data, but I haven't worked with these Authorize APIs enough to be confident I'm not missing something simple.

[Renaissance]

@kreynen

If this pertains to the past few days, auth.net appears to be having issues. I have ran transactions on very well tested urls to confirm, and we are getting nothing.

[Renaissance]

And I get an empty array too, although I am not sure I have had genuine failures yet. It hasn’t been 48 hours on mine since I confirmed this issue .

[kreynen]

So everyone else is just ignoring all the green on https://status.authorize.net/ and posting here when they see issues?    

[Renaissance]

I suppose.  Do you have a server side log of the IP addresses that hit your App? That is the place to go.