Webhook Failed to respond in timely manner

I have a webhook that is failing, with the reason "Failed to respond in timely manner.". The intent to receive works when I re-activate the Webhook, but fails almost immediately following the OK status. The strange thing is that the server logs see the requests for the intent to receive, but not any subsequent request. I've put additional logging in place on my end to send info about the request to Slack, but I'm receiving nothing.

There isn't a history/log on the Xero side that I can use to debug this, but everything that I'm seeing suggests the requests aren't event hitting my server.

The webhook has been active for about 3 years, and as soon as the request comes in, it is saved for processing via a background job. I've not actually witnesses a request take 5sec to process - most are sub 100ms.

Answer

Potential Causes and Debugging Steps Firewall or Network Restrictions

Ensure there haven't been any recent changes to your network or firewall that might be blocking incoming requests.

Check if the IP addresses of Xero's webhook requests are being filtered.

Rate Limits or Request Blocking by Xero

Xero might have introduced new rate limits or validation requirements.

If your webhook is failing immediately after activation, Xero could be deactivating it due to consecutive failures.

SSL/TLS Certificates or DNS Issues

Confirm that your SSL certificates haven't expired or changed recently.

Run a DNS lookup to check if the webhook’s endpoint is resolving correctly.

Background Job Processing Logic

You mentioned that webhook requests are saved for background processing. Could there be a delay or misconfiguration that prevents the server from acknowledging receipt within Xero’s expected timeframe?

Webhook Request Format or Authentication

Verify if Xero has updated the request payload or authentication method required.

Try setting up a temporary endpoint that simply logs requests to verify what Xero is sending.