We have a few Magento customers using SagePay.
At around 1900 on Sunday 19th November we started receiving reports from our eCommerce customers that their sites were receiving timeout issues when their customers were pressing the all-important Place Order button.
The message on screen and in logs should resemble something like:
2017-11-20 16:54:19.000000 (1511196859.8531) ALERT: Operation timed out after 90000 milliseconds with 0 bytes received 2017-11-20 16:57:09.000000 (1511197029.2547) ALERT: Operation timed out after 90001 milliseconds with 0 bytes received
Our initial prognosis was that this was a routing issue between the eComm installation and Sagepay servers, similarly SagePay’s own support portal at https://www.sagepay.co.uk/support/system-monitor reported an intermittent issue –
We are currently experiencing intermittent interruptions with MySagePay and the Live Gateway. Our investigations are underway to establish the root cause and to resolve this. Further updates to follow.
However as of the afternoon of Monday 20th November, we find that the issue is still ongoing.
We have been incontact with eBizmarts (the provided of SagePay integration solutions https://store.ebizmarts.com/extensions/sage-pay-suite-pro.html) and confirmed that the independent efforts of the community have tracked this down to a routing issue with some Networking changes (DNS and routing!) implemented by SagePay (L5 cannot confirm or deny this – it would be consistent with what we’re seeing).
The best way to diagnose this on your own eCommerce Server would be to ask your host to perform a traceroute to live.sagepay.com, and flush all local DNS caches.
The IP address we are consistently seeing for live.sagepay.com is now 195.170.169.9
In addition we see the following traceroute output:
3 akamai.prolexic.com (195.66.236.31) 9.703 ms 9.732 ms 9.890 ms 4 a72-52-60-202.deploy.static.akamaitechnologies.com (72.52.60.202) 9.090 ms a72-52-60-194.deploy.static.akamaitechnologies.com (72.52.60.194) 9.015 ms 8.815 ms 5 a72-52-60-197.deploy.static.akamaitechnologies.com (72.52.60.197) 9.789 ms a72-52-60-205.deploy.static.akamaitechnologies.com (72.52.60.205) 9.502 ms a72-52-60-197.deploy.static.akamaitechnologies.com (72.52.60.197) 9.404 ms 6 a209-200-176-172.deploy.static.akamaitechnologies.com (209.200.176.172) 11.757 ms 11.862 ms 11.113 ms 7 * * * 8 * * *
And those last entries repeat ad infinitum.
This confirms issues with SagePay’s recent routing changes, and depending on the ISP backbone and routing in play on your eComm infrastructure and host, determines whether you’re going to hit this problem.
As at 2000 on Monday 20th Nov, there has been no official word from SagePay on this, but rest assured the eBizmarts team and independent eComm implemenation outfits are weighing in and presenting this evidence to try and get this resolved.
We will post up a workaround using static routing / DNS here as soon as we have reliable tech details.
In the meantime feel free to watch this twitter thread as the guys at @thePixel are hammering it too: https://twitter.com/Duncan_ThePixel/status/932675891708284928
We can confirm that https://t.co/bir6yX5jDN (which sagepay added over the weekend) is the source of the problem. Using a host file and ssh forwarding to a server that doesn't route through that host (e.g. Linode) solves the issue.
— Duncan Ball (@Duncan_ThePixel) November 20, 2017
Update 21st November
From SagePay Support:
21 November 2017 – 13:37
As part of ongoing improvements following the interruption to payment processing on Saturday 18th November, our technology teams performed a fix yesterday evening around 11pm.
This fix has resolved the intermittent issues which a small number of merchants were encountering.