NTP server that traveled back in time caused massive Aussie mobile outage

Summary

networks NTP server that traveled back in time caused massive Aussie mobile outage Telstra skipped a patch, didn’t record changes, had no idea it was an accident waiting to happen Australian telco Telstra has revealed the cause of the recent incident that caused widespread connectivity problems across its mobile networks, inculding outages to Australia's 000 emergency services line, plus outages to electronic payments services and transport networks.The carrier explained itself in a submission [PDF] to a Senate inquiry into outages affecting Australia’s emergency services which initially investigated an outage at Telstra's main rival, Optus. The Optus incident is linked to multiple deaths after people could not reach emergency services.Telstra's submission reveals that the carrier's attempt to address a known resilience problem was the instigating incident. That problem was a faulty backup power feed in the chassis used to house a network time protocol (NTP) server. A few minutes before midnight on July 7, a Telstra techie started work to replace that chassis. By 3:38 AM on July 8, the worker had finished the job and powered up the server.The NTP server included a GPS card that Telstra says “did not operate as expected” when the server came back online. “We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault [which] had not been properly documented,” the submission states. “This meant the maintenance team was not aware of the way the device would behave when restarted.”Telstra also admitted it had not applied a software update to the device, despite knowing of its availability in early 2026.“Had that software update been completed or had the design change been properly reviewed and documented post the earlier incident, and reflected in the maintenance procedure, the outage may not have occurred,” the submission states.The outage did happen because once the NTP server came online it ...