AWS customer learns the hard way how even the smallest oversight can be mission-critical

Summary

off-prem AWS customer learns the hard way how even the smallest oversight can be mission-critical An expired card, overzealous spam filter, and broken authenticator create havoc with a very clear moral Nothing can ruin the end of a week like finding out that all of the websites and hosted email you’re responsible for are offline, which is exactly what happened to Christopher Bradbury and his web design and development firm, Digital Takumi last week. AWS has since resolved the situation, but Bradbury's mistakes can serve as a useful lesson to others of what not to do.As Bradbury explained to The Register in an email, he noticed last Thursday, July 16, that all the websites and Google Workspace email accounts connected to domains he manages were offline. All of those sites are hosted through Route 53, AWS’ combined DNS/hosting service, and none of them were resolving. Bradbury went digging through his emails to figure out if there was anything to point to the failures, and sure enough: In his spam folder were numerous messages from AWS telling him a payment card on file had expired, and warning him that the account used to host all those customer websites through Route 53 was going to be suspended unless he took action. He didn’t obviously, because he didn’t realize there was an issue. “AWS had been sending billing notifications, but unfortunately they had been filtered into a spam folder and, in some cases, were being delivered to an employee who had since left the business,” Bradbury told us. “I accept responsibility for missing those notifications,” he added, but that didn't help his customers' websites get back online. The situation could have been resolved sooner, but Bradbury had made some other mistakes as well:“The root account had MFA enabled using a software authenticator that had been stored on an older laptop which has since suffered a motherboard failure,” Bradbury explained. Without access to that authentication code generator, he was unable to get into ...