Quite like that first red one.
1 Like
Presumably if the channels are fed into Foxtel’s system, then the Foxtel slide would be displayed if the link with the channels source is lost.
1 Like
Coles are still struggling today, extra staff on hand, queues to checkout and multiple registers down.
4 Likes
2 Likes
Similar with a lot of Woolies stores. Ours canned the deliveries early on today for the rest of the day and they’ve rostered on a lot more staff for today and tomorrow for things.
This is where things will be interesting to see what happens. This is not just a small oops, the downtime and costs to businesses worldwide would be well into the billions and what happens there, who pays for it? Does cyber insurance cover this one? Do the companies have to wear it? Is there legal options available for the Crowdstrike customers - but then what about those indirectly affected because of companies further up the chain?
It wasn’t caused by a cyber attack so cyber insurance would not cover it. Most businesses may have other insurance that cover it but it would be pretty unlikely they could get much compensation from Crowdstrike maybe a few months of free service if they want to continue using it.
I think impacted companies are going to end up wearing it - Cyber insurance is unlikely to cover the impacts of what is effectively misconfigured software (its often aimed at the consequences of Data Breaches and Phishing). At best if you’ve got business disruption insurance you might get something
You’ll be lucky to get anything out of CrowdStrike too - they’ve more then likely got protections in the T&Cs for this
1 Like
4 Likes
It’s been a fucking pain to get bitlocker keys to every site and impacted device.
4 Likes
I can imagine poor techs that couldn’t make it in person and had to coach those through cmd 
6 Likes
A lot of video calls and long phone calls.
1 Like
Just as I thought but that’s a huge impact and cost to wear for many businesses. Easily the biggest tech disruption ever now and with no warning.
“safe mode? so wasn’t my PC safe this whole time?!”
in all seriousness though as an IT Tech, good luck to all those who have to deal with the CrowdStrike situation - thankfully we don’t use CrowdStrike in our operations, but my heart still goes out to the techs dealing with this situations
8 Likes
The Bitlocker keys in a lot of cases in Active Directory environments are stored on servers and as long as that server is up and running it should in theory allow the devices to boot into safe mode but optaining the key for that servers would not be easy as it’s a root device and would be stored elsewhere and only certain people would have access to it as its so sensitive.
Most DR systems are generally have the same software as the primary site but may take a different network path. It’s unlikely that a lot of businesses would have different endpoint protection software in DR as their primary site but that needs to change but I doubt that would have helped much in this situation. They need to figure out how a zero byte file was sent out in the first place.
3 Likes
good points
for me, it’s like why didn’t CrowdStrike test the deployment out in a sandbox before even deploying the patch publicly to their clients? Not to mention, that deploying on a Friday is usually frowned upon lol
2 Likes
My friend and her daughter are on holidays in Italy to catch up with family and do a bit of travelling.
Yesterday they were meant to fly from Barcelona to Ibiza. Everything had to be done manually which included them writing their names on a list of people going on their flight. When it was time to board there were no boarding passes, no proper screening and everyone was told to sit wherever they wanted. The trip is meant to take about an hour but this one took 12 hours.
On the flight over she changed all her money to euros in case she couldn’t use her credit cards. They finally arrived in Ibiza only to find that her luggage was still in Barcelona and her daughter’s luggage somehow ended up in Nice.
With no luggage they went to a shop and bought her daughter a dress and then went to a licensed cafe for lunch. The were approached by a father and son who were behaving like ‘sleazy Italians’. She didn’t want to be rude and was politely trying to get rid of them. Eventually they left - with her purse.
No luggage, no credit cards and no cash.
And to add to the fun her daughter now has covid.
9 Likes
They would need to push changes to handle threats very quickly but i find it hard to believe that it would not have gone though a suite of automated tests at a bare minimum. It’s likely for some reason it got through those tests and I am hoping the company provides a clear post incident report once a full investigation into it is complete.
I recently moved into a development role after working in IT support for many years and we run unit tests for everything but things can still get through and we even have a test team that tests everything before it goes to production.
The National Anti-Scam Centre is warning consumers and small businesses to be wary of unsolicited calls, emails or messages requesting they download a software patch or provide remote access to fix or protect their computer from the CrowdStrike/Microsoft outage. pic.twitter.com/IL31KDjKFK
— NASC Scamwatch (@Scamwatch_gov) July 20, 2024
CrowdStrike is an enterprise product so naturally only big business can afford it, but look how easy it was to take them all out. One bad update and a whole bunch of key services are crippled. And in a world setup for remote access this is one of those rare issues that required people to be on-site to fix (unless IT were stupid enough to actually get users to do it themselves).
This is the problem with IT these days. Management insist on cutting budgets because “IT costs us money” so everything gets outsourced to third party vendors. No testing of updates anymore, just leave auto updates on because IT are over-worked and under-appreciated and stopped caring. This should be a wake up call to the bean counters in big business that IT needs more resources and that any and all software updates need to be tested by IT before they get pushed out to every system.
This was also a lesson to the new generation of IT to stop running Windows on absolutely everything, because that’s all they know how to do. There’s not really much choice with desktop but with servers there is. We don’t have these ridiculous issues in the Linux world.
Microsoft need to put a stop to this as well. Cybersecurity and antivirus products should not be running as device drivers. That should never have been allowed to happen. They need their own layer that does not have the ability to bring down the OS. They are not drivers and shouldn’t be allowed to act as one.
Hopefully this does bring about some meaningful change to prevent someone else from doing this in the future.
Sorry to those who lost their Friday night and weekend to fix this.
21 Likes
the effects of the Crowdstrike outage are still being felt - ABCTV’s Landline program cannot run its advertised edition this week and a replay of the 30th Anniversary special from 2021 is airing in its place.
5 Likes