Nine Technical Issues -- March & April 2021

yep, I think that 9 are very lucky that this facility was developed in 2012 and so is based on more conventional broadcast technology.

… and I think you’re spot on with that … hopefully it will be a wake-up call for 7 as they start thinking about their move of news from Martin Place to Eveleigh …

1 Like

If they’ve lost data (that was appropriately located in the right system(s)) beyond last Saturday, that would be a significant worry.

3 Likes

Tonight’s coverage of the Super Rugby clash between the Rebels and the Reds is hosted at Melbourne’s AAMI Park instead of the North Sydney studio.

5 Likes

They blamed Covid for hosting and commentary from the studio but clearly a technical issue has forced them back to location with no issues.

4 Likes

I didn’t say they shouldn’t be connected to the internet…

Broadcast critical extends to news archives, edit suites and all the studio servers and automation. Not just pres.

1 Like

This bit here is a very important point - after something happens, it’s often the IT Team blamed and “why did you allow this to happen?” etc… ALL users now need to be vigilant and aware of risks, one slip, one failure and it can all come undone. The number of times users hate having to have complex passwords or multi-factor authentication etc… but these events are the reason why and will only become more and more common. We’ve heard about Nine being hit this week, there were probably hundreds of other Australian companies hit this week also to varying degrees.

1 Like

Editing suites may need access in order to get footage ie off YouTube. Unless they place the files downloaded to a central file server. Even doing that exposes the network to the internet and is a risk. But they should have endpoint randomware protection company wide, but it is possible it may have failed. But to still be affected after virtually a week is a bit of a worry and suggests some major issues with Nine’s network. As with appropriate backups one would think they would be back online mostly unaffected as far as a viewer would see by now.

This shows a (typical) misunderstanding of how slow it is to restore from backups.
Typical restorations are just small amounts of data, maybe one or two virtual machines.

Plus that restoration can’t really start until you’re sure you’ve cleaned off all malware from the network, otherwise if one infected computer (client or server) is powered on it re-infects the network & all the restoration work to-date is undone.

This is full-scale, and will take weeks, at least. Their focus will be on base infrastructure & critical business systems, with others behind delayed until later.

Trying to do too much at the same time will cause contention on the network, storage and servers, slowing down the entire process, plus there’s also a limit to how much can be done without making systems that have been restored unusably slow.

6 Likes

Sure but it should not be affecting day to day operations to the point where it is noticeable to the viewer for example 5pm weekend news not going ahead. That is why it sounds as if the malware got into backup infrastructure and may have been there for quite some time as if you had daily backups and you found when the system was compromised you can restore from the last full backup and then daily backups after that. For example if you do a monthly full backups, daily incremental backups and weekly differential backups you just need to determine the time it happened just say 28th March so you restore the monthly backup from March each differential backup up to the 28th and daily backups since then. It’s highly unlikely the previous backups ie March would be compromised as it would be on its own media or virtual media that would have been rotated. There is no need to access this backup on a daily basis.

But if it can’t be determined when the last backup was compromised then it’s much harder to be sure you aren’t restoring compromised files. It sounds to me their backups were compromised or some didn’t have adequate backups to begin with to have this affecting the viewer experience after a week. For a broadcast automation server wipe it ie reformat, restore from last monthly backup which should not be compromised and you should be back running again. But if you don’t have proper backups or the infection has been there a while then it’s harder to be sure. Also the server might be managed directly by the vender which means Nine would have little control over the backups.

Don’t know if this has been mentioned, but no Sports Sunday. Replaced with some Easter-related religious program.

Which is immediately pissed upon by an ad for MAFS, which should be considered an insult by anyone even remotely religious.

Watching AFL Footy Show for my sports fix of a Sunday morning, unitl I spotted live NBA on Viceland.

10 Likes

this has been mentioned in the 9 newtwork prorgams and shedules thread

Well the mods can move it if they want. I’d assume it’s due to the ongoing tech issues.

2 Likes

This is the direct quote from the Wide World OF Sports Facebook page:
“Due to ongoing technical difficulties, Sports Sunday is unable to air at this time.”

2 Likes

Is the NRL Footy Show on this morning?

It’s in the guide, and I think it’s coming from Newcastle, so I’d say yes.

UPDATE: It’s on like Donkey Kong. :wink:

To be honest… do they really need Sports Sunday?

Just have both NRL/AFL Footy Shows start at 10.00am after Weekend Today.

5 Likes

No 5PM news for Melbourne today on Easter Sunday? TV Guide says “Nine News: First at Five” should be on. Instead Territory Cops is on

1 Like

Local content quotas? It would be a pretty cheap program to make.

1 Like

Just had a promo in Adelaide at about 5:45pm advertising the renovation show that was on last night at 7:30pm, saying it was on tonight at 7:30pm.

3 Likes

Same thing happened on Friday night previewing ACA saying tomorrow night but it referenced the stories that aired on Friday.

4 Likes