… which most people who know and work in the free-to-air broadcasting business think is the wrong message …
I know that Nine hasn’t produced some shows lately because of the technical issues, but was wondering if anyone knows if “100% Footy” will air tonight or not? It didn’t last week, and Sports Sunday wasn’t on either for the last 2 weeks (neither the Late News too).
Well it’s becoming commonplace, US networks are now becoming part of broader “content” departments for their network and streaming services and cable networks. And network heads are switching between streaming and network all the time now. Not sure the exact issue, content is king.
Nobody gives a shit about channels anymore.
4 Likes
… what you say is absolutely true … but you’ll also notice that the “big” companies are becoming bigger while those not so big are being swallowed up or disappearing … in the world context Stan is less than a minnow and will eventually disappear so if you have a CEO that largely ignores profitable sectors of the business to chase a rainbow, shareholders will not be happy … meanwhile there’s still money to be made in free to air for the foreseeable future …
They’re doing a bit more than just reinstalling Microsoft Office.
4 Likes
Have a look at the split of how much money they’re making from digital and traditional media and come back to us.
1 Like
Yes it will air tonight around 10pm , Nines youtube nrl shows were done via handy cams this week nothing special but at least the put something to air.
1 Like
Quick Darwin update in the metro set -
2 Likes
There’s works in IT & there’s works in IT, you have no idea what you’re talking about when it comes to Broadcast infrastructure, you’re only making a fool out of yourself with all the comments.
I work one step back from this, & know a bit about what happened but not everything, & I wouldn’t divulge anything of note here even if I did know more, but if you saw the IT broadcast infrastructure of TCN or even 5 year old broadcast stuff, it would blow what you know sideways, to the point of you think you know it all, but wouldn’t know where to start if a minor issue arose, let alone a big one like this.
At the start the when was noticed something wasn’t quite right, NPC pulled the connection to TCN & have been working that way since, that’s how they escaped.
The IT systems at TCN are so complex & integrated with everything else, you can’t just stick a backup CD in the drive & start over, things will have to be done in a certain order at a certain time, & most of the critical stuff isn’t running windows 10, while it’s probably based on off the shelf software, many parts of it will likely be custom written, to suit their specific needs & what they want, & TCN Sydney might be the only place in the world who are using it in that configuration?
All these systems would’ve been years in planning & literally took months & months to install, setup, configure & test before use, you’re not going to just reinstall backups & go again in a couple of weeks after something this major.
There was a delay in moving into the new studios from Willoughby because of a number of teething problems & things not talking to other things as expected they should, the broadcast IT systems are so complex & intertwined with other things, it took the TCN engineers a good 3 weeks or more, to get the 2 way radio repeater at Artarmon (for the News helicopter) to talk to other computers properly at the studios, which is essentially micro part to the rest of the system, but that doesn’t indicate the Nine engineers are dumb, just that the system is so complex.
We have some extremely complex control systems across the country on the transmission side too, that are custom designed & built & there are literally 2 engineers in the world who know how they work & how to fix them should something happen, & I’ve no doubt Nine would’ve flown some pretty high up IT engineering staff in from around the world to fix this ASAP for the same reason.
Just because you’ve seen one IT network or software program don’t think you’ve seen them all.
Even if you work in IT in a big office, I think I’d be safe in saying, you don’t know, just how much you don’t know, when it comes to what Nine is dealing with here.
It’s quite possible some equipment has actually been bricked & can’t be restored, & will have to be custom rebuilt from scratch & shipped from OS to replace, which could in itself take months, before everything can be restored to as new or as was?
76 Likes
Thank you for the reply. I’m sure most members here appreciate the insight someone in your position can provide. No doubt the usual suspect will see this and reply with the same long-winded bullshit that we’ve been subject to for the past couple of days. All I can say is, don’t feed it. Just ignore it.
27 Likes
Great insight. Cheers.
19 Likes
Thank you for posting this detail. Really shines the light on the “I know better” people and how they have no bloody clue.
8 Likes
It’s one thing to see a helicopter crashed in a tree and go “I’m not a pilot but I know that isn’t right”. It’s another thing entirely to go “I’m not a pilot but I can pinpoint the exact thing that the pilot did wrong and how the design of the helicopter was flawed and contributed to the situation and I know how it can be fixed.”
4 Likes
is 9 news local returned to its statewide formate ?? i haven’t seen anything on Facebook or Twitter promoting it today
… a couple of points in your top notch description scare the bejesus out of me …
and …
… now I’ve been in broadcasting all my working life (and still am to a point even in retirement) and one mantra that was drummed into me over that entire period was that, where on-air broadcast systems are concerned never go for the “cutting edge” because, if it fails, you’re in deep doo-doo 
5 Likes
Excellent post.
To many of your points… so true.
7 Likes
Hopefully this is the time for one of the IT guys to go “I know this one! I know where to change this and get this one fixed and be a hero… then go back to being completely out of their depth with the rest of the mess!”
I’d like them to at least get one win, even if it is a small one.
Thanks @RFBurns for the insight. There’s the saying - the more you know about something, the more you realise you don’t know. IT and cyber-security are one of those big ones! Luckily things were noticed early enough as being odd that a connection could be taken down. I remember a couple of jobs where there’s a couple of unusual things happening, completely different symptoms to what the final cause was but trying to track and piece things together and then having to take down systems - they’re not decisions done lightly.
While I hope they’re up and running within the next couple of months (or sooner hopefully) I’d imagine the ramifications will be going on for years, whether it’s just the full reports of how and what happened through to multiple checklists to see what can be improved for next time… and unfortunately, there quite probably will be a next time the way things are happening.
4 Likes
The first part I’ve edited in my original post to clear up any possible misconceptions, don’t get me wrong it’s very complex & custom, but probably isn’t entirely custom written, & to be clear I’m not privy to exactly what percentage of custom it all is, but can say, you can’t just go down to any IT store & get stuff off the shelf for replacement.
The second part yes concerns us too, as it’s close to end of life, it will be replaced with something simpler.
5 Likes
No, the CEO is also responsible, and the board for overseeing management.
I don’t know how good or bad Nine’s CTO has been, so speaking generally…
It’s long past time that the blame isn’t just shifted down to someone who may (or may not) have advocated for more to be done on security but may have been ignored.
As you’ve said IT is business critical so it can’t just be left to the head of IT to try to do their best with whatever budget & other constraints they’re given.
It’s so important the highest level of management has to be responsible.
Unfortunately it’s all too common that a (below-CEO) role is only as responsible as they’re allowed to be. If their recommendations are ignored, then it’s unreasonable for that person to be sacked but not the CEO, or perhaps also the board.
If a building flooded, would it be the COO, or the head of finance who paid for the building, or the CEO who should be responsible for deciding to build/lease in a floodplain?
Why shouldn’t it?
How much is worth spending to entirely duplicate all necessary infrastructure, and how much disruption to put up with to regularly test a disaster recovery, for what may have been estimated to be a very low chance event?
It’s a risk/judgement call, and plenty of businesses, and governments, get it wrong (in hindsight).
And who (outside of Nine) cares which city news programmes are being controlled from?
I’d like more spent on properly highly available systems, but if a business decides they have a higher priority investment elsewhere & take the chance, then they can live with the consequences.
Nine will indeed have lost money, and for some years they may increase IT & security spending, but that’s Nine’s problem & Nine’s decision.
Centralisation and outsourcing has advantages, otherwise companies wouldn’t do it, but of course there’s the risks too.
As others have suggested you seem to have very optimistic view about how easily & quickly a ransomware attack can be cleaned up. How many ransomware cleanups, of a large company, have you been involved with?
And thanks @RFBurns for the insight.
Someone’s got to go first with the latest tech; while unfortunately that adds some risk until its well bedded down, it also means you get the advantages in productivity, efficiency, etc. the newest stuff provides, and with Nine’s move to new facilities in a new building, it - from the outside - appears logical to do so at such a time.
(I’m not suggesting using the latest had anything to do with what happened. I don’t know.)
4 Likes
Nine sent an email to staff this afternoon.
https://twitter.com/sophieelsworth/status/1378994065337769987?s=19
8 Likes