Brian Desmond's Blog

Inherits Network.Admin
Implements IOneManBand

More rant on MSBlaster

Jerry Dennanny rants on MS-Blaster

“This is a pretty easy statement to make when you are responsible for 1-10 machines, and patching pretty much means hitting windows update.

However, life isn't that simple for everyone.  In addition to my developer hat, I also have the (mis?)-fortune of being the IT manager for my company's site of ~200 nodes, with about a dozen production servers and a similar number of dev & qa servers.  We are part of a bigger, global enterprise network consisting of about 60,000 nodes.”

I couldn’t agree more. I’m responsible for about 800 nodes. The Chicago Public Schools System has about 60,000 computers (to which I am connected & represent a small fraction). From what I’ve been told, this blaster worm is already running rampant on the WAN. The technical competency of the techs at the schools isn’t the highest, and coupled with the fact that it’s summer break, the abundance of the worm on the instructional WAN doesn’t surprise me.. The majority of them are computer teachers. There are definitely some very technical ones out there, but I’d say the majority can’t sit down and figure out a way to script removal of the virus from their LAN, if they even have the resources to deploy such a thing. Fortunately for me, I do. Nonetheless, as a veritable one man band (I make the network run, write the software & web apps, fix printers, and take out the trash), making all this happen in the few hours I was in today was not easy. To make matters worse, my software update services server (windows update for the lan) completely crapped out a couple of days ago. I’m rebuilding it tomorrow L.

Here’s my point: If you want to rant about how everybody should have been all patched up by now, figure this out:

v     QA the patch on all their major platforms (hardware and software)

v     Deploy the in such a way that all of the affected computers receive it (note, no assigning software to Windows NT computers via group policy, so you’ll need a different way to deploy to them)

v     You’re going to need to install the proper patch on the proper version of windows

v     If you’ve got a few thousand (maybe a few tens of thousands) of computers, you’ll need to figure out a way to get the patch distributed throughout the organization to the proper file servers. Don’t forget to make computers download & install from their local file server. 50 computers at a remote office downloading all downloading at the same time over a slow link isn’t going to work.

v     The install is going to require a reboot, so you’ll have to get everything rebooted to make your changes effective.

Now, let’s get to the fact that some computers always straggle and don’t get patched. Cleanup time:

v     You’ll need to assess how many (and which) computers have the worm

v     You’ll need to clean the affected machines

v     You’ll need to deploy the patch to the affected machines

v     These machines will ned to be rebooted. If scheduled maintenance is on Tuesday, and it’s Thursday now, rebooting people’s computers might not be an option.

v     Oh, and you’d better have a good explanation for who exactly the NT authority is when your users call asking why their computers are being rebooted continuously couretesy of NT AUTHORITY, or why they even have the worm if your users read the newspaper or watch CNN (both are covering this issue).

An here’s a bonus point:

v     Your clients can’t connect to a major portion of the network because it’s located in New York City.

Have Fun!

Comments

Martin Spedding said:

I feel you pain. As I suggested my weblog (http://weblogs.asp.net/mspedding) I think the lessons learned from Windows 2003 Server need to be applied to the client. The client should be locked down and you have to take actions to open ports. Also a firewall should be configured automatically. The problem is the network is only as strong as it's weakess part. My only suggestion is lock down and ensure the firewalls and virus scanning are operating correctly. As you say the roll out of the patches is also a pain, in addtion to their frequency. Any chance of an upgrade from NT ?
# August 15, 2003 3:13 AM

Frans Bouma said:

Erm.. the worm ENTERS via port 135. If you have exposed that port to the internet, you are a fool. Besides that, if ONE MONTH is too long for you to patch servers, what do you do all day?

If you have a lot of servers to admin, ever heard of SMS? Push the install of the patch via the distribution system and off you go. Pointing the finger to Microsoft because they left a flaw in their software is very easy. I suggest you point the finger to yourself and start patching.
# August 15, 2003 3:38 AM

Jerry Dennany said:

Frans:
It's pretty easy to say to simply push the patch and you're off, and that one month is a long time, but you've left out testing. For example, we've discovered that the patches may cause core dumps in certain versions of SAP. Can't run without SAP, and can't run without the patch. Damned if you do, damned if you don't.
Sure, one month is a long time to execute your deployment plan, but not to create and test the plan. And what about the cost? As you can well see, bdesmond doesn't have an enormous budget where he can approve lots of overtime to get this cleaned up after hours. He's got to do this in scheduled maintenance windows, etc.
All the non-admins around here seem to think that this is a lot simpler than really it is.
# August 15, 2003 9:12 AM

TrackBack said:


Darrell Norton's Blog
# August 15, 2003 9:14 AM
Leave a Comment

(required) 

(required) 

(optional)

(required)