Why CodingHorror is horribly wrong about Blacklists and Virus Scanners

Jeff and I had an interesting debate on virus scanners a few weeks ago. He posted his take on the conversation yesterday, and (surprise!) we both think we won the argument. I believe the difference of opinion really comes down to a few different assumptions about the problem we're trying to solve:

• Different classes of Anti-Virus Software (Quality AV Software vs. Bundleware)
• Antivirus Effectiveness (Is it really just 33% effective?)
• The Goal of Virus Protection (Risk Management vs. Invincibility)
• Modeling the Threat (Known malware vs. New malware)
• Approach to Security (Practical vs. Theoretical)
• Productivity Tradeoffs (Virus Scanning vs. Running As Administrator)

So, let's go through the list:

Quality Anti-Virus Software

Anti-virus software gets a bad name because most people are used to really poor AV software. If your AV software came pre-installed with your computer, it's almost certainly junk. Let's take a quick look at the performance impact study Jeff previously quoted:

Now, we'll talk about the performance tradeoffs later, but for now let's just look at the difference between the top and bottom of the list. I've used AVG for years, partly because they have a version that's free for non-commercial use, and was happy to see that their free product offered vastly better speed - AND better virus scanning  performance - than most of the well-known brands which are installed by default.

So, let's start by ditching the straw man attack on AV as a technology based on some poor implementations.

Antivirus Effectiveness

Jeff opened the debate by linkslapping me with his Anti-Anti-Virus post. I'd read it before, and I re-read it. I remembered being shocked that AV software was only 33% effective, and decided to dig into the article he quoted. High level summary: that study was complete bunk.

Let's start with the clear conflict of interest in that test, due to the fact that the one virus scanner that did best in the test happened to be sold by... the company that funded the paper to begin with.

Let's ignore all the mathematical errors and get right to the real problem: the test says nothing about the actual effectiveness of virus scanners. So they took 32 infected files which had already got by a virus scanner and used them to test other virus scanners. Awesome. I'll bring 50 dead people to the hospital, see how many of them the doctors can save, and then proclaim that medical science is a sham. This test says nothing about the tens of thousands of infected files the scanners caught, or the effectiveness of the scanning software at all; it merely says that if an infected file gets past one scanner, a second probably won't help. Thanks, good to know.

It's interesting to note that the (ethically challenged) company which scored highest at 50% was likely not the most effective AV product; it just did a good job at catching what its competitors had missed. That doesn't tell me how it would do against the thousands of files that the competitors had caught. Say, for instance that one of those theoretical doctors managed to revive a cadaver. While that's impressive (okay, really impressive), it doesn't indicate that she'd be as effective as the other doctors on their general caseload.

All that to say that the 33% statistic isn't just a little off, sort of true, at least cautionary, etc. It's a completely worthless statistic.

Now skeptical, I decided to see how effective virus scanners really are.  AV Comparitives shows that several AV programs are more than 97% effective; for instance, AVG protects against 97.75% of malware.

Approach To Security

Now, I'll admit AV software doesn't offer us invincibility. 97.75% is not 100%, even for very large values of 97.75%. On the day this test was run, there were known viruses that AV software would not have stopped.

Worse, as Jeff points out, the numbers are not near as good for the "zero-day" threat posed by a new, unknown virus. Now, AV software does employ some sophisticated techniques for detecting malware based on suspicious behavior. AVG, for instance, says they use:

- Scanning - searching for character strings that are characteristic of a given virus
- Heuristic analysis - dynamic emulation of the scanned object’s instructions in a virtual computer environment
- Generic detection - detection of instructions characteristic of the given virus/group of viruses

However, virus authors are fiendish and relentless, and trying to come up with a way to stop them all is like playing whack-a-mole. So, yes, I'll agree that AV software doesn't (and can't) promise that I'll be impervious to virus attack. Does that mean it's a failure? Yeah, sorry. I've been deliberately wasting your time.

Or, rather, no. See, we're asking the wrong question here. Let's take a step back and talk about risk management.

It's about controlling, not eliminating risk

Cooks don't kill 100% of the germs when they was their hands, but we'd shut down their restaurant if we found that they didn't. Why? Because, by observing some basic kitchen sanitation, we lower the odds of transmitting food-borne illness to an acceptable level. Yes, it slows them down form all the spicy cooking they've got on their mind, but not nearly so much as having their restaurant closed.

Birth control is not 100% effective. Airbags aren't 100% effective. You get the point - a safety feature is worth while if the cost and inconvenience is outweighed by the reduction (never the elimination) of risk exposure.

Controlling contagious disease spread

Lowering the rate of spread of a virus effectively kills it. Sometimes the difference between the fringe and an epidemic is a just a small change in transmission effectiveness. So, thinking beyond my desktop for a second, limiting the transmission rates of malware is a very worthwhile cause. What if no one ran anti-virus software?

So, if we focus on controlling the spread of malware, AV software which is 97% effective is very worthwhile. With that in mind, telling the general public not to use AV software seems something like - well, killing kittens. A lot of kittens.

Modeling the Threat

A new virus doesn't instantly appear on my hard drive; it takes some time to travel. They're not slowed by transmission speed, since broadband has become so prevalent. Rather, since virus transmission generally requires some sort of user action (open an e-mail or an e-mail attachment, visit a dodgy website, download a file, etc.), the virus is waiting on us to do something to help it spread.

This is all a numbers game, right? In order to be 100% protected, I'd need to have the AV patches instantly, etc. But to reduce the odds of infection to an insignificant level, I just need to be reasonably up to date. That's because it takes a while for a new virus to get to me, and I just need to be protected by the time it gets to me. Sure, a mass e-mail attachment virus could potentially (1)get to me under 24 hours and (2) trick me, but the odds pretty low. I'm more likely to get a virus by downloading a file that's been affected, and as the report shows, I'm 99.77% likely to be protected. I don't need to be 100% protected, I just want to significantly reduce the odds that I'll get a virus. This works.

And it's not hard, either. My AVG virus definitions download at 4 AM daily and don't require a reboot. Install and forget.

The fact is that most malware¹ take weeks to build up a critical mass to start spreading rapidly, while AV companies usually have a virus definition available within a day of the first public detection. The overwhelming majority of infected files which I might download have a virus that's over one month old. They don't go away, they just keep floating around forever. The malware I'm most likely to deal with has been around for a while. So, my AV software is much more effective than 97% because malware isn't evenly distributed.

So that's why I think Jeff's missing the point when he says this:

We could appeal to the data. Of the top 5 threats on the virus radar, only one is younger than six months. However, the youngest dates from December 4th, a mere eight days ago. And it only takes one. If anything gets through your anti-virus software, you're just as compromised as you would be if you were running no anti-virus software at all.

Now that we've decided that we're trying to lower our risk, and that our actual risk is from known threats, the real value is in protecting ourselves against known threats. And that's what blacklists are for. Now, I'll agree that blacklists (by themselves) don't work against comment spam. That's because anyone can crank out new, unique comment spam all day. On the other hand, unique types of malware take time to develop. The numbers speak for themselves - as I said before, an updated AV program protects against more than 97% of the types of malware you're likely to encounter. By volume, of course, the protection percentage is much higher since the majority of infected files carry well-known threats.

Chocolate syrup doesn't work for hot dogs, but it does work for ice cream. Blacklists don't work for comment spam, but they do work for malware protection.²

Note: Many people make poor assumptions in their use of odds. I recently talked to a guy on a plane who told me that he knows the crash percentages on planes, and with all the flying he's done he's sure his number is coming up soon. That's an example of how we're inclined to think failure rates are additive. A more correct way to view this is that every time he steps on a plane, the overwhelming odds say he'll land safely. If I encounter 100 infected files, the statistics do not say that two or three will affect me.

Real-world Security

Here's where we really differ. Jeff says that we should throw out AV protection because it will never be as safe as just running in virtual machines as standard (non-administrative) users. I say that's a red herring, because it doesn't work. I totally grant that AV is not as effective as running as a non-admin. It doesn't matter, though. It's a red herring. No one runs as a non-admin because you can't actually use a computer when running as a non-admin, so it's completely irrelevant. Non-administrative computing - especially software development - does not exist. Nonexistent solutions to the malware problem don't help the problem.

The Non-administrative User Red Herring

I run as an administrative user. Jeff runs as an administrative user. Just about everyone on the Windows platform that's doing something besides security evangelism runs as an administrative user. That's because Windows without Admin privileges, in my experience, is unusable. And not just when I tried to use it as a software developer, either - my kids have a standard user account on our family computer, and it's rare to find software (even Flash games) that work there, even when installed with admin privileges. I'm sure I'll get some comments on how to run as an administrator, but in my experience it's just about impossible to be productive on the Windows platform using standard software without administrative privileges. I can get better results by just unplugging my computer. It's even more secure, and about as usable.

Here's a quote from one of Jeff's readers who had a similar experience trying to run games without admin privileges:

i tried running as a non-administrator in windows xp for quite a while. my xp partition is solely for gaming and all my email, picture printing, browsing, etc. is done in vista. i really wanted to play games as a non-administrator. really. i created an administrator account called "installer" and a different "limited account" to run my games. [...] some games won't even run if you aren't an administrator.

others will exhibit very odd behavior. [...] only if i played the game with an administrator account would i see the game how it was intended to be seen. after about 6 months i gave up, deleted the installer account and made my gaming account an administrator account. [cowgod(ed. ???) on December 12, 2007 10:00 AM]

Plus, malware can still do bad things when running without administrator permissions: delete or alter files, send e-mail, etc. To most users, loss of their files is one of the worst things that can happen to their computer.

The Virtual Machine Red Herring

VM's are nice, but don't protect what's really important - your files and your network. My VM gets infected, so do I delete the VM and lose all my files? If I want those potentially files, I'm in the same boat I'd be in if I wasn't using a VM. And a VM that's got network access can still launch DDOS attacks, send spam, all those things that malware does. It's just harder to track down, because it only runs when I run the VM. In fact, VM's make things worse - running more machines (real and virtual) means that any one of them is less likely to have the latest operating system patches.

So, let's wrap this up with a bit of reality: alternatives to AV software are only valid in this discussion if we're actually going to use them. Dismissing AV software because it's less effective than a theoretical, unworkable alternative doesn't make any sense.

Productivity Tradeoffs

Let's end with a discussion of the productivity tradeoffs of AV software vs. Jeff's other solutions. Which brings me to an interesting dichotomy in Jeff's dismissal of AV software due to the performance implications. Another reader noticed it, too:

Jeff,

I don't know how VMWare works, but, in my experience, Virtual PC 2007 from Microsoft is much slower than running on the real machine (even using the Core 2 Duo's virtualization capability). If you're worried about the performance hit of running malware-protection software, then VPC seems to be out of the running. Also, VPC is limited to, I believe, 16 bit graphics and has no USB support (i.e., in all probability, no printing or scanning). From some of the comments I've seen, it looks like network data flow isn't very smooth through the VPC, either.

To paraphrase from your RAID 0 blog entry, is it worth greatly increasing your risk for the sake of a small increase in speed?

Also, I have BIG problems with the methodology used to come up with the performance hits in the test you mentioned. The author of the test talks about it quite a bit. But, the fact is, he's running those tests in a VPC. Plus, the VPC is limited to just one of his CPU cores and just 512MB or RAM. Dollars for donuts, if he ran those tests on the actual hardware (dual core AMD 64 X2 4800+ and 2GB RAM), I'd bet his perfomance hits would have been an order of magnitude smaller. [David A. Lessnau on March 1, 2007 10:40 AM]

Exactly! Why is a minor AV hit (<1% CPU, 19% disk) unacceptable, but a much slower virtual machine solution which does nothing to protect my files is just fine?

Picking the Easy Solution

1. Do nothing (run as admin user with no AV software) - fastest, assuming you'll never encounter infected software. When that happens, productivity (yours and anyone you infect) goes out the window. This is equivalent to running an unsanitary restaurant.
2. Running in a VM with a limited user account - very slow computer performance, and most of the software doesn't work. Advantage: you may get frustrated, and go to an amusement park.
3. Install a free, quality AV program - minor computer performance impact, greatly improved odds of virus-free computing, no kittens killed.

¹ I'm not talking about protection from worms, which attempt to replicate across a network, usually without the need of any human involvement. "Successful" worms have become more rare as server administrators have become aware of the need to apply patches in a timely manner. While AV software helps combat worms, the real solutions are in securing servers so they don't allow remote code execution. This is best done by regularly applying security patches (e.g. Patch Tuesday).

² Firefox is sticking with a blacklist for phishing protection, too. They say that blacklists aren't perfect, but they're a lot better than nothing at all. See Johnathan Nightingale’s Mozilla24 presentation Beyond the Padlock [ASX video link].