Although sometimes tests are deliberately biased in order to get the “right” results, it is far more common for evaluators to inadvertently influence results by making inappropriate assumptions based on an incomplete understanding of the threats and anti-malware technologies. ESET strongly recommends that evaluators define and understand their own objectives and methodologies before starting the test.
“You can’t treat a comparative test in exactly the same way as a scientific experiment, but you do have to start with an objective and plan carefully,” says David Harley, Director of Malware Intelligence. “There are many ways to screw up a comparative test, but someone is always coming up with a way that I haven’t seen before and would never have thought of – human creativity, I suppose.”
Top Ten Mistakes:
1. Using samples received via email or on a honeypot machine, without checking that they really are malicious software.
2. Using one of the programs to be tested to validate the samples.
3. Assuming any sample detected by two or more scanners as malicious to be valid. This may bias the test in favour of products that flag everything that meets very broad criteria as suspicious, and against products that are more discriminating and fastidious about false positives.
4. Using VirusTotal or a similar service to check the samples and assume that any product that doesn’t report them as malicious can’t detect them. This will once again give the advantage to scanners that flag everything as “suspicious”, and will also disadvantage scanners that use some form of dynamic or behavioural analysis.
5. Using the default settings for detection testing, without trying to configure each product to the same level of paranoia.
6. Using default settings for scanning speed. This may bias products that get their speed advantage by cutting corners on detection.
7. Asking vendors to supply samples. This may allow the vendor to bias the results in their own favour by including samples that other companies are unlikely to have access to, and to the disadvantage of companies who consider it unethical to share samples outside their web of trust.
8. Categorising samples incorrectly, leading to possible errors in configuration. For instance, not all products flag certain kinds of “greyware” (described by some vendors as “possibly unwanted applications” or similar) as malware by default.
9. Too much self belief. If, when testing two products that use the same version of the same engine, they score completely differently, it is unsafe to assume that there must be something wrong with the lower-scoring product. It is just as likely to be a problem with the setup or methodology.
10. Not including a contact point or allowing any right to reply. Be open in the methodology used and the objective of the evaluation, to allow others to verify.
ESET recommends that aspiring testers and concerned consumers alike with an interest in raising testing standards should check out the documents available at the Anti-Malware Testing Standards Organization website - www.amtso.org.
Anti-Malware Testing: Good And Bad Practice is the title of the seminar given by David Harley in the Business Theatre at Infosecurity Europe 1.20pm Wednesday 29th April 2009.
ESET develops software solutions that deliver comprehensive protection against evolving computer security threats. ESET pioneered and continues to lead the industry in proactive threat detection. ESET NOD32 Antivirus, its flagship product, consistently achieves the highest accolades in all types of comparative testing and is the foundational product that extends the ESET product line to include ESET Smart Security. Both products have an extremely efficient code base that avoids the unnecessary large footprint found in some solutions. This means faster scanning that doesn’t slow down computers or networks.
Sold in more than 110 countries, ESET has worldwide production headquarters in Bratislava, SK and worldwide distribution headquarters in San Diego, U.S. ESET also has offices in UK, Argentina and Czech Republic and is globally represented by an extensive partner network. For more information, visit www.eset.co.uk or call 0845 838 0832.
Marylebone Media Relations
+44 (0) 7968 626838 (mobile)
- Contact Name:
- Sara Claridge
- Marylebone Media Relations
- Contact Email:
- click to reveal e-mail
- Contact Phone:
- Company Website:
Related press releases
- Monitoring your data centre has never been this easy
- AbilityNet launches RSI workplace survey
- Economic green shoots provide opportunity for Bristol SEO agency
- 25-34 year olds most savvy online shoppers according to new research
- Flourish: New Technology Facilitates One-to-One Coaching and Executive Development