In it’s latest meeting in Helsinki, The Anti-Malware Testing Standards Organisation (AMTSO) has adopted two new sets of guidelines for testing the perforance of security software and for testing entire security suites.
Security software testing organisations such as AV-Test.org and Virus Bulletin who test and rank security software for its effectiveness are often at odds with the security companies who produce the software. It is hoped that the agreement forged between leading security companies and the researchers will put an end to these long standing arguments.
Part of a series of documents on ATMSO’s website, the guidelines aim to introduce a set of standards widely accepted within the industry to help rank security software by effectivenessm, and althought they are no mandatory, many testing organisations have already agreed to adopt them including AV-Test.org, AV-Comparatives, Virus Bulletin, ICSA Labs and West Coast Labs.
Asked what he thought of the outcome, ATMSO board member and director of malware intellengence for ESSET, David Harley had this to say:
We’re just trying to get people to think harder their methodologies so that they actually make sense. It doesn’t mean you can’t do things different ways, it just means you have to try and conform to a rationality.
John Hawes from Virus Bulletin also commented on the guidelines:
We’ve already started implementing some of the ideas developed while discussing and designing this document, with some major expansions to the performance data we report in our comparatives in recent months and more improvements on the way.
We’re also hard at work developing a new style of test which will allow us to measure the full range of features in many of today’s security solutions.
Although the ATMSO has come to an agreement with these leading vendors, it has been critised in the past by others in the industry for lack of transparancy because many of the organisations members are comprised of security software vendors and may not entirely end the fueding between parties. AMTSO chairman and manager of Sophos’ threat lab commented that:
What we are trying to do is get everyone involved to think more widely about decisions made in designing tests. In this case we are talking about testing a whole product and not just part of a product, which can be very misleading, and also about measuring performance where it is so easy to create a performance test that is really just meaningless.