Malicious programs can be very sneaky. When they get access to a system, most of the time when you least expect it and stay hidden until the security software actually detects it. But, when the culprit is finally caught, the damage has already been done since it was active for some time and you have no idea of what sort of information has been stolen from your computer.
To avoid hardships like this, most of us got cyber security-oriented software like antivirus, anti-malware, Host Intrusion Prevention Systems (HIPS) installed on our system to protect our computer from known or unknown threats. The real question is, how to know if the antivirus software or the anti-malware software is actually protecting your computer?
Every antivirus guard or a cybersecurity package comes with a promise to protect your personal and financial information from cybercriminals. These software products also promise to offer you a hassle-free online and offline experience. The program would probably state that your computer is protected and the protection is enabled.
Yet, how can we be sure that these products live up to their claims? An interesting question isn’t it? How can you be sure that the antivirus or anti-malware program is actually running and that the antivirus or virus definition database hasn’t been tampered with?
When evaluating a cybersecurity product, we have to put these products to the test in multiple ways. So, every antivirus product that we review has gone through these tests. Also, more importantly, we have mentioned hands-on experience with the product. It is important to note that we cannot recommend all these tests for every product due to technical limitations.
Many antivirus programs include protection against spam filtering, anti-phishing, ransomware protection, VPN networks to maintain safer online presence, but some don’t add these features in their basic packages. Whatever the feature that is included in a given product could be tested with the help of the correct methodology.
How to test the real time antivirus protection?
Believe it or not, but in the good old days, the cyber security experts had few malware infested computers at their disposal to test each product’s ability to remove existing malware. Advances in malware programming, practicality and technical limitations put an end to this, but they came up with many methodologies to test the real time protection of antivirus programs.
Every decent antivirus offering currently on the market comes with an on-demand scanner to identify and destroy existing malware and virus infestations and a real time monitor to quarantine new attacks. Most antivirus software vendors do finish their yearly update cycles on or before early spring. Security experts gather new collections of malware samples to test these yearly updates.
To test real time antivirus protection, we gather a list of latest malware hosting URLs, and also downloads handpicked samples of malware from a cloud storage. Real time protection usually identifies the possible threats immediately, but it some cases they don’t. The identified percentage is then listed down. Also, any registry or system corruption done by a possible malware activity is also noted down with the help of several software programs.
Any background processes that were created are also listed down. If an antivirus guard prevents installation of all the malware programs, it earns a stellar rating, depending on how well it managed to prevent wide spread system corruption.
If a product prevents the installation of a malware but failure at prevention of corrupting the system, then it gets half the credits.
If a couple of malware processes are running in the background, then the rating is even lower.
Failure to prevent installation, malware processes running in the background plus system corruption gets no points at all.
Average of all these scores become the final verdict of real time antivirus protection.
Another way to test the real time protection feature of an antivirus software is using the EICAR test file. The EICAR test file is a product of the labor of several antivirus researchers, a harmless file that is detected as if it were actually a virus. The EICAR test file does nothing and it is completely harmless even if it is run on the computer. This eliminates the risk of actually exposing your system to malware. The EICAR file can be created with a simple code editor like the Notepad. Save it as an exe file and run it.
What this software does is generating false positives in antivirus software and online antivirus scanners. Failure at identifying this software means that something is seriously wrong with your antivirus software, meaning that you should take a look on real time protection settings. If the features are set as it should be, then try rebooting the system to fix the issue. If the reboot isn’t solving the problem, reinstall the antivirus software.
System Shutdown Simulator
System Shutdown simulator is a leak test software that has the ability to create the EICAR test file with a simple touch of a button. It comes with extra features such as letting you test if the EICAR can be detected when an antivirus program would have been closed when a system shutdown is being initiated. It also creates an auto start up entry in the registry to test HIPS feature and also a stealth download and an autorun of an executable file for firewall testing.
The System Shutdown Simulator is easy to use as the software itself comes with a self-explanatory user interface. To use it, run the program as the administrator. Then click on “Intercept System Shutdown” button as the first step. Click on the “Shutdown Computer” button where your computer will attempt to shut down. This will prompt a warning with a notification of sorts about an application running in background preventing you from signing out. If you cancel the shutdown, you will be back in desktop.
Most of the time, you will notice that the antivirus program icon that you continuously see at the notification area is no longer there. Meaning that it is possible that it is not running its services in the background. Now click on “Create EICAR test file” to check whether the antivirus software is able to warn you that it detected EICAR test file.
Click on “Create Auto Start Registry Key” to test whether the HIPS services are running correctly. Click on “Download and execute test file” button to see the firewall is running real time without any vulnerability.
Using Trojan Simulator to check everything is working properly.
The Trojan Simulator is a small program that is simulating the behavior of a trojan program being installed on a computer by adding a startup entry in the registry at the HKEY_LOCAL_MACHINE section. This program then proceeds to run a harmless little TSServ.exe application in background. This is the common behavior of a simple trojan virus. Using the Trojan Simulator program, we can make sure the antivirus feature of an antivirus guard is running correctly. But, the more sophisticated trojan files are using more complex techniques such as rootkit installation.
*To avoid “Failed to set data for Trojan Simulator” message prompt, run the program with Administrator privileges.
Many antivirus guards can detect the Trojan Simulator and not being able to download it or the program getting quarantined on or before the download is completed is a sign of your antivirus software running correctly.
SpyShelter is a system security test tool with many built in actions such as sound recording, screenshot capturing, webcam capturing, keylogging and clipboard monitoring. It comes with various methods to fulfil each action. Antivirus program won’t register the SpyShelter as a suspicious program most of the time due to not triggering enough actions to get flagged. The antivirus programs are designed to offer hassle free experience, so they are designed to prompt alerts only when it is necessary to do so. Activating a certain action or two is not enough to be considered as a threat.
Antivirus security Performance and its impact on system resources.
When the antivirus security program is running real time for detecting malware, to take actions against network intrusions, malicious URL blocking, it is definitely using your systems resources to get the job done.
The antivirus guards developed a bad reputation for draining too much system resources back in the day, to a level that your experience with the net was affected completely, whether you like it or not. It was a decision in between keeping the antivirus guard installed or taking the risk of removing it to improve system performance.
But now things are a lot better than it used to be. To have an idea on the impact that the virus guards are having on the system, we run some simple tests.
The requirements that an antivirus guard test must fulfil are as follows.
The antivirus software must load as early as possible in the boot up process. The time it took for a reboot is measured for five reboots and the average time that it takes for a reboot is taken. This average time is then compared to the average time it took for a reboot before installing that specific antivirus guard.
Looking for a significant slowdown of the systems, we have to take a look at the average times it took to open certain multimedia files or documents. The overall user experience is then noted down to understand the overall system impact.
Real time file scanning process can take up a big amount of processing power and hard disk file transferring speed. To determine this, we zip and unzip a collective of various kinds of files (4GB) before installing the antivirus guard and after installing it.
*a small script is used to accurately measure the times it took to process a certain process.
The average slowdowns in between these tests can be anywhere in between 1 to 30%. So, it is safer to say that you may experience real slow downs when it comes to more heavy-duty antivirus software.
The methodology used to rank the success rate of a firewall can be different from the vendors point of view. But there are several tests that you can complete to see the behavior of any given firewall despite the vendor.
A firewall has two main jobs.
Ensuring that programs installed in your computer are not misusing the network connection nor transmitting personal information to a third party.
Protecting the computer from outside attack.
To measure the success of a firewall, we have to use a computer with direct access to internet, ideally a physical computer which is connected to the internet through a router’s DMZ port.
Tests like port scans and other web-based tests can be used to find out whether the firewall is actually steal thing ports to completely hide the test system from attacks. The stock firewall feature that comes with Windows OS is capable of putting all the ports in stealth but some cyber security experts such as Kaspersky consider that it is not necessary to stealth the ports as long as they are closed and the firewall is actively preventing attacks.
In the earlier firewall programs, every time a program is trying to access the network, the user is greeted with a query asking whether or not to allow access. This approach is extremely hands on and isn’t very effective since the average user never has no idea on what kind of action they must take. Some allow everything while the others just click on whatever the thing without a second guess. Some people block every program until it affects the performance of their system or their daily online experience, after that they keep allowing every program.
Some 3rd party applications attempt to get through the firewall by masquerading as a trusted program. Usually a user whose got used to program control feature is allowing these applications the access to the network. This is why the cyber security experts are using leak tests to test the skills of an old-fashioned firewall.
To get this done, they code a small program and runs it to get access to the network through the firewall. Then they do a leak test. These programs use the same techniques used by the malicious programs, but without infecting the system without any malicious infestations.
The best firewalls are capable of automatically configuring the network permissions for known programs form trusted vendors while eliminating known bad programs and programs by bad vendors. They keep surveillance on unknown programs to see whether there are any attempts at a suspicious connection, at that point the firewall steps in and interrupts the connection.
Also, the best firewalls usually intercept exploit attacks at the network level before they even reach your computer. This is done with the help of behavioral control. Even for those that do not scan at the network level ends up quarantining the malware payload.
Whether it is an operating system or a simple image editor, any software comes with flaws and security holes. This is why, the black hat hackers are continuously working around the clock to get access to a certain system through an already existing backdoor or by using any vulnerability that they could find. When this happens, the vendor of the software product, releases a new security patch to address the issues, but until that you’re vulnerable.
The CORE Impact penetration tool kit is world renown as one of the most success pen testing products to identify firewall vulnerabilities. This tool is capable of attacking the system with more than two dozen of recent exploits and record how well the security system is protecting the system against those attacks.
Some malwares are capable of disabling the firewall by editing the registry. Windows firewall was vulnerable to this type of attacks back in the 2016. But now it has evolved in to a serious performer which is almost un-hackable.
To test this, we can code a small program to try to turn off the network firewall by editing the registry entries. Then we can try to eliminate the security service processes and essential Windows services using the Task Manager
Comodo leak tests to check the firewall performance
The Comodo Leak test program is created by COMODO, a cyber security firm which is famous for its COMODO Dragon web browser (based on chrome) and the Ice Dragon web browser (based on Firefox). COMODO is also renowned for their free antivirus software which is also allowed to be used for commercial purposes.
The Comodo leak test tool is one of the best ways to test for vulnerabilities in a firewall and HIPS programs. This can be used to check for the vulnerabilities present in antivirus feature of a cyber security program, as most of the antivirus programs are nowadays have behavioral analysis to detect if an unknown program is behaving in a way that can pose a risk for the security of a system.
All you have to do is simple yet effective. Run the program that you can download from here and click on the test button which will activate a sequence of 34 different tests ranging from invasion, SQL injection, rootkit installation, system hijacking, ransomware etc.
How to test malicious URL blocking feature?
Though it is possible to clean a system after an infestation, the best way to protect a system is preventing malware even before it reaches your computer. To get this done, many antivirus programs comes with browser extensions to keep you away from malicious program hosting websites. Even though there is a chance that the system won’t detect a malicious URL at once, there’s an opportunity to quarantine the malware files during or right after the download procedure is complete.
To check this feature, we get a collection of newest identified malicious URLs from the MRG-Effitas, an English company specialized in cyber security. Then we manually check the URLs that are not older than a week. All the URLs that are not working are removed from the list and the ones that were blocked from accessing or downloading something is also noted down. We skip the domains pointing to the same domain or downloadable files that already appeared in the test. At least three dozen verified malicious URLs are then checked before arriving the final verdict.
The final score for this test depends on the percentage of malicious URLs that were blocked from access or downloading malware.
Anti-Phishing detection and prevention
Con artists create websites that are mimicking financial websites, entertainment websites and sometimes even social media websites. These websites are designed to steal personal information and financial details by baiting the user to submit information. This information is then used to commit fraud. Phishing is platform independent and works on any operating system with the access to internet browsing ability.
Websites that are mimicking famous social media websites are getting blacklisted sooner but fake websites mimicking small financial organizations and small companies are not getting noticed before it is too late. This is why we are using only the latest phishing URLs for testing.
These URLs are gathered from 3rd party researchers who are sniffing after the reported yet not verified websites. Using the latest URLs forces the antivirus program to rely on real time analysis rather than relying on old school blacklisting.
To prevent malicious URLs, extensions for Chrome, Safari, Firefox and Edge are provided by the antivirus guard. These extensions are constantly monitoring the URLs and website information attached to the URLs.
All the URLs that doesn’t actively capture the user data are getting discarded. All the URLS returning an error message are also discarded. All the other URLs and whether they were detected by the antivirus guards or not is also noted down.
Parental control and monitoring is a feature that covers a wide range of programs and features.
This feature is used to keep the underage kids from accessing certain not suitable websites, or to monitor their internet usage. This feature also lets the parent to determine when and how long their kids are using the internet as well as how long they are allowed to use the internet each day. Other features can range from limiting chat contacts and keeping them away from certain social media topics and posts.
The best way to test this feature is by trying to access some porn websites through a browser when the parental control feature is on. Most of the products excel at blocking the porn sites.
The content filtering ability must be browser independent. To make sure that we use Firefox, Edge, Explorer, Chrome, and Opera. Some of the parental control features could be turned off with a simple network commend, and we try to turn this down using this command. And we can make sure whether we can evade the filtering process by using a secure VPN plugin or a proxy website.
These measures to make sure the internet activities on a children’s computer are effective if the kids cannot modify the time scheduling feature. To make sure that the time scheduling feature is working, we can modify date and time of our system. Most of the products do not fall into this trick.
Testing the features that the programs claim to have must be done. For an example, if it promises to filter out certain curse words from the emails or the messaging apps, then we can add a certain new word to see whether it is getting filtered out or not. To see whether it is capable of limiting chat contacts, we can try banning an account that we created for this test.
Whether you are using Gmail, Yandex Mail, outlook or yahoo as your primary email service provider, all of these high-ranking companies are providing a Spam filtering service to keep your inbox cleaner and safer. This feature can either be built on or done by a utility service running on the email server.
According to a test done by the AV Comparatives, one of the leading cyber security firms, Microsoft Outlook was able to block 89.87% of the Spam. 3rd party Spam filtering providers like ESET, G Data, SuperSpamKiller were able to filter almost 100% of the Spam messages. This article, which was originally published back in the March of 2016, further mentions that USA and Brazil are the most Spammed countries in the world. Over 127,000 Spam emails were used for this test.
The AV Comparatives, does conclude the test that several cyber security vendors are thinking of remove the anti-Spam feature from their products, probably due to the advances of built in anti-Spam features provided by the email service providers.
In the back of the day, cyber security analysts used to run anti-Spam tests by using a real-world account that gets both Spam and valid emails. But manually processing emails and analyzing the contents of the emails in Inbox and the Spam folder is time consuming and impractical. So, this methodology is not used nowadays.
But the ability to provide anti-spam support for POP3 email accounts, IMAP, Exchange and Web based email services are considered and carefully analyzed before our final verdict.
Antivirus Lab tests
Most of the independent testers do not have the resources to run heavy duty antivirus tests performed by the independent labs around the world. So, we pay close attention to their monthly reports.
AV Test Institute, a German company, continuously puts antivirus programs through a wide range of tests. ICSA Labs and West Coast Labs offer a wide variety of security certification tests, that we follow for malware detection and malware removal. These Labs are experts in the industry and they are actually getting paid by the cyber security product vendors to get their products tested to make sure that they are fool proof. Which also means that the product is significant enough with a significant user base and the vendor is willing to pay big money to get their products tested.
To test protection against exploits, the researchers expose each product to AV-Test’s reference set of over thousands of examples, and to several thousand widespread samples. The antivirus products are getting points for preventing the attacks at any stage, whether it is blocking access to a malicious URL or detecting a malware by using signatures or by quarantining it after entering the system. The best of the best gets almost 100% protection ratings in a test like this.
The AV-Test also make sure that performance of the antivirus product is not interfering with our day to day computer activities. To get this done, they measure the time difference between performing 13 common system actions with and without the security product is present. These actions can vary from downloading files from the internet, copying files both locally and across the network and running average programs like web browsers and heavy-duty graphic designing software. Averaging multiple runs of the same action helps them to identify the overall system impact that each product has.
AV-Comparatives, an Austrian cyber security company, is using a simple file detection test static test that checks each antivirus software against more than 100,000 malware samples with a false positives test to ensure accuracy. They also do performance test to measure the impact that it has on system performance. AV-Comparatives’ dynamic product test is much more significant when it comes to simulating an actual user experience as possible, allowing all features of an antivirus program to take action against malware.
The AV-Comparatives also test a cyber security program by putting it through a remediation test by challenging it to restore a malware infested system, to see whether it is capable of completely removing the malware.
SE Labs capture real world malware hosting URLs and use a replay technique to make sure that each antivirus software is encountering precisely the same web-based attacks or malware payloads. This procedure is way more practical and realistic, yet can be time and resource consuming. This is why they don’t usually report on more than 10 cyber security products.
The researches at SE Labs awards three points for blocking one of these attacks. If it takes action after the attack began but managing to remove the malware completely is worth of two points. If the attack is terminated without a full clean up, the antivirus gets only a one point. If the malware is running on the system without any interreference, then the product gets a minus five-point rating. Due to this reason, it is possible to see that some products scoring below zero.
They also evaluate how well each product refrains from falsely identifying valid software as malicious, and how much of an impact the false positive identification would have, and the capability of the program to prevent a malicious attack. They then combine the results of these tests and certify products at one of five certificate levels. The top performing ones to the least performing get AAA, AA, A, B and C respectively.
The MRG-Effitas is famous for malicious URL blocking test. They also release results for two tests every three months. One of the tests involve simulating real world protection against currently identified malware, which is similar to the dynamic real-world test done by AV-Comparatives. A product that completely eliminates any infestation receives Level 1 certificate, a product that managed to eliminate all the traces and planted files of a malware infestation after a reboot gets a Level 2 certificate.
The online banking certificate offered by MRG-Effitas is specifically testing for protection against financial malware.
Usability testing is a way of measuring the usability problems that occur when an antivirus program flags a legitimate software as a malicious or a suspicious software due to an error. To test this, researchers install and run a collection of popular programs to note any sort of an odd behavior by the antivirus guard. A separate scan only test is then done to make sure that the antivirus software doesn’t flag a list of more than 600,000 legitimate files as malware.
AV-Comparatives, an Austria based cyber security firm works closely with the University of Innsbruck to regularly releasing many tests in this area. Security software that pass the test receive a Standard certification while that failed ones are designed as merely tested. If a security program performs really well, then it receives an Advanced or Advanced + certificate.
Ease of Use
Whether the antivirus program is easy to get used to is tested in here. An antivirus guard with an intuitive graphical user interface is a lot more welcoming than an interface cluttered with lots of features.