Malware thrives on systems that can’t detect it. It needs to be able to evade detection, which usually takes the form of a blacklist or a crude signature-based prevention system. This ruse can take little effort to pull off, because if you change the code enough, your malware will become undetectable to conventional hash matchers.
In order to be able to reliably identify a file repeatedly, there’s a tried-and-tested method that generally gives good results. You take an (unlimited) number of bytes from your chosen file. With this you create a unique, fixed-size number called a hash value. This hash value is specific to the file you’re targeting, meaning that you’ll be able to spot it on your system regardless of how it appears on the outside. Using an SHA1, SHA256 or MD5 hash of a file, you have a signature that can be matched against future files to check for a match.
Smashing the hash
In an effort to make their malware more successful, black hats know that they can continue to change the code until it becomes unrecognisable to this method of detection. If you make enough alterations to the code, it won’t show up using conventional systems that rely on this hash method to keep malware out – it’s the equivalent of changing a disguise daily in order to evade detection.
This is now a pretty routine tactic employed by black hats and hackers to keep their malware alive. Sometimes all it takes is a small change to the code itself before they recompile it into an ostensibly completely different file.
The Entropy Near-Match Analyser
The Entropy Near-Match Analyser was created in response to this problem. Security engineers needed a tool that took a more in-depth look at a file, using a more sophisticated method but without taking too much time or absorbing too many resources.
The concept of ‘entropy’ is a central tenet of thermodynamics. It refers to the orderliness of a system, basically differentiating the ‘order’ from what 19th Century German physicist Clausius called ‘chaos’.
This idea of order and chaos is useful outside the realm of theoretical physics, though. It’s this ability to differentiate between an orderly file and a disorderly one that makes Entropy useful – it measures the amount of disorder within a closed system.
Better than theoretical physics
In the context of thermodynamics, entropy can only ever be an approximation – there’s no way of knowing all possible states, so the result is more of an educated guess than a real-world measurement. But in our setting, Entropy can actually analyse the closed system – we know the content of each file, so the number and probability of each state are known absolutely.
Like signature-based tools, Entropy can produce accurate matches in a way that can be leveraged and used by your security systems as part of automation. But it’s way more accurate, responsive and useful than signature-based tools, because (for example) you can tune it to respond to matches within a certain degree of tolerance – it works on a confidence level.
Brittleness and accuracy
IOCs, or Indicators of Compromise, are popular in malware detection systems because they’re less ‘brittle’ than signatures. Entropy is an almost intuitive system – if you have a malware sample in a question from packet captures, for example, Entropy won’t necessitate the creation of a separate definition. Instead, you just take the measurement and – very quickly – Entropy will do its work. And because you don’t need to wait for a signature-based system to crunch through a hash in a linear way, Entropy outperforms these systems in speed too.
Recently, a prospective customer of Guidance Software in Asia called Entropy’s bluff. The customer changed a single line of code in a programme, compiled it, and then dared the sales engineer to spot the malware. Entropy immediately found every instance of the malware – something that competitive products evaluated in the same way were incapable of.
For more information on Guidance and their products, visit our website.
For further reading,