Using machine learning to detect the file type

One of the techniques hackers use to sneak their malicious files into security systems is to obfuscate their file types. For example, a (malicious) PowerShell script is expected to have an extension, .ps1. A system administrator can aim to combat the execution of all PowerShell scripts on a system by preventing the execution of all files with the .ps1 extension. However, the mischievous hacker can remove or change the extension, rendering the file's identity a mystery. Only by examining the contents of the file can it then be distinguished from an ordinary text file. For practical reasons, it is not possible for humans to examine all text files on a system. Consequently, it is expedient to resort to automated methods. In this chapter, we will demonstrate how you can use machine learning to detect the file type of an unknown file. Our first step is to curate a dataset.