Thething Documentation

Transcript

TheThing Documentation Release 0.0.1 Alberto Geniola Jun 27, 2017 Contents: 1 TheThing: automated installer analyzer 1.1 Infrastructure components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Supported topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Single tier configuration, using Virtualbox. 2.1 Topology description . . . . . . . . . . 2.2 Host Controller Installation . . . . . . 2.3 Guest installation . . . . . . . . . . . . 2.4 Sniffer Installation . . . . . . . . . . . 2.5 Configuring the infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 7 8 14 18 21 Two tiers configuration, using Openstack. 3.1 Topology description . . . . . . . . . 3.2 Host Controller Installation . . . . . 3.3 Sniffer Installation . . . . . . . . . . 3.4 Guest installation . . . . . . . . . . . 3.5 Configuring the infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 23 24 28 31 34 4 Multitier baremetal configuration. 4.1 Topology description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Notes on the iSCSI diskless boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Requirements and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 35 36 36 5 Indices and tables 39 3 . . . . . 3 4 5 i ii TheThing Documentation, Release 0.0.1 In short, TheThing is an automated analysis sandbox, mainly oriented at analyzing freeware installers on Windows platform. As such, its main goal is to automate the installation procedure of software, while collecting information about the underlying changes happening to the underlying system. During each analysis the system monitors the disk, registry and network, and keeps track of every action performed by the software being analyzed. Therefore, a report is produced, summarizing all the information collected during the analysis, together with installation screenshots and a description of interactions performed against the installer user interface (if any). If you are familiar with malware sandbox analysis system (MSAS), you might think this software applies similar principles to a complete different problem, which has not been addressed by any other MSAS at the moment. In particular, the strength of this software stands in the combination of UI automation and dynamic analysis of software. Such approach is particularly relevant when dealing with Potentially Unwated Applications (PUP) installers. This class of installers do indeed require human interaction in order to trigger the PUP installation. In fact, classic MSAS tend to fail the analysis of such PUP droppers because they are not able to click through the installation procedure. TheThing was developed precisely to offer a valid tool for collecting information that can be later used for discriminating PUP droppers from clean software. Contents: 1 TheThing Documentation, Release 0.0.1 2 Contents: CHAPTER 1 TheThing: automated installer analyzer TheThing is an automated sandbox analyzer developed precisely for analyzing PUP installers. As such, it has three main objectives: • To click through the installation process, simulating a human user • To monitor and collect information regarding system changes during installation process • To store collected data into a database for later study To achieve such goals, TheThing is implemented via a distributed architecture, composed by: • Crawler(s) (optional): collects jobs from the Internet and save them into the database. • A central database: stores jobs and results. Also stores some state for ongoing analysis, used internally by host controllers. • Host controller(s): handles the life-cycle of guests via specific hypervisor API and performs some postprocessing on collected data. • Guest agent(s): represents a machine used as Sandbox in which the analysis occurs. It can be either virtual or physical. • Network sniffer(s): implements a gateway, providing Internet access to guests. Controlled by host controllers, it is in charge of sniffing traffic flowing among guests. Ideally, each part of TheThing runs on a distinct node, that can be either physical or virtual. However, the architecture is flexible enough to support many different configurations, with variable numbers of nodes and network topologies. For insance, the user may opt for a single-tier architecture, which implements the entire distributed architecture on a single hardware node. On the other hand the user may also opt for a two tiers configuration, separating the HostController from the hypervisor, or even using a N-Tier architecture, where each entity of the architecture runs on a separate bare-metal node. An important advantage of TheThing is the flexibility to run with many different hypervisors/configuration. The current version of the infrastructure supports both VirtualBox, Openstack and even bare-metal node analysis. This means that the user can chose the best configuration for her needs, considering hardware capabilities of his infrastructure. From a very general point of view, a simple scheme of the architecture is represented by the following figure. 3 TheThing Documentation, Release 0.0.1 Guests (sandboxes) are equipped with a single network interface (eth0), connected to a private network. On the same network there must be a sniffer instance, which acts as a gateway, providing basic DHCP/DNS/NAT capabilities to the guests. The sniffer is equipped with at least two network interfaces: eth0 is connected to external Internet, while eth1 is connected to the private network. Therefore, traffic from/to eth1 is routed to eth0, enabling guests to access the Internet via NAT. Infrastructure components Before going through installation and configuration of the infrastructure, it is really important to understand how TheThing works from an high perspective. More specifically, it is worth understanding which are and how the nodes of the infrastructure communicate each others. Central DB The central database consists in the synchronization point of the entire infrastructure. It holds results, jobs and status of current analysis. In general, both crawlers and HostControllers need to communicate with the central database. Therefore, in simple cases, it is usually a good idea to condensate the DB instance, the HostController and the crawlers on a single physical server. By doing so, network communication between HostController and Database happens locally via the loopback interface. Nevertheless, such configuration enables a more secure setup of the infrastructure, because DB access can be limited exclusively to the loopback domain (127.0.0.1). In general, the DB must be reachable by crawlers and HostControllers. When using multiple distributed HostController, the DB must allow incoming connection from IPs where HostControllers and crawlers are running. We strongly recommend to not expose the DB on the Internet, but connect HostControllers and DB via a private LAN (using a switch). HostController A HostController is a service, running on a node, that is in charge of handling the life-cycle of sandboxes. To do so, it uses specific APIs, depending on the sandbox type it has to handle. For example, a HostController might use 4 Chapter 1. TheThing: automated installer analyzer TheThing Documentation, Release 0.0.1 Openstack API for orchestrating the analysis of binaries via an Openstack cloud. Similarly, it can use Virtualbox API to do the same. Nevertheless, it can handle the life-cycle of bare-metal machines. From network perspective, each HostController must be able to communicate with the central DB and should be reachable by the sandboxes it administrates. Ideally, it uses a private NIC (or a local socket) to communicate with the DB while listening on a public IP for requests incoming from GuestAgents. Guests A guest is a sandbox, which can either be physical or virtual, in which binaries are executed. Current version of this infrastructure only supports Windows 7 SP1 32 bit operating system. Guests’ life-cycle is handled by a HostController. That means that the HostController decides when to spawn a Guest and what jobs the guest should be assigned. Guests are configured in such a way that a specific software module, the GuestAgent, runs at startup. That software module is in charge of retrieving the binary to analyze, automate its installation and collect resource accesses performed by the binary during its analysis. All this information is stored locally and then transmitted back to the HostController. From the networking perspective, the GuestAgent needs to connect to a specific HostController, via a TCP connection on port 9000 (by default). This means that the HostController must be reachable by guests. The easiest way to ensure such a requirement is to publish the HostController over the Internet, binding a public IP. Other approaches are possible, but we don’t discuss them here. Sniffer A sniffer is a virtual or physical entity that provides basic network services to a number of guests, while implementing sniffing capabilities. Sniffers are controlled by HostController via a web-service interface. This essentially means that HostControllers should be able to communicate with sniffers via a web interface. Again, the most compatible (and also most insecure) way of providing such capability is to assign the sniffer a public IP. Prerequisites Each type of node has different prerequisites. • HostController: developed in Python, can be installed on Linux or Windows hosts (Linux 16.04 LTS and Windows Server 2012 have been tested). It requires Python 2.7 to be installed on the system. • Central Database: can be one of the technologies supported by SqlAlchemy. We recommend to use PostgreSql, since it is the only dbms we tested. • Crawlers: developed in Python, can be installed on Linux or Windows hosts. We do recommend to install it alongside one of the HostController instances. • Guest: only Windows 7 SP1 operating systems are supported. • Sniffer: currently is implemented as a software router on the top of a standard Linux distribution (Ubuntu Server 16.04 LTS). Requires at least two network interfaces. All the nodes of the infrastructure can either be physical or virtual, depending on the user’s specific needs. Supported topologies Being a distributed system, the deployment of TheThing is quite time consuming. Moreover, TheThing might be configured in a number of ways, depending on the specific sandbox system used (hypervisor/baremetal). In this 1.2. Supported topologies 5 TheThing Documentation, Release 0.0.1 document we provide two distinct tutorials for deploying the thing: a single tier configuration using VirtualBox as hypervisor and a two-tiers configuration using a remote Openstack cloud infrastructure. Both of them are briefly described below. Single-tier configuration, using VirtualBox This is the easiest configuration possible, recommended for first time users. It only requires a single node to work and implements all networking communication locally. In fact, all the components of the infrastructure are deployed on a single node. The virtualization system used for this purpose is VirtualBox. Although the 1-tier configuration is the easiest to start from, it cannot scale. We suggest to start using this configuration and once confident, move to a more scalable configuration. Installation and Deployment instructions for such topology are available HERE Two-tiers configuration, using Openstack The two tiers configuration uses a node for hosting the database, the crawlers and a HostController. In this specific case, Openstack cloud is used as virtualization option. Such configuration enables better scalability, in accordance with resources available on the specific Openstack cloud. Installation and Deployment instructions for such topology are available HERE Multitier Baremetal The multitier baremetal configuration is implemented via a number of hardware nodes. In particular, the database, the HostController and crawlers usually reside on a single server. Another hardware node is used as dedicated sniffer. Eventually, a (potentially large) number of dedicated nodes are used as bare-metal sandboxes. Installation and Deployment instructions for such topology are available HERE Mixed multitier, [todo] Installation and Deployment instructions for such topology are available HERE. 6 Chapter 1. TheThing: automated installer analyzer CHAPTER 2 Single tier configuration, using Virtualbox. A single tier configuration consists in installing all the mandatory parts of the distributed system on a single hardware node. Obviously, such a configuration frustrates the advantages of a distributed system (scalability, robustness, etc.). On the other hand, it enables a simpler configuration scheme and simplifies some security issues affecting more complex topologies. In general, we do recommend this configuration if one or more of the following conditions are met: • This is the first time the user is experimenting the infrastructure. • Only a single, robust and high-performance server can be used for deploying the infrastructure. • Security is a must. • The number of installers to be analyzed is small or analysis time can be long. In this tutorial, the user will learn how to deploy the central DB, an HostController Agent and the Virtualbox virtualization system both on a Windows Server 2012 64bit operating system and Linux Ubuntu 16.04 LTS. Each one of the following sections is indeed splitted in two parts: the first one describes install instructions for Windows, while the second one regards Linux. Topology description A single tier topology condensates all the mandatory modules of the infrastructure on a single hardware node. While such node can either be virtual or physical, we recommend using a physical server, dedicated to such infrastructure, so that maximum performance can be achieved. The goal of this tutorial is to obtain the configuration showed in the following figure. 7 TheThing Documentation, Release 0.0.1 The host node has three distinct network interfaces. The first one is its physical NIC, used as main adapter when accessing the network. This interface will bind an IP address, marked as XXX.XXX.XXX.XXX, depending on how the user’s network is configured. A second (virtual) network adapter, based on the Virtualbox Host Only driver, is installed on the Host. Such interface binds 192.168.56.1 and its goal is to enable local communication (within the host) between Host Controller Agent and the Virtual Sniffer. Therefore, the Host Controller Agent will explicitely bind 192.168.56.1 and ETH2 of Sniffer will bind 192.168.56.2. A third network is emulated via the VirtualBox Nat Engine. Such network routes traffic incoming from the sniffer to the external internet, via NAT. The Virtual Sniffer acts as a gateway, by forwarding traffic from 192.168.0.0/24 to 172.16.0.1/24, which is in turn routed to the Internet via the physical NIC of the server. This configuration is pretty heavy and involves cascading NAT. Therefore it does not scale nor perform well. However, since all NIC use paravirtualized network, this configuration is still suitable for a small number of sandboxes and requires no networking hardware to be implemented. In the end, each sandbox has a single NIC, using DHCP to acquire an IP address. The sniffer uses dnsmasq service to provide both DNS realy and DHCP services on 192.168.0.0/24. Host Controller Installation In this section the user will learn how to configure the HostController for the single tier topology presented in the introduction. In particular, he will learn how to install the central database and the HostController binaries, alongside all the necessary software dependency. Installing the database The current version of the infrastructure uses SQLAlchemy as ORM. Although SQLAlchemy suports a variety of DBMS, we only tested the infrastructure with Postgresql, therefore we only provide instructions to install that database. Windows Download and install the appropriate Postgres DB version from here (we recommend Postgres 9.4). During the installation, stick with the defaults (e.g. use 5432 as listening port). Remember to note down the superuser password for using the databse. Now we need to create a database named “analyzer” and an user called “host_controller_agent”. Open a command prompt and navigate to postgres bin directory (usually C:\program files\postgreSQL\9.4bin\). Then execute the following commands: C:\> createdb -U postgres analyzer <> C:\> createuser -U postgres host_controller_agent <> C:\> psql -U postgres <> postgres#: alter user host_controller_agent with encrypted password '<>'; postgres#: grant all privileges on database analyzer to host_controller_agent; 8 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 to exit We have now created a database named analyzer and an user named host_controller_agent with password <>. Obviously, the user has to specify a meaningful password different from the placeholder used in this tutorial. Also, note the trailing semicolon used in both the SQL commands. To test if everything has worked, just try to log-in as the newly created user: C:\> psql -U host_controller_agent analyzer <> If login was successful, the db has been correctly configured. You can move to the next section. Linux Let’s download the necessary binaries $ sudo apt-get update $ sudo apt-get install postgresql It’s now time to create the database we will use for our infrastructure. $ sudo -u postgres createdb analyzer Now let’s create an user named host_controller_agent $ sudo -u postgres createuser host_controller_agent It is then time to create a password for the new user and assign to it all privileges for operatring on analyzer DB. $ sudo -u postgres psql postgres postgres=#: alter user host_controller_agent with encrypted password '<>'; postgres=#: grant all privileges on database analyzer to host_controller_agent; At this point, db configuration should be over. Let’s try to login and see if everything is working as expected. $ psql -h 127.0.0.1 -U host_controller_agent -d analyzer <> <> If login was successful, the db has been correctly configured. You can move to the next section. Installing Virtualbox Current version of the infrastructure has been tested with Virtualbox 5.1. Although newer versions of Virtualbox might be available, we do recommend to stick with the version 5.1 that has been proved working at the time of writing. If supported by the host, we also suggest to stick with the 64 bit version of Virtualbox, which might take full advantage of resources exposed by the underlying system. 2.2. Host Controller Installation 9 TheThing Documentation, Release 0.0.1 Windows Download and install the x64 bit Windows binaries of Virtualbox 5.1 from here. Just run the installer and follow the installation wizard. Linux The first step consists in downloading the appropriate version of Virtualbox for Linux system. Detailed instructions for installing Virtualbox on your favourite Linux distribution are available here. In this tutorial we will only provide instructions for installing Virtualbox 5.1.20 x64 on Linux Ubuntu 16.04 LTS. The following code snippet describes how to add the Virtualbox official repository to our apt sources and how to download and install Virtualbox 5.1. $ wget -q -O - https://www.virtualbox.org/download/oracle_vbox.asc | sudo apt-key add ˓→$ echo "deb http://download.virtualbox.org/virtualbox/debian vivid contrib" | sudo ˓→tee /etc/apt/sources.list.d/oracle-vbox.list $ sudo apt-get update $ sudo apt-get install dkms virtualbox-5.1 Installing Python This software uses python 2.7 to work. Python 3 is not supported, yet. Both x32 and x64 versions work correctly. However, when using VirtualBox as hypervisor, the python architecture must match the virtualbox architecture. In other terms, if using a 32 bit version of Virtualbox, the user has to stick with python 2.7 32bit; on the contrary, when using Virtualbox 64bit, HostController should use python 2.7 64 bits. This is a limtation introduced by COM binding libraries currently available for Virtualbox SDK. We assume the user opted to use a 64bit version of Virtualbox on a 64bit OS (either Windows or Linux). Windows ####### #. Download (from here) and install python 2.7 into a specific location, e.g. c:\python27_64 1. Open a command prompt and let’s create a virtualenv in C:\InstallAnalyzer for this project. C:\> c:\python27_64\scripts\pip.exe install virtualenv C:\> cd c:\ C:\> c:\python27_64\scripts\virtualenv InstallAnalyzer 2. It is now time to install windows’ building libraries. Python 2.7 uses VisualStudio 2008 building engine. Download and install it from here . After installation, make sure VS90COMNTOOLS environment variable has been set correctly. If this does not happen (or the user wants to use a more recent build environment), manually setting the path to a VC compiler should do the trick. Have a look at here. 3. Install prebuilt binaries. Windows requires some special care in order to build everything up. In particular, this software depends on lxml which needs development packages for libxml2 and libnxslt which are not trivial to build on windows. So, we advise to use prebuilt binaries on windows. Download and install the appropriate lxml and pyxml version from here and here. More precisely, we need to download the following two binary packages, precompiled for python 2.7 x64: • http://www.lfd.uci.edu/~gohlke/pythonlibs/tuth5y6k/lxml-3.7.3-cp27-cp27m-win_amd64.whl • http://www.lfd.uci.edu/~gohlke/pythonlibs/tuth5y6k/PyXML-0.8.4-cp27-none-win_amd64.whl Once downloaded, use a prompt to perform the installation: C:\> C:\InstallAnalyzer\scripts\pip install lxml-3.7.3-cp27-cp27m-win_ ˓→amd64.whl C:\> C:\InstallAnalyzer\scripts\pip install PyXML-0.8.4-cp27-none-win_ ˓→amd64.whl 10 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 Linux Python installation on Linux is pretty straight forward. In fact, most Linux versions already bundle a Python interpreter. Anyways, open a terminal and issue the following commands in order to install python2.7 alongside some software dependencies needed by the HostController Agent. $ sudo apt-get install python-pip python2.7-dev python2.7 unzip python-lxml python˓→zsi python-cffi python-ssdeep We now want to create a virtualenv for our HostController Agent in /home/ubuntu/InstallAnalyzer, so we run the following commands: $ sudo pip2 install virtualenv $ cd /home/ubuntu $ virtualenv InstallAnalyzer We also need to install lxml libraries in our virtualenv. $ cd InstallAnalyzer/bin $ pip2 install lxml ssdeep Install Virtualbox SDK API The HostController Agent relies on Virtualbox API bindings in order to programmatically control virtual machines. Hence, it requires the Virtualbox SDK to be installed on the host, matching the Virtualbox version installed. Windows ####### At first, download the appropriate version of Virtualbox SDK in accordance with the Virtualbox version installed previously. In our specific case, we need to download the SDK version for Virtualbox 5.1.10, from here. Once downloaded, extract the sdk content somewhere, cd into sdk/installer and run C:\> C:\InstallAnalyzer\scripts\python vboxapisetup.py install Then, copy the vboxwebservice bindinds: C:\> cd ../bindings/webservice/python/lib C:\> copy *.py C:\InstallAnalyzer\lib\site-packages\vboxapi\ Linux First of all we need to download and unzip the correct version of Virtualbox SDK, that is 5.1.20 at the time of writing. $ cd /home/ubuntu $ wget http://download.virtualbox.org/virtualbox/5.1.20/VirtualBoxSDK-5.1.20-114628. ˓→zip $ unzip VirtualBoxSDK-5.1.20-114628.zip Then we install the SDK into our virtualenv by issuing the following commands. $ cd sdk/installers $ export VBOX_INSTALL_PATH=/usr/lib/virtualbox $ sudo -E /home/ubuntu/InstallAnalyzer/bin/python2.7 vboxapisetup.py install 2.2. Host Controller Installation 11 TheThing Documentation, Release 0.0.1 Installing HostController binaries At this stage, all the “hard” dependencies should be ok. It’s time to download and install the HostController Agent. Windows First, let’s clone the git repository of HostController Agent C:\> git clone https://[email protected]/aaltopuppaper/hostcontroller.git Now we need to build the distributable version and install it via PIP command. C:\> C:\> C:\> C:\> cd hostcontroller C:\InstallAnalyzer\scripts\python setup.py sdist cd dist C:\InstallAnalyzer\scripts\pip install HostController-0.1.zip Linux Let’s download the HostController Agent binaries from the official git repository. $ cd /home/ubuntu $ git clone https://github.com/albertogeniola/HostController1.1_python.git Now let’s build and install those binaries into our virtualenv $ $ $ $ cd /home/ubuntu cd HostController1.1_python /home/ubuntu/InstallAnalyzer/bin/python2.7 setup.py sdist sudo /home/ubuntu/InstallAnalyzer/bin/pip2.7 install dist/HostController-0.1.tar.gz ˓→--upgrade Creating HostOnly adapter via VirtualBox According to the network topology presented in [Introduction](1_Introduction.md], the Host controller must provide a HostOnly Adapter interface. Such interface must be allocated manually via VBoxManage interface. In general, Virtualbox automatically creates a HostOnly adapter at installation time. However this might change in the future, so the user has not to rely on such assumption. Windows Open a command prompt with administrator rights, navigate into the virtualbox installation directory and check if any HostOnly adapter is available for use. C:\> cd "%PROGRAMFILES%\Oracle\Virtualbox" C:\> VBoxManage list -l hostonlyifs If no interface is listed, let’s create one, otherwise skip the next command. C:\> VBoxManage hostonlyif create Now let’s verify its configuration: 12 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 C...>VBoxManage list -l hostonlyifs C:\Program Files\Oracle\VirtualBox>VBoxManage list -l hostonlyifs Name: VirtualBox Host-Only Ethernet Adapter #2 GUID: cf51993c-1be7-43d2-8eea-707c98ca3722 DHCP: Disabled IPAddress: 192.168.208.1 NetworkMask: 255.255.255.0 IPV6Address: fe80:0000:0000:0000:3814:453d:ba44:c6a2 IPV6NetworkMaskPrefixLength: 64 HardwareAddress: 0a:00:27:00:00:26 MediumType: Ethernet Status: Up VBoxNetworkName: HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter #2 The HostOnly nic has been given “VirtualBox Host-Only Ethernet Adapter #2” as name, but its network configuration needs to be adjusted to reflect our topology. So we run the following command: C...>VBoxManage hostonlyif ipconfig "VirtualBox Host-Only Ethernet Adapter #2" --ip ˓→192.168.56.1 --netmask 255.255.255.0 Let’s double check the new configuration: C...>VBoxManage list -l hostonlyifs Name: VirtualBox Host-Only Ethernet Adapter #2 GUID: cf51993c-1be7-43d2-8eea-707c98ca3722 DHCP: Disabled IPAddress: 192.168.56.1 NetworkMask: 255.255.255.0 IPV6Address: fe80:0000:0000:0000:3814:453d:ba44:c6a2 IPV6NetworkMaskPrefixLength: 64 HardwareAddress: 0a:00:27:00:00:26 MediumType: Ethernet Status: Up VBoxNetworkName: HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter #2 Linux Open a terminal and verify if there is any HostOnly interface available. $ VBoxManage list -l hostonlyifs If no interface is listed, let’s create one, otherwise skip the next command. C:\> VBoxManage hostonlyif create Now let’s verify its configuration: $ VBoxManage list -l hostonlyifs ... The HostOnly nic has been given “VirtualBox Host-Only Ethernet Adapter #2” as name, but its network configuration needs to be adjusted to reflect our topology. So we run the following command: $ VBoxManage hostonlyif ipconfig "..." --ip 192.168.56.1 --netmask 255.255.255.0 2.2. Host Controller Installation 13 TheThing Documentation, Release 0.0.1 Let’s double check the new configuration: $ VBoxManage list -l hostonlyifs ... Quick test TODO: make sure HostController can start Guest installation This section of the tutorial will guide the user through the preparation of a virtual disk image, which will be used as base disk for the virtual Sandbox. Although most of the preparation steps are common to any kind of Guest, this guide targets the case of single tier configuration for a Windows 7 SP1 32bit client, using Virtualbox as virtualization tool. In summary, the user needs to: 1. Create the virtual disk and virtual machines 2. Install a fresh Windows 7 SP1 32 bit OS 3. Install the Guest Agent 4. Save and store the VM image Creating the Virtual Machine The very first step consists in creating a temporary Virtual Machine, used only to prepare the virtual disk. This task can be achieved similarly, both in Windows and Linux. Windows The first step is to create a virtual disk. So, open a command prompt and type the following commands. C:\> cd \ C:\> cd "%PROGRAMFILES%\Oracle\Virtualbox" C:\> VBoxManage createhd --filename C:\InstallAnalyzer\Disks\guest_preparation.vdi -˓→size 25000 Now, we want to create a VM and attach the newly created disk to that. C:\> VBoxManage createvm --name "guest_preparation" --ostype "Windows7" --register C:\> VBoxManage storagectl "guest_preparation" --name "SATA Controller" --add sata -˓→controller IntelAHCI C:\> VBoxManage storageattach "guest_preparation" --storagectl "SATA Controller" -˓→port 0 --device 0 --type hdd --medium "C:\InstallAnalyzer\Disks\guest_preparation. ˓→vdi" In order to install Windows 7 on the machine, we will need a virtual CD/DVD drive, that will mount our installation media. Therefore: C:\> VBoxManage storagectl "guest_preparation" --name "IDE Controller" --add ide C:\> VBoxManage storageattach "guest_preparation" --storagectl "IDE Controller" -˓→port 0 --device 0 --type dvddrive --medium /path/to/windows_7_SP1_32.iso 14 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 Finally, let’s configure some other settings, such as ram, vram and hardware acceleration C:\> VBoxManage C:\> VBoxManage ˓→boot4 none C:\> VBoxManage C:\> VBoxManage modifyvm "guest_preparation" --ioapic on modifyvm "guest_preparation" --boot1 dvd --boot2 disk --boot3 none -modifyvm "guest_preparation" --memory 2048 --vram 128 modifyvm "guest_preparation" --nic1 nat --nictype1 virtio At this point the virtual machine is ready to be started. Linux The first step is to create a virtual disk. So, open a command prompt and type the following commands. $ cd /home/ubuntu/InstallAnalyzer $ VBoxManage createhd --filename /home/ubuntu/InstallAnalyzer/Disks/guest_preparation. ˓→vdi --size 25000 Now, we want to create a VM and attach the newly created disk to that. $ VBoxManage createvm --name "guest_preparation" --ostype "Windows7" --register $ VBoxManage storagectl "guest_preparation" --name "SATA Controller" --add sata -˓→controller IntelAHCI $ VBoxManage storageattach "guest_preparation" --storagectl "SATA Controller" --port ˓→0 --device 0 --type hdd --medium "/home/ubuntu/InstallAnalyzer/Disks/guest_ ˓→preparation.vdi" In order to install Windows 7 on the machine, we will need a virtual CD/DVD drive, that will mount our installation media. Therefore: $ VBoxManage storagectl "guest_preparation" --name "IDE Controller" --add ide $ VBoxManage storageattach "guest_preparation" --storagectl "IDE Controller" --port 0 ˓→--device 0 --type dvddrive --medium /path/to/windows_7_SP1_32.iso Finally, let’s configure some other settings, such as ram, vram and hardware acceleration $ VBoxManage modifyvm $ VBoxManage modifyvm ˓→boot4 none $ VBoxManage modifyvm $ VBoxManage modifyvm "guest_preparation" --ioapic on "guest_preparation" --boot1 dvd --boot2 disk --boot3 none -"guest_preparation" --memory 2048 --vram 128 "guest_preparation" --nic1 nat --nictype1 virtio At this point the virtual machine is ready to be started. Install the Operating system The next step consists in installing Windows 7 SP1 32 bit on the vm. To do so, simply start the VM. To start the VM, just execute the following command from the command prompt (for both Windows and Linux): ...> VBoxManage startvm "guest_preparation" The virtual machine will start its boot sequence. The windows installation procedure should begin shortly after the boot up. The user now needs to follow the installation wizard like in a normal machine. 2.3. Guest installation 15 TheThing Documentation, Release 0.0.1 Once the installation is completed, the installation wizard will prompt the user to choose an username and a password. While the user can type any kind of username, it is strictly mandatory to set no password, so that the auto log-on is activated. Note that this system will not work in case a password is set. The virtual machine prepared in advance is equipped with a single network adapter using a paratirtualized network interface. While most of the Linux distributions do include a driver for the paravirtualized NIC, windows 7 does not. This means that we need to install a specific driver, manually. To do so, let’s first shut down the VM we created. Then, from the host, let’s download the drivers from here. Once the download is complete, we need to mount that iso into the drive of the virtual machine. Once again, open a terminal/command prompt, navigate within the Virtualbox path, and type the following commands: ...> VBoxManage storageattach "guest_preparation" --storagectl "IDE Controller" -˓→port 0 --device 0 --type dvddrive --medium /path/to/downloaded_virtio_driver.iso Then start the VM again and wait for the OS to complete the bootstrap. ...> VBoxManage startvm "guest_preparation" Once the boot is complete, open a command prompt from within the VM, and type the following command: C:\> mmc devmgmt.msc Navigate among the devices and look for the unrecognized Ethernet Controller under “Other devices” category. Then right click on it and chose “Update driver”. A driver installation wizard will begin. Select “Look for a driver in a specific location” and browse the D: drive, where the virtio iso has been mounted. The system will automatically recognize and will prompt confirmation for installing the needed driver. Be sure to accept the driver installation. Windows update and activation At this stage the Guest is capable of surfing the Internet via its own virtual adapter (which is natted behind the network of the host). Therefore, the user can now activate the operating system. We also suggest to update the system via Windows Update. Once the updates have been correctly installed, it is strongly advised to disable windows automatic updates. Doing so, the system will not try to update the OS during the analysis. On the contrary, in case automatic updates are left enabled, there is a chance that they will trigger during the analysis, impacting on performance and potentially biasing sniffed network traffic. Once the system has been correctly activated and updates have been performed, the user can then proceed with the installation of the GuestAgent bootstrapper. Install .NET 4.0, VC Redist 2013 and VC Redist 2015 Before installing the GuestAgent Bootstrapper, the system must comply with some software package dependency. In particular, the agents rely on both Visual Studio 2013/2015 Visual C++ redistributable packages and .NET 4.0. Thus, download and install the 32bit version of the .NET framework from the Microsoft website. The correct version of the .NET framework can be found here. Then, do the same for the VC++ 2013 and VC++ 2015 Redistributable Packages. The VC++ 2013 32 bit version is available here. The VC++ 2015 32 bit version is available her. Note:: It may happens that those software packages fail the installation when the system is not updated. In case of installation failure, we do recommend to update the system and perform the installation once again. 16 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 Install the GuestAgent Bootstrapper From within the Virtual Machine, open a browser and navigate to this URL. Then, execute the installation of the bootstrapper, by simply double clicking on it. Then, follow the wizard to complete the installation. The installer will take care of downloading the needed python environment, necessary dependencies and will also install the bootstrap autostart task. To double check the bootstrapper installation, reboot the virtual machine. Just after Windows loads up, the bootstrapper program should automatically start, complaining about no response from any sniffer. If that is the case, the bootstrapper is correctly working. Now close the bootstrapper and shut down the virtual machine correctly. Pack the image The image is finally ready. We just need to save it to a known location and make it “immutable”. To do so, we need to shut down the virtual machine, release the disk from the storage controller, make it immutable and then remove the guest_preparation machine. Windows Open a prompt and execute the following: C:\> cd "%PROGRAMFILES%\Oracle\Virtualbox" C:\> vboxmanage storageattach "guest_preparation" --storagectl "SATA Controller" -˓→port 0 --device 0 --medium none C:\> vboxmanage modifymedium "C:\InstallAnalyzer\Disks\guest_preparation.vdi" --type ˓→immutable --compact Now we want to remove the virtual machine used to setup the virtual disk. C:\> vboxmanage unregistervm "guest_preparation" --delete Finally, make the virtual disk readonly. This step is not mandatory. However it is recommended toi make the disk readonly so that there is no change for Virtualbox to write on it. C:\> attrib +R "C:\InstallAnalyzer\Disks\guest_preparation.vdi" Linux Similarly to the windows commands, let’s open a terminal and release the disk from the virtual machine. $ vboxmanage storageattach "guest_preparation" --storagectl "SATA Controller" --port ˓→0 --device 0 --medium none $ vboxmanage modifymedium /home/ubuntu/InstallAnalyzer/Disks/guest_preparation.vdi" -˓→type immutable --compact Then remove the virtual machine: $ vboxmanage unregistervm "guest_preparation" --delete Finally, make the virtual disk readonly. This step is not mandatory. However it is recommended toi make the disk readonly so that there is no change for Virtualbox to write on it. $ chmod 555 "/home/ubuntu/InstallAnalyzer/Disks/guest_preparation.vdi" 2.3. Guest installation 17 TheThing Documentation, Release 0.0.1 Sniffer Installation A sniffer is a machine (which might be physical or virtual) acting as a gateway between the sandboxes and the Internet. In this tutorial we will refer to it with both gateway and sniffer terms. It implements some basic network services for guest machines, alongside a backend web interface, used by the HostController to manage sniffing sessions. In accordance with the single tier topology, we will virtualize the sniffer alongside the sandboxes. The user might opt for one of the following options in order to install the prepare the sniffer: • Download and use the pre-configured VDI image for Virtualbox Emulated Sniffer from here (recommended for first-time users) • Manually configure from scratch a Sniffer VM (needed for custom network configuration) The first option is advised for users who approach to the system for the first time or are not used to any linux environment. In fact, the pre-configured image has been customized in accordance with the network configuration presented in Introduction. If the user has stuck with the topology presented in the previous steps, this image will work out-ofthe-box. On the contrary, if some network configuration has been customized, it is necessary to adjust or configure the image from scratch. Option A: Download preconfigured image The pre-configured image of Sniffer is [available here](). Such image is a compressed dynamically allocated VDI, of about 1.5 GB. Once downloaded, extract the image into a known location, e.g. C:\InstallAnalyzer. The given image configures the first NIC, eth0, of the gateway as WAN access. In our specific configuration, this interface is given the static address 172.16.0.2. Its traffic flows through the Virtualbox NAT netowrk, which is internally controlled by the HostController Agent. The second nic, eth1, is then attached to the internal ant connecting guests to the sniffer. A DHCP service is bound to that NIC and serves addresses from 192.168.2.2/24 up to 192.168.2.250/24. The provided image also contains the last version of the MiddleRouter software, that is used for sniffing traffic from guest instances. Credentials used by that image are: - Username: ubuntu - Password: reverse Such image is ready to be used in test environments. Before using such image in a production environment we strongly advise to change username/password credentials. This can be done in a number of ways. The easiest is to mount the image on a VM, run it, and change the password of ubuntu user. That’s it for the sniffer. Just proceed to the next step of the tutorial. Option B: Create the Sniffer virtual machine In case the user wants to configure the sniffer by herself, she needs to perform the following steps. Note that this guide only covers the case of installation on Linux Host (in either virtual or physical hosts). VM Setup To start, let’s create a VM with Virtualbox. Use the Virtualbox wizard to create an Ubuntu virtual machine. Make sure the VM is given at least 2 Gb of ram memory and a (virtual) disk of 25 Gb of free space. Once the VM has been created, edit it by enabling both the second and the third network adapters. Make sure all the three NICs use paravirtualized network (virtio-net) emulation mode. Then attach Adapter 1 to NAT and Adapter 3 to Host-Only adapter created during HostConstroller configuration. Leave Adapter 2 unattached. Then, download and install your favourite Linux distribution on the virtual machine. Ubuntu 16.04LTS 64bit is suggessted, since it is the one we tested in this tutorial. 18 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 MiddleRouter installation The next step is to install the MiddleRouter software on the fresh linux installed OS. This operation is pretty straight forward thanks to the setup script available here . Thus, reboot the vm, log into a terminal and then issue the following commands. $ cd /tmp $ git clone https://github.com/albertogeniola/TheThing-Sniffer.git $ cd middlerouter $ sudo chmod +x install_script.sh $ sudo -H ./install_script.sh << follow the wizard >> When prompted to install the sniffer agent type “Y”. Optionally, you may want to install the pcap analyzer on the same host. If that is the case, answer “y” when prompted. In our specific case, being a single tier node, we also install the network analyzer on the sniffer. Configure network dapters We now must configure the network connectivity of the sniffer to reflect our network topology as presented in the topology description. In particular: • ETH0 is used as external network connectivity and statically acquires 172.16.0.2, routing traffic to 172.16.0.1 (VboxNat gateway). • ETH1 is used as interface for internal network (TestNet), which guests connect to. In our case we bind it to 192.168.0.1/24. Both DHCP and DNS services will insist on this interface. • ETH2 is connected directly to the Host via the Virtualbox HostOnly adapter. The sniffer binds 192.168.56.2 while the Host binds 192.168.56.1. Let’s edit /etc/network/interface so that it matches the following: # -------------------------------------------------# Loopback device auto lo iface lo inet loopback # WAN/Internet network, where traffic will be routed auto eth0 iface eth0 inet static address 172.16.0.2 netmask 255.255.255.0 gateway 172.16.0.1 dns-nameservers 172.16.0.1 # Internal NIC, where guests are attached auto eth1 iface eth1 inet static address 192.168.0.1 netmask 255.255.255.0 # HostOnly adapter, used to talk with HostController Agent auto eth2 iface eth2 inet static address 192.168.56.2 netmask 255.255.255.0 2.4. Sniffer Installation 19 TheThing Documentation, Release 0.0.1 # -------------------------------------------------- NOTE: Interface names might be different on new Linux distributions, because of the new predictable interface naming implementation. To avoid any issue we suggest to disable such feature by adding a custom switch in the default grub configuration file. To do so, edit your /etc/default/grub changing the line from: GRUB_CMDLINE_LINUX="" to: GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" and, finally, update grub. $ sudo update-grub At next reboot, interfaces will be given classical ethX names. Configure network services The first step is to adjust the configuration regarding the DHCP service and the DNS relay. Those are provided by dnsmasq. So, let’s edit /etc/dnsmasq.conf, making it look like the following: # -------------------------------------------------interface=eth1 interface=lo dhcp-range=192.168.0.2,192.168.0.250,255.255.255.0,12h # -------------------------------------------------- This configuration tells to dnsmasq to provide DNS relay and DHCP service on both the loopback interface (lo) and on eth1 (InternalNat). Is also enables the DHCP server, that assignings IPs in the range between 192.168.0.2 and 192.168.0.250. We now need to enable forwarding among interfaces. Edit /etc/sysctl.conf and adjust the forwarding option to match 1. # -------------------------------------------------net.ipv4.ip_forward = 1 # -------------------------------------------------- Then make changes permanent by running $ sudo sysctl -p /etc/sysctl.conf In order to provide external connectivity to guests, the gateway needs to enable the NAT service. This is done by leaveraging _iptables_. Our objective is to forward packets from internal network, on /dev/eth1, to the external network on /dev/eth0. Therefore we issue the following commands. $ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE $ sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ˓→ACCEPT $ sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT $ sudo iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE 20 Chapter 2. Single tier configuration, using Virtualbox. TheThing Documentation, Release 0.0.1 $ sudo iptables -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ˓→ACCEPT $ sudo iptables -A FORWARD -i eth1 -o eth2 -j ACCEPT Since IPTABLES rules are not automatically persisted, we need to save them. $ sudo apt-get install iptables-persistent $ sudo iptables-save > /etc/iptables/rules.v4 Security Enforcements TODO: limiting netowrk capabilities of guests (i.e. avoid them to access LAN where HOST is connected). TODO: make the sniffer gateway to bind only 192.168.56.2 and block traffic from 192.168.0.x/24 to 192.168.56.2. Check everything is ok Upon successful reboot, make sure the sniffer service is correctly running, by typing: $ sudo reboot <> $ sudo service sniffer status <> $ sudo service analyzer status <> In case it is not running, check the output log for the cause: $ cat /var/log/sniffer.err <> $ cat /var/log/analyzer.err <> Finally, the very last test that confirms the correct installation of the sniffer and its good network configurationFrom the Host controller, make sure the web-service interface is working. Open a browser and type: http://192.168.56.1:8080 Image storage At this point the image of the sniffer is ready. However, it a good idea to clone it into C:\InstallAnalyzer\. To do so, ensure the sniffer is powered off, then open a terminal on the host and type the following command: C:\> cd "%PROGRAMFILES%\Oracle\VirtualBox" ..\> VBoxManage clonehd "C:\InstallAnalyzer\sniffer.vdi" --format ˓→VDI Configuring the infrastructure TODO Now that a minimum set of components has been put in place, it is time to configure the HostController Agent. Each HostController Agent relies on a specific configuration file, __controller.conf__, located in its installation folder. Therefore, we will find the _controller.conf_ under C:InstallAnalyzerLib 2.5. Configuring the infrastructure 21 TheThing Documentation, Release 0.0.1 22 Chapter 2. Single tier configuration, using Virtualbox. CHAPTER 3 Two tiers configuration, using Openstack. A two tiers configuration uses two distinct nodes to implement the analysis infrastructure. In particular, a local node will host the central database and the host controller, while a remote cloud infrastructure (i.e. Openstack) is in charge of running the sandboxes and the sniffers. In contrast with the Single tier configuration, this one has the advantage of offloading the heavy virtualization task to a remote infrastructure (e.g. Openstackcloud). In general, we do recommend this configuration if one or more of the following conditions are met: • The user can leverage a powerful Openstack infrastructure (which might be remote or in his intranet) • The numberof installers to be analyzed is great and throughput should be high • The user trusts the remote cloud infrastructure This tutorial will guide the user in the installation of Central Database, Host Controller and in the preparation of the sniffer instance to be used on the openstack cloud. Moreover, the user will learn how to deploy the infrastructure on an Openstack Cloud. Topology description A two tiers topology consists in two nodes: a local node handling the central database together with the Host Agent, and a remote cloud infrastructure. The following figure depicts the configuration goal of the two tiers topology that the user will deploy following this tutorial. Openstack/img.png As already mentioned, there are two major entities in game. A local server (tier 1) hosts the database, the crawlers and the HostController Agent while the remote openstack cloud hosts sniffers and sandboxes. Communication between 23 TheThing Documentation, Release 0.0.1 HostController and the database happens locally to the tier 1. This means that the DB instance will only bind the loopback address. On the contrary, communication among sniffer, HostController Agent and Sandboxes happens via Internet/Intranet, depending on the user’s specific topology. Tier 1 For sake of simplicity, we assume the user is using public Internet. This means that a gateway exists bnetween the Tier 1 and the ISP, which usually implements both L3/L4 services such as routing, nat and firewall. For this reason, the GuestController agent needs to bind the private address provided by the gateway (either via DHCP or statically assigned). Moreover the user must configure by herself the necessary firewall/natting rules to allow communication between remote cloud and GuestAgent. In particular the user must redirect incoming traffic to port TCP 9000 to the HostController Agent. Referring to the picture, the rule should be defined like the following: External IP 0.0.0.0 External Port Start 9000 External Port End 9000 Internal IP X.X.X.X Internal Port 9000 Tier 2 The Openstack network configuration is based on the coexistence of two networks: an Internal Network and an External Network. Sandboxes live within the Internal Network. They receive the network address via Openstack DHCP service. Moreover, the DHCP service also configures the default gateway of sandboxes, setting it to the address of the Sniffer. Therefore, the sniffer acts as a gateway between the Internal Network (192.168.0.0/24) and the External Network (192.168.1.0/24). The sniffer is configured to acquire the address of the public network via DHCP, while statically binding 192.168.0.1 on the Internal Network. Moreover, the sniffer must also acquire a floating public IP, so that it can be easily reachable by the HostController via the public Internet. What needs to be done manually Most of the network configuration of the remote openstack cloud is performed automatically by the HostController Manager, based on its configuration file. However, there are a few things that cannot be automated. The user has to perform such tasks based on her specific network topology. In particular: • Open port TCP 9000 on tier 1 via port mapping, allowing incoming connections from the Internet (or from the specific openstack neutron node) • Set the HostController Agent to listen on the private IP address where TCP 9000 has been mapped to (X.X.X.X) • Allocate the External Network on Openstack cloud (we assume its CIDR is 192.168.1.0/24) • Create a Router on the External Network that routes traffic to the Internet The rest of networking configuration is handled automatically by the HostController Agent, which uses the Openstack API directly to setup the Internal Network, connect the required interfaces, provide routing capabilities to the Sniffer Instance. Host Controller Installation In this section the user will learn how to configure the HostController for the two tiers topology presented in the introduction (network topology here). In particular, he will learn how to install the central database and the HostController binaries, alongside all the necessary software dependency. 24 Chapter 3. Two tiers configuration, using Openstack. TheThing Documentation, Release 0.0.1 Installing the database The current version of the infrastructure uses SQLAlchemy as ORM. Although SQLAlchemy suports a variety of DBMS, we only tested the infrastructure with Postgresql, therefore we only provide instructions to install that database. Windows Download and install the appropriate Postgres DB version from here (we recommend Postgres 9.4). During the installation, stick with the defaults (e.g. use 5432 as listening port). Remember to take note of the superuser password used during the installation process of the database (this will be needed later on). Now we need to create a database named “analyzer” and an user called “host_controller_agent”. Open a command prompt and navigate to postgres bin directory (usually C:\program files\postgreSQL9.4bin\). Then execute the following commands: C:\> createdb -U postgres analyzer <> C:\> createuser -U postgres host_controller_agent <> C:\> psql -U postgres <> postgres#: alter user host_controller_agent with encrypted password '<>'; postgres#: grant all privileges on database analyzer to host_controller_agent; to exit We have now created a database named analyzer and an user named host_controller_agent with password <>. Obviously, the user has to specify a meaningful password different from the placeholder used in this tutorial. Also, note the trailing semicolon used in both the SQL commands. To test if everything has worked, just try to log-in as the newly created user: C:\> psql -U host_controller_agent analyzer <> If login was successful, the db has been correctly configured. You can move to the next section. Linux Let’s download the necessary binaries $ sudo apt-get update $ sudo apt-get install postgresql Time to create the database we will use for our infrastructure. $ sudo -u postgres createdb analyzer Now let’s create an user named host_controller_agent 3.2. Host Controller Installation 25 TheThing Documentation, Release 0.0.1 $ sudo -u postgres createuser host_controller_agent It is then time to create a password for the new user and assign to it all privileges for operatring on analyzer DB. $ sudo -u postgres psql postgres postgres=#: alter user host_controller_agent with encrypted password '<>'; postgres=#: grant all privileges on database analyzer to host_controller_agent; At this point, db configuration should be over. Let’s try to login and see if everything is working as expected. $ psql -h 127.0.0.1 -U host_controller_agent -d analyzer <> <> If login was successful, the db has been correctly configured. You can move to the next section. Installing Python This software uses python 2.7 to work. Python 3 is not supported, yet. Both x32 and x64 versions work correctly. We assume the user opted to use a 64bit OS (either Windows or Linux). Windows 1. Download (from here) and install python 2.7 into a specific location, e.g. C:\python27_64\ 2. Open a command prompt and let’s create a virtualenv in C:\InstallAnalyzer for this project. C:\> c:\python27_64\scripts\pip.exe install virtualenv C:\> cd c:\ C:\> c:\python27_64\scripts\virtualenv InstallAnalyzer 3. It is now time to install windows’ building libraries. Python 2.7 uses VisualStudio 2008 building engine. Download and install it from here . After installation, make sure VS90COMNTOOLS environment variable has been set correctly. If this does not happen (or the user wants to use a more recent build environment), manually setting the path to a VC compiler should do the trick. Have a look at here . 4. Install prebuilt binaries. Windows requires some special care in order to build everything up. In particular, this software depends on lxml which needs development packages for libxml2 and libnxslt which are not trivial to build on windows. So, we advise to use prebuilt binaries on windows. Download and install the appropriate lxml and pyxml version from here and here . More precisely, we need to download the following two binary packages, precompiled for python 2.7 x64: • http://www.lfd.uci.edu/~gohlke/pythonlibs/tuth5y6k/lxml-3.7.3-cp27-cp27m-win_amd64.whl • http://www.lfd.uci.edu/~gohlke/pythonlibs/tuth5y6k/PyXML-0.8.4-cp27-none-win_amd64.whl Once downloaded, use a prompt to perform the installation: C:\> C:\InstallAnalyzer\scripts\pip install lxml-3.7.3-cp27-cp27m-win_ ˓→amd64.whl C:\> C:\InstallAnalyzer\scripts\pip install PyXML-0.8.4-cp27-none-win_ ˓→amd64.whl 26 Chapter 3. Two tiers configuration, using Openstack. TheThing Documentation, Release 0.0.1 Linux Python installation on Linux is pretty straight forward. In fact, most Linux versions already bundle a Python interpreter. Anyways, open a terminal and issue the following commands in order to install python2.7 alongside some software dependencies needed by the HostController Agent. $ sudo apt-get install python-pip python2.7-dev python2.7 unzip python-lxml python˓→zsi python-cffi python-ssdeep We now want to create a virtualenv for our HostController Agent in /home/ubuntu/InstallAnalyzer, so we run the following commands: $ sudo pip2 install virtualenv $ cd /home/ubuntu $ virtualenv InstallAnalyzer We also need to install lxml libraries in our virtualenv. $ cd InstallAnalyzer/bin $ pip2 install lxml ssdeep Installing HostController binaries At this stage, all the “hard” dependencies should be ok. It’s time to download and install the HostController Agent. Windows First, let’s clone the git repository of HostController Agent C:\> git clone https://[email protected]/aaltopuppaper/hostcontroller.git Now we need to build the distributable version and install it via PIP command. C:\> C:\> C:\> C:\> cd hostcontroller C:\InstallAnalyzer\scripts\python setup.py sdist cd dist C:\InstallAnalyzer\scripts\pip install HostController-0.1.zip Linux Let’s download the HostController Agent binaries from the official git repository. $ cd /home/ubuntu $ git clone https://github.com/albertogeniola/HostController1.1_python.git Now let’s build and install those binaries into our virtualenv: $ $ $ $ cd /home/ubuntu cd HostController1.1_python /home/ubuntu/InstallAnalyzer/bin/python2.7 setup.py sdist sudo /home/ubuntu/InstallAnalyzer/bin/pip2.7 install dist/HostController-0.1.tar.gz ˓→--upgrade 3.2. Host Controller Installation 27 TheThing Documentation, Release 0.0.1 Sniffer Installation A sniffer is a machine (which might be physical or virtual) acting as a gateway between sandboxes and the Internet. In this tutorial we will refer to it with both gateway and sniffer terms. It implements some basic network services for guest machines, alongside a backend web interface, used by the HostController to manage sniffing sessions. In accordance with the two tiers topology, the sniffer will be implemented via a virtual machine in the remote cloud (on Tier 2). The user might opt for one of the following options in order to install the sniffer virtual image: • Download and use the pre-configured VDI image for Openstack Emulated Sniffer [from here]() (recommended for first-time users) • Manually configure the Sniffer VM from scratch Please note that, usage of the pre-configured image is subjected to the implementation of exact network topology presented here here. On the contrary, if the underlying network topology in place is different from the one described previously, it is necessary to configure the image from scratch. Option A: Download preconfigured image This approach consists in using a pre-configured linux image which reflects all the configuration needed to work with a two tier architecture via remote Openstack cloud. Still, the user must change the default access credentials to the machine, preventing others to obtain unlegitimate access to such instance. So, the first step is to download the pre-configured image, which is available here . Once downloaded, extract the image, use your favourite virtualization system to create a virtual machine and mount such image as primary disk. Hence, boot the VM and log in with the following credentials: • Username: ubuntu • Password: reverse Once logged in, change the default username/password credentials with the ones you prefer, issuing the following commands: sudo passwd ubuntu <> <> Now gracefully shutdown the machine and unmount the disk image. The last step is to upload the image to the Openstack cloud. This can be done via the Horizon web interface (if available) or via the image service CLI (a.k.a. Glance). If using the CLI, open a terminal and type the following (wither in Windows or Linux): glance image-create --name SnifferImage --disk-format VDI --file --progress That command will upload the image located at to the Glance image service, assigning to it the name SnifferImage and showing the progress of the operation. That’s it for the sniffer. Just go to the next step. Option B: Create the Sniffer virtual machine In case the user wants to configure the sniffer by herself, she needs to perform the following steps. Note that this guide only covers the case of installation on Linux Host (in either virtual or physical hosts). 28 Chapter 3. Two tiers configuration, using Openstack. TheThing Documentation, Release 0.0.1 VM Setup To start, let’s create a VM with Virtualbox. Use the Virtualbox wizard to create an Ubuntu virtual machine. Make sure the VM is given at least 2 Gb of ram memory and a (virtual) disk of 25 Gb of free space. Once the VM has been created, edit it by enabling both the second and the third network adapters. Make sure all the three NICs use paravirtualized network (virtio-net) emulation mode. Then attach Adapter 1 to NAT and Adapter 3 to Host-Only adapter created during HostConstroller configuration. Leave Adapter 2 unattached. Then, download and install your favourite Linux distribution on the virtual machine. Ubuntu 16.04LTS 64bit is suggested, since it is the one we tested in this tutorial. MiddleRouter installation The next step is to install the MiddleRouter software on the fresh linux installed OS. This operation is pretty straight forward thanks to the setup script available here . Thus, reboot the vm, log into a terminal and then issue the following commands. $ cd /tmp $ git clone https://github.com/albertogeniola/TheThing-Sniffer.git $ cd middlerouter $ sudo chmod +x install_script.sh $ sudo -H ./install_script.sh << follow the wizard >> When prompted to install the sniffer agent type “Y”. Optionally, you may want to install the pcap analyzer on the same host. If that is the case, answer “y” when prompted. In our specific case, being a single tier node, we also install the network analyzer on the sniffer. Configure network dapters We now must configure the network connectivity of the sniffer to reflect our network topology as presented in here. In particular: • ETH0 is used as external network connectivity and dynamically acquires the ip address. • ETH1 is used as interface for internal network (TestNet), which guests connect to. In our case we bind it to 192.168.0.1/24. Both DHCP and DNS services will insist on this interface. Let’s edit /etc/network/interface so that it matches the following: # -------------------------------------------------# Loopback device auto lo iface lo inet loopback # WAN/Internet network, where traffic will be routed auto eth0 iface eth0 inet dhcp # Internal NIC, where guests are attached auto eth1 iface eth1 inet static address 192.168.0.1 netmask 255.255.255.0 # -------------------------------------------------- 3.3. Sniffer Installation 29 TheThing Documentation, Release 0.0.1 NOTE: Interface names might be different on new Linux distributions, because of the new predictable interface naming implementation. To avoid any issue we suggest to disable such feature by adding a custom switch in the default grub configuration file. To do so, edit your /etc/default/grub changing the line from: GRUB_CMDLINE_LINUX="" to: GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" and, finally, update grub. $ sudo update-grub At next reboot, interfaces will be given classical _ethX_ names. Configure network services The first step is to adjust the configuration regarding the DHCP service and the DNS relay. Those are provided by dnsmasq. So, let’s edit /etc/dnsmasq.conf, making it look like the following: # -------------------------------------------------interface=eth1 interface=lo # -------------------------------------------------- This configuration tells to dnsmasq to provide only DNS relay on both the loopback interface (lo) and on eth1 (InternalNat). Is does not enable the DHCP server: this task is up to the Openstack networking service. We now need to enable forwarding among interfaces. Edit /etc/sysctl.conf and adjust the forwarding option to match 1. # -------------------------------------------------net.ipv4.ip_forward = 1 # -------------------------------------------------- Then make changes permanent by running $ sudo sysctl -p /etc/sysctl.conf In order to provide external connectivity to guests, the gateway needs to enable the NAT service. This is done by leaveraging iptables. Our objective is to forward packets from internal network, on /dev/eth1, to the external network on /dev/eth0. Therefore we issue the following commands. $ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE $ sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ˓→ACCEPT $ sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT Since IPTABLES rules are not automatically persisted, we need to save them. $ sudo apt-get install iptables-persistent $ sudo iptables-save > /etc/iptables/rules.v4 30 Chapter 3. Two tiers configuration, using Openstack. TheThing Documentation, Release 0.0.1 Check everything is ok Upon successful reboot, make sure the sniffer service is correctly running, by typing: $ sudo reboot <> $ sudo service sniffer status <> $ sudo service analyzer status <> In case it is not running, check the output log for the cause: $ cat /var/log/sniffer.err <> $ cat /var/log/analyzer.err <> Finally, the very last test that confirms the correct installation of the sniffer and its good network configurationFrom the Host controller, make sure the web-service interface is working. Open a browser and type: http://192.168.56.1:8080 Image upload At this point the image of the sniffer is ready. We can proceed with the upload into the Openstack cloud. First, ensure the local virtual machine is powered off, then unmount the disk image. Then, if Horizon service is available, the user can simply proceed with the image upload via the web interface. If command line interface is preferred, the user can do the same by issuing the following command (both in Linux or Windows): glance image-create --name SnifferImage --disk-format VDI --file --progress That command will upload the image located at to the Glance image service, assigning to it the name SnifferImage and showing the progress of the operation. Guest installation This section of the tutorial will guide the user through the preparation of a virtual disk image, which will be used as base disk for the virtual Sandbox in the Openstack cloud. Although most of the preparation steps are common to any kind of Guest, this guide targets the case of single tier configuration for a Windows 7 SP1 32bit client, to be used with an Openstack KVM service. In summary, the user needs to: 1. Creating the base image via Virtualbox or QEMU 2. Installing the Guest Agent 3. Saving and storing the VM image The creation of the base image consists of creating a virtual machine and installing Windows 7 32bit on it. Such task can be performed both on a Windows or a Linux host. In this tutorial we’ll only explain how to do that from within a Windows Host. If needing to prepare the Guest Image from a Linux Host, refer to the good tutorial maintained by 3.4. Guest installation 31 TheThing Documentation, Release 0.0.1 Openstack Documentation . In scu case, the user has to prepare the image via QEMU and then install the Guest Agent binaries before uploading the image to the Glance image service. Creating the base image on Windows This tutorial shows how to create the base Guest Image by using Virtualbox as hypervisor and then injecting the needed drivers via DISM utility. The first step is to create a virtual disk. Since we will then need to inject drivers into the disk, we use a disk format that is compatible with DISM utility, i.e. a VHD disk. So, open a command prompt as an administrator and type the following commands: C:\> cd "%PROGRAMFILES%\Oracle\Virtualbox" C:\> VBoxManage createhd --filename "C:\InstallAnalyzer\Disks\guest_preparation.vhd" ˓→-size 25000 --fromat VHD Now, we want to create a VM and attach the newly created disk to that. C:\> VBoxManage createvm --name "guest_preparation" --ostype "Windows7" --register C:\> VBoxManage storagectl "guest_preparation" --name "SATA Controller" --add sata -˓→controller IntelAHCI C:\> VBoxManage storageattach "guest_preparation" --storagectl "SATA Controller" -˓→port 0 --device 0 --type hdd --medium "C:\InstallAnalyzer\Disks\guest_preparation. ˓→vhd" In order to install Windows 7 on the machine, we will need a virtual CD/DVD drive, that will mount our installation media. Therefore: C:\> VBoxManage storagectl "guest_preparation" --name "IDE Controller" --add ide C:\> VBoxManage storageattach "guest_preparation" --storagectl "IDE Controller" -˓→port 0 --device 0 --type dvddrive --medium /path/to/windows_7_SP1_32.iso Finally, let’s configure some other settings, such as ram, vram and hardware acceleration C:\> VBoxManage C:\> VBoxManage ˓→boot4 none C:\> VBoxManage C:\> VBoxManage modifyvm "guest_preparation" --ioapic on modifyvm "guest_preparation" --boot1 dvd --boot2 disk --boot3 none -modifyvm "guest_preparation" --memory 2048 --vram 128 modifyvm "guest_preparation" --nic1 nat --nictype1 82540EM At this point the virtual machine is ready for the installation of the operating system, so let’s start it right away: C:\> VBoxManage startvm "guest_preparation" The virtual machine will start its boot sequence. The windows installation procedure should begin shortly after the boot up. The user now needs to follow the installation wizard like in a normal machine. Once the installation is completed, the installation wizard will ask the user to choose an username and a password. While the user can type any kind of username, it is strictly mandatory to set no password, so that the auto log-on is activated. Note that this system will not work in case a password is set. System updates At this stage the Guest is capable of surfing the Internet via its own virtual adapter (which is natted behind the network of the host). Therefore, the user can now activate the operating system. If needed, the user can also update the system via Windows Update. Once the updates have been correctly installed, ww strongly advise to disable the 32 Chapter 3. Two tiers configuration, using Openstack. TheThing Documentation, Release 0.0.1 automatic update feature. Doing so, the system will not try to update the OS during the analysis. On the contrary, in case automatic updates are left enabled, there is a chance that they will trigger during the analysis, impacting on performance and potentially biasing sniffed network traffic. Once the system has been correctly activated and updates have been performed, the user can then proceed with the installation of the GuestAgent bootstrapper. Install .NET and dependencies Before installing the GuestAgent Bootstrapper, the system must comply with some software package dependency. In particular, the agents rely on both Visual Studio 2013/2015 Visual C++ redistributable packages and .NET 4.0. Thus, download and install the 32bit version of the .NET framework from the Microsoft website. The correct version of the .NET framework can be found here ‘_ <. Then, do the same for the VC++ 2013 and VC++ 2015 Redistributable Packages. The VC++ 2013 32 bit version is available here . The VC++ 2015 32 bit version is available here . Install the GuestAgent Bootstrapper From within the Virtual Machine, open a browser and dowload the precompiled installation package for the guest agent at this URL . Then, execute the installation of the bootstrapper, by simply double clicking on it. Then, follow the wizard to complete the installation. The installer will take care of downloading the needed python environment, necessary dependencies and will also install the bootstrap autostart task. To double check the bootstrapper installation, reboot the virtual machine. Just after Windows loads up, the bootstrapper program should automatically start, complaining about no response from any sniffer. If that is the case, the bootstrapper is correctly working. Now close the bootstrapper and shut down the virtual machine correctly. User’s defined customization At this stage, the user might apply some specific customization to the image. For instance, he might want to install a new browser or some flash player plugin. He can also install common software usually available on desktop computers, such as Java runtime or Microsoft Office. If planning to analyze evasive binaries, the user should also surf the web and create fake social network accounts, so that cookies are left on the system. In our tutorial we do not perform any of these operations. Disable startup repair on unclean reboot It might happen that a VM reboots unexpectedly. When this happens, Microsoft Windows OS tend to start the Startup Recovery process, which require human actions to be completed. In our case, such recovery process cannot be applied, therefore we need to disable it. To do so, open a command prompt as administrator and type the following command: C:\> bcdedit /set {default} recoveryenabled No 3.4. Guest installation 33 TheThing Documentation, Release 0.0.1 Installing virt-io specific drivers Openstack virtualization driver might differ from the one used by Virtualbox. Most of the times Openstack is configured to uses libvirt and KVM for its guests. In such case, we need to make our guests compatible with KVM module. Thus, we need to manually inject some virtio drivers into the Windows installation. To do so, let’s gracefully shutdown the virtual machine used so far (Start->shutdown from within the Guest). Then, let’s download the VIRTIO signed drivers provided by RedHat at this address . Once done, mount or extract the ISO into a specific location. We will refer to that as VIRTIO_DIR. It is not time to open an elevated command prompt from within the host. Then, we can mount the VHD image via the DISM utility. C:\> Dism /Mount-Image /ImageFile:C:\InstallAnalyzer\Disks\guest_preparation.vhd / ˓→Index:1 /MountDir:C:\test Hence, let’s add the storage, network, USB and PCI drivers with the following commands: C:\> C:\> C:\> C:\> Dism Dism Dism Dism /Image:C:\test /Image:C:\test /Image:C:\test /Image:C:\test /Add-Driver /Add-Driver /Add-Driver /Add-Driver /Driver:VIRTIO_DIR/viostor/w7/x86 /Driver:VIRTIO_DIR/NetKVM/w7/x86 /Driver:VIRTIO_DIR/vioserial/w7/x86 /Driver:VIRTIO_DIR/Balloon/w7/x86 Finally, commit the changes and unmount the image: C:\> Dism /Unmount-Image /MountDir:C:\InstallAnalyzer\Disks\guest_preparation.vhd / ˓→Commit Uploading the image to the Glance Service Once the image has been prepared, we can now safely upload it to the Openstack cloud. This can be done via Horizon web service (if available) or via CLI. We choose the CLI, so we issue the following command: glance image-create --name SandboxImage --disk-format vhd --container-format bare -˓→file C:\InstallAnalyzer\Disks\guest_preparation.vhd --progress This concludes the preparation of the Sandbox image. Configuring the infrastructure TODO Now that a minimum set of components has been put in place, it is time to configure the HostController Agent. Each HostController Agent relies on a specific configuration file, __controller.conf__, located in its installation folder. Therefore, we will find the _controller.conf_ under C:InstallAnalyzerLib 34 Chapter 3. Two tiers configuration, using Openstack. CHAPTER 4 Multitier baremetal configuration. Most of the Malware Sandbox Analysis System freely available nowadays only rely on virtualization for performing the analysis. In fact, any virtualization system facilitates the analysis by implementing snapshots and rollback functions, which are particularly useful to revert the status of the sandbox after every analysis. Moreover, most hypervisors also export a series of APIs which can be used to automate the control the sandboxes. In the end, virtualization also optimizes the hardware usage: a powerful machine can host many small virtual sandboxes. On the other hand, binary analysis performed on bare metal nodes becomes relevant in some circumstances. For instance, bare metal hosts do not suffer from hypervisor detection which is sometimes used by evasive malware. In particular, bare metal analysis can be used as a comparision meter when compared with analysis happening on a virtual system. This represents an interesting use-case that justifies the need of bare-metal sandboxing. TheThing has been developed to support bare metal guests from the very beginning. Moreover, bare metal support has been developed in respect with both scalability and performance. In particular, TheThing takes full advantage of diskless boot together with iSCSI protocol in order to implement centralized image management and fast rollback. Topology description As already mentioned, the topology implemented by bare-metal guests includes at least 3 distinct hardware nodes and at least one smart plug: • HostController: implementing the central DB, the HostController Agent, the iSCSI service and crawlers; • Sniffer: a dedicated machine used as network gateway with sniffing capabilities; • Sandboxes: one or more dedicated machines used as sandboxes for the analysis; • SmartPlug: one for each Sandbox taking part in the topology. At the moment, only TP-Link HS100 is supported. BareMetal/img-details.png 35 TheThing Documentation, Release 0.0.1 From the picture above we identify two distinct network domains. The first one is identified by a private LAN connecting together HostController, Sandboxes and the Sniffer. The second one simply connects the sniffer to the external Internet. Like in the other topologies, the sniffer behaves as a gateway: it routes traffic incoming on ETH1 (LAN) to ETH1(WAN) and vice versa, implementing NAT. Therefore, the sniffer is configured to automatically receive an IP address on its ETH0. On the contrary, it will statically acquire 192.168.0.1 on ETH1. The sniffer will then serve DHCP and DNS requests on ETH1, assigning IPs from 192.168.0.2 up to 192.168.0.128. The HostController resides in the same LAN where Sandboxes are. Therefore, it binds the address 192.168.0.251. Again, since we plan to use a single HostController, the database is hosted locally to the HostController and will bind the loopback address (127.0.0.1). Sandboxes reside in the private LAN and acquire an IP address directly from the sniffer, via DHCP. Each sandbox is connected to the power line through a smart plug, which joins the private LAN too. The sniffer will then be configured to assign specific IP addresses to smart plugs so that each of them will be leased an known IP address via DHCP. Such configuration is necessary in order to create a binding between each smart plug and its connected sandbox. More details are available in the Sniffer configuration section. Notes on the iSCSI diskless boot BareMetal/img.png Bare metal sandboxes take advantages of diskless boot over iSCSI protocol in order to implement clean state rollback and centralized image management. The idea is to use diskless sandboxes, which boot over the network via a PreXecution Environment (PXE). Each sandbox is configured to boot via a permanent USB stick containing a custom version of iPXE. When the sandbox boots up, the iPXE script will perform a special HTTP request against the HostController Service. When this happens, the HostController will allocate a new differencing disk upon a base virtual disk. Then it exposes the newly created disk via an iSCSI target. Then, the web request performed by the sandbox is responded with the iSCSI boot parameter that the specific sandbox needs to apply. As a result, the Sandbox will hook the specified SAN and will boot up. Every IO against the disk will go trough iSCSI and will only affect the differencing disk attached to that iSCSI LUN. When the analysis is over, the HostController will hard reboot the Sandbox using the associated power plug. Therefore the sandbox will boot again in the iPXE environment and the process reiterates. The only difference with the previous step is that the differencing disk will be deleted and created again (such operation takes almost no time). Requirements and limitations Despite other different topologies, the current one necessitates a Windows Server OS to host the HostController service. Such requirement is due to the need of a scriptable iSCSI service, alongside the support of virtual disks and differencing virtual disks (VHD and VHDX). While there is no current plan to support Linux hosts for this specific topology, the implementation of a Linux compatible service would not be impossible. The current implementation also assumes that the guest machines are equal in terms of hardware (same mainboard CPU). Such limitation depends on the usage of a single “base image”, which includes all the basic configuration and 36 Chapter 4. Multitier baremetal configuration. TheThing Documentation, Release 0.0.1 clean state for a single sandbox. In fact, different sandboxes would probably require different drivers and configurations, thus many different base-image for each hardware configuration. Such requirement would vanish the benefits of maintaining a single central disk image. In order to implement machine hard control, we use smart plugs. Unfortunately, the system only supports TP-Link HS100 smart plugs at the current stage. However, it is not diffucult to implement specific software layer driver for different smart plugs. Expert users or pyuthon programmers could adapt the code in order to work with different smart plugs. 4.3. Requirements and limitations 37 TheThing Documentation, Release 0.0.1 38 Chapter 4. Multitier baremetal configuration. CHAPTER 5 Indices and tables • genindex • modindex • search 39