Death to the Hypervisors as we know them

In next few years, hypervisors will be very thin due to Multicore processors and SR-IOV features in the IO devices.

Let us examine the hypervisors role today:

• Scheduling of multiple virtual images on a single or dual processors.  That is, multiple virtual images can be run in one CPU.
• Linux based hypervisors takes advantage of Linux scheduler to schedule different virtual images by running them as user processes of the Linux operating system.  Some hypervisors have modified the Linux kernel scheduler to provide fair and configured share of the CPU across multiple virtual images.
• Virtualization of  IO devices - It is done in following way:
•  Hypervisor owns the physical device.
•  Based on defined criteria (eg:  VLAN in case of Ethernet) it exposes multiple virtual drivers. 
• Provide Paravirtual capability such that VM treat the virtual IO device as its physical device.
• Data movement happens through the hypervisor between physical device and VMs.
• Dedicated IO devices to individual VMs:  Hypervisor does not pay much role in this except for allocation/de-allocation of IO devices to VMs and ensure that two or more VMs don't use the same device.
• Management of Virtual Images:  Hypervisors always come with some management software which does 
• Bring up/down the Virtual Images.
• Allocation of virtual IO devices to different Virtual images.
• Monitoring

Trends in Multicore technology and SR-IOV would reduce the value of hypervisors. Majority of hypervisor functionality is going into the hardware either in Multicore processors or in IO devices.

• Scheduling:  With multiple cores in the chip, there is no need for running multiple VMs in one single processor.  Eight core processors will be common.  Eight different virtual images can be run on these Multicore processors. So scheduling is effectively will disappear from the Hypervisor.  Running multiple images on a single processor does not perform well anyway.  I believe with Multicore processors, only one VM would be run for performance reasons.
• Driver Demultiplexing will also go away from the Hypervisors as SR-IOV is going to be popular. PCI SR-IOV provides multiple virtual functions within one single physical function.  Each virtual function can be assigned to individual VMs and these VMs can get hold of data directly from the hardware.  This not only simplifies the hypervisor dramatically, but also increases the performance by avoiding hypervisor layer in data movement.
• As far as Dedicated IO is concerned, Hypervisors are not doing anything much anyway.

What is it Hypevisor needs to do in future ideal world:   Just the Management. 

Comments?

Disaster Recovery for Network security devices uisng Virtualization

Network security appliance running as a Virtual appliance on a dedicated physical machine has many advantages. One advantage is LAN Bypass functionality.

Another advantage of Virtualization in network security appliances is clearly seen in Disaster recovery and Business continuity scenarios (DR/BC).

Every network component including network security devices msut be considered in Disaster Recovery (DR) planning. There are two key concepts used in DR world - Recovery Point Objective (RPO) and Recovery Time Objective (RTO). They are not self explanatory. If you are confused by those terms, you are not alone. RPO is the amount of time between last backup/replication point and failure point. RTO is the time between failure and recovery. DR efficiency is typically measured by RPO and RTO. Lesser values indicate better DR efficiency.

There are multiple approaches to achieve DR/BC. Selection of an approach is typically evaluated based on business requirements. Products are selected based on the approach chosen by businesses. Some of the approaches, typically used, in physical security appliances are

• Backup and Restore : Security appliances provide facility for administrators to take backup of the configuration. They also provide facility to restore the previously backed configuration. Since configuration change is done by administrator, he/she can take backup immediately, there by making RPO value 0. When there is any disaster, recovery involves getting new appliance and restoring the configuration. RTO is directly depends on the amount of time it takes to get the new appliance. It could be days to a week. It is also possible that the vendor may not be selling those appliances. In those cases recovery involves buying new product and doing fresh configuration. In these cases, it can take even more time for complete recovery.
• Replication: Replication is similar to Backup, but replication typically happens without user intervention. In security appliances, replication approach may not provide any added advantage as configuration is only thing that needs to be kept in safe place. Typically configuration is done by administrator. So, providing manual option of taking backup and putting in a safe place is good enough. Replication require duplicate setup in a safe place and it is expensive. There is no doubt that replication is needed for business Servers, where the content keeps changing with the time and the content is very important. Think of merchant applications where the ordering information is critical. Even in security appliances too, if log messages are important for business, then replication method can be applied to log database.
  

As discussed above, network security appliance recovery might involve procuring new box. Many complications can come in play. Security vendor might have 'End Of Life'ed the product with newer hardware/software. Security vendor company might have closed the shop. Security vendor might have updated the product with newer firmware. In all these cases, the saved configuration might not be useful. It may require procuring newer product and reconfiguration of the product from scratch. This increases the recovery time.

Network security appliance as a VA running on standard hardware eliminates above problems and makes RTO very small.

VMWare/Xen technology allows backing up the complete VA. VA includes OS, security application and configuration. VA can be backed up at the safe place. Disaster recovery involves:

• Procuring generic hardware (PC based hardware normally).
• Getting VA from secure place.
• Bringing up.

Summary: Network security VA provides one DR/BC advantage over physical security appliance.

Citrix XenEnterprise Vs VmWare ESX - Performance comparision

Xen and VmWare are two most popular server virtualization hypervisors. It is always interesting to know how they compare with each other. I came across this video.

Test setup consists of 1U dell PE 1950, Intel Quad Cores with 8Gbytes of RAM and external iSCSI storage DELL PE2850. Two kinds of tests were done - In the first test, Windows terminal servers is run as VM and terminal setup rate was measured. In the second test three more virtual machines are used - Web server, File Server containing home directories of terminal server users and Domain Controller for authenticating the user. It appears that any GET on webserver gets the file from the file server.

It appears that XenEnterprise 3 and ESX 3.0 is used in performance measurements.

According to the video, there is not much performance difference between Xen and ESX in the first test case. In the second test case, ESX performed 3 times better. Test results and analysis is done appear to be pointing disk IO bandwidth.

It seems that CPU and memory utilization by Xen and ESX are almost same. ESX is slightly better. Disk IO seems to the bottleneck on Xen side.

These results are big surprise to me. Xen with Paravirtualization should give equal, if not better performance. It was not clear from the video presentation that whether PV enabled OS was used in the tests.

I suggest to look at the video and make your own conclusions. I will post here if I come across any performance comparisons in future.

Simple way to make Virtual appliances

Ubuntu community released ubuntu JeOS (Just Enough OS) to make virtual appliance. Here is the is the announcement. I am pasting part of announcement here.

"

Ubuntu Server Edition JeOS

Just enough OS for your virtual appliance

Ubuntu Server Edition JeOS (pronounced "Juice") is an efficient variant of our server operating system, configured specifically for virtual appliances. Currently available as a CD-Rom ISO for download, JeOS is a specialized installation of Ubuntu Server Edition with a tuned kernel that only contains the base elements needed to run within a virtualized environment.

Users deploying virtual appliances built on top of JeOS will benefit from:

• better perfomances on the same hardware compared to a full non-optimized OS,
• smaller footprint of the virtual appliance on their valuable disk space,
• fewer updates and therefore less maintenance than a full server installation.

ISVs looking to develop virtual appliances will have a compelling platform in Ubuntu JeOS, an OS optimised for virtualization that greatly reduces the complexity and maintenance overhead normally associated with general purpose operating systems. Ubuntu JeOS Edition has been tuned to take advantage of key performance technologies of the latest virtualization products from VMware. This combination of reduced size and optimized performance ensures that Ubuntu JeOS Edition delivers a highly efficient use of server resources in large virtual deployments.

Without unnecessary drivers, and only the minimal required packages, ISVs can configure their supporting OS exactlyas they want. Ubuntu provides only the security and enhancement updates needed for operating in a virtualised environment with no extraneous overhead.

"

Technical specification of JeOS are given as

"

Tech Specs:

• 151M iso image
• Kernel 2.6.22
• Optimized for VMWare ESX, VMWare Server
• Intel or AMD x86 architecture
• Minimum memory 128Mo
• No graphical environment preloaded as it is aimed at Server virtual appliance
• Working knowledge of linux administration, dpkg and aptitude recommended to start building your own appliance

Download

Start testing it right now, download the image today from http://cdimage.ubuntu.com/jeos/

"

Step by Step instructions to make virtual appliance from JeOS with your application are given here. Pretty well written article indeed. I have not tried it myself and it looks very easy. Some extracts from this article.

"

Installation of JeOS

Installation of JeOS is done the same way you would install any other OS in VMware, but you'll need to consider a few things:

• To reduce the size of JeOS, SCSI drivers have not been included in the kernel shipped with JeOS. Make sure that you instruct VMware to use an IDE drive instead.

• If you plan on shipping a virtual appliance, do not assume that the end-user will know how to extend disk size to fit their needs. So, either plan for a large virtual disk to allow for your appliance to grow, or provide the user with adequate documentation on allocating more space.

• Given that it's much easier to change the amount of RAM allocated to your VM, set the RAM to whatever you think is a safe minimum for your appliace. The user can change this if/when necessary.

• Even though LVM setup is proposed by the installer, it doesn't work at this time-- so if you need LVM, it will need to be set up post-install.

"

Install your application. Sample is given in the article.

Once your application is installed, clean this up as per the article.

Then final step in preparing virtual appliance as per article is :
"

Now it's time to shut down the system and do the final cleanup. Once it's shut down, go to your virtual machine settings in the VMware management console and select the virtual disk. Defragment it, which should reclaim all of the space you've freed, which will make the appliance smaller. Your virtual appliance is ready to ship! "

Nice work indeed from Ubuntu community.

Security among VMs - A thought.

Some physical deployments provide port based security. In physical world, it is implemented by replacing typical traditional L2 switch with L2 security switch. Each port is secured in the sense that each port belongs to a security zone. Security switches are typically provide firewall, IPS and in some cases provide Anti Virus capabilities. Policies can be setup among zones.

In virtual world, you deploy multiple VMs. If all VMs belong to one particular zone, then external security switch is good enough. But if VMs belong to multiple security zones, then packets may go among VMs without packets being seen outside of the box having VMs. If one VM is compromised, it gives chance for attackers to snoop the packets on its virtual adapter and might even become launching pad to compromise other machines.

In addition to security problems, there is no visibility of traffic patterns among VMs.

Like in physical world, there are virtual switches within Virtualized environment. It is logical to replace virtual switches with virtual security switches. Unfortunately, it is not as simple as it in physical world. Virtual switches are part of Hypervisor and there are only two possibilities:

• Enhance Host OS to add security functions. In Xen, it is simple. It uses Linux bridge to implement virtual switches. It can be enhanced with security functions.
• Advantages
• There is no additional logical connectivity required.
• No additional virtual switches.
  
• Disadvantages
• All security functions run in the context of hypervisor. Performance is limited by number of CPUs allocated for hypervisor machine.
• Any problem in security function brings down the entire physical system.
• Image Updates could be a problem
• It is recommended that Host OS has very minimal software to ensure that it is not vulnerable.
  
• Make some VMs as security switches. That is, don't replace virtual switches. Work along with virtual switches. VMs based on zone they belong to connect via these security VMs.
  
• Advantages:
• Secuirty VM can do many functions.
• Security VM can be restarted at any time if there is any problem.
• Security VM can be updated like any VM.
• Disadvantages:
• Additional connectivity is required to redirect the traffic via these security VMs. Hence, more virtual switches may be required.

Security VMs must provide following functions to provide security services for server VMs.

• Security VMs must support Layer 2 operation (transparent mode)
  
• Security VMs must support multiple zones.
• Security VMs provide very good log and reporting functions.

LAN Bypass using Xen/Vmware kind of virtualization technology

Many inline security infrastructure products such as IPS, Web application firewalls, Anti Virus and Anti Spam gateways make use of a feature called 'LAN Bypass'. This feature is implemented in hardware using watch dog timer functionality. If the software fails to reset the watch dog timer, it interconnects the LAN ports there by bypassing any software functions running on the device. Basically, if software running on device fails to run due to hangs and crash, the connectivity does not suffer. Many businesses prefer bypassing security functions over non availability of network.

LAN bypass hardware is expensive and typically works with two Ethernet ports. Xen/Vmware based virtualization technology eliminates the need for having this expensive hardware by running security software as VM and having some kind of monitor in host OS (domain 0) which monitors the health of security VM. When the VM does not respond, the host can create software bridge among all physical Ethernet ports.

As a appliance vendor, one should follow these steps to make a security appliance: None of these steps are required to be done by end user.

• Use Xen or Vmware : My preference is Xen as it has paravirtualized Ethernet drivers, which is fast compared to full virtualization.
  
• Run the security software as VM.
• Have as many virtual switches as number of physical Ethernet ports.
• Create as many virtual adapters on VM as number of physical ports.
• Connect virtual adapter of VM and physical adapter in separate virtual switches (bridges).
• Security software anyway creates a bridge among virtual adapters to work in transparent mode.
• Have monitor software running in host operating system and have some monitor counterpart in security VM to monitor the applications in VM.
• When host monitor detects the problem in security VM, it can try restarting it. If it does not restarts, then the host monitor can remove all virtual switches and create a bridge connecting all physical Ethernet ports.

With this, we can eliminate the need for LAN Bypass hardware.

What are the limitations with above approach?

• Performance of VM is typically 20% less than running software in non virtual environment.
  
• Though it solves problems of software crashes and hangs, this is not solution for hardware problems.
  
• This only works for transparent security appliances.
  

As I understand from several administrators, around 70% of cases, problems are observed in the software application than the hardware. Hence this solution should work fine and I feel that many security vendors would be doing this in future.

By the way, at Intoto we do this in IntruPro-IPS and UTM products.

Virtualization in Network security - Present and future

Service providers are increasingly providing security services at their edge for their customers. Firewall, IPsec VPN, IPS and P2p rate throttling are some of the security services provided by service providers. SPs are providing this functionality using virtualization capability provided by many security appliance vendors. Virtualization provided is at the level of instantiating configuration and run time states. That is, for each customer, there is separate routing table, firewall session table, configuration tables, Ipsec SA table etc.. But the code and data segments are shared. With role based management, different customer accounts are handled by different admin personnel of service provider.

Here is the question. Do Xen/VmWare/KVM based virtualization technology replace current security virtualization? Note that in this virtualization, there would be as many VM instances as number of virtual instances (customers) required. Each virtual instances has its own OS and security application in addition to configuration and run time states.

Advantages of Xen/VmWare/KVM based virtualization:

• It provides isolation : If there is any software problem, only current instance is effected. In case of traditional virtualization provided by security vendors, any software problem brings down the entire system.
• It provides mechanism to assign CPU for each virtual system.
• It provides mechanism to control memory used by each virtual system.
• It provides mechanism to control bandwidth used by each virtual system.
• Basically, it is true virtualization.

Disadvantages of this type of virtualization over traditional virtualization:

• It requires as many virtual machines as number of customers.
  
• Memory requirements are too high
  
• CPU utilization is very high.
• System Performance would be way less
• High cost.

In summary, if the number of virtual instances needed are more than few dozen, then traditional security virtualization is the choice for service providers. Traditional virtualization provides optimal cost and performance over Xen/VmWare based virtualization. I have a feeling that Vmware/Xen based virtualization becomes popular in Enterprise environments where it can replace multiple security appliances. But, in case of service providers market, traditional virtualization would be used for some more years to come.