MindShaRE: Analysis of VMware Workstation and ESXi Using Debug Symbols from Flings
January 07, 2021 | Reno RobertThe availability of debug symbols greatly assists a researcher in understanding a software architecture, performing live debugging or static analysis. An end-to-end black box analysis of a closed source hypervisor is a time-consuming process. Microsoft has made this work easier by publishing debug symbols for most of the Hyper-V components. However, there is still no debug info available for VMware Workstation (WS) or ESXi. Considering this, the Project Zero blog posts on Adobe Reader symbols greatly inspired me to carry out a similar analysis for VMware.
This blog details how VMware Flings can be useful in obtaining some of the symbol information stripped from VMware WS or ESXi. Flings are free, short-term projects released without support by VMware mostly as an enhancement to some of the existing products. The two Flings of interest for this analysis are VNC Server and VNC Client (archived link) and ESXi ARM Edition, the former having DWARF debug information for SVGA device implementation and the latter having function names of many other components of the vmx worker process.
SVGA symbols in VNC Server and VNC Client Fling
VNC Server and VNC Client Fling released in February 2016. It is a cross-platform VNC implementation with code leveraged from VMware Workstation. The Fling has VNC server and client binaries for all major operating systems – Windows, Linux, and Mac. The Windows binary is not accompanied by a corresponding PDB debug file, but the Mac and Linux binaries have the embedded debug information in them.
In order to understand the code shared by VNC Fling and WS, I decided to compare the binaries having debug information against vmware-vmx. For the comparison to be effective, it is best to choose a WS version released around the same timeline as that of the Fling. The idea behind this is to increase the likelihood of having a similar code base as well as build environment. Since the Fling was released on February 25, 2016, the following list of WS and Fusion releases seemed ideal for analysis:
|
WS version |
Fusion version |
Release date |
|
12.0.1 |
8.0.2 |
29 Oct 2015 |
|
12.1.0 |
8.1.0 |
08 Dec 2015 |
|
12.1.1 |
8.1.1 |
21 Apr 2016 |
|
12.5.0 |
8.5.0 |
08 Sep 2016 |
IDA’s F.L.I.R.T. (version 7.5) was my first choice for performing signature matching between executables. To generate the pattern file, I modified the IDB2PAT script published by FireEye to support 64-bit mode RIP relative addressing. In RIP relative addressing, 32-bit signed displacements (+/- 2GB) are used to reference code or data. These 4 bytes of displacement are treated as variable bytes during signature generation. Below is essential part of the patch applied to find_ref_loc() function:
Three binaries are under consideration: mksVNCServer for Linux, mksVNCClient for Linux, and mksVNCServer for macOS. The mksVNCServer binary returned the best results during signature matching and also had a superset of functions available in mksVNCClient. Moreover, the availability of DWARF debug information provides rich details regarding source code, structure declarations, function inlining and other optimizations. The WS version 12.1.0 released couple of months before the Fling turned out to be the most promising one. Here is the summary of FLIRT signature matching:
|
WS version |
FLIRT hits |
|
12.0.1 |
40041 |
|
12.1.0 |
43283 |
|
12.1.1 |
43231 |
|
12.5.0 |
42998 |
After narrowing down the version of interest, I relied on the symbol porting feature in BinDiff to import the function and variable names from mksVNCServer.
Figure 1 - vmware-vmx after porting symbols using BinDiff
Anyone who has previously looked into VMware’s SVGA attack surface will recognize where these functions originate. If you are new to this, Wandering through the Shady Corners of VMware Workstation/Fusion and Straight outta VMware [PDF] are excellent references to start with.
What more essential information can be ported to WS from the Fling? The type information. IDA can export typeinfo as C header from mksVNCServer, which can be then loaded in vmware-vmx. There are some caveats in this approach. The exported C header needs a few fixes, like renaming variables with C++ keywords (new, template, class and private), rewriting of certain variadic function definitions and so forth to be successfully parsed by IDA. Once the typeinfo is imported, function prototypes can be ported, too. To accomplish this, first extract each prototype from mksVNCServer as a key value pair of function_name:function_type, then iterate through the extracted type information and apply it to vmware-vmx having symbols.
Figure 2 - vmware-vmx after porting function prototypes
At this point, it is convenient to analyze vmware-vmx and mksVNCServer side-by-side. Moreover, there are couple other of the dwarves tools [PDF] that I find useful in static analysis of available DWARF information: pahole and pfunct.
pahole was originally developed to inspect alignment holes in structures using DWARF debug information. Considering that mksVNCServer is compiled with debug information, pahole provides a way to analyze data structures, their size, and their usage information in the source file. It is possible to either query a particular structure by name using -C or dump everything including anonymous structures using -a and then grep for information.
Similarly, pfunct provides great insights about functions. This is especially useful in recovering details regarding inlined function definitions and local variable optimizations. Consider the below case of StateFFP_TranslateSM4(), where pfunct allows us to statically map a code block from 0xe8610 - 0xe864c (60 bytes) to AddOutputDecl().
Figure 3 - Block of inlined code belonging to AddOutputDecl()
Now what? Can we put together all this information for a better understanding of past vulnerabilities or research? Yes - the first thing that comes to mind is shader translation. In fact, StateFFP_TranslateSM4() analyzed using pfunct is one of those vulnerable functions.
WS 12.5.5 released in March 2017 fixed some vulnerabilities in shader translation. We are not going to dive into the details of the bugs again. Wandering through the Shady Corners of VMware Workstation/Fusion provides a very detailed walkthrough of shader attack surface, the vulnerabilities found in opcode handling, and the proof-of-concepts to trigger them. I was more curious to check what the vulnerable code looks like after porting all the symbols and type information to WS 12.1.0.
Figure 4 - Vulnerabilities in StateFFP_TranslateSM4()
Clearly, the decompiled code has more information than previously available from vmware-vmx-debug. This being the tip of iceberg, a lot more shader bugs got fixed over the years. In the current state of GPU virtualization, shaders are probably the JavaScript of hypervisors. Given the reality of this complex and ever-growing attack surface, VMware has now introduced a sandboxed graphics renderer as a security enhancement.
At this point, one might wonder if this debug information from the Fling is still relevant, given it was released 5 years back? I strongly believe that it is. Despite all the changes due to bug fixes and feature additions, the core design and APIs have not changed drastically. Also, this can be a great addition to the paper Straight outta VMware [PDF] for anyone interested in analyzing VMware’s SVGA implementation.
Symbols in ESXi ARM edition
The next Fling of choice is the more recent ESXi ARM edition released on October 6, 2020. Since ESXi ARM is bound to share a lot of code with ESXi x86, this is an easy pick for analysis when compared to the VNC Fling. But how do we set up ESXi ARM? The easily available options are installation on a Raspberry Pi or emulation with QEMU. However, a more convenient option for static analysis is to just extract the vmx executable from the ISO image. To get this working, install ESXi x86 7.0 (available for free download as a guest VM) then extract the ESXi ARM vmx executable using the vmtar utility available in ESXi x86. Note that the vmx mentioned in this section has nothing to do with Virtual Machine Extension (VMX) but refers to the VMware worker process executable.
After successfully extracting the vmx aarch64 ELF, things did not go as I hoped. The binary was completely stripped of debug information. However, the dynamic section had a lot more entries than one would generally see in an executable. A quick line count of readelf -s returned a number as high as ~25k. Below is a rough comparison of the number of entries in the dynamic symbol table of ESXi for x86 and ARM (Fling version 1.1):
|
Executable |
Entries in x86 |
Entries in ARM |
|
vmx |
820 |
25200 |
|
vmx-debug |
845 |
25434 |
|
vmx-stats |
822 |
30496 |
It looks like the aarch64 executables are compiled with the linker flag --export-dynamic/-E, which has exported all non-static functions and global variables into the dynamic symbol table. Let’s do a quick grep for a known attack surface, say the virtual XHCI USB controller recently patched by VMware.
The results are surprisingly good. In case of a virtual device, these function names can help us identify a code block emulating a certain hardware specification. There are also symbols available for many other low-level interfaces such as the PhysMem family of functions mentioned in the patent for Transparent Page Sharing [TPS]. Even if a virtual device has minimal dynamic symbols (UHCI, EHCI, etc.), the presence of symbols for other low-level APIs makes it easier to understand them.
Once the initial analysis is over, we can port the symbols from ESXi ARM to ESXi x86. Since BinDiff has the ability to compare executables from two different CPU architectures, this is a very realistic use case to try out this feature.
Figure 5 - Symbols ported to ESXi x86 from ESXi ARM using BinDiff
In fact, the results turned out to be very satisfying. We never had so many symbols for ESXi before, and this provides a good start for side-by-side analysis. Moreover, with the availability of symbols for vmx executable, one can understand its communication with Virtual Machine Monitor (VMM) much better.
In regards to the VMM, a couple of observations have already been made. An embedded VMM ELF in the vmx executable is loaded by a kernel driver (Hypervisor Framework [PDF]). Also, the embedded ELF has symbols (Wandering through the Shady Corners of VMware Workstation/Fusion). Dumping the VMM is a two-stage process: a loader vmmblob ELF followed by another embedded vmmmods ELF.
Figure 6 - Embedded vmmblob loader code
Figure 7 - Embedded vmmmods VMM code
These symbols are not only available in ESXi ARM edition but across ESXi x86 and WS. What I really wanted to check was how much of the code from VMM overlaps with that of vmx. Can symbols in VMM be ported to vmx? Since the ARM edition has symbols for both vmx and VMM, it is an ideal choice to perform this comparison. We are particularly interested in BinDiff matches based on “name hash matching”. Though around 100 entries were found, only few had high similarity. Most other functions differ in their implementations, making it hard to port the symbols from the VMM.
Figure 8 - PhysMem_Get - vmx (left) vs VMM (right)
Porting the symbols from VMM is not a concern anyway, since the vmx executable in ESXi ARM already has them. Evidently, the time spent on searching and matching the Flings have provided us with useful debug information beyond vmx-debug or the VMM. It also demonstrates how some less significant pieces of software can carry significant amount of information about production code.
Conclusion
Going forward, Flings can be a great addition for anyone analyzing WS/ESXi or other VMware products. They certainly proved to be helpful in obtaining some of the symbol information stripped from VMware WS or ESXi. Understanding these debug symbols is key to understanding how the program works and where vulnerabilities may be found. Hopefully, Flings will help your research into VMware vulnerabilities as well.
You can find me on Twitter @RenoRobertr, and follow the team for the latest in exploit techniques and security patches.