Альтернативный интерфейс жесткого диска Windows VM через Proxmos vE 5 для VirtIO

Guide to Slipstream VirtIO Drivers into Windows ISO’s

Introduction

This guide will help manually add VirtIO drivers to Windows ISO’s so that the drivers are automatically installed and available to a Windows VM on Proxmox. This way the VirtIO ISO will not need to be attached to a virtual machine on Proxmox and there will be no need to manually load the drivers 1-by-1 in Windows Setup at install time. While these ISO’s should work with any KVM based virtualization, I am only testing them using a Proxmox host.

By slipstreaming these drivers direct into an ISO, it drastically reduces the work required to setup new Windows based VM’s by removing the need to manually install these drivers for each machine.

Tested Windows Versions

Just because I have not tested a version of Windows does not mean the methods outlined here will not work. They may work as is, with modification or not at all. Feel free to try other versions as needed.

Disclaimers

I also am not covering how to inject updates into Windows, though it is possible using similar methods. Should you want to also inject Windows updates I suggest you research the topic and incorporate the additional setup and commands into your workflow.

What are VirtIO Drivers?

In brief, VirtIO drivers are drivers designed to allow Windows VM’s to perform better when used with KVM based virtualization, like Proxmox currently uses. This includes but is not limited to: display drivers, storage drivers, networking drivers, etc. The VirtIO ISO also contains the qemu guest agent which allows the KVM host (Proxmox) to have better control over a guest OS, in this case Windows. This allows functions like properly shutting down a Windows VM from Proxmox’s controls. The ISO also contains the Balloon service files as well as the driver, which when installed allow a Windows guest to work with the memory ballooning feature of KVM/Proxmox. Generally speaking, the drivers make the VM run more smoothly, with more control and less issues.

Resources

Tips

• Keep original copies of your ISO’s in a safe place, especially if they are not simply downloads from Microsoft’s site (like a retail ISO).
• Prior to doing any of these steps, ensure your original ISO works for installation.
• You may consider keeping copies of the modified install.wim and boot.wim in case you need to inject additional drivers or updates to them at another time.
• Read over the entire guide before attempting the steps. Understand what each step does and why you are doing it.

Preparation

1. Setup a Windows based machine to prepare the ISO’s on. This can be another VM or a physical machine. I recommend using either Windows 10 Pro or Windows Server 2019 to do this on. Install Windows, update it, etc. Make sure to have enough space to handle your ISO’s + spare space. I would generally recommend ~25GB free space per ISO plus at least 32GB for Windows and programs to install.
2. Install Windows ADK. The only required selection from the checklist of install options is «Development Tools». If PXE booting Windows images you may also choose «Windows Preinstall Environment (Windows PE)». This will give us access to commands like dism and commands in PowerShell needed for working with the ISO’s.
3. Download the VirtIO ISO and have whatever Windows ISO(s) you plan to work with on this machine.
4. In file explorer, change the settings to show hidden files and folders and to display file extensions. Generally on the View Tab, 2 check boxes.

1. iso-win# — This directory will contain all the files from the ISO of a specific version of Windows
2. iso-virtio — This directory will contain all the files from the VirtIO ISO
3. drv-win# — This directory will contain only the drivers needed for a specific version of Windows
4. wim-win# — This directory will contain the current install.esd, install.wim and/or boot.wim to be modified
5. mount — The directory we will mount the .wim files to modify them.

For entries with win# replace «#» with whatever version/abbreviation you like. Ex: iso-win10 or drv-win2k19.

After making the above directories, we need to copy files from the ISO’s to the proper locations.

1. Mount the Windows ISO.
2. Select all the files and folders on the Windows ISO and copy them. Paste them to the iso-win# directory.
3. Unmount the Windows ISO and Mount the VirtIO ISO.
4. Select all the files and folders on the VirtIO ISO and copy them. Paste them to the iso-virtio directory.
5. Unmount the VirtIO ISO.

The VirtIO contents is explained here. The gist is there is a folder for each type of driver (display, network, etc.). Within each folder are multiple folders, one for each version of Windows. Within each of those folders is a folder for the CPU architecture like amd64, ARM64 or x86. We only need to inject the drivers that match the same version of Windows being used with the same architecture planned (generally speaking only «amd64»).

I copied all the files/folders in iso-virtio to drv-win#. I then went into each folder and deleted any folder representing a version of windows other than the one matching the drv-win# folder, the version of Windows we are currently working on injecting drivers in. Ex: if working with a Windows 10 ISO, in the NetKVM directory select all folder, ctrl+click the w10 folder to de-select it and then delete. This will leave only the w10 folder in the NetKVM folder. Repeat this for every folder under drv-win#.

While neither file in guest-agent is a driver, I would keep at least the version matching the architecture (typically the x86_64.msi file). It will prove helpful post install.

I then went back to the drv-win# folder.

1. Click in the search bar at the top right of the file explorer window.
2. From the «Search» tab, select «Kind» and select «folder». This inserts «kind:=folder» in the search box.
3. Put a single space after that entry and then type in «ARM» and hit enter or wait.
4. Once complete, select all the results and delete them. This should delete all of the «ARM» subfolders in drv-win# as we don’t need those drivers.
5. Repeat steps 2-3 but instead of «ARM», type in «x86». Only do this if you do not plan to install Windows as 32bit/x86 (in some cases it’s not supported anyway, like in Windows Server). Then repeat step 4 and delete all the resulting folders. This deletes all the «x86» subfolders from the drv-win# folder.

This should leave just the folders/drivers needed for the version of Windows we are injecting the drivers into. This is important to help reduce the size of an image and to avoid having incompatible drivers loaded/installed.

Optional but Recommended
I recommend taking the drv-win# folder and copying it to the iso-win# folder. Doing so will help make post install steps easier.

Now we need to copy/move the install.esd or install.wim (depends on Windows version) file from the Windows ISO to the wim-win# directory. We also need to copy the boot.wim file.

1. Copy install.esd or install.wim from C:\Users\myuser\Downloads\iso-win#\sources to C:\Users\myuser\Downloads\wim-win#
2. Copy boot.wim from C:\Users\myuser\Downloads\iso-win#\sources to C:\Users\myuser\Downloads\wim-win#

This should leave us with:

• Contents of the Windows ISO in iso-win#
• Contents of the VirtIO ISO in iso-virtio
• Only the folders/files of the drivers needed for the specific version of Windows being worked on in drv-win#
• 2 files in wim-win#, install.wim or install.esd and boot.wim
• An empty mount folder.
• Windows ADK installed

Slipstream VirtIO Drivers into Windows ISO

This process is repetitious. .wim files (and .esd) often have multiple indexes. So when you boot from a Windows 10 ISO for example, and you have options to install «Windows 10 Home», «Windows 10 Pro», etc. each is a different index in the same install.wim/install.esd on the same ISO. This means we need to inject the driver into each option we intend to keep/use. We also need to inject the driver into the boot.wim, which also has indexes (generally 1 for Win PE and 2 for the Windows Setup). Unless PXE booting and using the Win PE portion, we only need to inject into the Windows Setup portion for the boot.wim.

To cut down on this, I recommend removing any index (option) that you do not intend to use ever. For example if you will never be using «Windows 10 Education» from a Windows 10 ISO, better to remove it then to spend time injecting drivers to it. We will cover how to do this soon.

1. Getting Wim Info

This command will display information about a given image file (.wim or .esd). Primarily we want to know how many indexes there are, and which index(s) we will use and which we won’t.

2. (Situational) Remove Unused Indexes

This command will remove the designated index from the image. If you are unsure, do NOT delete an index. Only use this command to delete options from an ISO you are sure you will not use. I also highly recommend to re-run the command from step 1 above after removing each index. This is because options are re-indexed after each removal. IE: if index 3 was «Windows 10 Home» and index 4 was «Windows 10 Pro», after removing index 3, «Windows 10 Pro» now becomes index 3 instead of 4. AKA: Dont assume the indexes are the same after removing another.

3. (Situational) Convert .esd to .wim

This step can also take a relatively long time (like 5-15 mins on a modern machine). Just let it run to completion.

Per Index Steps

Perform the steps 4-7 in order for a single image/index before repeating them for the next.

Replace install.wim with boot.wim when injecting drivers into the setup image.

4. Mount an Image

In this step we mount an image/index to inject the drivers into. This is similar in concept to mounting an ISO except that we are mounting a specific index within the install.wim or boot.wim image.

5. Inject/Slipstream Drivers

In the above commands, -Recurse has the command look through all the folders and subfolders of our drv-win# directory for drivers. The -ForceUnsigned switch removes the requirement for drivers to be digitally signed to be installed which may be required for some drivers in the VirtIO collection.

6. Verify Drivers Have Been Added

7. Save Changes and Un-mount the Image

Once the drivers have been injected, we need to save the changes to the image while we un-mount it.

Final One Time Steps (per ISO)

The proceeding step is meant to be done after steps 4-7 have been repeated for each file/index required. Be sure that you have done steps 4-7 for the setup index of boot.wim and at least one index of the install.wim.

8. Copy Updated .wim Files Back to ISO Folder

We need to take the modified install.wim and boot.wim and place them back in the ISO file/folder structure. These steps can be done in File Explorer.

1. Copy install.wim from the C:\Users\myuser\Downloads\wim-win# directory.
2. Paste install.wim to C:\Users\myuser\Downloads\iso-win#\sources\ directory.
3. Copy boot.wim from the C:\Users\myuser\Downloads\wim-win# directory.
4. Paste boot.wim to C:\Users\myuser\Downloads\iso-win#\sources\ directory.

9. Rebuild Windows ISO

Our last step takes everything we have done and rebuilds a new ISO using the updated images with the drivers contained within them. The resulting ISO will be what we want to install Windows VM’s from going forward.

This particular step doesn’t have ADK/PS variations, it’s just a single command. As far as I am aware, the same command can be run from either ADK (cmd) or PS.

Replace new-iso-name.iso with the desired name of the new Windows ISO. I recommend this not be the same as the original ISO. The above command also assumes UEFI bootable media, I am personally not concerned with BIOS booting Windows.

This will allow for ISO images larger than 4.7GB single layer DVD’s and beyond the approx. 4GB install.wim limit for an ISO.

Post Install Using the New ISO

After installing Windows in a Proxmox VM using your new ISO, there are still a few steps that you need to take to get the most out of it. Earlier in the guide, I recommended copying our drv-win# folder to the iso-win# folder. I am going to assume in the steps below this was done. If not, you will need to mount the VirtIO ISO/disc for the current Windows VM to finish the post install steps.

1. Install QEMU Guest Agent

The Windows ISO (and the VirtIO ISO) which should be still mounted post Windows Install as a CD/DVD drive/disc contains a folder drv-win#\guest-agent which should contain at least a single .msi file (if not 2 files) named qemu-ga-i386.msi or qemu-ga-x86_64.msi (for 32bit/64bit Windows OS’s respectively). This is similar to the VMWare/ESXi guest agent allowing the host (in this case Proxmox) to have more accurate information of and better control over its guest VM’s.

To install this on the Windows guest VM, in File Explorer:

1. Navigate to the Windows CD/DVD (the ISO mounted within the VM that we installed from), let’s assume D:\ and go to the guest-agent folder D:\drv-win#\guest-agent.
2. Double click on the appropriate .msi for your architecture of Windows (i386 = 32bit and x86_64 = 64bit).

A command prompt window will open briefly then vanish, that’s it. It should be installed.

2. Enable the Balloon Service

The Balloon driver and service allow KVM guests like our Windows VM to work with the memory ballooning feature of KVM. While our previous steps have added the driver to the ISO and thus our install, we still need to setup and activate the service for ballooning in our Windows VM.

To install/activate the ballooning service:

1. Create a new folder C:\Program Files\Balloon.
2. Navigate to the Windows CD/DVD (the ISO mounted within the VM that we installed from), let’s assume D:\ and go to the proper balloon directory D:\drv-win#\Balloon\w10\amd64. Replace w10 with the folder for your version of Windows and amd64 for your architecture of Windows.
3. Copy all the files in the proper sub directory of Balloon\w10\amd64\*. There are roughly 6 files in this folder, one of which should be blnsvr.exe.
4. Paste these files into the created directory from step 1 at C:\Program Files\Balloon.
5. Open cmd (command prompt) as an Admin from C:\Program Files\Balloon or cd to C:\Program Files\Balloon.
6. Run the command blnsvr.exe -i. This should give a few lines of output in cmd about starting the service.

3. Reboot the VM

To finalize the post install configuration, reboot the Windows VM. To verify that the guest agent is working, you can look at the Summary of the Windows VM in Proxmox. Specifically under «Bootdisk size» should be the «IPs» entry listing the IP addresses of the VM with a button that says «More». The «More» button displays further guest agent network info.

You can confirm that the Balloon service is running from the Windows guest using the PS command: get-service -name ballo*. It should return a result stating the Balloon Service is running.

Conclusion

This guide should provide the means to:

• Create a Windows ISO (or ISO’s) with VirtIO drivers baked in.
• Implement post install steps for Windows VM’s in Proxmox to take advantage of qemu guest agent and the balloon driver/service.
• Reproduce the process of injecting the VirtIO drivers into Windows media using ADK (cmd) or PowerShell.
• Eliminate the need to manually install VirtIO drivers during installation or post install for Windows VM’s.

Hopefully you find this guide helpful. Please open an issue should you have any corrections, updates, questions or input on this guide. Thank you.

Guide to Slipstream VirtIO Drivers into Windows ISO’s

Introduction

This guide will help manually add VirtIO drivers to Windows ISO’s so that the drivers are automatically installed and available to a Windows VM on Proxmox. This way the VirtIO ISO will not need to be attached to a virtual machine on Proxmox and there will be no need to manually load the drivers 1-by-1 in Windows Setup at install time. While these ISO’s should work with any KVM based virtualization, I am only testing them using a Proxmox host.

By slipstreaming these drivers direct into an ISO, it drastically reduces the work required to setup new Windows based VM’s by removing the need to manually install these drivers for each machine.

Tested Windows Versions

Just because I have not tested a version of Windows does not mean the methods outlined here will not work. They may work as is, with modification or not at all. Feel free to try other versions as needed.

Disclaimers

I also am not covering how to inject updates into Windows, though it is possible using similar methods. Should you want to also inject Windows updates I suggest you research the topic and incorporate the additional setup and commands into your workflow.

What are VirtIO Drivers?

In brief, VirtIO drivers are drivers designed to allow Windows VM’s to perform better when used with KVM based virtualization, like Proxmox currently uses. This includes but is not limited to: display drivers, storage drivers, networking drivers, etc. The VirtIO ISO also contains the qemu guest agent which allows the KVM host (Proxmox) to have better control over a guest OS, in this case Windows. This allows functions like properly shutting down a Windows VM from Proxmox’s controls. The ISO also contains the Balloon service files as well as the driver, which when installed allow a Windows guest to work with the memory ballooning feature of KVM/Proxmox. Generally speaking, the drivers make the VM run more smoothly, with more control and less issues.

Resources

Tips

• Keep original copies of your ISO’s in a safe place, especially if they are not simply downloads from Microsoft’s site (like a retail ISO).
• Prior to doing any of these steps, ensure your original ISO works for installation.
• You may consider keeping copies of the modified install.wim and boot.wim in case you need to inject additional drivers or updates to them at another time.
• Read over the entire guide before attempting the steps. Understand what each step does and why you are doing it.

Preparation

1. Setup a Windows based machine to prepare the ISO’s on. This can be another VM or a physical machine. I recommend using either Windows 10 Pro or Windows Server 2019 to do this on. Install Windows, update it, etc. Make sure to have enough space to handle your ISO’s + spare space. I would generally recommend ~25GB free space per ISO plus at least 32GB for Windows and programs to install.
2. Install Windows ADK. The only required selection from the checklist of install options is «Development Tools». If PXE booting Windows images you may also choose «Windows Preinstall Environment (Windows PE)». This will give us access to commands like dism and commands in PowerShell needed for working with the ISO’s.
3. Download the VirtIO ISO and have whatever Windows ISO(s) you plan to work with on this machine.
4. In file explorer, change the settings to show hidden files and folders and to display file extensions. Generally on the View Tab, 2 check boxes.

1. iso-win# — This directory will contain all the files from the ISO of a specific version of Windows
2. iso-virtio — This directory will contain all the files from the VirtIO ISO
3. drv-win# — This directory will contain only the drivers needed for a specific version of Windows
4. wim-win# — This directory will contain the current install.esd, install.wim and/or boot.wim to be modified
5. mount — The directory we will mount the .wim files to modify them.

For entries with win# replace «#» with whatever version/abbreviation you like. Ex: iso-win10 or drv-win2k19.

After making the above directories, we need to copy files from the ISO’s to the proper locations.

1. Mount the Windows ISO.
2. Select all the files and folders on the Windows ISO and copy them. Paste them to the iso-win# directory.
3. Unmount the Windows ISO and Mount the VirtIO ISO.
4. Select all the files and folders on the VirtIO ISO and copy them. Paste them to the iso-virtio directory.
5. Unmount the VirtIO ISO.

The VirtIO contents is explained here. The gist is there is a folder for each type of driver (display, network, etc.). Within each folder are multiple folders, one for each version of Windows. Within each of those folders is a folder for the CPU architecture like amd64, ARM64 or x86. We only need to inject the drivers that match the same version of Windows being used with the same architecture planned (generally speaking only «amd64»).

I copied all the files/folders in iso-virtio to drv-win#. I then went into each folder and deleted any folder representing a version of windows other than the one matching the drv-win# folder, the version of Windows we are currently working on injecting drivers in. Ex: if working with a Windows 10 ISO, in the NetKVM directory select all folder, ctrl+click the w10 folder to de-select it and then delete. This will leave only the w10 folder in the NetKVM folder. Repeat this for every folder under drv-win#.

While neither file in guest-agent is a driver, I would keep at least the version matching the architecture (typically the x86_64.msi file). It will prove helpful post install.

I then went back to the drv-win# folder.

1. Click in the search bar at the top right of the file explorer window.
2. From the «Search» tab, select «Kind» and select «folder». This inserts «kind:=folder» in the search box.
3. Put a single space after that entry and then type in «ARM» and hit enter or wait.
4. Once complete, select all the results and delete them. This should delete all of the «ARM» subfolders in drv-win# as we don’t need those drivers.
5. Repeat steps 2-3 but instead of «ARM», type in «x86». Only do this if you do not plan to install Windows as 32bit/x86 (in some cases it’s not supported anyway, like in Windows Server). Then repeat step 4 and delete all the resulting folders. This deletes all the «x86» subfolders from the drv-win# folder.

This should leave just the folders/drivers needed for the version of Windows we are injecting the drivers into. This is important to help reduce the size of an image and to avoid having incompatible drivers loaded/installed.

Optional but Recommended
I recommend taking the drv-win# folder and copying it to the iso-win# folder. Doing so will help make post install steps easier.

Now we need to copy/move the install.esd or install.wim (depends on Windows version) file from the Windows ISO to the wim-win# directory. We also need to copy the boot.wim file.

1. Copy install.esd or install.wim from C:\Users\myuser\Downloads\iso-win#\sources to C:\Users\myuser\Downloads\wim-win#
2. Copy boot.wim from C:\Users\myuser\Downloads\iso-win#\sources to C:\Users\myuser\Downloads\wim-win#

This should leave us with:

• Contents of the Windows ISO in iso-win#
• Contents of the VirtIO ISO in iso-virtio
• Only the folders/files of the drivers needed for the specific version of Windows being worked on in drv-win#
• 2 files in wim-win#, install.wim or install.esd and boot.wim
• An empty mount folder.
• Windows ADK installed

Slipstream VirtIO Drivers into Windows ISO

This process is repetitious. .wim files (and .esd) often have multiple indexes. So when you boot from a Windows 10 ISO for example, and you have options to install «Windows 10 Home», «Windows 10 Pro», etc. each is a different index in the same install.wim/install.esd on the same ISO. This means we need to inject the driver into each option we intend to keep/use. We also need to inject the driver into the boot.wim, which also has indexes (generally 1 for Win PE and 2 for the Windows Setup). Unless PXE booting and using the Win PE portion, we only need to inject into the Windows Setup portion for the boot.wim.

To cut down on this, I recommend removing any index (option) that you do not intend to use ever. For example if you will never be using «Windows 10 Education» from a Windows 10 ISO, better to remove it then to spend time injecting drivers to it. We will cover how to do this soon.

1. Getting Wim Info

This command will display information about a given image file (.wim or .esd). Primarily we want to know how many indexes there are, and which index(s) we will use and which we won’t.

2. (Situational) Remove Unused Indexes

This command will remove the designated index from the image. If you are unsure, do NOT delete an index. Only use this command to delete options from an ISO you are sure you will not use. I also highly recommend to re-run the command from step 1 above after removing each index. This is because options are re-indexed after each removal. IE: if index 3 was «Windows 10 Home» and index 4 was «Windows 10 Pro», after removing index 3, «Windows 10 Pro» now becomes index 3 instead of 4. AKA: Dont assume the indexes are the same after removing another.

3. (Situational) Convert .esd to .wim

This step can also take a relatively long time (like 5-15 mins on a modern machine). Just let it run to completion.

Per Index Steps

Perform the steps 4-7 in order for a single image/index before repeating them for the next.

Replace install.wim with boot.wim when injecting drivers into the setup image.

4. Mount an Image

In this step we mount an image/index to inject the drivers into. This is similar in concept to mounting an ISO except that we are mounting a specific index within the install.wim or boot.wim image.

5. Inject/Slipstream Drivers

In the above commands, -Recurse has the command look through all the folders and subfolders of our drv-win# directory for drivers. The -ForceUnsigned switch removes the requirement for drivers to be digitally signed to be installed which may be required for some drivers in the VirtIO collection.

6. Verify Drivers Have Been Added

7. Save Changes and Un-mount the Image

Once the drivers have been injected, we need to save the changes to the image while we un-mount it.

Final One Time Steps (per ISO)

The proceeding step is meant to be done after steps 4-7 have been repeated for each file/index required. Be sure that you have done steps 4-7 for the setup index of boot.wim and at least one index of the install.wim.

8. Copy Updated .wim Files Back to ISO Folder

We need to take the modified install.wim and boot.wim and place them back in the ISO file/folder structure. These steps can be done in File Explorer.

1. Copy install.wim from the C:\Users\myuser\Downloads\wim-win# directory.
2. Paste install.wim to C:\Users\myuser\Downloads\iso-win#\sources\ directory.
3. Copy boot.wim from the C:\Users\myuser\Downloads\wim-win# directory.
4. Paste boot.wim to C:\Users\myuser\Downloads\iso-win#\sources\ directory.

9. Rebuild Windows ISO

Our last step takes everything we have done and rebuilds a new ISO using the updated images with the drivers contained within them. The resulting ISO will be what we want to install Windows VM’s from going forward.

This particular step doesn’t have ADK/PS variations, it’s just a single command. As far as I am aware, the same command can be run from either ADK (cmd) or PS.

Replace new-iso-name.iso with the desired name of the new Windows ISO. I recommend this not be the same as the original ISO. The above command also assumes UEFI bootable media, I am personally not concerned with BIOS booting Windows.

This will allow for ISO images larger than 4.7GB single layer DVD’s and beyond the approx. 4GB install.wim limit for an ISO.

Post Install Using the New ISO

After installing Windows in a Proxmox VM using your new ISO, there are still a few steps that you need to take to get the most out of it. Earlier in the guide, I recommended copying our drv-win# folder to the iso-win# folder. I am going to assume in the steps below this was done. If not, you will need to mount the VirtIO ISO/disc for the current Windows VM to finish the post install steps.

1. Install QEMU Guest Agent

The Windows ISO (and the VirtIO ISO) which should be still mounted post Windows Install as a CD/DVD drive/disc contains a folder drv-win#\guest-agent which should contain at least a single .msi file (if not 2 files) named qemu-ga-i386.msi or qemu-ga-x86_64.msi (for 32bit/64bit Windows OS’s respectively). This is similar to the VMWare/ESXi guest agent allowing the host (in this case Proxmox) to have more accurate information of and better control over its guest VM’s.

To install this on the Windows guest VM, in File Explorer:

1. Navigate to the Windows CD/DVD (the ISO mounted within the VM that we installed from), let’s assume D:\ and go to the guest-agent folder D:\drv-win#\guest-agent.
2. Double click on the appropriate .msi for your architecture of Windows (i386 = 32bit and x86_64 = 64bit).

A command prompt window will open briefly then vanish, that’s it. It should be installed.

2. Enable the Balloon Service

The Balloon driver and service allow KVM guests like our Windows VM to work with the memory ballooning feature of KVM. While our previous steps have added the driver to the ISO and thus our install, we still need to setup and activate the service for ballooning in our Windows VM.

To install/activate the ballooning service:

1. Create a new folder C:\Program Files\Balloon.
2. Navigate to the Windows CD/DVD (the ISO mounted within the VM that we installed from), let’s assume D:\ and go to the proper balloon directory D:\drv-win#\Balloon\w10\amd64. Replace w10 with the folder for your version of Windows and amd64 for your architecture of Windows.
3. Copy all the files in the proper sub directory of Balloon\w10\amd64\*. There are roughly 6 files in this folder, one of which should be blnsvr.exe.
4. Paste these files into the created directory from step 1 at C:\Program Files\Balloon.
5. Open cmd (command prompt) as an Admin from C:\Program Files\Balloon or cd to C:\Program Files\Balloon.
6. Run the command blnsvr.exe -i. This should give a few lines of output in cmd about starting the service.

3. Reboot the VM

To finalize the post install configuration, reboot the Windows VM. To verify that the guest agent is working, you can look at the Summary of the Windows VM in Proxmox. Specifically under «Bootdisk size» should be the «IPs» entry listing the IP addresses of the VM with a button that says «More». The «More» button displays further guest agent network info.

You can confirm that the Balloon service is running from the Windows guest using the PS command: get-service -name ballo*. It should return a result stating the Balloon Service is running.

Conclusion

This guide should provide the means to:

• Create a Windows ISO (or ISO’s) with VirtIO drivers baked in.
• Implement post install steps for Windows VM’s in Proxmox to take advantage of qemu guest agent and the balloon driver/service.
• Reproduce the process of injecting the VirtIO drivers into Windows media using ADK (cmd) or PowerShell.
• Eliminate the need to manually install VirtIO drivers during installation or post install for Windows VM’s.

Hopefully you find this guide helpful. Please open an issue should you have any corrections, updates, questions or input on this guide. Thank you.

Ultimate Beginner’s Guide to Proxmox GPU Passthrough

mirror of The Ultimate Beginner’s Guide to GPU Passthrough (Proxmox, Windows 10)
by /u/cjalas

Before We Begin (Credits)

Disclaimer: In no way, shape, or form does this guide claim to work for all instances of Proxmox/GPU configurations. Use at your own risk. I am not responsible if you blow up your server, your home, or yourself. Surgeon General Warning: do not operate this guide while under the influence of intoxicating substances. Do not let your cat operate this guide. You have been warned.

Let’s Get Started (Pre-configuration Checklist)

It’s important to make note of all your hardware/software setup before we begin the GPU passthrough. For reference, I will list what I am using for hardware and software. This guide may or may not work the same on any given hardware/software configuration, and it is intended to help give you an overall understanding and basic setup of GPU passthrough for Proxmox only.

Your hardware should, at the very least, support: VT-d, interrupt mapping, and UEFI BIOS.

My Hardware Configuration:

Motherboard: Supermicro X9SCM-F (Rev 1.1 Board + Latest BIOS)

CPU: LGA1150 Socket, Xeon E3-1220 (version 2) ^(1)

Memory: 16GB DDR3 (ECC, Unregistered)

GPU: 2x GTX 1050 Ti 4gb, 2x GTX 1060 6gb ^(2)

My Software Configuration:

Latest Proxmox Build (5.3 as of this writing)

Windows 10 LTSC Enterprise (Virtual Machine) ^(3)

¹On most Xeon E3 CPUs, IOMMU grouping is a mess, so some extra configuration is needed. More on this later.

²It is not recommended to use multiple GPUs of the same exact brand/model type. More on this later.

³Any Windows 10 installation ISO should work, however, try to stick to the latest available ISO from Microsoft.

Configuring Proxmox

This guide assumes you already have at the very least, installed Proxmox on your server and are able to login to the WebGUI and have access to the server node’s Shell terminal. If you need help with installing base Proxmox, I highly recommend the official «Getting Started» guide and their official YouTube guides.

Step 1: Configuring the Grub

Assuming you are using an Intel CPU, either SSH directly into your Proxmox server, or utilizing the noVNC Shell terminal under «Node», open up the /etc/default/grub file. I prefer to use nano, but you can use whatever text editor you prefer.

nano /etc/default/grub

Look for this line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

Then change it to look like this:

For Intel CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

For AMD CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

IMPORTANT ADDITIONAL COMMANDS

You might need to add additional commands to this line, if the passthrough ends up failing. For example, if you’re using a similar CPU as I am (Xeon E3-12xx series), which has horrible IOMMU grouping capabilities, and/or you are trying to passthrough a single GPU.

These additional commands essentially tell Proxmox not to utilize the GPUs present for itself, as well as helping to split each PCI device into its own IOMMU group. This is important because, if you try to use a GPU in say, IOMMU group 1, and group 1 also has your CPU grouped together for example, then your GPU passthrough will fail.

Here are my grub command line settings:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

A. Disabling the Framebuffer: video=vesafb:off,efifb:off

B. ACS Override for IOMMU groups: pcie_acs_override=downstream,multifunction

When you finished editing /etc/default/grub run this command:

update-grub

Step 2: VFIO Modules

You’ll need to add a few VFIO modules to your Proxmox system. Again, using nano (or whatever), edit the file /etc/modules

nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Then save and exit.

Step 3: IOMMU interrupt remapping

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Step 4: Blacklisting Drivers

We don’t want the Proxmox host system utilizing our GPU(s), so we need to blacklist the drivers. Run these commands in your Shell:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Step 5: Adding GPU to VFIO

Run this command:

lspci -v

Your shell window should output a bunch of stuff. Look for the line(s) that show your video card. It’ll look something like this:

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Make note of the first set of numbers (e.g. 01:00.0 and 01:00.1). We’ll need them for the next step.

Run the command below. Replace 01:00 with whatever number was next to your GPU when you ran the previous command:

lspci -n -s 01:00

Doing this should output your GPU card’s Vendor IDs, usually one ID for the GPU and one ID for the Audio bus. It’ll look a little something like this:

01:00.0 0000: 10de:1b81 (rev a1)

01:00.1 0000: 10de:10f0 (rev a1)

What we want to keep, are these vendor id codes: 10de:1b81 and 10de:10f0.

Now we add the GPU’s vendor id’s to the VFIO (remember to replace the id’s with your own!):

echo "options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1"> /etc/modprobe.d/vfio.conf

Finally, we run this command:

update-initramfs -u

reset

Now your Proxmox host should be ready to passthrough GPUs!

Configuring the VM (Windows 10)

Now comes the ‘fun’ part. It took me many, many different configuration attempts to get things just right. Hopefully my pain will be your gain, and help you get things done right, the first time around.

Step 1: Create a VM

Making a Virtual Machine is pretty easy and self-explanatory, but if you are having issues, I suggest looking up the official Proxmox Wiki and How-To guides.

For this guide, you’ll need a Windows ISO for your Virtual Machine. Here’s a handy guide on how to download an ISO file directly into Proxmox. You’ll want to copy ALL your .ISO files to the proper repository folder under Proxmox (including the VirtIO driver ISO file mentioned below).

Example Menu Screens

General => OS => Hard disk => CPU => Memory => Network => Confirm

IMPORTANT: DO NOT START YOUR VM (yet)

Step 1a (Optional, but RECOMMENDED): Download VirtIO drivers

Step 2: Enable OMVF (UEFI) for the VM

Boot Order: CD-ROM, Disk (scsi0)
SCSI Controller: VirtIO SCSI Single
BIOS: OMVF (UEFI)

Don’t Forget: When you change the BIOS from SeaBIOS (Default) to OMVF (UEFI), Proxmox will say something about adding an EFI disk. So you’ll go to your Hardware Tab/Window and do that. Add > EFI Disk.

Step 3: Edit the VM Config File

Going back to the Shell window, we need to edit /etc/pve/qemu-server/.conf, where is the VM ID Number you used during the VM creation (General Tab).

nano /etc/pve/qemu-server/<vmid>.conf

In the editor, let’s add these command lines (doesn’t matter where you add them, so long as they are on new lines. Proxmox will move things around for you after you save):

machine: q35
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

Save and exit the editor.

Step 4: Add PCI Devices (Your GPU) to VM

Look at all those GPUs

Under the VM’s Hardware Tab/Window, click on the Add button towards the top. Then under the drop-down menu, click PCI Device.

Look for your GPU in the list, and select it. On the PCI options screen, you should only need to configure it like so:

All Functions: YES
Rom-Bar: YES
Primary GPU: NO
PCI-Express: YES (requires 'machine: q35' in vm config file)

Here’s an example image of what your Hardware Tab/Window should look like when you’re done creating the VM.

Oopsies, make sure “All Functions” is CHECKED.

Step 4a (Optional): ROM File Issues

In the off chance that things don’t work properly at the end, you MIGHT need to come back to this step and specify the ROM file for your GPU. This is a process unto itself, and requires some extra steps, as outlined below.

Download your GPU’s ROM file

Dump your GPU’s ROM File:

cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/<GPURomFileName>.bin
echo 0 > rom

Alternative Methods to Dump ROM File:

a. Using GPU-Z (recommended)

b. Using NVFlash

You can use SFTP for this, or directly through Windows’ Command Prompt:

scp /path/to/<romfilename>.rom myusername@proxmoxserveraddress:/usr/share/kvm/<romfilename>.rom

Step 4a3: Add the ROM file to your VM Config (EXAMPLE):

hostpci0: 01:00,pcie=1,romfile=<GTX1050ti>.rom

Example of the Hardware Tab/Window, Before Windows 10 Installation.

Step 5: START THE VM!

We’re almost at the home stretch! Once you start your VM, open your noVNC / Shell Tab/Window (under the VM Tab), and you should see the Windows installer booting up. Let’s quickly go through the process, since it can be easy to mess things up at this junction.

Final Setup: Installing / Configuring Windows 10

Step 1: VirtIO Driver Installation

Simply go to your VM’s Hardware Tab/Window (again), double click the CD-ROM drive file (it should currently have the Windows 10 ISO loaded), and switch the ISO image to the VirtIO ISO file.

Tabbing back to your noVNC Shell window, click Browse, find your newly loaded VirtIO CD-ROM drive, and go to the vioscsi > w10 > amd64 sub-directory. Click OK.

Now the Windows installer should do its thing and load the Red Hat VirtIO SCSI driver for your hard drive. Before you start installing to the drive, go back again to the VirtIO CD-Rom, and also install your Network Adapter VirtIO drivers from NetKVM > w10 > amd64 sub-directory.

IMPORTANT #1: Don’t forget to switch back the ISO file from the VirtIO ISO image to your Windows installer ISO image under the VM Hardware > CD-Rom.

When you’re done changing the CD-ROM drive back to your Windows installer ISO, go back to your Shell window and click Refresh. The installer should then have your VM’s hard disk appear and have windows ready to be installed. Finish your Windows installation.

IMPORTANT #2: When Windows asks you to restart, right click your VM and hit ‘Stop’. Then go to your VM’s Hardware Tab/Window, and Unmount the Windows ISO from your CD-Rom drive. Now ‘Start’ your VM again.

Step 2: Enable Windows Remote Desktop

If all went well, you should now be seeing your Windows 10 VM screen! It’s important for us to enable some sort of remote desktop access, since we will be disabling Proxmox’s noVNC / Shell access to the VM shortly. I prefer to use Windows’ built-in Remote Desktop Client. Here’s a great, simple tutorial on enabling RDP access.

Step 3: Disabling Proxmox noVNC / Shell Access

To make sure everything is properly configured before we get the GPU drivers installed, we want to disable the built-in video display adapter that shows up in the Windows VM. To do this, we simply go to the VM’s Hardware Tab/Window, and under the Display entry, we select None (none) from the drop-down list. Easy. Now ‘Stop’ and then ‘Start’ your Virtual Machine.

NOTE: If you are not able to (re)connect to your VM via Remote Desktop (using the given internal IP address or computer name / hostname), go back to the VM’s Hardware Tab/Window, and under the PCI Device Settings for your GPU, checkmark Primary GPU**. Save it, then ‘Stop’ and ‘Start’ your VM again.**

Step 4: Installing GPU Drivers

At long last, we are almost done. The final step is to get your GPU’s video card drivers installed. Since I’m using NVIDIA for this tutorial, we simply go to http://nvidia.com and browse for our specific GPU model’s driver (in this case, GTX 10XX series). While doing this, I like to check Windows’ Device Manager (under Control Panel) to see if there are any missing VirtIO drivers, and/or if the GPU is giving me a Code 43 Error. You’ll most likely see the Code 43 error on your GPU, which is why we are installing the drivers. If you’re missing any VirtIO (usually shows up as ‘PCI Device’ in Device Manager, with a yellow exclamation), just go back to your VM’s Hardware Tab/Window, repeat the steps to mount your VirtIO ISO file on the CD-Rom drive, then point the Device Manager in Windows to the CD-Rom drive when it asks you to add/update drivers for the Unknown device.

Sometimes just installing the plain NVIDIA drivers will throw an error (something about being unable to install the drivers). In this case, you’ll have to install using NVIDIA’s crappy GeForce Experience(tm) installer. It sucks because you have to create an account and all that, but your driver installation should work after that.

Congratulations!

After a reboot or two, you should now be able to see NVIDIA Control Panel installed in your Windows VM, as well as Device Manager showing no Code 43 Errors on your GPU(s). Pat yourself on the back, do some jumping jacks, order a cake! You’ve done it!

Multi-GPU Passthrough, it CAN be done!

Credits / Resources / Citations

Thank you everyone!