Using Freebsd As A Server
I have used FreeBSD as a server on and off for a couple years. After recently corrupting a research server running Ubuntu server, I found it imperative to migrate back to FreeBSD, setup ZFS snapshots and segment hosted services using the systems native hypervisor to improve my systems recoverability. If you have not checked out sanoid, I would highly recommend it. This post has been developed to capture that process and point to helpful resources that I have found along the way.
Freebsd, similar to Linux, has its own native hypervisor, bhyve. This allows for running virtual machines at close to native performance. For more information on bhyve and FreeBSD, you can check out 3 and 2 Now, I love setting things up manually just as much as any other tech enthusiast but I have found the CLI tool, cbsd to save me much time. There does exist a wide breadth of jail and host management utilities, but this is my current go to. The setup process is simple and captured in the quick start tutorial provided 1. There is also a FreeBSD distribution, clonos that leverages cbsd and provides a web GUI for simplified management.
Note After setting up the cbsd tool, I have yet to figure out how to configure permissions to enable the tool to be used by an administrative user without the use of sudo or doas. I have a suspicion that it is related to the location that the jails are initially setup. If I figure that out, I will revisit this post and make an amendment.
Getting Started
There are a couple changes made to the /etc/rc.conf fill but for the most part, the pkg or port package will provide you with the changes that need to be made post installation. Just be sure to not install all the service based packages you need at once unless you are using a GUI environment or redirecting the install output to a log file.
Helpful programs typically installed post installation pre-reboot. I will append any helpful commands that I found to add the setup.
pkg install git vim tailscale does tmux
service tailscaled enable
service tailscaled start
sudo tailscale up
sudo sysrc tailscaled_tun_dev="tailscale0"
The next step is the most important, the firewall setup. For that, I will currently direct you to 4 as I improve my understanding on the firewall options offered and the best configuration for a home server.
PCI Passthrough
If you are an AI researcher or enjoying cracking passwords in your free time, it will save you allot of grief to figure out how to access the devices native GPU in your nearly segmented environment. Some of this guidance is still experiemental, but my current working configuration requires identifiy the pci device in question with pciconf -lv. In my case, I am using pciconf -lv | grep “nvidia” -i -C5 to find my nvidia gpu. My output is the following:
ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x24b0 subvendor=0x10de subdevice=0x14ad
vendor = 'NVIDIA Corporation'
device = 'GA104GL [RTX A4000]'
class = display
subclass = VGA
With the pci device bus/slot/function information, we can now mask the device to allow it to be passed to a bhyve host. More information on this can be found 5. In my case, I will append pptdevs=”1/0/0” to my */boot/loader.conf file. I can then use the following command to pass the gpu device to the running or soon to be running bhyve host. You will also need to append hw.vmm.amdvi.enable=”1” to /boot/loader.conf.
cbsd bhyve-ppt mode=attach ppt=1/0/0 jname=[host name]
cbsd bset bhyve_wire_memory=1 jname=research
# verifiable at
cat /usr/jails/vm/[host name]/bhyve.conf
A better explain and walk through of GPU passthrough on bhyve was given at EuroBSD2023 by Corvin Köhne.
Now if you made it through this whole post and are only interested in running FreeBSD as a workstation, there are additional programs you will want to install. checking out the forums is a good starting point, software recommendation thread.