Tuning FreeBSD loader.conf for ZFS to prevent kmem_map too small kernel panic 2

 in FreeBSD / Networking  tagged FreeBSD / ZFS by

Been using FreeBSD 8 and 8.1 for over a year now, love it.  Have root on the ZFS mirror, works great!  However, I’ve struggled to find a good way to set the /boot/loader.conf variables for kmem_size and zfs.arc_max. Everything I tried didn’t seem to work, so I ended up leaving all that out and just let the default “auto-tuning” to its thing.  But whenever I tried to copy large files, like 20+ GB files, it would crash the server.  It would give me that “kmem_map too small” kernel panic.

In addition, I have a GB network and fairly new 500GB drives in my array, which, are not high-end, speedy drives, but should operate fairly well.  I was getting network transfer speeds of about 45MB/s to 55MB/s, varying.  But usually managed to stick to about 50 fairly regularly.  I am not using an scientific method of measuring this, only the Windows 7 transfer window.  So it’s not something I’d rely on, but it was close enough to see *about* what we’re doing.

I finally did some reading, and found that setting my vfs.zfs.arc_max to about one-half my RAM, minus 512MB, and setting vm.kmem_size to 1.5x my RAM, it would stop giving me the kmem_map panic.  I found a wide range of suggestions out there, but in general, there were more suggestions to set the kmem_size higher, and keep the arc_max lower than half the ram.   Also, I found the general thought was you would be best having 8G of RAM in the first place and auto-tuning seemed to work best.

For my server, running 8.1-RELEASE, I have an i3 cpu on an  Intel server board, with 4G RAM.  My ZFS is a mirror on the Root, plus my main storage using 6 drives, 500G  ea. (varying makes), in a RAIDz2 array. So I have no UFS, only ZFS.  Once I set those two parameters, I jumped from about 50MB/s to 72MB/s.  And my 48G transfer test of files moved over no prob.  FINALLY!! I love it!

Here’s my relevant loader.conf variables:

vm.kmem_size_max="8G"
vm.kmem_size="6G"
vfs.zfs.arc_min="1024M"
vfs.zfs.arc_max="1500M"
vfs.zfs.prefetch_disable=0

Also, I have two 16G swap partitions. (I know, overkill, but I wanted to pre-allocate those on my mirror and leave it for when I upgrade the RAM)

I am sure someone out there with a better understanding would recommend better values, but, it works good enough for me and so far its’ stable.

By the way this server also shares lots of media, VirtualBox VM’s, and also some Postgres and Mysql with Apache/PHP.  It’s not under any load, just a few of us access this system, but we do read/write large files from time to time.

I hope this helps others out there!  Good luck!