Return to Level1Techs.com

Building Custom Kernel with ZFS Built In

Great stuff!! :+1:

Hey, I’m trying to build zfs builtin kernel in CentOS. Should I be seeing the usual zfs.ko spl.ko, etc. files in /usr/lib/modules? With dkms, they’re usually in extra, and kmod they’re in fs/zfs. I haven’t seen them anywhere in any of the builtin kernels I’ve compiled, and sometimes when I see the make process have an error it’s got a gpl-compatibility error that stops the process.

There’s multiple ways for module code to work:

  • Built-in
  • Using initramfs (linked as a blob in the same file)
  • Using initramfs in initrd
  • Loaded from /lib/modules/…

The bottom 3 will have .o files linked into .ko files and stored on the filesystem for dynamic linking and loading at some point in the boot process, or perhaps at leisure.

The first one will see the .o file linked alongside other .o files (like the scheduler, the networking, the process and memory management code) into the same binary blob/executable.


It’s strange you’re getting the “GPL compatibility errors” as an end user you’re allowed to link code together between GPL and CDDL (or even with proprietary code), as long as you don’t intend to publish it linked together online.

1 Like

If I install the headers and devel packages, in /usr/src/kernels I appear to have good signs

My Module.symvers has this:

$ grep zfs Module.symvers 
0x00000000      zfs_set_userquota       vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_multilist__remove     vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_visible        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__rrwfastpath__exitmiss    vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_to_name        vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_set_xattr        vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_get_xattr        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__evict    vmlinux EXPORT_SYMBOL
0x00000000      zfs_set_version vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zio__delay__skip      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__iodone vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_default_string vmlinux EXPORT_SYMBOL
0x00000000      zfs_close       vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_remove__free__unvisited       vmlinux EXPORT_SYMBOL
0x00000000      zfs_id_overobjquota     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_free__long__range     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_multilist__insert     vmlinux EXPORT_SYMBOL
0x00000000      zfs_userquota_prop_prefixes     vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_valid_keylocation      vmlinux EXPORT_SYMBOL
0x00000000      zfs_vget        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zio__delay__miss      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_remove__free__inflight        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_dbuf__state_change    vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_set_scanstamp    vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_get_scanstamp    vmlinux EXPORT_SYMBOL
0x00000000      zfs_zpl_version_map     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zio__delay__hit       vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_delegatable    vmlinux EXPORT_SYMBOL
0x00000000      zfs_inactive    vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_taskq_ent__start      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_blocked__read vmlinux EXPORT_SYMBOL
0x00000000      zfs_is_readonly vmlinux EXPORT_SYMBOL
0x00000000      zfs_special_devs        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__wait__for__eviction      vmlinux EXPORT_SYMBOL
0x00000000      zfs_dataset_name_hidden vmlinux EXPORT_SYMBOL
0x00000000      zfs_readlink    vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_dsl_pool_sync__done   vmlinux EXPORT_SYMBOL
0x00000000      zfs_write       vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_fini      vmlinux EXPORT_SYMBOL
0x00000000      zfs_symlink     vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_readonly       vmlinux EXPORT_SYMBOL
0x00000000      zfs_userspace_many      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__ace__allows      vmlinux EXPORT_SYMBOL
0x00000000      zfs_tmpfile     vmlinux EXPORT_SYMBOL
0x00000000      zfs_statvfs     vmlinux EXPORT_SYMBOL
0x00000000      zfs_ereport_post_checksum       vmlinux EXPORT_SYMBOL
0x00000000      zfs_setsecattr  vmlinux EXPORT_SYMBOL
0x00000000      zfs_getsecattr  vmlinux EXPORT_SYMBOL
0x00000000      zfs_remove      vmlinux EXPORT_SYMBOL
0x00000000      zfs_space       vmlinux EXPORT_SYMBOL
0x00000000      zfs_zstd_decompress_level       vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_readlink vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_get_table      vmlinux EXPORT_SYMBOL
0x00000000      zfs_name_to_prop        vmlinux EXPORT_SYMBOL
0x00000000      zfs_mkdir       vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zil__process__normal__itx     vmlinux EXPORT_SYMBOL
0x00000000      zfs_suspend_fs  vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_encryption_key_param   vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__async__upgrade__sync     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_txg__opened   vmlinux EXPORT_SYMBOL
0x00000000      zfs_umount      vmlinux EXPORT_SYMBOL
0x00000000      zfs_userspace_one       vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zil__process__commit__itx     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__rrwfastpath__rdmiss      vmlinux EXPORT_SYMBOL
0x00000000      zfs_getpage     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__delete   vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__fastpath__execute__access__miss  vmlinux EXPORT_SYMBOL
0x00000000      zfs_ereport_is_valid    vmlinux EXPORT_SYMBOL
0x00000000      zfs_post_state_change   vmlinux EXPORT_SYMBOL
0x00000000      zfs_acl_byteswap        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__hit    vmlinux EXPORT_SYMBOL
0x00000000      zfs_znode_byteswap      vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_userquota      vmlinux EXPORT_SYMBOL
0x00000000      zfs_map vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_dbuf__evict__one      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_txg__synced   vmlinux EXPORT_SYMBOL
0x00000000      zfs_fsync       vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_default_numeric        vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_user   vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_inheritable    vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_dnode__move   vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_string_to_index        vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_reduce    vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zrlock__reentry       vmlinux EXPORT_SYMBOL
0x00000000      zfs_prune       vmlinux EXPORT_SYMBOL
0x00000000      zfs_resume_fs   vmlinux EXPORT_SYMBOL
0x00000000      zfs_obj_to_path vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_symlink  vmlinux EXPORT_SYMBOL
0x00000000      zfs_create      vmlinux EXPORT_SYMBOL
0x00000000      zfs_component_namecheck vmlinux EXPORT_SYMBOL
0x00000000      zfs_getattr_fast        vmlinux EXPORT_SYMBOL
0x00000000      zfs_create_fs   vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_written        vmlinux EXPORT_SYMBOL
0x00000000      zfs_spa_version_map     vmlinux EXPORT_SYMBOL
0x00000000      zfs_history_event_names vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__read   vmlinux EXPORT_SYMBOL
0x00000000      zfs_zstd_cache_reap_now vmlinux EXPORT_SYMBOL
0x00000000      zfs_id_overblockquota   vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_txg__quiesced vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_init      vmlinux EXPORT_SYMBOL
0x00000000      zfs_attr_table  vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_index_to_string        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_txg__syncing  vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__dprintf  vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_txg__quiescing        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__miss   vmlinux EXPORT_SYMBOL
0x00000000      zfs_putpage     vmlinux EXPORT_SYMBOL
0x00000000      zfs_allocatable_devs    vmlinux EXPORT_SYMBOL
0x00000000      zfs_deleg_canonicalize_perm     vmlinux EXPORT_SYMBOL
0x00000000      zfs_deleg_whokey        vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_valid_for_type vmlinux EXPORT_SYMBOL
0x00000000      zfs_deleg_verify_nvlist vmlinux EXPORT_SYMBOL
0x00000000      zfs_readdir     vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_get_type       vmlinux EXPORT_SYMBOL
0x00000000      zfs_oldacl_byteswap     vmlinux EXPORT_SYMBOL
0x00000000      zfs_rmdir       vmlinux EXPORT_SYMBOL
0x00000000      zfs_zstd_decompress     vmlinux EXPORT_SYMBOL
0x00000000      zfs_link        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__hit      vmlinux EXPORT_SYMBOL
0x00000000      zfsvfs_create   vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_setonce        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_delay__mintime        vmlinux EXPORT_SYMBOL
0x00000000      zfs_lookup      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_remove__free__synced  vmlinux EXPORT_SYMBOL
0x00000000      zfs_preumount   vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_taskq_ent__birth      vmlinux EXPORT_SYMBOL
0x00000000      zfs_post_remove vmlinux EXPORT_SYMBOL
0x00000000      zfs_id_overquota        vmlinux EXPORT_SYMBOL
0x00000000      zfs_setattr     vmlinux EXPORT_SYMBOL
0x00000000      zfs_ereport_post        vmlinux EXPORT_SYMBOL
0x00000000      zfs_max_dataset_nesting vmlinux EXPORT_SYMBOL
0x00000000      zfs_remount     vmlinux EXPORT_SYMBOL
0x00000000      zfs_fid vmlinux EXPORT_SYMBOL
0x00000000      zfsvfs_free     vmlinux EXPORT_SYMBOL
0x00000000      zfs_prop_init   vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_new_state__mfu        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_new_state__mru        vmlinux EXPORT_SYMBOL
0x00000000      zfs_dirty_inode vmlinux EXPORT_SYMBOL
0x00000000      zfs_post_autoreplace    vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__write  vmlinux EXPORT_SYMBOL
0x00000000      zfs_domount     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zfs__ace__denies      vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__demand__hit__predictive__prefetch        vmlinux EXPORT_SYMBOL
0x00000000      zfs_zstd_compress       vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_enter     vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_taskq_ent__finish     vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_upgrade_txholds  vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_arc__miss     vmlinux EXPORT_SYMBOL
0x00000000      zfs_read        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_zil__commit__io__error        vmlinux EXPORT_SYMBOL
0x00000000      trace_zfs_l2arc__evict  vmlinux EXPORT_SYMBOL
0x00000000      zfs_rename      vmlinux EXPORT_SYMBOL
0x00000000      zfs_access      vmlinux EXPORT_SYMBOL
0x00000000      zfs_sa_upgrade  vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_tryenter  vmlinux EXPORT_SYMBOL
0x00000000      zfs_rangelock_exit      vmlinux EXPORT_SYMBOL
0x00000000      zfs_open        vmlinux EXPORT_SYMBOL

and then fs/zfs and include/zfs folders exist. The only problem is all I see is .h and .c files, there’s no .o or .ko files at all.

I guess I’ll try this out and see what happens…

oh wow, I tried building a kernel in Ubuntu in a chroot environment, and all of the sudden there’s all these .o files being built after a basic make.

I’m not sure what’s going on with my rhel environment, but apparently it is not conducive to building kernels. Something is wrong.

Update: Nevermind, I guess I just wasn’t searching for them properly. I see the zfs .o files, but the kernel builtin rpms still aren’t working.

What’s more, rhel /usr/src/kernel/ doesn’t have any .o files, while the Ubuntu src folder has all of the ones I would expect.

lol I went back to Ubuntu this afternoon and have built 3 zfs-builtin kernels already. Way too much fun. So much easier on Ubuntu.