Derp Proof FIO

This is an FIO job file I wrote up with a built in manual for folks who aren’t familiar with FIO. It gives a rundown for the most common options. Includes logging.

Have fun. Mods, if this is in the wrong place please move it to the proper spot!

I just wanted to post a long overdue update. If you want to run some easier, smaller and more manageable jobs you can group things together.

ie.

[job1]
rw=read
filename=/dev/nvme0n1:/dev/nvme1n1

I also completely forgot to include directory=%%%%%, which is updated below.

    [global]
    log_avg_msec=5
    write_bw_log=bandwidth.results
    write_iops_log=iops.results
    write_lat_log=lat.results


    # bs= blocksize for test. Common sizes are 4k, 64k,128k,256k, 512k, 1M

    bs=512k

    # iodepth = queue depth. Common depths are 32, 64, 126, 256, 512
    #emulate a real world run by setting to 1
    iodepth=32

    # no reason to change direct
    direct=1

    # only change if you need to, ioengine is fine for linux. Change for your OS.
    ioengine=libaio

    # do not change randrepeat, this gives validation to test and makes sure you're not just writing on m.2 DRAM buffer stored data.
    randrepeat=1

    # reports all jobs into the same output. No need to change unless you think there's a single m.2 that's under-performing.
    group_reporting=1

    # sets the jobs to run as a time based run.
    time_based

    # runtime is how long the test will run in seconds.
    runtime=120

    # filesize sets the size of the test file plot.
    filesize=10G

    # cpus_allowed will set what CPU thread to use for FIO. CSV for certain threads.
    # For performance, find what numa your drive is on and bind those cpus to your job.
    # You can add cpus_allowed= to any job, allowing you bind cores to 
    # whatever numa the drive(s) in that job are on.
    # Separate jobs for separate NUMA.  
    # You should always comment out the global value set here if you do that.
    # Dash for a span of cpus. ie 0,1,2,3  or 0-24
    cpus_allowed=0-24

    # cpus_allowed_policy changes how the threads are locked. 
    # This can be changed to shared or split. 
    # Split dedicates one thread to one job
    # shared will allow more than one job to run a thread. Tweak for performance.
    # Setting policy to split will almost always give the best performance but shared will give you a more real-world test.
    cpus_allowed_policy=shared

    # numjobs sets the amount of threads to run per job. ie. if 
    # numbjobs=2 than each job will run two tests on the m.2.
    # Change value to see where performance is the best.  My finding is that 2 is the best performance.
    numjobs=2

    # do not change invalidate. If set to 0 your test will not be valid.
    invalidate=1

    # ramp_time sets the amount of time to warm up the drives.
    # Longer ramp times will fill the write buffer so you are able to log data for sustained writes only.
    ramp_time=10

    # How to use this job file:
    # Job structure is broken down into four values, job number defined as [job$], job type as defined by rw=$$, 
    # target for test is defined by filename=/dev/nvme$n$ and the name of the test as defined by name=$$
    # You can also define a target volume, (software raid/stripe) using directory=%%%%



    # Remove comment to switch test mode.
    # write = seq write
    # read = seq read
    # rw = seq read and write
    # randread random reads
    # randwrite = random writes
    # randrw = random reads and writes
    # trimwrite = before writing the block will be trimmed and then 
    # written. Good for testing file system performance.

    # Remove jobs and change "filename" to suit your target(s). 
    # This test is currently setup for 4 nvme.  If you want to test 1 drive 
    # remove or comment out jobs 2,3 and 4. 
    # If you want to test more drives or perform multiple tests on one 
    drive,
    # change the filenames as required. If you want to test more drives, 
    # add more jobs or if your running the same test on multiple drives 
    # you can separate drives with a " : ". 
    # ie /dev/nvme0n1:/dev/nvme1n1

    [job1]


    rw=read
    # rw=write
    # rw=rw
    # rw=randread
    # rw=randwrite
    # rw=randrw
    # rw=trimwrite

    filename=/dev/nvme0n1
    name=raw=test

    [job2]

    rw=read
    # rw=write
    # rw=rw
    # rw=randread
    # rw=randwrite
    # rw=randrw
    # rw=trimwrite

    filename=/dev/nvme1n1
    name=raw=test

    [job3]

    rw=read
    # rw=write
    # rw=rw
    # rw=randread
    # rw=randwrite
    # rw=randrw
    # rw=trimwrite

    filename=/dev/nvme2n1
    name=raw=test

    [job4]

    rw=read
    # rw=write
    # rw=rw
    # rw=randread
    # rw=randwrite
    # rw=randrw
    # rw=trimwrite

    filename=/dev/nvme3n1
    name=raw=test
3 Likes

Worth noting that this config won’t work on FreeBSD-based systems such as FreeNAS without changing the ioengine. libaio is a Linux-native alternative to POSIX aio (Linux doesn’t have POSIX aio in the kernel, it’s in glibc and it is not great). posixaio is fine on FreeBSD though.

2 Likes