Performance testing with bcache

[Andrew Martin]

Hello,

I am working on setting up bcache on Ubuntu 13.10 (with kernel 3.11.0-17)
for read performance testing in writethrough mode. I've written a test script
which uses dd to write data to the bcache device, read it back in, and then
attempt to read it again (to measure cache speed):
# write the data to the disk
dd if=/dev/zero of=/path/to/bcache/mount/file.raw bs=512 count=2

# read the data (using iflag=nocache to make sure it gets into bcache's
# cache and doesn't use the kernel's cache)
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

# now perform the read, which should read from bcache's cache
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
before and after performing the read, the value is always 100. Would using
iflag=direct be better than iflag=nocache in this case? Is there a better
way to force data into the cache for this type of performance benchmarking?

Thanks,

Andrew Martin

[Sitsofe Wheeler]

Post by Andrew Martin
I am working on setting up bcache on Ubuntu 13.10 (with kernel 3.11.0-17)
for read performance testing in writethrough mode. I've written a test script
which uses dd to write data to the bcache device, read it back in, and then
# write the data to the disk
dd if=/dev/zero of=/path/to/bcache/mount/file.raw bs=512 count=2

This will only tell you sequential speeds of small blocks. You may want
to use a tool like fio to let you generate different types of IO
patterns to see what the impact is. For example, the difference between
an hdd's and an sdd's top large block sequential speeds isn't nearly as
dramatic as the difference between their top small block random
speeds...

Post by Andrew Martin
# read the data (using iflag=nocache to make sure it gets into bcache's
# cache and doesn't use the kernel's cache)
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache
# now perform the read, which should read from bcache's cache
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

This may not perform as you expect it to do. My testing shows that the
read cache bcache is a bit quirky and to see it being used you have to
ensure that the blocks being read the second time are the exact same
size as the blocks that were read the first time. I think that
particular issue is alluded to in TROUBLESHOOTING PERFORMANCE on
http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt?h=bcache-dev#n126
(Still getting cache misses, of the same data). I would advise you to
warm the cache by doing writes as that doesn't seem to show the problem
but also see below.

Post by Andrew Martin
However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
before and after performing the read, the value is always 100. Would using
iflag=direct be better than iflag=nocache in this case? Is there a better
way to force data into the cache for this type of performance benchmarking?

The amount free isn't as important as the cache hits/misses ratio.
Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
section mentioned above and posting your findings here because the
bcache defaults often defeat first attempt synthetic benchmarking...

[Andrew Martin]

----- Original Message -----
...

Post by Sitsofe Wheeler

Post by Andrew Martin
However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
before and after performing the read, the value is always 100. Would using
iflag=direct be better than iflag=nocache in this case? Is there a better
way to force data into the cache for this type of performance benchmarking?

The amount free isn't as important as the cache hits/misses ratio.
Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
section mentioned above and posting your findings here because the
bcache defaults often defeat first attempt synthetic benchmarking...

Thanks, I will try fio with these recommended steps!

Andrew

[Andrew Martin]

----- Original Message -----

Post by Andrew Martin
----- Original Message -----
...

Post by Sitsofe Wheeler

Post by Andrew Martin
However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
before and after performing the read, the value is always 100. Would using
iflag=direct be better than iflag=nocache in this case? Is there a better
way to force data into the cache for this type of performance benchmarking?

The amount free isn't as important as the cache hits/misses ratio.
Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
section mentioned above and posting your findings here because the
bcache defaults often defeat first attempt synthetic benchmarking...

Thanks, I will try fio with these recommended steps!

Sitsofe,

Do you have an example fio job file that has worked well for testing bcache?

Thanks,

Andrew

[Sitsofe Wheeler]

Post by Andrew Martin
Do you have an example fio job file that has worked well for testing bcache?

Try these as a starting point:
benchmark-seq.fio:
; Script to gather general sequential write performance statistics
; Basic usage:
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio

[global]
filename=/dev/${DEV}
runtime=${RT}
bs=${BS}
iodepth=4
ioengine=libaio
direct=1
stonewall
size=${SIZE}
randseed=${SEED}
[write]
rw=write

benchmark-rand.fio:
; Script to gather general random write performance statistics
; Basic usage:
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-rand.fio

[global]
filename=/dev/${DEV}
runtime=${RT}
bs=${BS}
iodepth=4
ioengine=libaio
direct=1
stonewall
size=${SIZE}
randseed=${SEED}
[randwrite]
rw=randwrite

After making the above files do
export RT=30; export SEED=21; export SIZE="10G"; export DEV="bcache0"
for b in "4" "16" "64"; do export BS="${b}k"; for job in "benchmark-seq" "benchmark-rand"; do fio $job.fio; done; done

The first time round get the stats where the cache is not being used.
Next time round set the appropriate cache options (easier said than done
:-), run a single benchmark-seq.fio job (I'd recommend 64k blocksize) to
warm the cache then rerun the above.

Let us know what you did and how it went!

[Andrew Martin]

Sitsofe,

----- Original Message -----

Sent: Saturday, March 15, 2014 12:52:15 PM
Subject: Re: Performance testing with bcache

Post by Andrew Martin
Do you have an example fio job file that has worked well for testing bcache?

; Script to gather general sequential write performance statistics
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio

I ran the example job files with the commands you posted, but the output
bandwidth seems extremely low:
http://pastebin.com/13RBYwwp
http://pastebin.com/bsp9KbDq

fio job file:
http://pastebin.com/zfVJdjMQ

Am I doing something wrong when running fio?

Thanks,

Andrew

[Kent Overstreet]

How much data is being bypassed?

Post by Andrew Martin
Sitsofe,
----- Original Message -----

Sent: Saturday, March 15, 2014 12:52:15 PM
Subject: Re: Performance testing with bcache

Post by Andrew Martin
Do you have an example fio job file that has worked well for testing bcache?

; Script to gather general sequential write performance statistics
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio

I ran the example job files with the commands you posted, but the output
http://pastebin.com/13RBYwwp
http://pastebin.com/bsp9KbDq
http://pastebin.com/zfVJdjMQ
Am I doing something wrong when running fio?
Thanks,
Andrew
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Sitsofe Wheeler]

Post by Kent Overstreet
How much data is being bypassed?

Post by Andrew Martin
Am I doing something wrong when running fio?

Andrew,

There's not enough information to be able to say if something is going
wrong at this stage as I don't know:
a) The speed of the backing device (also the block device name (e.g.
sda/md0 helps later)) with a given IO pattern
b) The speed of the front device (block name also helps later) with a
given IO pattern
c) What bcache settings you have set
d) What type of IO was done to the bcache device and at which points
(e.g. was writeback happening at that point?)
e) What statistics bcache is reporting

Initially while you're getting used to this I'd stick to just running
one block size (4k). So something like:
export RT=30; export SEED=21; export SIZE="5G"; export DEV="bcache0"; BS="4k"

Ensure you have read and implemented the appropriate pieces mentioned in
the TROUBLESHOOTING PERFORMANCE section of
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/bcache.txt#n130
otherwise results can be unexpected when benchmarking...

Something like the following would be easier to follow:
Set the cache_mode to none. Run (and record)
grep -r . /sys/block/bcache0/bcache/
Clear the bcache total stats (see clear_stats)
View and record the bcache backing device total stats then run
fio benchmark-rand.fio
View and record the bcache backing device total stats
Set the cache_mode to writeback. Run (and record)
grep -r . /sys/block/bcache0/bcache/
Clear the bcache total stats then run
View and record the bcache backing device total stats then run
fio benchmark-rand.fio
View and record the bcache backing device total stats

If everything is working (and the front device is faster for random
writes) I'd expect the with writeback fio job to be noticibly faster
than none fio job (how much faster is setup dependent). Only if my
expectation was correct would I repeat the above with the
benchmark-seq.fio job...