bcache on NBD block devices

[Juan Antonio Martinez]

Hi all

(This is my first post on this list. I'm still a novice on bcache and
proper nettiquete on this list, so apologize for my mistakes)

Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
kernel 3.11.0-15 on Ubuntu 13.10

/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device

- I've created and registered /dev/sda1 as cache device without problems
- To create nbd file to be exported I've typed following sequence:

# create an empty file
root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0
# create a tiny fs on it
root# mksquashfs /usr/local /dev/bcache0 -noappend

Notice the squashfs: I only need to export fs in read-only mode

At this point I can mount file throught bcache w/o problems,
so unregister, de-loopize and copy raw bcache_test.img file to nbd
server host to be exported.

nbd-server is properly configure: I can list, connect and use file:

root# nbd-client server -N bcache_test /dev/nbd1 -persist -timeout 60
Negotiation: ..size = 64MB
bs=1024, sz=67108864 bytes
timeout=60

Using bcache-super-show tells me that file is ok for bcache

root# bcache-super-show /dev/nbd1
sb.magic ok
sb.first_sector 8 [match]
sb.csum AC160BF8DFFB808D [match]
sb.version 1 [backing device]

dev.label (empty)
dev.uuid 936f8be4-7a90-4d2b-b118-5a45a1e37603
dev.sectors_per_block 1
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 0 [writethrough]
dev.data.cache_state 1 [clean]

cset.uuid 479b0df0-56d3-4f31-951f-ffb69d8d233b

But.... At this point /dev/bcache0 is not created so cannot register it
manually with "echo <CSET-UUID> > /sys/block/bcache0/bcache/attach"

Item worse: NBD doesn't provide bcache directory under /sys/block/nbd1,
so can't execute "echo 1 > /sys/block/nbd1/bcache/running"

Notice that If I "dd" /dev/nbd1 to a local file and "losetup" on
it, /dev/bcache0 appears automagically attached to /dev/loop0

¿Any ideas?
¿Does NBD supports bcache in kernel 3.11?
¿If doesn't, any idea on the best way to provide local disk cache for
NBD remote devices?

Thanks in advance
Juan Antonio

[Paul Clements]

On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez

Post by Juan Antonio Martinez
Hi all
(This is my first post on this list. I'm still a novice on bcache and
proper nettiquete on this list, so apologize for my mistakes)
Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
kernel 3.11.0-15 on Ubuntu 13.10
/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device
- I've created and registered /dev/sda1 as cache device without problems
# create an empty file
root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0

have you tried doing this step from the client machine instead, i.e.:

make-bache -B /dev/nbd1

after the nbd connection is up? does that work?

[Paul Clements]

On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez

Post by Juan Antonio Martinez
Hi all
(This is my first post on this list. I'm still a novice on bcache and
proper nettiquete on this list, so apologize for my mistakes)
Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
kernel 3.11.0-15 on Ubuntu 13.10
/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device
- I've created and registered /dev/sda1 as cache device without problems
# create an empty file
root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0

have you tried doing this step from the client machine instead, i.e.:

make-bache -B /dev/nbd1

after the nbd connection is up? does that work?

[Juan Antonio Martinez]

Post by Paul Clements
On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez

Post by Juan Antonio Martinez
Hi all
(This is my first post on this list. I'm still a novice on bcache and
proper nettiquete on this list, so apologize for my mistakes)
Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
kernel 3.11.0-15 on Ubuntu 13.10
/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device
- I've created and registered /dev/sda1 as cache device without problems
# create an empty file
root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0

make-bache -B /dev/nbd1
after the nbd connection is up? does that work?

Just tested: works fine.... but only in the LTSP client where I did the
changes, and had to tell nbd-server to export imagefile in read-write
mode

Remember that my configuration consist in several (>250!) LTSP-NBD
clients sharing _the_same_ mounted squashfs image file. I cannot for
obious reasons make the image file to be shared read-write on every
clients at the same time... and seems that bcache needs write access to
the backing block device to work

¿Is this correct? If so, bcache is not for me :-( ¿Any alternatives?

Perhaps I coult try to create bcache'd image file, and then modify NBD
server permissions to export in copy-on-write-mode... but this would
work if bcache only writes in bcache superblock ( my image files are
typically 8Gb size each) as this will create one cow file per client on
server

Cheers
Juan Antonio

[Juan Antonio Martinez]

I answer mySelf: exporting NBD image file in copy-on-write mode, and
starting several clients at the same time I can see that diff file is
only 4096 bytes long ( the size of backing device bcache superblock) an=
d
seems not to increase size. So I can continue working with bcache :-)

Thanks for your time. Cheers
Juan Antonio

El mi=C3=A9, 26-02-2014 a las 09:31 +0100, Juan Antonio Martinez escrib=

Post by Paul Clements
On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez

Post by Juan Antonio Martinez
Hi all
(This is my first post on this list. I'm still a novice on bcache=

and

Post by Paul Clements

Post by Juan Antonio Martinez
proper nettiquete on this list, so apologize for my mistakes)
Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardwa=

re

Post by Paul Clements

Post by Juan Antonio Martinez
kernel 3.11.0-15 on Ubuntu 13.10
/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device
- I've created and registered /dev/sda1 as cache device without p=

roblems

Post by Paul Clements

Post by Juan Antonio Martinez
- To create nbd file to be exported I've typed following sequence=
# create an empty file
root# dd if=3D/dev/zero of=3Dbcache_test.img bs=3D1M count=3D64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0

=20

Post by Paul Clements
have you tried doing this step from the client machine instead, i.e=
make-bache -B /dev/nbd1
after the nbd connection is up? does that work?

=20
Just tested: works fine.... but only in the LTSP client where I did t=

he

changes, and had to tell nbd-server to export imagefile in read-write
mode
=20
Remember that my configuration consist in several (>250!) LTSP-NBD
clients sharing _the_same_ mounted squashfs image file. I cannot for
obious reasons make the image file to be shared read-write on every
clients at the same time... and seems that bcache needs write access =

to

the backing block device to work
=20
=C2=BFIs this correct? If so, bcache is not for me :-( =C2=BFAny alte=

rnatives?=20

=20
Perhaps I coult try to create bcache'd image file, and then modify NB=

D

server permissions to export in copy-on-write-mode... but this would
work if bcache only writes in bcache superblock ( my image files are
typically 8Gb size each) as this will create one cow file per client =

on

server
=20
Cheers
Juan Antonio