1*4882a593Smuzhiyun******************************** 2*4882a593SmuzhiyunRDMA Network Block Device (RNBD) 3*4882a593Smuzhiyun******************************** 4*4882a593Smuzhiyun 5*4882a593SmuzhiyunIntroduction 6*4882a593Smuzhiyun------------ 7*4882a593Smuzhiyun 8*4882a593SmuzhiyunRNBD (RDMA Network Block Device) is a pair of kernel modules 9*4882a593Smuzhiyun(client and server) that allow for remote access of a block device on 10*4882a593Smuzhiyunthe server over RTRS protocol using the RDMA (InfiniBand, RoCE, iWARP) 11*4882a593Smuzhiyuntransport. After being mapped, the remote block devices can be accessed 12*4882a593Smuzhiyunon the client side as local block devices. 13*4882a593Smuzhiyun 14*4882a593SmuzhiyunI/O is transferred between client and server by the RTRS transport 15*4882a593Smuzhiyunmodules. The administration of RNBD and RTRS modules is done via 16*4882a593Smuzhiyunsysfs entries. 17*4882a593Smuzhiyun 18*4882a593SmuzhiyunRequirements 19*4882a593Smuzhiyun------------ 20*4882a593Smuzhiyun 21*4882a593Smuzhiyun RTRS kernel modules 22*4882a593Smuzhiyun 23*4882a593SmuzhiyunQuick Start 24*4882a593Smuzhiyun----------- 25*4882a593Smuzhiyun 26*4882a593SmuzhiyunServer side: 27*4882a593Smuzhiyun # modprobe rnbd_server 28*4882a593Smuzhiyun 29*4882a593SmuzhiyunClient side: 30*4882a593Smuzhiyun # modprobe rnbd_client 31*4882a593Smuzhiyun # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \ 32*4882a593Smuzhiyun /sys/devices/virtual/rnbd-client/ctl/map_device 33*4882a593Smuzhiyun 34*4882a593Smuzhiyun Where "sessname=" is a session name, a string to identify the session 35*4882a593Smuzhiyun on client and on server sides; "path=" is a destination IP address or 36*4882a593Smuzhiyun a pair of a source and a destination IPs, separated by comma. Multiple 37*4882a593Smuzhiyun "path=" options can be specified in order to use multipath (see RTRS 38*4882a593Smuzhiyun description for details); "device_path=" is the block device to be 39*4882a593Smuzhiyun mapped from the server side. After the session to the server machine is 40*4882a593Smuzhiyun established, the mapped device will appear on the client side under 41*4882a593Smuzhiyun /dev/rnbd<N>. 42*4882a593Smuzhiyun 43*4882a593Smuzhiyun 44*4882a593SmuzhiyunRNBD-Server Module Parameters 45*4882a593Smuzhiyun============================= 46*4882a593Smuzhiyun 47*4882a593Smuzhiyundev_search_path 48*4882a593Smuzhiyun--------------- 49*4882a593Smuzhiyun 50*4882a593SmuzhiyunWhen a device is mapped from the client, the server generates the path 51*4882a593Smuzhiyunto the block device on the server side by concatenating dev_search_path 52*4882a593Smuzhiyunand the "device_path" that was specified in the map_device operation. 53*4882a593Smuzhiyun 54*4882a593SmuzhiyunThe default dev_search_path is: "/". 55*4882a593Smuzhiyun 56*4882a593Smuzhiyundev_search_path option can also contain %SESSNAME% in order to provide 57*4882a593Smuzhiyundifferent device namespaces for different sessions. See "device_path" 58*4882a593Smuzhiyunoption for details. 59*4882a593Smuzhiyun 60*4882a593Smuzhiyun============================ 61*4882a593SmuzhiyunProtocol (rnbd/rnbd-proto.h) 62*4882a593Smuzhiyun============================ 63*4882a593Smuzhiyun 64*4882a593Smuzhiyun1. Before mapping first device from a given server, client sends an 65*4882a593SmuzhiyunRNBD_MSG_SESS_INFO to the server. Server responds with 66*4882a593SmuzhiyunRNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol 67*4882a593Smuzhiyunversion for backward compatibility. 68*4882a593Smuzhiyun 69*4882a593Smuzhiyun2. Client requests to open a device by sending RNBD_MSG_OPEN message. This 70*4882a593Smuzhiyuncontains the path to the device and access mode (read-only or writable). 71*4882a593SmuzhiyunServer responds to the message with RNBD_MSG_OPEN_RSP. This contains 72*4882a593Smuzhiyuna 32 bit device id to be used for IOs and device "geometry" related 73*4882a593Smuzhiyuninformation: side, max_hw_sectors, etc. 74*4882a593Smuzhiyun 75*4882a593Smuzhiyun3. Client attaches RNBD_MSG_IO to each IO message send to a device. This 76*4882a593Smuzhiyunmessage contains device id, provided by server in his rnbd_msg_open_rsp, 77*4882a593Smuzhiyunsector to be accessed, read-write flags and bi_size. 78*4882a593Smuzhiyun 79*4882a593Smuzhiyun4. Client closes a device by sending RNBD_MSG_CLOSE which contains only the 80*4882a593Smuzhiyundevice id provided by the server. 81*4882a593Smuzhiyun 82*4882a593Smuzhiyun========================================= 83*4882a593SmuzhiyunContributors List(in alphabetical order) 84*4882a593Smuzhiyun========================================= 85*4882a593SmuzhiyunDanil Kipnis <danil.kipnis@profitbricks.com> 86*4882a593SmuzhiyunFabian Holler <mail@fholler.de> 87*4882a593SmuzhiyunGuoqing Jiang <guoqing.jiang@cloud.ionos.com> 88*4882a593SmuzhiyunJack Wang <jinpu.wang@profitbricks.com> 89*4882a593SmuzhiyunKleber Souza <kleber.souza@profitbricks.com> 90*4882a593SmuzhiyunLutz Pogrell <lutz.pogrell@cloud.ionos.com> 91*4882a593SmuzhiyunMilind Dumbare <Milind.dumbare@gmail.com> 92*4882a593SmuzhiyunRoman Penyaev <roman.penyaev@profitbricks.com> 93