1*4882a593Smuzhiyun=========================== 2*4882a593SmuzhiyunInfiniBand Midlayer Locking 3*4882a593Smuzhiyun=========================== 4*4882a593Smuzhiyun 5*4882a593Smuzhiyun This guide is an attempt to make explicit the locking assumptions 6*4882a593Smuzhiyun made by the InfiniBand midlayer. It describes the requirements on 7*4882a593Smuzhiyun both low-level drivers that sit below the midlayer and upper level 8*4882a593Smuzhiyun protocols that use the midlayer. 9*4882a593Smuzhiyun 10*4882a593SmuzhiyunSleeping and interrupt context 11*4882a593Smuzhiyun============================== 12*4882a593Smuzhiyun 13*4882a593Smuzhiyun With the following exceptions, a low-level driver implementation of 14*4882a593Smuzhiyun all of the methods in struct ib_device may sleep. The exceptions 15*4882a593Smuzhiyun are any methods from the list: 16*4882a593Smuzhiyun 17*4882a593Smuzhiyun - create_ah 18*4882a593Smuzhiyun - modify_ah 19*4882a593Smuzhiyun - query_ah 20*4882a593Smuzhiyun - destroy_ah 21*4882a593Smuzhiyun - post_send 22*4882a593Smuzhiyun - post_recv 23*4882a593Smuzhiyun - poll_cq 24*4882a593Smuzhiyun - req_notify_cq 25*4882a593Smuzhiyun 26*4882a593Smuzhiyun which may not sleep and must be callable from any context. 27*4882a593Smuzhiyun 28*4882a593Smuzhiyun The corresponding functions exported to upper level protocol 29*4882a593Smuzhiyun consumers: 30*4882a593Smuzhiyun 31*4882a593Smuzhiyun - rdma_create_ah 32*4882a593Smuzhiyun - rdma_modify_ah 33*4882a593Smuzhiyun - rdma_query_ah 34*4882a593Smuzhiyun - rdma_destroy_ah 35*4882a593Smuzhiyun - ib_post_send 36*4882a593Smuzhiyun - ib_post_recv 37*4882a593Smuzhiyun - ib_req_notify_cq 38*4882a593Smuzhiyun 39*4882a593Smuzhiyun are therefore safe to call from any context. 40*4882a593Smuzhiyun 41*4882a593Smuzhiyun In addition, the function 42*4882a593Smuzhiyun 43*4882a593Smuzhiyun - ib_dispatch_event 44*4882a593Smuzhiyun 45*4882a593Smuzhiyun used by low-level drivers to dispatch asynchronous events through 46*4882a593Smuzhiyun the midlayer is also safe to call from any context. 47*4882a593Smuzhiyun 48*4882a593SmuzhiyunReentrancy 49*4882a593Smuzhiyun---------- 50*4882a593Smuzhiyun 51*4882a593Smuzhiyun All of the methods in struct ib_device exported by a low-level 52*4882a593Smuzhiyun driver must be fully reentrant. The low-level driver is required to 53*4882a593Smuzhiyun perform all synchronization necessary to maintain consistency, even 54*4882a593Smuzhiyun if multiple function calls using the same object are run 55*4882a593Smuzhiyun simultaneously. 56*4882a593Smuzhiyun 57*4882a593Smuzhiyun The IB midlayer does not perform any serialization of function calls. 58*4882a593Smuzhiyun 59*4882a593Smuzhiyun Because low-level drivers are reentrant, upper level protocol 60*4882a593Smuzhiyun consumers are not required to perform any serialization. However, 61*4882a593Smuzhiyun some serialization may be required to get sensible results. For 62*4882a593Smuzhiyun example, a consumer may safely call ib_poll_cq() on multiple CPUs 63*4882a593Smuzhiyun simultaneously. However, the ordering of the work completion 64*4882a593Smuzhiyun information between different calls of ib_poll_cq() is not defined. 65*4882a593Smuzhiyun 66*4882a593SmuzhiyunCallbacks 67*4882a593Smuzhiyun--------- 68*4882a593Smuzhiyun 69*4882a593Smuzhiyun A low-level driver must not perform a callback directly from the 70*4882a593Smuzhiyun same callchain as an ib_device method call. For example, it is not 71*4882a593Smuzhiyun allowed for a low-level driver to call a consumer's completion event 72*4882a593Smuzhiyun handler directly from its post_send method. Instead, the low-level 73*4882a593Smuzhiyun driver should defer this callback by, for example, scheduling a 74*4882a593Smuzhiyun tasklet to perform the callback. 75*4882a593Smuzhiyun 76*4882a593Smuzhiyun The low-level driver is responsible for ensuring that multiple 77*4882a593Smuzhiyun completion event handlers for the same CQ are not called 78*4882a593Smuzhiyun simultaneously. The driver must guarantee that only one CQ event 79*4882a593Smuzhiyun handler for a given CQ is running at a time. In other words, the 80*4882a593Smuzhiyun following situation is not allowed:: 81*4882a593Smuzhiyun 82*4882a593Smuzhiyun CPU1 CPU2 83*4882a593Smuzhiyun 84*4882a593Smuzhiyun low-level driver -> 85*4882a593Smuzhiyun consumer CQ event callback: 86*4882a593Smuzhiyun /* ... */ 87*4882a593Smuzhiyun ib_req_notify_cq(cq, ...); 88*4882a593Smuzhiyun low-level driver -> 89*4882a593Smuzhiyun /* ... */ consumer CQ event callback: 90*4882a593Smuzhiyun /* ... */ 91*4882a593Smuzhiyun return from CQ event handler 92*4882a593Smuzhiyun 93*4882a593Smuzhiyun The context in which completion event and asynchronous event 94*4882a593Smuzhiyun callbacks run is not defined. Depending on the low-level driver, it 95*4882a593Smuzhiyun may be process context, softirq context, or interrupt context. 96*4882a593Smuzhiyun Upper level protocol consumers may not sleep in a callback. 97*4882a593Smuzhiyun 98*4882a593SmuzhiyunHot-plug 99*4882a593Smuzhiyun-------- 100*4882a593Smuzhiyun 101*4882a593Smuzhiyun A low-level driver announces that a device is ready for use by 102*4882a593Smuzhiyun consumers when it calls ib_register_device(), all initialization 103*4882a593Smuzhiyun must be complete before this call. The device must remain usable 104*4882a593Smuzhiyun until the driver's call to ib_unregister_device() has returned. 105*4882a593Smuzhiyun 106*4882a593Smuzhiyun A low-level driver must call ib_register_device() and 107*4882a593Smuzhiyun ib_unregister_device() from process context. It must not hold any 108*4882a593Smuzhiyun semaphores that could cause deadlock if a consumer calls back into 109*4882a593Smuzhiyun the driver across these calls. 110*4882a593Smuzhiyun 111*4882a593Smuzhiyun An upper level protocol consumer may begin using an IB device as 112*4882a593Smuzhiyun soon as the add method of its struct ib_client is called for that 113*4882a593Smuzhiyun device. A consumer must finish all cleanup and free all resources 114*4882a593Smuzhiyun relating to a device before returning from the remove method. 115*4882a593Smuzhiyun 116*4882a593Smuzhiyun A consumer is permitted to sleep in its add and remove methods. 117