|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|
|
NAME | SYNOPSIS | DESCRIPTION | USAGE | EXAMPLES | NOTES | SEE ALSO | AUTHORS | COLOPHON |
|
|
|
mlx5dv_wr_mkey_configure(3) mlx5dv_wr_mkey_configure(3)
mlx5dv_wr_mkey_configure - Create a work request to configure an
MKEY
mlx5dv_wr_set_mkey_access_flags - Set the memory protection
attributes for an MKEY
mlx5dv_wr_set_mkey_layout_list - Set a memory layout for an MKEY
based on SGE list
mlx5dv_wr_set_mkey_layout_interleaved - Set an interleaved memory
layout for an MKEY
#include <infiniband/mlx5dv.h>
static inline void mlx5dv_wr_mkey_configure(struct mlx5dv_qp_ex *mqp,
struct mlx5dv_mkey *mkey,
uint8_t num_setters,
struct mlx5dv_mkey_conf_attr *attr);
static inline void mlx5dv_wr_set_mkey_access_flags(struct mlx5dv_qp_ex *mqp,
uint32_t access_flags);
static inline void mlx5dv_wr_set_mkey_layout_list(struct mlx5dv_qp_ex *mqp,
uint16_t num_sges,
const struct ibv_sge *sge);
static inline void mlx5dv_wr_set_mkey_layout_interleaved(struct mlx5dv_qp_ex *mqp,
uint32_t repeat_count,
uint16_t num_interleaved,
const struct mlx5dv_mr_interleaved *data);
The MLX5DV MKEY configure API and the related setters
(mlx5dv_wr_set_mkey*) are an extension of IBV work request API
(ibv_wr*) with specific features for MLX5DV MKEY.
MKEYs allow creation of virtually-contiguous address spaces out of
non-contiguous chunks of memory regions already registered with
the hardware. Additionally it provides access to some advanced
hardware offload features, e.g. signature offload.
These APIs are intended to be used to access additional
functionality beyond what is provided by mlx5dv_wr_mr_list() and
mlx5dv_wr_mr_interleaved(). The MKEY features can be optionally
enabled using the mkey configure setters. It allows using
different features in the same MKEY.
To use these APIs a QP must be created using mlx5dv_create_qp(3)
which allows setting the MLX5DV_QP_EX_WITH_MKEY_CONFIGURE in
send_ops_flags.
The MKEY configuration work request is created by calling
mlx5dv_wr_mkey_configure(), a WR builder function, followed by
required setter functions. num_setters is a number of required
setters for the WR. All setters are optional. num_setters can be
zero to apply attr only. Each setter can be called only once per
the WR builder.
The WR configures mkey and applies attr of the builder function
and setter functions’ arguments for it. If mkey is already
configured, the WR overrides some mkey properties depends on
builder and setter functions’ arguments (see details in setters’
description). To clear configuration of mkey, use ibv_post_send()
with IBV_WR_LOCAL_INV opcode or ibv_wr_local_inv().
Current implementation requires the IBV_SEND_INLINE option to be
set in wr_flags field of ibv_qp_ex structure prior to builder
function call. Non-inline payload is currently not supported by
this API. Please note that inlining here is done for MKEY
configuration data, not for user data referenced by data layouts.
Once MKEY is configured, it may be used in subsequent work
requests (SEND, RDMA_READ, RDMA_WRITE, etc). If these work
requests are posted on the same QP, there is no need to wait for
completion of MKEY configuration work request. They can be posted
immediately after the last setter (or builder if no setters).
Usually there is no need to even request a completion for MKEY
configuration work request.
If completion is requested for MKEY configuration work request it
will be delivered with the IBV_WC_DRIVER1 opcode.
Builder function
mlx5dv_wr_mkey_configure()
Post a work request to configure an existing MKEY. With
this call alone, it is possible to configure the MKEY and
keep or reset signature attributes. This call may be
followed by zero or more optional setters.
mqp The QP to post the work request on.
mkey The MKEY to configure.
num_setters
The number of setters that must be called after this
function.
attr The MKEY configuration attributes
MKEY configuration attributes
MKEY configuration attributes are provided in
mlx5dv_mkey_conf_attr structure.
struct mlx5dv_mkey_conf_attr {
uint32_t conf_flags;
uint64_t comp_mask;
};
conf_flags
Bitwise OR of the following flags:
MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR
Reset the signature attributes of the MKEY. If not
set, previously configured signature attributes will
be kept.
comp_mask
Reserved for future extension, must be 0 now.
Generic setters
mlx5dv_wr_set_mkey_access_flags()
Set the memory protection attributes for the MKEY. If the
MKEY is configured, the setter overrides the previous
value. For example, two MKEY configuration WRs are posted.
The first one sets IBV_ACCESS_REMOTE_READ. The second one
sets IBV_ACCESS_REMOTE_WRITE. In this case, the second WR
overrides the memory protection attributes, and only
IBV_ACCESS_REMOTE_WRITE is allowed for the MKEY when the WR
is completed.
mqp The QP where an MKEY configuration work request was
created by mlx5dv_wr_mkey_configure().
access_flags
The desired memory protection attributes; it is
either 0 or the bitwise OR of one or more of flags
in enum ibv_access_flags.
Data layout setters
Data layout setters define how data referenced by the MKEY will be
scattered/gathered in the memory. In order to use MKEY with RDMA
operations, it must be configured with a layout.
Not more than one data layout setter may follow builder function.
Layout can be updated in the next calls to builder function.
When MKEY is used in RDMA operations, it should be used in a
zero-based mode, i.e. the addr field in ibv_sge structure is an
offset in the total data.
mlx5dv_wr_set_mkey_layout_list()
Set a memory layout for an MKEY based on SGE list. If the
MKEY is configured and the data layout was defined by some
data layout setter (not necessary this one), the setter
overrides the previous value.
Default WQE size can fit only 4 SGE entries. To allow
more, the QP should be created with a larger WQE size that
may fit it. This should be done using the max_inline_data
attribute of struct ibv_qp_cap upon QP creation.
mqp The QP where an MKEY configuration work request was
created by mlx5dv_wr_mkey_configure().
num_sges
Number of SGEs in the list.
sge Pointer to the list of ibv_sge structures.
mlx5dv_wr_set_mkey_layout_interleaved()
Set an interleaved memory layout for an MKEY. If the MKEY
is configured and the data layout was defined by some data
layout setter (not necessary this one), the setter
overrides the previous value.
Default WQE size can fit only 3 interleaved entries. To
allow more, the QP should be created with a larger WQE size
that may fit it. This should be done using the
max_inline_data attribute of struct ibv_qp_cap upon QP
creation.
As one entry will be consumed for strided header, the MKEY
should be created with one more entry than the required
num_interleaved.
mqp The QP where an MKEY configuration work request was
created by mlx5dv_wr_mkey_configure().
repeat_count
The data layout representation is repeated
repeat_count times.
num_interleaved
Number of entries in the data representation.
data Pointer to the list of interleaved data layout
descriptions.
Interleaved data layout is described by
mlx5dv_mr_interleaved structure.
struct mlx5dv_mr_interleaved {
uint64_t addr;
uint32_t bytes_count;
uint32_t bytes_skip;
uint32_t lkey;
};
addr Start address of the local memory buffer.
bytes_count
Number of data bytes to put into the buffer.
bytes_skip
Number of bytes to skip in the buffer before the
next data block.
lkey Key of the local Memory Region
Signature setters
The signature attributes of the MKEY allow
adding/modifying/stripping/validating integrity fields when
transmitting data from memory to network and when receiving data
from network to memory.
Use the signature setters to set/update the signature attributes
of the MKEY. To reset the signature attributes without
invalidating the MKEY, use the
MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR flag.
mlx5dv_wr_set_mkey_sig_block()
Set MKEY block signature attributes. If the MKEY is
already configured with the signature attributes, the
setter overrides the previous value. See dedicated man
page for mlx5dv_wr_set_mkey_sig_block(3).
Crypto setter
The crypto attributes of the MKey allow encryption and decryption
of transmitted data from memory to network and when receiving data
from network to memory.
Use the crypto setter to set/update the crypto attributes of the
MKey. When the MKey is created with
MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO it must be configured with
crypto attributes before the MKey can be used.
mlx5dv_wr_set_mkey_crypto()
Set MKey crypto attributes. If the MKey is already
configured with crypto attributes, the setter overrides the
previous value. see dedicated man page for
mlx5dv_wr_set_mkey_crypto(3).
Create QP and MKEY
Code below creates a QP with MKEY configure operation support and
an indirect mkey.
/* Create QP with MKEY configure support */
struct ibv_qp_init_attr_ex attr_ex = {};
attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
struct mlx5dv_qp_init_attr attr_dv = {};
attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS;
attr_dv.send_ops_flags = MLX5DV_QP_EX_WITH_MKEY_CONFIGURE;
ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
mkey_attr.create_flags = MLX5DV_MKEY_INIT_ATTR_FLAGS_INDIRECT;
struct mlx5dv_mkey *mkey = mlx5dv_create_mkey(&mkey_attr);
List data layout configuration
Code below configures an MKEY which allows remote access for read
and write and is based on SGE list layout with two entries. When
this MKEY is used in RDMA write operation, data will be scattered
between two memory regions. The first 64 bytes will go to memory
referenced by mr1. The next 4096 bytes will go to memory
referenced by mr2.
ibv_wr_start(qpx);
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_INLINE;
struct mlx5dv_mkey_conf_attr mkey_attr = {};
mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
struct ibv_sge sgl[2];
sgl[0].addr = mr1->addr;
sgl[0].length = 64;
sgl[0].lkey = mr1->lkey;
sgl[1].addr = mr2->addr;
sgl[1].length = 4096;
sgl[1].lkey = mr2->lkey;
mlx5dv_wr_set_mkey_layout_list(mqpx, 2, sgl);
ret = ibv_wr_complete(qpx);
Interleaved data layout configuration
Code below configures an MKEY which allows remote access for read
and write and is based on interleaved data layout with two entries
and repeat count of two. When this MKEY is used in RDMA write
operation, data will be scattered between two memory regions. The
first 512 bytes will go to memory referenced by mr1 at offset 0.
The next 8 bytes will go to memory referenced by mr2 at offset 0.
The next 512 bytes will go to memory referenced by mr1 at offset
516. The next 8 bytes will go to memory referenced by mr2 at
offset 8.
ibv_wr_start(qpx);
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_INLINE;
struct mlx5dv_mkey_conf_attr mkey_attr = {};
mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
struct mlx5dv_mr_interleaved data[2];
data[0].addr = mr1->addr;
data[0].bytes_count = 512;
data[0].bytes_skip = 4;
data[0].lkey = mr1->lkey;
data[1].addr = mr2->addr;
data[1].bytes_count = 8;
data[1].bytes_skip = 0;
data[1].lkey = mr2->lkey;
mlx5dv_wr_set_mkey_layout_interleaved(mqpx, 2, 2, &data);
ret = ibv_wr_complete(qpx);
A DEVX context should be opened by using mlx5dv_open_device(3).
mlx5dv_create_mkey(3), mlx5dv_create_qp(3),
mlx5dv_wr_set_mkey_sig_block(3), mlx5dv_wr_set_mkey_crypto(3)
Oren Duer ⟨oren@nvidia.com⟩
Sergey Gorenko ⟨sergeygo@nvidia.com⟩
Evgenii Kochetov ⟨evgeniik@nvidia.com⟩
This page is part of the rdma-core (RDMA Core Userspace Libraries
and Daemons) project. Information about the project can be found
at ⟨https://github.com/linux-rdma/rdma-core⟩. If you have a bug
report for this manual page, send it to
linux-rdma@vger.kernel.org. This page was obtained from the
project's upstream Git repository
⟨https://github.com/linux-rdma/rdma-core.git⟩ on 2025-08-11. (At
that time, the date of the most recent commit that was found in
the repository was 2025-08-04.) If you discover any rendering
problems in this HTML version of the page, or you believe there is
a better or more up-to-date source for the page, or you have
corrections or improvements to the information in this COLOPHON
(which is not part of the original manual page), send a mail to
man-pages@man7.org
mlx5dv_wr_mkey_configure(3)
|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|