糖尿病康复 > 【RDMA】ibv_modify_qp()

【RDMA】ibv_modify_qp()

时间：2023-05-20 04:01:52

描述

ibv_modify_qp()修改队列对的属性。

更改的属性描述了QP的发送和接收属性。在UC和RC QP中，这意味着将QP与远程QP连接。

在Infiniband中，其中一个应该执行到子网管理员(SA)的路径查询(one should perform path query to the Subnet Administrator (SA))，以便确定应该为QP配置哪些属性或作为最佳解决方案，请使用Communication Manager(CM)或通用RDMA CM代理(CMA)连接QP。但是，有些应用程序倾向于自己连接QP，并通过sockets交换数据来决定要使用哪些QP属性。

在RoCE中，其中一个应该为连接的QP配置在QP属性中的GRH，或者为UD QP配置在地址句柄(Address Handle中)的GRH。

在iWARP中，只能使用通用RDMA CM代理(CMA)连接QP。

struct ibv_qp_attr描述了队列对的属性。

struct ibv_qp_attr描述QP的属性

struct ibv_qp_attr {enum ibv_qp_stateqp_state;enum ibv_qp_statecur_qp_state;enum ibv_mtupath_mtu;enum ibv_mig_statepath_mig_state;uint32_tqkey;uint32_trq_psn;uint32_tsq_psn;uint32_tdest_qp_num;intqp_access_flags;struct ibv_qp_capcap;struct ibv_ah_attrah_attr;struct ibv_ah_attralt_ah_attr;uint16_tpkey_index;uint16_talt_pkey_index;uint8_ten_sqd_async_notify;uint8_tsq_draining;uint8_tmax_rd_atomic;uint8_tmax_dest_rd_atomic;uint8_tmin_rnr_timer;uint8_tport_num;uint8_ttimeout;uint8_tretry_cnt;uint8_trnr_retry;uint8_talt_port_num;uint8_talt_timeout;};

struct ibv_qp_attr的完整说明

关于QP属性中某些特定值的一些警告：

在开发过程中，将attr.retry_cnt和attr.rnr_retry设置为0是一个好习惯。如果代码中存在竞争，最好在开发阶段检测它们attr.timeout中的值0表示一直等待ACK或NACK。这意味着，如果消息中的任何数据包丢失并且没有发送ACK或NACK，则不会发生任何重试，并且QP只会停止发送数据attr.rnr_retry中的值7表示在远程端发送RNR Nack时，发送消息重试次数无上限

struct ibv_qp_cap描述QP的size(用于发送和接收队列)。

struct ibv_qp_cap {uint32_tmax_send_wr;uint32_tmax_recv_wr;uint32_tmax_send_sge;uint32_tmax_recv_sge;uint32_tmax_inline_data;};

struct ibv_qp_cap: 说明

struct ibv_ah_attr描述了QP的地址向量

struct ibv_ah_attr:说明

struct ibv_global_routeAH中描述将要发送的数据包的GRH中使用的值

struct ibv_global_route属性说明:

attr_maskspecifies the QP attributes to be modified. It is either 0 or the bitwise OR of one or more of the following flags:

attr_mask指定要修改QP的哪些属性。它是0或以下一个或多个标志的按位或：

The following table specify the full supported transition for QP with service typeIBV_QPT_UDand the attributes it accepts:

下表指定了服务类型为IBV_QPT_UD的QP的支持的状态转换及其接受的属性：

The following table specify the full supported transition for QP with service typeIBV_QPT_UCand the attributes it accepts:

下表指定了服务类型为IBV_QPT_UD的QP的支持的状态转换及其接受的属性：

The following table specify the full supported transition for QP with service typeIBV_QPT_RCand the attributes it accepts:

下表指定了服务类型为IBV_QPT_RC的QP的支持的状态转换及其接受的属性：

参数

返回值

示例

1) Modify aUDQP from the RESET state to the RTS state:

将UDQP从RESET状态修改为RTS状态：

2) Modify aUCQP from the RESET state to the RTS state:

将UCQP从RESET状态修改为RTS状态：

3) Modify an RC QP from the RESET state to the RTS state:

将RCQP从RESET状态修改为RTS状态：

常见问题

为什么需要调用ibv_modify_qp()？

您需要调用ibv_modify_qp()才能将QP修改为可以接收和发送数据的状态

我可以将QP移到RTR状态并保持在该状态吗？

是的你可以。如果QP只是接收者

我如何知道要配置QP的哪些属性？

In Infiniband, you should perform path query to the SA or use CM or CMA, in iWARP, you should use the CMA

在Infiniband中，应该执行到SA的路径查询，或者使用CM或CMA；在iWARP中，应该使用CMA。

ibv_modify_qp()失败了，该QP的属性现在是什么？

与调用ibv_modify_qp()之前完全相同，未对QP进行任何更改。尽管该失败可能是由于某个错误导致RDMA设备将QP移至SQE或ERROR状态引起的

ibv_modify_qp()在通过RoCE已连接QP 进行INIT-> RTR转换时失败，这是什么原因？

使用RoCE时，必须配置GRH。

GRH在连接的QP中：作为QP属性的一部分

GRH在UD QP中：作为“地址句柄”属性的一部分

Igor R. says:February 11,

如果您能阐明以下主题，我将不胜感激。
根据“ Infinibeand网络体系结构”(*参见下面的引用)，在RDMA读取request期间，允许重试时从内存中重新读取响应者的RQlogic。是否可以通过设置retry_cnt = rnr_cnt = 0来完全禁用此功能？
换句话说，可以确保在这种情况下，单个RDMA 读取请求永远不会导致对任何远程位置的双重读取吗？ (如果这是设备内存，则这可能至关重要，并且访问可能会产生副作用。)
(I'd appreciate if you could shed some light on the following subject.
As per "Infinibeand Network Architecture"(* see quotations below), during RDMA Read request, the responder's RQ logic is allowed to re-read from memory when re-trying. Can we totally disable this functionality by setting retry_cnt=rnr_cnt=0?
In other words, can one be sure in such a case, that a single RDMA Read request will never cause double read of any remote location? (This can be crucial if it's a device memory, and an access might have side effects.
Thanks!)
* Infiniband Network Architecture, p.34:(Infiniband网络体系结构，第34页：)
<<
This service is referred to as "reliable" for the following reasons:
-- The destination QP's RQ Logic verifies the PSN in each request packet to ensure that all of the message's request packets are received in order, that none are missing, and, if any duplicate request packets are received, that they are only processed once. There is one exception; upon receipt of a duplicate memory read, the data is read from memory again.
>>
And in details (p.394):(还有详细说明(第394页)：)
<<
Effects of Retry on Requester and Responder The following list defines the sequence of events:
6. RQ Logic receives duplicate copy of request. When the second copy (the retried one) of the request packet eventually arrives at the RQ Logic, it has the same PSN as the original request packet. Its PSN, therefore, falls into the Ack’ d PSN region (also referred to as the Duplicate region), thereby identifying it as a duplicate request packet. The actions taken by the RQ Logic are determined by the request type:
— If it’ s a duplicate RDMA Read:
– The RQ Logic re-executes the read from its local memory.
– It sends back the requested read data in a series of one or more RDMA Read response packets.
– The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
– The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
– After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN

.ReplyIgor R. says:February 12,

好吧，这里的重试计数器似乎是“红鲱鱼”，因为真正的问题是远程HCA无法缓冲读取的数据。因此，在发生故障的情况下，即使不会发生自动重传，也将需要手动进行一次-以便无论如何将数据读取两次。因此，仅禁用自动重试是不够的，并且需要更复杂的方法。
Well, it seems that the retry counter is a "red herring" here, as the real problem is that remote HCA doesn't buffer the read data. So, in case of failure, even if the automatic retransmission won't occur, a manual one will be required - so that the data will be read twice anyway. Thus, solely disabling automatic retries is not sufficient, and more complicated approach is needed.

ReplyDotan Barak says:February 12,

我必须承认，我不太了解您要执行的操作...
是的，如果retry_cnt为0，则不会发生重传，
QP将在请求方进入错误状态。
这是您所需的预期行为吗？
如果您必须从一侧读取数据而无需从设备读取任何数据，
也许RDMA读取不是正确的方法
(也许将数据缓存在本地内存中并执行RDMA写操作会更好。)
谢谢
多坦
I must admit that I don't really understand what you are trying to perform...
Yes, if the retry_cnt will be 0 retransmission won't happen,
and the QP will get into the error state in the requester side.
Is this is the expected behavior that you need?
If you must read the data from one side without any read from the device,
maybe RDMA Read isn't the right approach
(maybe cache the data in local memory and perform RDMA Write is better..)
Thanks
Dotan

Igor R. says:February 12,

感谢您回复。
在我的情况下，响应方(设备所在的位置)必须尽可能保持被动状态。我只是想考虑各种可能性并了解其后果...
Thanks for the input.
In my case the responding side (where the device is) must remain as passive as possible. I'm just trying to consider various possibilities and to understand their consequences...

Alan says:August 29,

Hi Dotan,

Do we have to memset the ibv_qp_attr structure before using it in the ibv_modify_qp() function? I am thinking there are flags in the function API to indicate which fields are being used why we still need to memset it?

Thanks.

Dotan Barak says:August 29,

Hi Alan.

In general - you are correct (and in most cases it will work without any problem).

However:

* Most ibv_* structu don't have bitmask to specify the valid attributes

* The struct ibv_qp_attr *may* be extended (although the developers try hard to keep backward binary compatibility), and in that case calling memset() on the struct will keep the behavior as it was in the past

I believe, as a developer, in defensive programming and this is a good practice to handle API which is evolving.

I hope that I answered your question.

Thanks

Dotan

Yanfang Le says:October 12,

Hi Dotan,

I am trying to flush the work requests in qp, so I first turn the qp state to error state, and then convert to the RTS state. During this process, I don't know where can I get the ah_attr information. Could you tell me how can I get the ah_attr information? Thanks.

I am trying to copy the information from the old qp (before turn the state to error), however, it doesn't work.

Dotan Barak says:October 17,

Hi.

When the QP is in ERROR state,

you cannot trust the attributes which were queried from it, expect for the QP state.

If you must keep the AH attributes, maybe you can:

1) Save a local copy of the attributes as a function of the QP number

2) Use the qp_context pointer to hold a buffer which holds the AH attributes.

Thanks

Dotan

Haomai says:January 21,

Hi, if I make a connected qp to error state, is it possible that peer side can detect this.

For example, if we use tcp socket to communicate, if we close one side's socket, other side will get "0" when read. So peer side can know peer socket already closed. Is it possible that rdma can achieve this?

Thanks!

Dotan Barak says:January 23,

Hi Haomai.

In RDMA there aren't any keep-alive messages that send automatically,

if needed you need to perform this in your application.

If you close one side, only if you are using rdmacm, the remote side will be informed on this;

but not in "standard" data verbs.

If the other side (which didn't move to Error) won't send any message - it won't know that its remote

QP moved to Error.

Thanks

Dotan

ReplyAdrian says:March 23,

Hi Dotan,

I've just started with verbs and when I try to modify a qp from INIT to RTR state, a 101 error is returned. The 101 error means #define ENETUNREACH 101 /* Network is unreachable */. The values are exchanged via TCP sockets (lid=4 qp_num=116 psn=9105394 from client to server and lid=3 qp_num=116 psn=14618275 from server to client). I don't know what is happening and there is difficult to obtain information.

Thanks

ReplyDotan Barak says:March 24,

Hi Adrian.

Are you using explicit ibv_modify_qp(() or do you call librdmacm function?

Are you using IB or RoCE?

Thanks

Dotan

ReplyAdrian says:March 24,

I'm using IB and explicit ibv_modify_qp()Dotan Barak says:March 24,

It is weird; since this return value isn't really returned ...

Which QP transport type are you using?

Can you please write explicit the values that you set + the used mask in the ibv_modify_qp()?

Thanks

DotanAdrian says:March 24,

Hi Dotan,

the code is

attr.qp_state = IBV_QPS_RTR;

attr.path_mtu = mtu;

attr.dest_qp_num = rmsg.qp_num[0];

attr.rq_psn = rmsg.psn[0];

attr.ah_attr.dlid = rmsg.lid;

attr.max_dest_rd_atomic = 1;

attr.min_rnr_timer = 12;

attr.ah_attr.is_global = 0;

attr.ah_attr.sl = 0;

attr.ah_attr.src_path_bits = 0;

if (ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |

IBV_QP_DEST_QPN | IBV_QP_RQ_PSN | IBV_QP_MIN_RNR_TIMER |

IBV_QP_MAX_DEST_RD_ATOMIC)){

cerr << "Failed to modify QP 1 to RTR." << endl;

abort();

}

lid, qp_num and psn are sent via TCP sockets.

Thanks

ReplyDotan Barak says:March 26,

Hi.

I assume that you are using RC QP.

What about attr.ah_attr.port_num?

Thanks

Dotan

ReplyAdrian says:March 31,

Yes, I use RC QP and the port number is 1. Finally, we reinstall the OFED drivers and it works.

Thank you for your time.Dotan Barak says:March 31,

Sure.

Maybe there were some inconsistencies between the various libraries..

I'm glad at the end everything worked for you

DotanSangwook says:April 14,

Hi Adrian & Dotan. Same error happens in my code too.

Explicit ibv_modify_qp call to change QP state to RTR fails and returns 101. (My code is based on ibv_rc_pingpong program, an example from libibverb library. And ibv_rc_pingpong also fails in my environment.)

You said the problem is solved after reinstalling OFED drivers.. Can you share your system environment and OFED version?

(I installed OFED ver 2.4-1.0.4 on Intel Xeon E5-2670 + MLNX ConnectX-3 machine running Ubuntu 14.04 with kernel 3.13.0-48.)

ReplySangwook says:April 14,

Hi, I wrote a comment to ask for Adrian's system environment. But I figured out the cause of my problem- I had to add "gid-index" option in my program because I'm using RoCE. My program worked fine with that option. Anyway, thanks!

ReplyDotan Barak says:April 16,

I'm glad that everything is fine.

I'll add this note on RoCE, so people won't fail on it again...

Thanks

DotanJesus Camachosays:May 5,

Hi Dotan!

I am running the ib_write_lat microbenchmark with OpenSM.

I am disabling a link while running them and then OpenSM reconfigures all the network and the benchmark continues working without problems.

If have reduce the number of retries in the QPs to 1 or 2, then ib_wirte_lat crash in some cases now.

However, if I change the timeout of the QPs, I have the same behaviour.

I expected to have less retries if I increase the QP timeout. I understand that every retry is triggered after the QP timeout expires.

Am i right?

If not, how is it affecting the QP timeout to the number of retries?

Thanks for your help!

Best regards,

Jesus Camacho

ReplyDotan Barak says:May 5,

Hi Jesus.

Yes, you are right: every retry is triggered after a QP timeout expires.

There is an internal clock in the RDMA device, and at least after the timeout expires

(+ a delta which depends on this clock resolution), there is a retransmission.

If (in your experience) the link will be stable until the last retransmission,

ib_write_lat will continue working..

Thanks

Dotan

If until the

What is "important" is the

ReplyJesus Camachosays:May 6,

Hi Dotan,

I am replacing a LFT instead of switching on the link back.

It takes less than 2 ms from the time I send the new block to the LFT and I receive its ACK later. Then the new path should be ready in this short period of time.

In one of my experiments I am using only one retry and timeout 10 (4194.304 usec).

Although I spend only 4 ms until the next retransmission, I have to spend up to 0.5 seconds for a successful completion when polling the completion queue (ibv_poll_cq using RC).

Do you know why this takes all this time?

Thank you!

JesusDotan Barak says:May 7,

Hi Jesus.

What does LFT mean? (I don't really familiar with this term)

Are you performing Automatic Path Migartion?

What are the values of the timeout/retry_count?

Thanks

DotanJesus Camachosays:May 8,

Hi Dotan,

instead of using Automatic Path Migration, I am replacing the forwarding tables (trough OpenSM) in some of the switches to avoid the broken link (I just say the switches to take another port to avoid the link that is down). This takes about 2 milliseconds.

I am using this values in the ib_write_lat microbenchmark:

timeout 10

retry_cnt 1

This seems to work (I am using a small network). Then I understand that the retransmission is OK in less than timeout+timeout.

I am measuring the time to send a packet from one node to another and this takes usually several milliseconds.

However, it takes longer after the fault. And most of the time is spent polling the completion queue -ibv_poll_cq- (up to 0.5 seconds).

Do you know what could be the reason for this?

Thanks for your time :-)

Jesus

ReplyDotan Barak says:May 8,

You are welcome

I don't have a solid answer here (from knowledge), but I have an assumption:

RDMA is very fine tune to handle good flow (since this happen most of the time).

In case of error flow starts (i.e. retransmission, violation, etc.), maybe this flow isn't fully tuned.

Another option is that this happening because of other issues

(for example: is the NUMA node is local or remote? do you use CPU affinity to the running process? etc.)

I would suggest to use a sniffing tool and try to measure this (when the message is send, when an ack is returned, and when there is a Work Completion available in the CQ) or send an email to the support team of the HW vendor about this.

BTW, making the SM change the subnet may cause the subnet to start recovery flows

(since *maybe* client reregister events are created in the subnet's nodes).

Thanks

Dotan

Anonymous says:October 1,

Hi Dotan,

When I tried to modify retry_cnt to 7 using ibv_modify_qp,

I got "Invalid argument". Do you have any suggestion ?

I listed my code here:

struct ibv_qp_attr attr;

int qret;

attr.retry_cnt = (uint8_t)7;

attr.timeout = (uint8_t)14;

attr.rnr_retry = (uint8_t)7;

qret = ibv_modify_qp(id->qp, &attr, IBV_QP_RETRY_CNT|IBV_QP_TIMEOU T|IBV_QP_RNR_RETRY);

Thanks

ReplyDotan Barak says:October 1,

Hi.

From which QP state to which QP state did you try to do this?

I suspect that you need more flags in the QP transition that you made.

Thanks

Dotan

ReplyJunhyun says:August 12,

Hi Dotan, I recently built an event-driven RDMA send/recv IO engine for multiplexing multiple QP events and so far I worked out most kinks, but it suffers when more than one connection is introduced.

What frustrates me is that the connection works perfectly until I get the WC for my first WR! (QP transition and post_send doesn't return any errors, even with the second connection)

The second connection keeps throwing me status 12 (RETRY_EXC_ERROR) WCs while the first connection keeps on exchanging data smoothly. When I retry the post_send, I get status 5, vendor_err 249. I took it to mean that the QP has already transitioned to an error state.

Environment:

I use dual-port Mellanox CX-3 EN 10GBe on RoCE. Port 1 acts both as the port for my shell connection and RoCE.

I have made it so that all QPs from the same device share device context and pd, but they use disjoint MRs.

Here's the code I use for creating and transitioning QPs.

Every created QPs follow this exact path.

I wish I could find more explanation as to why this keeps failing.

Please tell me if I'm doing anything wrong.

conn.port is the local port I chose for use, and remote_info members are obtained from the other host over TCP connection. (byte-reordered)

Again, thanks for maintaining such a helpful site.

struct ibv_qp_init_attr qp_init_attr;

memset(&qp_init_attr, 0, sizeof(qp_init_attr));

qp_init_attr.qp_type = IBV_QPT_RC;

qp_init_attr.sq_sig_all = 1;

qp_init_attr.send_cq = conn.cq;

qp_init_attr.recv_cq = conn.cq;

qp_init_attr.cap.max_send_wr = DEFAULT_CQ_SIZE / 2;

qp_init_attr.cap.max_recv_wr = DEFAULT_CQ_SIZE / 2;

qp_init_attr.cap.max_send_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);

qp_init_attr.cap.max_recv_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);

CHECK_PTR(conn.qp = ibv_create_qp(conn.pd, &qp_init_attr), "Failed to create QP for device " << device << ", port " << port << ", error=" << strerror(errno) << ", pd=" < INIT

struct ibv_qp_attr to_init;

int flags;

memset(&to_init, 0, sizeof(to_init));

to_init.qp_state = IBV_QPS_INIT;

to_init.port_num = conn.port_num;

to_init.pkey_index = 0;

to_init.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE;

flags = IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS;

CHECK_EQ0(ibv_modify_qp(conn.qp, &to_init, flags), "QP transition failed (RESET -> INIT): " <RTR

struct ibv_qp_attr to_rtr;

int flags;

memset(&to_rtr, 0, sizeof(to_rtr));

to_rtr.qp_state = IBV_QPS_RTR;

to_rtr.path_mtu = DEFAULT_QP_MTU;

to_rtr.dest_qp_num = remote_info.qp_num;

to_rtr.rq_psn = 0;

to_rtr.max_dest_rd_atomic = 0;

to_rtr.min_rnr_timer = 12;

to_rtr.ah_attr.dlid = remote_info.lid;

to_rtr.ah_attr.sl = 0;

to_rtr.ah_attr.src_path_bits = 0;

to_rtr.ah_attr.port_num = conn.port_num;

// Begin RoCE specific conf (currently universal)

to_rtr.ah_attr.is_global = 1;

memcpy(&to_rtr.ah_attr.grh.dgid, remote_info.gid, sizeof(remote_info.gid));

to_rtr.ah_attr.grh.flow_label = 0;

to_rtr.ah_attr.grh.hop_limit = 1;

to_rtr.ah_attr.grh.sgid_index = 0;

to_rtr.ah_attr.grh.traffic_class = 0;

flags = IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |

IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER;

CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rtr, flags), "QP transition failed (INIT -> RTR): " < RTS

struct ibv_qp_attr to_rts;

int flags;

memset(&to_rts, 0, sizeof(to_rts));

to_rts.qp_state = IBV_QPS_RTS;

to_rts.timeout = 14;

to_rts.retry_cnt = 7;

to_rts.rnr_retry = 7;

to_rts.sq_psn = 0;

to_rts.max_rd_atomic = 0;

flags = IBV_QP_STATE | IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT |

IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC;

CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rts, flags), "QP transition failed (RTR -> RTS)" << getQPDescription(local_info));

ReplyDotan Barak says:August 18,

Hi.

Please make the following:

1) verify that the DEFAULT_QP_MTU is less than the interface MTU

(i.e. if the network I/F MTU is 1500, this value should be 1024).

2) All remote attributes are correct

3) Make sure that there is a synchronization between both sides

(to prevent a case where the sender send a message and the receiver isn't in RTR state,

or in Error state).

Thanks

Dotan

Arvind says:April 10,

Why does the state have to go to RTR and then to RTS. Can't we just switch from Init to RTS?

ReplyDotan Barak says:April 11,

Hi.

* When transitioning to RTR - only the receive attributes are filled (the QP can stay at this state, if it will only receive messages)

* When transitioning to RTS - only the send attributes are filled

At the end of the day, this is the what the InfiniBand spec specifies...

Thanks

Dotan

Replyqiuhaonan says:April 19,

Hello Dotan.

I have met an interesting problem. At first, I connected two QPs by exchanging the meta info(QPN, GID and etc) and modifying them to RTS. Then exchanging some data on them. Finally, I closed one QP by ibv_destroy_qp and the other QP got errors when sending data to the closed QP, but when I tried to modify the error QP to RTS, it could return to the RTS state and got errors when trying to send data again. The curious thing was why the error QP could be modified to RTS state when the remote target QP had been destroyed. Aren't there some SYN-like packets to be exchanged between the two QPs? Thanks.

Haonan

ReplyDotan Barak says:April 19,

Hi.

I have a feeling that you used Reliable Connected QP.

When you got the error in the QP, you transitioned it back to RTS;

however, there wasn't any remote side - so you got (again) transport error.

No, there isn't any "SYN" like packets - they are replaced with Communication Manager

(but you didn't use the Communication Manager, you replaced it with a socket).

The behavior is as expected

Thanks

Dotan

Replyqiuhaonan says:April 19,

Hi Dotan.

Yes, I used RC QP and verbs for this test.

Thanks.

ReplyWeizhi Liao says:December 30,

Hi Dotan：

After reading this blog，I feel confused about the required and optional attributes of QP on status transitions 。 the Linux manpages 、rdma_aware_network_programming_user_manual and your blog have different description 。which is right? Thanks

ReplyDotan Barak says:February 22,

Hi.

What do you mean?

Can you give me an example?

Thanks

Dotan

Replyraphael says:May 21,

Hi Dotan

I will use an RC queue pair and Write verb from an FPGA device (w/o RDMA network card, 100GbE MAC). For simulation purpose with a workstation and a connectx5 I use python / Scapy to build a RoCEv2 Write_IMM packet from the data and receiver qp_num, rkey, addr, etc. The receiver is also a Connectx-5.

it works, I get the data and the Immediate value, but only for 8 transfers. The last RC acknowledge from receiver is a RNR Nak.

Is it possible to configure the Receiver HCA to continuously recive the data ?

thanks

ReplyDotan Barak says:July 10,

Hi.

RNR Nak means that the "Receiver Not Ready", this means that there isn't any Receiver Request in that (responder) QP.

So, I suggest that you'll post more Receive Requests for that side.

Thanks

Dotan

ReplyXiangyu Zhang says:July 13,

Hi Dotan:

These days I encounter an error:after run ibv_post_send to issue an RDMA READ operation, there is nothing happened, this means that the RDMA READ operation don't appears in CQ. After to read "Help, I've posted and Send Request and it wasn't completed with a corresponding Work Completion. What happened?" in "ibv_post_send" page, I decide to try advise in it. The problem is I can't modify qp attribute, the ibv_modity_qp return -22 all the time, below is my code:

int ret = 0;

rdma_addrinfo* addrinfo;

rdma_addrinfo hints = {};

struct ibv_qp_attr qp_attr;

hints.ai_port_space = RDMA_PS_TCP;

struct ibv_qp *client_qp = nullptr;

if (rdma_getaddrinfo(const_cast(host.c_str()),

const_cast(port.c_str()), &hints, &addrinfo)) {

return errors::InvalidArgument(

strerror(errno), ": ", "cannot connect to rdma://", host, ":", port);

}

ibv_qp_init_attr init_attr = {};