What is the hardware queue pair?

The hardware queue pair is the best way to understand nvme/spdk. Only by deep understanding can we make good use of nvme.

From the perspective of nvme controller registers

As the name implies, it is a queue composed of some hardware registers.

Empty queue

Full queue

• Question:
  Can you join and leave the team concurrently? No

submission hardware queue entry

Each entry is shown in the following table:

• Entry:

host software submission Command to the tail entry, by operating the submission hardware queue tail doorbell register;

• Dequeue:
  hardware takes the head entry in turn to process

completion hardware hardware queue entry

each The main fileds of entry are as follows:

• Join the team
  After the hardware has processed the nvme command obtained above, put the corresponding completed information in the completion queue head entry;

• Dequeue
  After the host software is interrupted or actively polled until there are new entries added on it, go to completion queue tail to get the processing result of the most recently submitted command.

This is mainly to operate the completion queue doorbell register to achieve:

queue pair

There are multiple submission hardware queues and multiple completion hardware queues inside the nvme controller. When submitting and executing NVME commands, you need to use the entries in the above two hardware queues and use them in pairs.

From the perspective of nvme controller architecture

The hardware queue pair is a waterwheel for extracting “nvme command”. Design to the submission/execution and completion of NVME instructions.

Overview

Here is a classic picture:

NVME command submission

NVME has finished processing the command

Package of hardware queue pair by spdk

IO request submission function

• spdk_nvme_ns_cmd_read

1. Production request
   _nvme_ns_cmd_rw(ns, qpair, &payload, ….)

2. Submit a request
   nvme_qpair_submmit_request()

   2.1 nvme_transport_qpair_submit_request(qpair, request)
   2.1.1 nvme_pcie_qpair_submit_request (qpair, request)
   2.1.1.1 nvme_pcietig_qrequest, request, qpair, );
   2.1.1.2 nvme_pcie_qpair_submit_tracker(qpair, tr)

nvme_pcie_qpair_submit_tracker(struct spdk_nvme_qpair *qpair, struct nvme _tracker *tr)
{
 struct nvme_request *req;
 struct nvme_pcie_qpair *pqpair = nvme_pcie_qpair(qpair);
 struct nvme_pcie_ctrlr *pctrlr = nvmer_pcie_ctrlr(qpair);pair
 br />
 req = tr->req;
 assert(req != NULL);
 req->timed_out = false;
 if (spdk_unlikely(pctrlr->ctrlr. timeout_enabled)) {
 req->submit_tick = spdk_get_ticks();
} else {
 req->submit_tick = 0;
 }

 pqpair- >tr[tr->cid].active = true;

 /* Copy the command from the tracker to the submission queue. */
 nvme_pcie_copy_command(&pqpair->cmd[pqpair-> sq_tail], &req->cmd);

 if (spdk_unlikely(++pqpair->sq_tail == pqpair->num_entries)) {
 pqpair->sq_tail = 0;
 }

 if (spdk_unlikely(pqpair->sq_tail == pqpair->sq_head)) {
 SPDK_ERRLOG("sq_tail is pa ssing sq_head!
");
 }

 spdk_wmb();
 if (spdk_likely(nvme_pcie_qpair_update_mmio_required(qpair,
 pqpair->sq_tail,
 pqpair->sq_shadow_tdbl,
 pqpair->sq_eventidx))) {
 g_thread_mmio_ctrlr = pctrlr;
 spdk_mmio_write_4(pqpair->sq_tdbl, pqpair->sq_tail); // <--- pqpair->sq_tail); /> g_thread_mmio_ctrlr = NULL;
 }
}

As you can see from the code path above, spdk nvme needs to operate hardware registers, and there is no lock and atomic operation all the way. Need upper protection.

IO completion query function

spdk_nvme_qpair_process_completions

The cq_hdbel corresponding to the same operation as above, no lock and atomic operation, requires upper layer protection to prevent concurrent modification of the doorbell register.

How to make good use of multiple hardware queue pairs

It is necessary to manage the correspondence between hardware queue pairs and threads and processor cores in practical applications to prevent multiple threads from accessing a hardware queue at the same time .