What is the hardware queue pair?
The hardware queue pair is the best way to understand nvme/spdk. Only by deep understanding can we make good use of nvme.
From the perspective of nvme controller registers
As the name implies, it is a queue composed of some hardware registers.
Empty queue
Full queue
- Question:
Can you join and leave the team concurrently? No
submission hardware queue entry
Each entry is shown in the following table:
- Entry:
host software submission Command to the tail entry, by operating the submission hardware queue tail doorbell register;
- Dequeue:
hardware takes the head entry in turn to process
completion hardware hardware queue entry
each The main fileds of entry are as follows:
p>
-
Join the team
After the hardware has processed the nvme command obtained above, put the corresponding completed information in the completion queue head entry; - Dequeue
After the host software is interrupted or actively polled until there are new entries added on it, go to completion queue tail to get the processing result of the most recently submitted command.
This is mainly to operate the completion queue doorbell register to achieve:
queue pair
There are multiple submission hardware queues and multiple completion hardware queues inside the nvme controller. When submitting and executing NVME commands, you need to use the entries in the above two hardware queues and use them in pairs.
From the perspective of nvme controller architecture
The hardware queue pair is a waterwheel for extracting “nvme command”. Design to the submission/execution and completion of NVME instructions.
Overview
Here is a classic picture:
NVME command submission
NVME has finished processing the command
Package of hardware queue pair by spdk
IO request submission function
- spdk_nvme_ns_cmd_read
-
Production request
_nvme_ns_cmd_rw(ns, qpair, &payload, ….) -
Submit a request
nvme_qpair_submmit_request()2.1 nvme_transport_qpair_submit_request(qpair, request)
2.1.1 nvme_pcie_qpair_submit_request (qpair, request)
2.1.1.1 nvme_pcietig_qrequest, request, qpair, );
2.1.1.2 nvme_pcie_qpair_submit_tracker(qpair, tr)
nvme_pcie_qpair_submit_tracker(struct spdk_nvme_qpair *qpair, struct nvme _tracker *tr)
{
struct nvme_request *req;
struct nvme_pcie_qpair *pqpair = nvme_pcie_qpair(qpair);
struct nvme_pcie_ctrlr *pctrlr = nvmer_pcie_ctrlr(qpair);pair
br />
req = tr->req;
assert(req != NULL);
req->timed_out = false;
if (spdk_unlikely(pctrlr->ctrlr. timeout_enabled)) {
req->submit_tick = spdk_get_ticks();
} else {
req->submit_tick = 0;
}
pqpair- >tr[tr->cid].active = true;
/* Copy the command from the tracker to the submission queue. */
nvme_pcie_copy_command(&pqpair->cmd[pqpair-> sq_tail], &req->cmd);
if (spdk_unlikely(++pqpair->sq_tail == pqpair->num_entries)) {
pqpair->sq_tail = 0;
}
if (spdk_unlikely(pqpair->sq_tail == pqpair->sq_head)) {
SPDK_ERRLOG("sq_tail is pa ssing sq_head! ");
}
spdk_wmb();
if (spdk_likely(nvme_pcie_qpair_update_mmio_required(qpair,
pqpair->sq_tail,
pqpair->sq_shadow_tdbl,
pqpair->sq_eventidx))) {
g_thread_mmio_ctrlr = pctrlr;
spdk_mmio_write_4(pqpair->sq_tdbl, pqpair->sq_tail); // <--- pqpair->sq_tail); /> g_thread_mmio_ctrlr = NULL;
}
}
As you can see from the code path above, spdk nvme needs to operate hardware registers, and there is no lock and atomic operation all the way. Need upper protection.
IO completion query function
spdk_nvme_qpair_process_completions
The cq_hdbel corresponding to the same operation as above, no lock and atomic operation, requires upper layer protection to prevent concurrent modification of the doorbell register.
How to make good use of multiple hardware queue pairs
It is necessary to manage the correspondence between hardware queue pairs and threads and processor cores in practical applications to prevent multiple threads from accessing a hardware queue at the same time .