In-depth understanding NVME Hardware Queue Pair

What is the hardware queue pair?

The hardware queue pair is the best way to understand nvme/spdk. Only by deep understanding can we make good use of nvme.

From the perspective of nvme controller registers

As the name implies, it is a queue composed of some hardware registers.

Empty queue

In-depth understanding of nvme hardware queue pair

Full queue

In-depth understanding of nvme hardware queue pair

  • Question:
    Can you join and leave the team concurrently? No

submission hardware queue entry

Each entry is shown in the following table:

In-depth understanding of nvme hardware queue pair

  • Entry:

host software submission Command to the tail entry, by operating the submission hardware queue tail doorbell register;
in-depth understanding of nvme hardware queue pair

  • Dequeue:
    hardware takes the head entry in turn to process

completion hardware hardware queue entry

each The main fileds of entry are as follows:

In-depth understanding of nvme hardware queue pair

  • Join the team
    After the hardware has processed the nvme command obtained above, put the corresponding completed information in the completion queue head entry;

  • Dequeue
    After the host software is interrupted or actively polled until there are new entries added on it, go to completion queue tail to get the processing result of the most recently submitted command.

This is mainly to operate the completion queue doorbell register to achieve:
In-depth understanding of nvme hardware queue pair

queue pair

There are multiple submission hardware queues and multiple completion hardware queues inside the nvme controller. When submitting and executing NVME commands, you need to use the entries in the above two hardware queues and use them in pairs.

From the perspective of nvme controller architecture

The hardware queue pair is a waterwheel for extracting “nvme command”. Design to the submission/execution and completion of NVME instructions.

Overview

Here is a classic picture:
In-depth understanding of nvme hardware queue pair

NVME command submission

In-depth understanding of nvme hardware queue pair

NVME has finished processing the command

In-depth understanding of nvme hardware queue pair

Package of hardware queue pair by spdk

IO request submission function

  • spdk_nvme_ns_cmd_read
  1. Production request
    _nvme_ns_cmd_rw(ns, qpair, &payload, ….)

  2. Submit a request
    nvme_qpair_submmit_request()

    2.1 nvme_transport_qpair_submit_request(qpair, request)
    2.1.1 nvme_pcie_qpair_submit_request (qpair, request)
    2.1.1.1 nvme_pcietig_qrequest, request, qpair, );
    2.1.1.2 nvme_pcie_qpair_submit_tracker(qpair, tr)

nvme_pcie_qpair_submit_tracker(struct spdk_nvme_qpair *qpair, struct nvme _tracker *tr)
{
struct nvme_request *req;
struct nvme_pcie_qpair *pqpair = nvme_pcie_qpair(qpair);
struct nvme_pcie_ctrlr *pctrlr = nvmer_pcie_ctrlr(qpair);pair
br />
req = tr->req;
assert(req != NULL);
req->timed_out = false;
if (spdk_unlikely(pctrlr->ctrlr. timeout_enabled)) {
req->submit_tick = spdk_get_ticks();
} else {
req->submit_tick = 0;
}

pqpair- >tr[tr->cid].active = true;

/* Copy the command from the tracker to the submission queue. */
nvme_pcie_copy_command(&pqpair->cmd[pqpair-> sq_tail], &req->cmd);

if (spdk_unlikely(++pqpair->sq_tail == pqpair->num_entries)) {
pqpair->sq_tail = 0;
}

if (spdk_unlikely(pqpair->sq_tail == pqpair->sq_head)) {
SPDK_ERRLOG("sq_tail is pa ssing sq_head! ");
}

spdk_wmb();
if (spdk_likely(nvme_pcie_qpair_update_mmio_required(qpair,
pqpair->sq_tail,
pqpair->sq_shadow_tdbl,
pqpair->sq_eventidx))) {
g_thread_mmio_ctrlr = pctrlr;
spdk_mmio_write_4(pqpair->sq_tdbl, pqpair->sq_tail); // <--- pqpair->sq_tail); /> g_thread_mmio_ctrlr = NULL;
}
}

As you can see from the code path above, spdk nvme needs to operate hardware registers, and there is no lock and atomic operation all the way. Need upper protection.

IO completion query function

spdk_nvme_qpair_process_completions

The cq_hdbel corresponding to the same operation as above, no lock and atomic operation, requires upper layer protection to prevent concurrent modification of the doorbell register.

How to make good use of multiple hardware queue pairs

It is necessary to manage the correspondence between hardware queue pairs and threads and processor cores in practical applications to prevent multiple threads from accessing a hardware queue at the same time .

WordPress database error: [Table 'yf99682.wp_s6mz6tyggq_comments' doesn't exist]
SELECT SQL_CALC_FOUND_ROWS wp_s6mz6tyggq_comments.comment_ID FROM wp_s6mz6tyggq_comments WHERE ( comment_approved = '1' ) AND comment_post_ID = 1361 ORDER BY wp_s6mz6tyggq_comments.comment_date_gmt ASC, wp_s6mz6tyggq_comments.comment_ID ASC

Leave a Comment

Your email address will not be published.