首页
关于
Search
1
PacketForge 博客部署文档
16 阅读
2
深入 SONiC 与 SAI:从命令行到芯片的完整数据之旅
14 阅读
3
DPU 与 CPO:AI 时代的网络基础设施革命
11 阅读
4
Broadcom芯片
10 阅读
5
DPDK高性能网络
10 阅读
默认分类
交换芯片
高性能网络
AI算力网络
WebSocket高性能项目
DPDK 成长路线
①性能工程基础
②DPDK 源码深读
③用户态协议栈
④软件交换与 SONiC
⑤PCIe、虚拟化与 DPU
⑥面试与复盘
登录
Search
标签搜索
DPDK→DPU 学习路线
P0
P1
P2
P3
P5
PCIe
P4
P6
libmtcp
NUMA
mTCP
OVS
SONiC
SAI
DPU
perf
gdb
EAL
mempool
PacketForge
累计撰写
53
篇文章
累计收到
3
条评论
首页
栏目
页面
关于
最新文章
热门文章
评论最多
点赞最多
2026-08-18
P1-05 · PMD:把中断换成轮询,把每包分配换成预分配
对应路线图:Phase 1 (Day 22-49) | DPDK 24.11.6 LTS | 内核对照 v6.6代码出处:source/dpdk-stable-24.11.6/drivers/net/ixgbe/,引用均标 文件:行号上一篇:P1-04 mbuf | 下一篇:P1-06 NUMA一句话定位ethdev 是抽象层,PMD 是轮询驱动。热路径只有一次函数指针间接调用,没有其他分支。热路径真实代码(lib/ethdev/rte_ethdev.h:6261-6294)static inline uint16_t rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, struct rte_mbuf **rx_pkts, const uint16_t nb_pkts) { uint16_t nb_rx; struct rte_eth_fp_ops *p; void *qd; // ...(DEBUG 编译下的参数检查,省略)... /* fetch pointer to queue data */ p = &rte_eth_fp_ops[port_id]; qd = p->rxq.data[queue_id]; // ... nb_rx = p->rx_pkt_burst(qd, rx_pkts, nb_pkts); // ... return nb_rx; }关键结构 rte_eth_fp_ops:每端口一份、按 cache line 对齐的函数指针表,声明在 lib/ethdev/rte_ethdev_core.h:129,实例在 lib/ethdev/rte_ethdev.c:42:struct rte_eth_fp_ops rte_eth_fp_ops[RTE_MAX_ETHPORTS];rx/tx 热路径只碰这张表,不碰 rte_eth_dev 大结构——热路径和配置路径的数据彻底分离。PMD 启动时把函数填进来(ixgbe 注册点,ixgbe_ethdev.c:1112):eth_dev->rx_pkt_burst = &ixgbe_recv_pkts;ixgbe 还会按特性换实现(ixgbe_rxtx.c:4940/4947/4954):启用 vector → ixgbe_recv_pkts_vec;启用 bulk alloc → ixgbe_recv_pkts_bulk_alloc;默认 → ixgbe_recv_pkts。RX 路径逐步走:DD 位协议 + refill + RDT 回写① 轮询 DD 位(ixgbe_rxtx.c:1812-1836,节选) while (nb_rx < nb_pkts) { /* * The order of operations here is important as the DD status * bit must not be read after any other descriptor fields. * rx_ring and rxdp are pointing to volatile data so the order * of accesses cannot be reordered by the compiler. If they were * not volatile, they could be reordered which could lead to * using invalid descriptor fields when read from rxd. * ... */ rxdp = &rx_ring[rx_id]; staterr = rxdp->wb.upper.status_error; if (!(staterr & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD))) break; /* * Use acquire fence to ensure that status_error which includes * DD bit is loaded before loading of other descriptor words. */ rte_atomic_thread_fence(rte_memory_order_acquire); rxd = *rxdp;三个层次:volatile 约束编译器不重排;acquire fence 约束 CPU 不把 DD 位之外的字段提前读进来;DD=0 立即 break——没包就收工,绝不自旋空等。② refill:上一轮就挂好下一轮的 mbuf(ixgbe_rxtx.c:1870-1903,节选) nmb = rte_mbuf_raw_alloc(rxq->mb_pool); if (nmb == NULL) { // ... 计数 + break:描述符留着,下轮再试 break; } nb_hold++; rxe = &sw_ring[rx_id]; rx_id++; if (rx_id == rxq->nb_rx_desc) rx_id = 0; /* Prefetch next mbuf while processing current one. */ rte_ixgbe_prefetch(sw_ring[rx_id].mbuf); // ... rxm = rxe->mbuf; // 取走旧的 rxe->mbuf = nmb; // 挂上新的 dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); rxdp->read.hdr_addr = 0; rxdp->read.pkt_addr = dma_addr; // 新 mbuf 地址写给 NICmbuf 不是"收到包才分配",而是本轮补下一轮:从 mempool 拿新 mbuf 写回描述符,旧 mbuf 直接交给应用。这就是 mempool 存在的意义闭环——把分配移出热路径。③ mbuf 字段初始化(ixgbe_rxtx.c:1918-1945,节选) pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.wb.upper.length) - rxq->crc_len); rxm->data_off = RTE_PKTMBUF_HEADROOM; rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off); rxm->nb_segs = 1; rxm->next = NULL; rxm->pkt_len = pkt_len; rxm->data_len = pkt_len; rxm->port = rxq->port_id; pkt_info = rte_le_to_cpu_32(rxd.wb.lower.lo_dword.data); // ... pkt_flags = rx_desc_status_to_pkt_flags(staterr, vlan_flags); pkt_flags = pkt_flags | rx_desc_error_to_pkt_flags(staterr, (uint16_t)pkt_info, rxq->rx_udp_csum_zero_err); pkt_flags = pkt_flags | ixgbe_rxd_pkt_info_to_pkt_flags((uint16_t)pkt_info); rxm->ol_flags = pkt_flags; rxm->packet_type = ixgbe_rxd_pkt_info_to_pkt_type(...); if (likely(pkt_flags & RTE_MBUF_F_RX_RSS_HASH)) rxm->hash.rss = rte_le_to_cpu_32( rxd.wb.lower.hi_dword.rss);硬件 offload 信息(校验和/VLAN/RSS 哈希/packet type)从描述符直接灌进 mbuf——零软件解析开销。④ RDT 回写:把描述符还给 NIC(ixgbe_rxtx.c:1959-1982,节选) rxq->rx_tail = rx_id; /* * If the number of free RX descriptors is greater than the RX free * threshold of the queue, advance the Receive Descriptor Tail (RDT) * register. * Update the RDT with the value of the last processed RX descriptor * minus 1, to guarantee that the RDT register is never equal to the * RDH register, which creates a "full" ring situation from the * hardware point of view... */ nb_hold = (uint16_t) (nb_hold + rxq->nb_rx_hold); if (nb_hold > rxq->rx_free_thresh) { // ... rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold;三个细节:① RDT 故意写 rx_id - 1——保证 RDT ≠ RDH,否则硬件误判 ring 满停收(PMD 最经典的 bug);② 攒批回写——nb_hold 累计超过 rx_free_thresh 才写一次寄存器,不逐包写 MMIO;③ _WC_WRITE——写合并(Write Combining)放宽序,批量 MMIO 写更快。批量预分配版本(ixgbe_rxtx.c:1629-1670,节选) /* allocate buffers in bulk directly into the S/W ring */ alloc_idx = rxq->rx_free_trigger - (rxq->rx_free_thresh - 1); rxep = &rxq->sw_ring[alloc_idx]; diag = rte_mempool_get_bulk(rxq->mb_pool, (void *)rxep, rxq->rx_free_thresh); if (unlikely(diag != 0)) return -ENOMEM; rxdp = &rxq->rx_ring[alloc_idx]; for (i = 0; i < rxq->rx_free_thresh; ++i) { /* populate the static rte mbuf fields */ mb = rxep[i].mbuf; if (reset_mbuf) { mb->port = rxq->port_id; } rte_mbuf_refcnt_set(mb, 1); mb->data_off = RTE_PKTMBUF_HEADROOM; /* populate the descriptors */ dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(mb)); rxdp[i].read.hdr_addr = 0; rxdp[i].read.pkt_addr = dma_addr; }一次 rte_mempool_get_bulk 预分配 rx_free_thresh 个 mbuf 直接铺进软件环——P1-02 学的 mempool 批量 API,在这里兑现成"收包零分配"。vector PMD:SIMD 批处理ixgbe_recv_pkts_vec(ixgbe_rxtx_vec_sse.c:599 起):一次循环处理 8/16 个描述符,SIMD 批量搬运字段,movemask 一条指令提取多个 DD 位。本质:把逐包的分支和标量搬运变成定长批量操作——DPDK "burst" 哲学的硬件层实现。选择逻辑在 ixgbe_rxtx.c:4940。🐧 内核对照:同一块硬件,两种活法真实内核代码(ixgbe_main.c:2306-2345,节选): while (likely(total_rx_packets < budget)) { // budget 上限! // ... /* return some buffers to hardware, one at a time is too slow */ if (cleaned_count >= IXGBE_RX_BUFFER_WRITE) { ixgbe_alloc_rx_buffers(rx_ring, cleaned_count); cleaned_count = 0; } rx_desc = IXGBE_RX_DESC(rx_ring, rx_ring->next_to_clean); size = le16_to_cpu(rx_desc->wb.upper.length); if (!size) break; /* This memory barrier is needed to keep us from reading * any other fields out of the rx_desc until we know the * descriptor has been written back */ dma_rmb(); rx_buffer = ixgbe_get_rx_buffer(rx_ring, rx_desc, &skb, size, &rx_buffer_pgcnt); // ... XDP 处理 ... skb = ixgbe_construct_skb(rx_ring, rx_buffer, &xdp, rx_desc); // ... ixgbe_rx_skb(q_vector, skb); // ... total_rx_packets++; }环节内核 ixgbe(软中断里)DPDK ixgbe PMD(lcore 里)触发硬中断(MSI-X 每队列)→ NAPI纯轮询预算budget 限制(防饿死其他 CPU)无预算,burst 打满屏障dma_rmb()acquire fence(等价物)补缓冲ixgbe_alloc_rx_buffers 页引用rte_mbuf_raw_alloc mbuf 池包容器构造 skb(ixgbe_construct_skb)预分配 mbuf 直接换指针协议完全一样(DD 位 + RDT),差异在运行模型:内核在软中断里受预算约束地做,DPDK 在应用线程里无约束地做。10G 线速(14.88 Mpps)下,内核单核百万级 PPS,DPDK 单核可达线速——差距就是中断、skb 构造和协议栈的代价。面试 3 问描述 DD 位与 RDT 回写的协议。 → ① NIC 写 DD 通知就绪(单向);② 软件写 RDT 归还描述符(单向);③ RDT 写 rx_id-1 防等于 RDH(ixgbe_rxtx.c:1965-1968 注释)。为什么收包路径的 mbuf 是"上一轮"分配的? → refill 提前挂 mbuf 进描述符,收包时只换指针(ixgbe_rxtx.c:1898-1903);批量版一次 mempool_get_bulk 铺 rx_free_thresh 个(ixgbe_rxtx.c:1641-1642)。rte_eth_fp_ops 表解决什么问题? → 热路径只碰每端口对齐的小表(rte_ethdev_core.h:129),与配置路径的 rte_eth_dev 大结构分离,避免每次收包都踩冷数据。🔧 动手实验testpmd 跑 io / mac / macswap / rxonly / txonly 五种 forward mode 记录 PPS;再对比 scalar 与 vector PMD 的小包 PPS。产出 perf-data/P1/W1-testpmd-modes.md。下一篇:P1-06 · NUMA 与 lcore 调度
2026年08月18日
1 阅读
0 评论
0 点赞
第 1/1 页 · 本页 1 条 · 共 1 条