Node.js 中 stream.pipeline 与 async 迭代器的混合使用：避免背压失控和内存泄漏
在处理大文件上传、日志转发或者数据库游标导出时，很多人会把 `fs.createReadStream` 和 `fs.createWriteStream` 直接通过 `.pipe()` 连起来。这种写法在数据量小的时候跑得很顺，但一旦下游写入变慢或者中间需要插入自定义转换逻辑，问题就暴露出来了：可读流继续按自己的节奏生产数据，可写流的内部缓冲区越堆越高，最终进程内存飙升，甚至因为 `write after end` 或者未捕获的 error 事件直接崩掉。`stream.pipeline` 本身解决了错误传递和流生命周期管理，但它和 async 迭代器混用时，背压的处理方式很容易被误解。这篇内容围绕一个真实的日志采集场景，梳理 `pipeline`、`for await...of` 以及手动 `highWaterMark` 配合时的行为差异，给出能直接落地的代码写法。 先看一个常见的错误写法。假设要把一个不断追加的日志文件按行过滤后转发到另一个可写流，中间做一下脱敏。有人会这样写： ```js const { createReadStream, createWriteStream } = require('fs'); const { pipeline } = require('stream/promises'); const { Transform } = require('stream'); const reader = createReadStream('/var/log/app.log'); const writer = createWriteStream('/data/clean.log'); const masker = new Transform({ transform(chunk, enc, cb) { const text = chunk.toString().replace(/token=\w+/g, 'token=***'); cb(null, text); } }); await pipeline(reader, masker, writer); ``` 这个写法本身没问题，`pipeline` 会在任意一环出错时销毁所有流并 reject。但问题出在 masker 的 `transform` 里直接 `chunk.toString()`：如果日志里有多字节字符恰好被切在 chunk 边界上，就会出现乱码或替换错位。更隐蔽的是，当 writer 的下游是网络套接字、写入延迟抖动大时，`pipeline` 确实会尊重背压，但 Transform 的 `highWaterMark` 默认 16KB，如果单条日志超过这个长度，`transform` 回调里处理完的字符串长度可能远超写入端的消费能力，缓冲区仍然会短暂堆积。 要手动控制节奏，可以改用 async 迭代器从可读流拉取数据，但别用 `for await (const chunk of reader)` 然后逐块 `writer.write()` 并忽略返回值。`writer.write()` 返回 false 时表示缓冲区已满，必须等待 `drain` 事件再继续。下面的写法把背压显式暴露出来： ```js const { once } = require('events'); async function copyWithBackpressure(reader, writer, mask) { for await (const chunk of reader) { const text = mask(chunk); if (!writer.write(text)) { await once(writer, 'drain'); } } writer.end(); } ``` 这段代码在 `write` 返回 false 时会挂起当前迭代，直到可写流排空，天然形成拉取式节奏。注意 `once(writer, 'drain')` 要放在 `write` 之后、下一次迭代之前，否则会丢掉一次 drain 信号。另外，如果 writer 在等待期间报错，`once` 不会自动处理 error 事件，需要配合 `AbortController` 或者在外层用 `Promise.race` 监听 `error`，否则进程会因未处理的 error 事件退出。 实际项目里更稳妥的做法是让 `pipeline` 和 async 迭代器各司其职：把需要异步拉取的上游（比如分页 API、数据库游标）封装成 `Readable.from`，再交给 `pipeline` 去对接下游转换和写入。这样背压由 `pipeline` 统一管理，又保留了 async 生成器的可读性： ```js const { Readable } = require('stream'); const { pipeline } = require('stream/promises'); async function* fetchPages(client) { let cursor = null; do { const page = await client.list({ cursor, limit: 200 }); for (const item of page.items) { yield JSON.stringify(item) + '\n'; } cursor = page.nextCursor; } while (cursor); } await pipeline( Readable.from(fetchPages(apiClient), { objectMode: false }), new Transform({ transform(chunk, enc, cb) { cb(null, chunk); } }), createWriteStream('/data/export.ndjson') ); ``` 这里 `Readable.from` 对异步生成器做了适配，生成器 `yield` 的节奏受下游消费速度约束；`pipeline` 负责在任意流出错时调用 `destroy()`，避免生成器里的游标资源泄漏。需要留心的一点是 `Readable.from` 默认 `objectMode: true`，如果 `yield` 出来的是字符串又不想让下游按对象模式处理，记得显式传 `objectMode: false`，否则 `createWriteStream` 收到的是对象而不是 Buffer，会直接抛类型错误。 排查这类问题时，`stream.pipeline` 的错误对象里会带上 `code` 和出错的流实例，但不会告诉你当时各个流的缓冲水位。可以在关键 Transform 上挂一个定时器打印 `readableLength` 和 `writableLength`，或者用 `process.memoryUsage().heapUsed` 在 chunk 到达时做采样。如果发现 `writableLength` 持续接近 `highWaterMark` 但从不触发 drain，检查一下是不是转换逻辑里忽略了 `write` 的返回值，或者把 `writer.write` 放在了 `setInterval` 之类的异步回调里，导致背压信号根本没被消费。 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1601.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1602.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1603.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1604.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1605.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1606.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1607.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1608.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1609.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1610.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1611.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1612.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1613.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1614.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1615.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1616.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1617.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1618.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1619.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1620.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1621.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1622.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1623.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1624.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1625.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1626.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1627.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1628.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1629.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1630.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1631.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1632.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1633.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1634.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1635.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1636.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1637.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1638.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1639.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1640.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1641.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1642.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1643.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1644.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1645.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1646.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1647.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1648.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1649.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1650.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1651.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1652.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1653.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1654.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1655.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1656.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1657.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1658.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1659.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1660.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1661.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1662.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1663.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1664.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1665.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1666.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1667.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1668.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1669.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1670.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1671.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1672.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1673.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1674.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1675.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1676.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1677.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1678.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1679.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1680.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1681.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1682.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1683.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1684.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1685.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1686.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1687.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1688.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1689.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1690.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1691.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1692.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1693.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1694.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1695.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1696.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1697.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1698.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1699.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1700.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1701.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1702.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1703.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1704.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1705.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1706.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1707.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1708.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1709.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1710.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1711.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1712.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1713.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1714.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1715.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1716.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1717.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1718.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1719.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1720.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1721.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1722.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1723.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1724.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1725.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1726.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1727.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1728.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1729.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1730.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1731.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1732.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1733.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1734.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1735.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1736.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1737.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1738.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1739.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1740.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1741.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1742.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1743.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1744.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1745.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1746.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1747.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1748.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1749.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1750.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1751.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1752.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1753.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1754.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1755.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1756.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1757.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1758.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1759.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1760.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1761.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1762.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1763.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1764.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1765.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1766.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1767.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1768.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1769.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1770.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1771.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1772.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1773.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1774.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1775.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1776.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1777.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1778.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1779.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1780.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1781.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1782.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1783.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1784.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1785.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1786.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1787.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1788.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1789.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1790.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1791.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1792.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1793.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1794.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1795.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1796.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1797.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1798.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1799.html 推荐查看天气：http://www.tianqi98.com/jizhuanwan/1800.html
