从第 09 节继续:线程要等的可能是一条条件

第 09 节的 Lock 保护共享状态,Event 表示开始或停止。现在消费者要等待的是“缓冲里有数据,或者生产者失败了”。这条条件由几个共享字段共同决定,而且消费者醒来后还要安全地取走数据。

threading.Condition 把一把锁与等待通知组合起来。使用它时,先说清楚要等哪个谓词为真,再安排修改状态、通知和消费的顺序。

wait 期间释放锁,返回前重新取得锁

先看一个语义骨架。items 是共享缓冲;所有检查和修改都使用同一个 condition 的锁。

with condition:
    while not items:
        condition.wait()
    item = items.pop(0)

进入 with condition 时取得底层锁。condition.wait() 先释放它,再让当前线程等待;收到通知后,线程还要重新取得这把锁,wait 才能返回。如果 wait 一直占着锁,生产者就无法进入临界区放入数据。Python 3.12 Condition 文档

生产者的对应工作是:

with condition:
    items.append("sample")
    condition.notify_all()

notify_all() 唤醒当前等在此 Condition 上的所有线程,但不释放锁。生产者继续持锁,直到退出 with;消费者随后竞争同一把锁,逐个重新检查自己的条件。调用 wait、notify_all 时必须已经持有底层锁。

时刻消费者生产者底层锁
1检查 items 为空尚未进入消费者持有
2调用 wait,开始等待可以进入wait 已释放
3被通知,等待重新取得锁放入数据,notify_all生产者仍持有
4wait 返回,重新检查并取数据退出 with消费者重新持有

通知只意味着“值得再检查一次”。多个消费者竞争时,别人可能先取走唯一的数据;通知也可能只表示失败或关闭。因此必须使用 while,不能醒来一次就假定一定有数据。

wait_for 把谓词循环写在一处

下面的写法把“反复检查,必要时 wait”交给 Condition:

with condition:
    ready = condition.wait_for(lambda: bool(items), timeout=5)
    if not ready:
        raise TimeoutError("no data")
    item = items.pop(0)

Condition.wait_for(predicate, timeout) 在持锁时调用谓词;谓词为假才释放锁等待,醒来重新取得锁后再检查。它返回最后一次谓词的值,超时后可能仍为假。谓词应快速读取共享状态,不要在里面做耗时 I/O。不要将它与 asyncio.wait_for(awaitable, timeout) 混淆:一个等线程共享状态,一个管理异步操作的超时。

回到 Slime:收集结果、暂停、失败和关闭共用通知

沿训练调用方进入 RolloutScheduler.generate():它设置需求量,通知收集线程,然后循环检查 ready。没有结果就 wait;有结果就消费,降低 demand,再通知收集方。收集方更新 ready 后同样 notify_all,因此通知连接的是两个方向的状态变化。

源码入口:generate 在结果为空时等待

RolloutScheduler.generate
slime/rollout/fully_async_distributed.py · L162–L186
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

        groups = []
        dropped_count = 0
        drop_reasons = Counter()
        with self.condition:
            if self.error is not None:
                raise self.error
            if self.demand:
                raise RuntimeError("A rollout batch is already being collected")
            # The previous batch is durable before the next collection starts.
            self.collecting.clear()
            self.rollout_id = rollout_id
            self.demand = self.args.rollout_batch_size
            self.prefetch = min(prefetch, self.capacity)
            self.paused = False
            if self.thread is None:
                self.thread = threading.Thread(target=self._run, name="rollout-collect", daemon=True)
                self.thread.start()
            self.condition.notify_all()
            while self.demand:
                if self.error is not None:
                    raise self.error
                if not self.ready:
                    self.condition.wait()
                    continue
                group, verdict = self.ready.popleft()

暂停沿同一把锁进入 pause(),但谓词换成了“控制请求及在途结果已经平息”。关闭时 close() 设置 closed 和 error,再通知所有等待者;等待者重新检查 error 才能抛出异常退出。仅仅通知而不改变退出条件,会让它们醒来后继续等待。

源码入口:暂停等待与关闭唤醒

RolloutScheduler.pause
slime/rollout/fully_async_distributed.py · L206–L223
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def pause(self, *, drain=True):
        """Stop admission; optionally wait for in-flight results to become durable."""
        with self.condition:
            was_paused = self.paused
            self.paused = True
            self.condition.notify_all()
            while self.error is None:
                unsettled = bool(self.controls) or any(
                    self.configurations.get(worker) != (True, self.rollout_id) for worker in self.running_workers
                )
                if drain:
                    unsettled |= bool(self.pending or self.running_workers)
                if not unsettled:
                    break
                self.condition.wait()
            if self.error is not None:
                raise self.error
            return was_paused

RolloutScheduler.close
slime/rollout/fully_async_distributed.py · L242–L250
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def close(self):
        with self.condition:
            self.closed = True
            # Unblock consumers and pause() callers waiting for in-flight work.
            if self.error is None:
                self.error = RuntimeError("Rollout scheduler is closed")
            self.condition.notify_all()
        if self.thread is not None:
            self.thread.join()

该版本四处 Condition 构造的职责如下。区别在谓词和状态拥有者,wait 的锁语义相同。

使用者等待或通知的真实含义
RolloutSchedulergenerate 等 ready 或 error;pause 等控制请求与在途结果;收集、错误和关闭均通知。_run 没有 RPC 可等时用带 0.05 秒超时的 wait,再回到循环检查
AsyncRolloutWorkerpause 等 admission_paused,以及 drain 时 active 归零;异步循环更新计数后通知,线程退出也通知。线程锁内只短暂交接状态,不能把阻塞式 Condition.wait 放进事件循环线程
RolloutBuffer显式让 Condition 复用已有 RLock;write 放数据后 notify_all。但当前 read 为空时立即返回空结果,没有调用 wait,不能从名字 not_empty 推断它一定阻塞
GenerationProbewait_for 等 error、done 或测试传入的事件谓词;醒来后再区分错误、提前结束、目标是否真正达成
源码入口:Worker、Buffer 与 Probe

AsyncRolloutWorker.pause
slime/rollout/fully_async_rollout.py · L109–L117
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def pause(self, *, drain=True):
        with self.condition:
            previous = self.paused
            self.paused = True
            while self.running and self.error is None and (not self.admission_paused or (drain and self.active)):
                self.condition.wait()
            if self.error is not None:
                raise self.error
            return previous

RolloutBuffer.__init__ / write / read
slime_plugins/rollout_buffer/buffer.py · L232–L253
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

        self.lock = threading.RLock()
        self.not_empty = threading.Condition(self.lock)
        self.total_written = 0
        self.total_read = 0
        self.task_type = task_type

    def write(self, data):
        with self.lock:
            self.buffer.append(data)
            self.total_written += 1
            self.not_empty.notify_all()
        return data

    def read(self):
        with self.not_empty:
            if len(self.buffer) == 0:
                return {"data": [], "meta_info": {}}

            # Don't clear temp_data for regular read operations
            result = self.buffer.get()
            self.total_read += len(result["data"])
            return result

GenerationProbe.wait
tests/pipeline_rl_test_helpers.py · L88–L93
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def wait(self, predicate):
        with self.condition:
            ready = self.condition.wait_for(lambda: self.error or self.done or predicate(self.events), timeout=180)
            if self.error:
                raise self.error
            assert ready and predicate(self.events), f"Probe failed on {self.url}: {self.events[-3:]}"

Worker 更新 active 并通知;Worker 线程退出时通知。读完这些分支,返回调用方的谓词循环,而不是把一次通知当作任务完成。

RLock 处理同一线程的嵌套加锁

普通 Lock 已由当前线程占有时,再去 acquire 同一把 Lock 也会等待。threading.RLock() 记录拥有它的线程和重入层数;同一线程可以再次取得,但每取得一次都必须释放一次。最外层释放之前,其他线程仍然进不来。通常写嵌套 with lock,由上下文管理器配对 acquire/release。RLock 文档

Slime 有一条具体的嵌套调用:get_accelerator() 取得 _SELECTION_LOCK 后调用 _register_builtin_backends(),后者调用 register_accelerator(),再次取得同一把锁。RLock 允许这个同线程调用链继续。

源码入口:沿嵌套调用检查同一把 _SELECTION_LOCK

get_accelerator
slime/utils/accelerator/__init__.py · L311–L319
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

def get_accelerator() -> Accelerator:
    global _ACCELERATOR
    if _ACCELERATOR is not None:
        return _ACCELERATOR
    with _SELECTION_LOCK:
        if _ACCELERATOR is not None:
            return _ACCELERATOR
        _register_builtin_backends()
        requested = _requested_name()

_register_builtin_backends
slime/utils/accelerator/__init__.py · L246–L259
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

def _register_builtin_backends() -> None:
    if "cuda" not in _REGISTRY:
        register_accelerator("cuda", CUDAAccelerator, _cuda_available, priority=100, communication_backends=("nccl",))
    if "musa" not in _REGISTRY:
        register_accelerator(
            "musa", MUSAAccelerator, is_musa_available, priority=200, communication_backends=("mccl",)
        )
    if "npu" not in _REGISTRY:
        register_accelerator("npu", NPUAccelerator, is_npu_available, priority=250, communication_backends=("hccl",))
    if "supa" not in _REGISTRY:
        register_accelerator(
            "supa", SUPAAccelerator, is_supa_available, priority=150, communication_backends=("bccl",)
        )

register_accelerator
slime/utils/accelerator/__init__.py · L53–L71
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

def register_accelerator(
    name: str,
    factory: Callable[[], Accelerator],
    is_available: Callable[[], bool] | None = None,
    priority: int = 0,
    communication_backends: tuple[str, ...] = (),
) -> None:
    """Register a lazily constructed backend without changing Slime core."""
    normalized = name.strip().lower()
    if not normalized or normalized in {"auto", "none"}:
        raise ValueError("Accelerator name must be a non-empty backend name")
    with _SELECTION_LOCK:
        _REGISTRY[normalized] = _BackendRegistration(
            factory=factory,
            is_available=is_available or (lambda: factory().is_available()),
            priority=priority,
            communication_backends=tuple(name.lower() for name in communication_backends),
        )

另外五处构造分别保护存储写入、归档、训练状态、读者租约和插件缓冲:

RLock 所在位置保护的共享状态
rollout_store返回给同一存储池写入路径共用的锁,调用方用 with lock 写入
RolloutArchive.save非共享归档建立自己的存储及锁,写入块与元数据时使用
RolloutQueueController._training_lock训练批次、游标与恢复状态;如 ready_batch 在持锁时调用同样加锁的 batch
QueueReader._lock读样本、归还与心跳线程共同访问的租约表
RolloutBuffer.lock缓冲及计数,也作为 not_empty 的底层锁
其余 RLock 的固定版本入口

存储池 · 归档 · 训练批次的嵌套加锁及 ready_batch 调用方 · 读者租约与心跳。插件缓冲见上方 source-5。

RLock 只解决同一线程重入同一把锁;多把锁取得顺序相反,仍可能死锁。Condition 默认建立 RLock;显式传入 RLock 时,wait 会完全释放其重入占有,返回前恢复相应状态,而不是只做一次普通 release。

Timer 是延迟执行一次回调的线程

threading.Timer(interval, function) 创建一个 Thread 子类实例;start() 才开始计时,到期在该线程中执行回调。join() 等待这个计时线程结束;cancel() 只在回调开始前阻止它运行,不能打断已经进入回调的代码。间隔不是精密时钟,也不要凭“立刻 cancel”推断没有竞态。Timer 文档

Slime 的存储恢复测试暂时破坏文件头,然后 Timer(0.05, restore).start() 延迟恢复,观察读取重试是否成功;finally 中 join,确保恢复线程已结束。这里没有使用 cancel,因为测试需要 restore 实际运行。cancel 是本节补充的清理语义。

源码入口:恢复文件头的计时线程

test_result_and_shelve_payloads_recover_at_the_storage_layer: restore and timer
tests/test_published_payload_retry.py · L29–L48
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def restore():
        with path.open("r+b") as f:
            f.seek(segment.offset)
            f.write(header)
            f.flush()
            os.fsync(f.fileno())

    timer = threading.Timer(0.05, restore)
    timer.start()
    try:
        if loader == "plain":
            result = transport.unpack_rollout_payload(ref)
        elif loader == "async":
            result = asyncio.run(transport.unpack_published_payload(ref))
        else:
            result = ref.load()
        assert result == value
    finally:
        timer.join()
    assert "Straw extent visibility retry" in capfd.readouterr().err

运行:一次无数据通知、一份数据、两种 Timer 结局

下载 example.py,运行 python3 -W error example.py。只使用标准库,推荐 Python 3.12,最低 Python 3.11。

这是同步原语的语义实验,不是给 Slime 的 Buffer 增加阻塞行为。消费者用 wait_for;控制方先只通知、不放数据,等它确实重新检查为假后再放数据。RLock 展示同线程嵌套。Timer 分别展示执行后 join,以及先 cancel 再 start 的无竞态取消案例。

notify while holding lock True
predicate still false True
received ['sample']
same thread entered twice
outer lock still held
timer called True
cancelled timer called False
展开完整可运行示例
"""Condition predicate, RLock recursion, and Timer lifecycle; Python 3.11+."""

import threading


def main():
    condition = threading.Condition()
    waiting = threading.Event()
    rechecked = threading.Event()
    notified = threading.Event()
    items = []
    received = []

    def has_data():
        # wait_for evaluates this while holding condition's lock.
        if not items:
            waiting.set()
            if notified.is_set():
                rechecked.set()
        return bool(items)

    def read():
        with condition:
            condition.wait_for(has_data)
            received.append(items.pop(0))

    reader = threading.Thread(target=read)
    reader.start()
    waiting.wait()
    with condition:
        notified.set()
        condition.notify_all()  # No data yet: the predicate stays false.
        print("notify while holding lock", not received)
    rechecked.wait()
    print("predicate still false", not received)
    with condition:
        items.append("sample")
        condition.notify_all()
    reader.join()
    print("received", received)

    lock = threading.RLock()
    lock.acquire()
    try:
        with lock:
            print("same thread entered twice")
        print("outer lock still held")
    finally:
        lock.release()

    called = threading.Event()
    timer = threading.Timer(0, called.set)
    timer.start()
    timer.join()
    print("timer called", called.is_set())

    cancelled_call = threading.Event()
    timer = threading.Timer(0, cancelled_call.set)
    # Cancel before start so this experiment cannot race with the callback.
    timer.cancel()
    timer.start()
    timer.join()
    print("cancelled timer called", cancelled_call.is_set())


if __name__ == "__main__":
    main()
复述检查

wait 放开锁,让生产者有机会改变状态;notify_all 不放锁;被唤醒的消费者重新加锁并检查谓词。RLock 的最后一次释放才让别的线程进入。Timer.cancel 无法强停已经运行的回调,Timer.join 负责等线程真正结束。

书籍核对与本节判断

书上怎么说: 第 19 章用线程 Event 说明协作式退出。[依据 A]

我的判断: Condition、RLock、Timer 的具体语义以 Python 3.12 官方文档及本节实验为依据。本地术语索引和中英全文检索 threading.Condition、RLock、threading.Timer 没有直接命中,书籍不承担这些方法的语义来源。

依据 A · 第 19 章《Python 并发模型》› Spinner with Threads,锚点 spinner-with-threads。英文 corpus/en/ch19.md:145–244,中文 corpus/zh/ch19.md:133–232。书籍基于 Python 3.10;本节验证环境 Python 3.12.3。

原文:“By design, there is no API for terminating a thread in Python.”

对照:“Python 故意没有提供终止线程的 API,如果想终止线程,则必须向线程发送相应的消息。”

本节对应的全部源码位置

下面把同类用法在 Slime 中的每个扫描位置接回本节讲解;点击文件名可查看完整源码快照。重复出现的 API 共用解释,具体输入与清理由所在函数决定。

展开本节 33 条源码记录
记录 / 讲解源码位置原代码
occ-000735slime/data/archive.py:66
RolloutArchive.save
codec, lock = SampleCodec(store, args=args), threading.RLock()
occ-000755slime/data/queue_data_source.py:153
RolloutQueueController.__init__
self._training_lock = threading.RLock()
occ-000799slime/data/queue_data_source.py:1034
QueueReader.__init__
self._lock = threading.RLock()
occ-000830slime/data/transport.py:105
rollout_store
_writers[key] = (store, threading.RLock())
occ-000912slime/rollout/fully_async_distributed.py:60
RolloutScheduler.__init__
self.condition = threading.Condition()
occ-000913
有接收者线索
slime/rollout/fully_async_distributed.py:91
RolloutScheduler._retire_worker
self.condition.notify_all()
occ-000915
有接收者线索
slime/rollout/fully_async_distributed.py:126
RolloutScheduler._run
self.condition.notify_all()
occ-000916
有接收者线索
slime/rollout/fully_async_distributed.py:127
RolloutScheduler._run
self.condition.wait(timeout=0.05)
occ-000917
有接收者线索
slime/rollout/fully_async_distributed.py:147
RolloutScheduler._run
self.condition.notify_all()
occ-000918
有接收者线索
slime/rollout/fully_async_distributed.py:151
RolloutScheduler._run
self.condition.notify_all()
occ-000922
有接收者线索
slime/rollout/fully_async_distributed.py:159
RolloutScheduler._run
self.condition.notify_all()
occ-000926
有接收者线索
slime/rollout/fully_async_distributed.py:179
RolloutScheduler.generate
self.condition.notify_all()
occ-000927
有接收者线索
slime/rollout/fully_async_distributed.py:184
RolloutScheduler.generate
self.condition.wait()
occ-000928
有接收者线索
slime/rollout/fully_async_distributed.py:198
RolloutScheduler.generate
self.condition.notify_all()
occ-000929
有接收者线索
slime/rollout/fully_async_distributed.py:211
RolloutScheduler.pause
self.condition.notify_all()
occ-000931
有接收者线索
slime/rollout/fully_async_distributed.py:220
RolloutScheduler.pause
self.condition.wait()
occ-000932
有接收者线索
slime/rollout/fully_async_distributed.py:228
RolloutScheduler.resume
self.condition.notify_all()
occ-000933
有接收者线索
slime/rollout/fully_async_distributed.py:248
RolloutScheduler.close
self.condition.notify_all()
occ-001027slime/rollout/fully_async_rollout.py:88
AsyncRolloutWorker.__init__
self.condition = threading.Condition()
occ-001034
有接收者线索
slime/rollout/fully_async_rollout.py:114
AsyncRolloutWorker.pause
self.condition.wait()
occ-001042
有接收者线索
slime/rollout/fully_async_rollout.py:171
AsyncRolloutWorker._thread_main
self.condition.notify_all()
occ-001052
有接收者线索
slime/rollout/fully_async_rollout.py:223
AsyncRolloutWorker._loop
self.condition.notify_all()
occ-001192slime/utils/accelerator/__init__.py:39
<module>
_SELECTION_LOCK = threading.RLock()
occ-001314slime_plugins/rollout_buffer/buffer.py:232
RolloutBuffer.__init__
self.lock = threading.RLock()
occ-001315slime_plugins/rollout_buffer/buffer.py:233
RolloutBuffer.__init__
self.not_empty = threading.Condition(self.lock)
occ-001316
有接收者线索
slime_plugins/rollout_buffer/buffer.py:242
RolloutBuffer.write
self.not_empty.notify_all()
occ-001407tests/pipeline_rl_test_helpers.py:46
GenerationProbe.__init__
self.condition = threading.Condition()
occ-001410
有接收者线索
tests/pipeline_rl_test_helpers.py:80
GenerationProbe.run
self.condition.notify_all()
occ-001411
有接收者线索
tests/pipeline_rl_test_helpers.py:86
GenerationProbe.run
self.condition.notify_all()
occ-001412
有接收者线索
tests/pipeline_rl_test_helpers.py:90
GenerationProbe.wait
ready = self.condition.wait_for(lambda: self.error or self.done or predicate(self.events), timeout=180)
occ-001867tests/test_published_payload_retry.py:36
test_result_and_shelve_payloads_recover_at_the_storage_layer
timer = threading.Timer(0.05, restore)
occ-001868
有接收者线索
tests/test_published_payload_retry.py:37
test_result_and_shelve_payloads_recover_at_the_storage_layer
timer.start()
occ-001870
有接收者线索
tests/test_published_payload_retry.py:47
test_result_and_shelve_payloads_recover_at_the_storage_layer
timer.join()

专题首页 · 上一节 · 下一节:线程池 Future 的结果与退出 · 源码与教程对照 · 全部用法