从第 09 节继续:线程要等的可能是一条条件
第 09 节的 Lock 保护共享状态,Event 表示开始或停止。现在消费者要等待的是“缓冲里有数据,或者生产者失败了”。这条条件由几个共享字段共同决定,而且消费者醒来后还要安全地取走数据。
threading.Condition 把一把锁与等待通知组合起来。使用它时,先说清楚要等哪个谓词为真,再安排修改状态、通知和消费的顺序。
wait 期间释放锁,返回前重新取得锁
先看一个语义骨架。items 是共享缓冲;所有检查和修改都使用同一个 condition 的锁。
with condition:
while not items:
condition.wait()
item = items.pop(0)
进入 with condition 时取得底层锁。condition.wait() 先释放它,再让当前线程等待;收到通知后,线程还要重新取得这把锁,wait 才能返回。如果 wait 一直占着锁,生产者就无法进入临界区放入数据。Python 3.12 Condition 文档
生产者的对应工作是:
with condition:
items.append("sample")
condition.notify_all()
notify_all() 唤醒当前等在此 Condition 上的所有线程,但不释放锁。生产者继续持锁,直到退出 with;消费者随后竞争同一把锁,逐个重新检查自己的条件。调用 wait、notify_all 时必须已经持有底层锁。
| 时刻 | 消费者 | 生产者 | 底层锁 |
|---|---|---|---|
| 1 | 检查 items 为空 | 尚未进入 | 消费者持有 |
| 2 | 调用 wait,开始等待 | 可以进入 | wait 已释放 |
| 3 | 被通知,等待重新取得锁 | 放入数据,notify_all | 生产者仍持有 |
| 4 | wait 返回,重新检查并取数据 | 退出 with | 消费者重新持有 |
通知只意味着“值得再检查一次”。多个消费者竞争时,别人可能先取走唯一的数据;通知也可能只表示失败或关闭。因此必须使用 while,不能醒来一次就假定一定有数据。
wait_for 把谓词循环写在一处
下面的写法把“反复检查,必要时 wait”交给 Condition:
with condition:
ready = condition.wait_for(lambda: bool(items), timeout=5)
if not ready:
raise TimeoutError("no data")
item = items.pop(0)
Condition.wait_for(predicate, timeout) 在持锁时调用谓词;谓词为假才释放锁等待,醒来重新取得锁后再检查。它返回最后一次谓词的值,超时后可能仍为假。谓词应快速读取共享状态,不要在里面做耗时 I/O。不要将它与 asyncio.wait_for(awaitable, timeout) 混淆:一个等线程共享状态,一个管理异步操作的超时。
回到 Slime:收集结果、暂停、失败和关闭共用通知
沿训练调用方进入 RolloutScheduler.generate():它设置需求量,通知收集线程,然后循环检查 ready。没有结果就 wait;有结果就消费,降低 demand,再通知收集方。收集方更新 ready 后同样 notify_all,因此通知连接的是两个方向的状态变化。
源码入口:generate 在结果为空时等待
RolloutScheduler.generateslime/rollout/fully_async_distributed.py · L162–L186
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
groups = []
dropped_count = 0
drop_reasons = Counter()
with self.condition:
if self.error is not None:
raise self.error
if self.demand:
raise RuntimeError("A rollout batch is already being collected")
# The previous batch is durable before the next collection starts.
self.collecting.clear()
self.rollout_id = rollout_id
self.demand = self.args.rollout_batch_size
self.prefetch = min(prefetch, self.capacity)
self.paused = False
if self.thread is None:
self.thread = threading.Thread(target=self._run, name="rollout-collect", daemon=True)
self.thread.start()
self.condition.notify_all()
while self.demand:
if self.error is not None:
raise self.error
if not self.ready:
self.condition.wait()
continue
group, verdict = self.ready.popleft()暂停沿同一把锁进入 pause(),但谓词换成了“控制请求及在途结果已经平息”。关闭时 close() 设置 closed 和 error,再通知所有等待者;等待者重新检查 error 才能抛出异常退出。仅仅通知而不改变退出条件,会让它们醒来后继续等待。
源码入口:暂停等待与关闭唤醒
RolloutScheduler.pauseslime/rollout/fully_async_distributed.py · L206–L223
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def pause(self, *, drain=True):
"""Stop admission; optionally wait for in-flight results to become durable."""
with self.condition:
was_paused = self.paused
self.paused = True
self.condition.notify_all()
while self.error is None:
unsettled = bool(self.controls) or any(
self.configurations.get(worker) != (True, self.rollout_id) for worker in self.running_workers
)
if drain:
unsettled |= bool(self.pending or self.running_workers)
if not unsettled:
break
self.condition.wait()
if self.error is not None:
raise self.error
return was_pausedRolloutScheduler.closeslime/rollout/fully_async_distributed.py · L242–L250
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def close(self):
with self.condition:
self.closed = True
# Unblock consumers and pause() callers waiting for in-flight work.
if self.error is None:
self.error = RuntimeError("Rollout scheduler is closed")
self.condition.notify_all()
if self.thread is not None:
self.thread.join()该版本四处 Condition 构造的职责如下。区别在谓词和状态拥有者,wait 的锁语义相同。
| 使用者 | 等待或通知的真实含义 |
|---|---|
RolloutScheduler | generate 等 ready 或 error;pause 等控制请求与在途结果;收集、错误和关闭均通知。_run 没有 RPC 可等时用带 0.05 秒超时的 wait,再回到循环检查 |
AsyncRolloutWorker | pause 等 admission_paused,以及 drain 时 active 归零;异步循环更新计数后通知,线程退出也通知。线程锁内只短暂交接状态,不能把阻塞式 Condition.wait 放进事件循环线程 |
RolloutBuffer | 显式让 Condition 复用已有 RLock;write 放数据后 notify_all。但当前 read 为空时立即返回空结果,没有调用 wait,不能从名字 not_empty 推断它一定阻塞 |
GenerationProbe | wait_for 等 error、done 或测试传入的事件谓词;醒来后再区分错误、提前结束、目标是否真正达成 |
源码入口:Worker、Buffer 与 Probe
AsyncRolloutWorker.pauseslime/rollout/fully_async_rollout.py · L109–L117
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def pause(self, *, drain=True):
with self.condition:
previous = self.paused
self.paused = True
while self.running and self.error is None and (not self.admission_paused or (drain and self.active)):
self.condition.wait()
if self.error is not None:
raise self.error
return previousRolloutBuffer.__init__ / write / readslime_plugins/rollout_buffer/buffer.py · L232–L253
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
self.lock = threading.RLock()
self.not_empty = threading.Condition(self.lock)
self.total_written = 0
self.total_read = 0
self.task_type = task_type
def write(self, data):
with self.lock:
self.buffer.append(data)
self.total_written += 1
self.not_empty.notify_all()
return data
def read(self):
with self.not_empty:
if len(self.buffer) == 0:
return {"data": [], "meta_info": {}}
# Don't clear temp_data for regular read operations
result = self.buffer.get()
self.total_read += len(result["data"])
return resultGenerationProbe.waittests/pipeline_rl_test_helpers.py · L88–L93
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def wait(self, predicate):
with self.condition:
ready = self.condition.wait_for(lambda: self.error or self.done or predicate(self.events), timeout=180)
if self.error:
raise self.error
assert ready and predicate(self.events), f"Probe failed on {self.url}: {self.events[-3:]}"Worker 更新 active 并通知;Worker 线程退出时通知。读完这些分支,返回调用方的谓词循环,而不是把一次通知当作任务完成。
RLock 处理同一线程的嵌套加锁
普通 Lock 已由当前线程占有时,再去 acquire 同一把 Lock 也会等待。threading.RLock() 记录拥有它的线程和重入层数;同一线程可以再次取得,但每取得一次都必须释放一次。最外层释放之前,其他线程仍然进不来。通常写嵌套 with lock,由上下文管理器配对 acquire/release。RLock 文档
Slime 有一条具体的嵌套调用:get_accelerator() 取得 _SELECTION_LOCK 后调用 _register_builtin_backends(),后者调用 register_accelerator(),再次取得同一把锁。RLock 允许这个同线程调用链继续。
源码入口:沿嵌套调用检查同一把 _SELECTION_LOCK
get_acceleratorslime/utils/accelerator/__init__.py · L311–L319
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def get_accelerator() -> Accelerator:
global _ACCELERATOR
if _ACCELERATOR is not None:
return _ACCELERATOR
with _SELECTION_LOCK:
if _ACCELERATOR is not None:
return _ACCELERATOR
_register_builtin_backends()
requested = _requested_name()_register_builtin_backendsslime/utils/accelerator/__init__.py · L246–L259
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def _register_builtin_backends() -> None:
if "cuda" not in _REGISTRY:
register_accelerator("cuda", CUDAAccelerator, _cuda_available, priority=100, communication_backends=("nccl",))
if "musa" not in _REGISTRY:
register_accelerator(
"musa", MUSAAccelerator, is_musa_available, priority=200, communication_backends=("mccl",)
)
if "npu" not in _REGISTRY:
register_accelerator("npu", NPUAccelerator, is_npu_available, priority=250, communication_backends=("hccl",))
if "supa" not in _REGISTRY:
register_accelerator(
"supa", SUPAAccelerator, is_supa_available, priority=150, communication_backends=("bccl",)
)
register_acceleratorslime/utils/accelerator/__init__.py · L53–L71
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def register_accelerator(
name: str,
factory: Callable[[], Accelerator],
is_available: Callable[[], bool] | None = None,
priority: int = 0,
communication_backends: tuple[str, ...] = (),
) -> None:
"""Register a lazily constructed backend without changing Slime core."""
normalized = name.strip().lower()
if not normalized or normalized in {"auto", "none"}:
raise ValueError("Accelerator name must be a non-empty backend name")
with _SELECTION_LOCK:
_REGISTRY[normalized] = _BackendRegistration(
factory=factory,
is_available=is_available or (lambda: factory().is_available()),
priority=priority,
communication_backends=tuple(name.lower() for name in communication_backends),
)
另外五处构造分别保护存储写入、归档、训练状态、读者租约和插件缓冲:
| RLock 所在位置 | 保护的共享状态 |
|---|---|
rollout_store | 返回给同一存储池写入路径共用的锁,调用方用 with lock 写入 |
RolloutArchive.save | 非共享归档建立自己的存储及锁,写入块与元数据时使用 |
RolloutQueueController._training_lock | 训练批次、游标与恢复状态;如 ready_batch 在持锁时调用同样加锁的 batch |
QueueReader._lock | 读样本、归还与心跳线程共同访问的租约表 |
RolloutBuffer.lock | 缓冲及计数,也作为 not_empty 的底层锁 |
其余 RLock 的固定版本入口
存储池 · 归档 · 训练批次的嵌套加锁及 ready_batch 调用方 · 读者租约与心跳。插件缓冲见上方 source-5。
RLock 只解决同一线程重入同一把锁;多把锁取得顺序相反,仍可能死锁。Condition 默认建立 RLock;显式传入 RLock 时,wait 会完全释放其重入占有,返回前恢复相应状态,而不是只做一次普通 release。
Timer 是延迟执行一次回调的线程
threading.Timer(interval, function) 创建一个 Thread 子类实例;start() 才开始计时,到期在该线程中执行回调。join() 等待这个计时线程结束;cancel() 只在回调开始前阻止它运行,不能打断已经进入回调的代码。间隔不是精密时钟,也不要凭“立刻 cancel”推断没有竞态。Timer 文档
Slime 的存储恢复测试暂时破坏文件头,然后 Timer(0.05, restore).start() 延迟恢复,观察读取重试是否成功;finally 中 join,确保恢复线程已结束。这里没有使用 cancel,因为测试需要 restore 实际运行。cancel 是本节补充的清理语义。
源码入口:恢复文件头的计时线程
test_result_and_shelve_payloads_recover_at_the_storage_layer: restore and timertests/test_published_payload_retry.py · L29–L48
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码
def restore():
with path.open("r+b") as f:
f.seek(segment.offset)
f.write(header)
f.flush()
os.fsync(f.fileno())
timer = threading.Timer(0.05, restore)
timer.start()
try:
if loader == "plain":
result = transport.unpack_rollout_payload(ref)
elif loader == "async":
result = asyncio.run(transport.unpack_published_payload(ref))
else:
result = ref.load()
assert result == value
finally:
timer.join()
assert "Straw extent visibility retry" in capfd.readouterr().err运行:一次无数据通知、一份数据、两种 Timer 结局
下载 example.py,运行 python3 -W error example.py。只使用标准库,推荐 Python 3.12,最低 Python 3.11。
这是同步原语的语义实验,不是给 Slime 的 Buffer 增加阻塞行为。消费者用 wait_for;控制方先只通知、不放数据,等它确实重新检查为假后再放数据。RLock 展示同线程嵌套。Timer 分别展示执行后 join,以及先 cancel 再 start 的无竞态取消案例。
notify while holding lock True
predicate still false True
received ['sample']
same thread entered twice
outer lock still held
timer called True
cancelled timer called False
展开完整可运行示例
"""Condition predicate, RLock recursion, and Timer lifecycle; Python 3.11+."""
import threading
def main():
condition = threading.Condition()
waiting = threading.Event()
rechecked = threading.Event()
notified = threading.Event()
items = []
received = []
def has_data():
# wait_for evaluates this while holding condition's lock.
if not items:
waiting.set()
if notified.is_set():
rechecked.set()
return bool(items)
def read():
with condition:
condition.wait_for(has_data)
received.append(items.pop(0))
reader = threading.Thread(target=read)
reader.start()
waiting.wait()
with condition:
notified.set()
condition.notify_all() # No data yet: the predicate stays false.
print("notify while holding lock", not received)
rechecked.wait()
print("predicate still false", not received)
with condition:
items.append("sample")
condition.notify_all()
reader.join()
print("received", received)
lock = threading.RLock()
lock.acquire()
try:
with lock:
print("same thread entered twice")
print("outer lock still held")
finally:
lock.release()
called = threading.Event()
timer = threading.Timer(0, called.set)
timer.start()
timer.join()
print("timer called", called.is_set())
cancelled_call = threading.Event()
timer = threading.Timer(0, cancelled_call.set)
# Cancel before start so this experiment cannot race with the callback.
timer.cancel()
timer.start()
timer.join()
print("cancelled timer called", cancelled_call.is_set())
if __name__ == "__main__":
main()
复述检查
wait 放开锁,让生产者有机会改变状态;notify_all 不放锁;被唤醒的消费者重新加锁并检查谓词。RLock 的最后一次释放才让别的线程进入。Timer.cancel 无法强停已经运行的回调,Timer.join 负责等线程真正结束。
书籍核对与本节判断
书上怎么说: 第 19 章用线程 Event 说明协作式退出。[依据 A]
我的判断: Condition、RLock、Timer 的具体语义以 Python 3.12 官方文档及本节实验为依据。本地术语索引和中英全文检索 threading.Condition、RLock、threading.Timer 没有直接命中,书籍不承担这些方法的语义来源。
依据 A · 第 19 章《Python 并发模型》› Spinner with Threads,锚点 spinner-with-threads。英文 corpus/en/ch19.md:145–244,中文 corpus/zh/ch19.md:133–232。书籍基于 Python 3.10;本节验证环境 Python 3.12.3。
原文:“By design, there is no API for terminating a thread in Python.”
对照:“Python 故意没有提供终止线程的 API,如果想终止线程,则必须向线程发送相应的消息。”
本节对应的全部源码位置
下面把同类用法在 Slime 中的每个扫描位置接回本节讲解;点击文件名可查看完整源码快照。重复出现的 API 共用解释,具体输入与清理由所在函数决定。
展开本节 33 条源码记录
| 记录 / 讲解 | 源码位置 | 原代码 |
|---|---|---|
| occ-000735 | slime/data/archive.py:66 RolloutArchive.save | codec, lock = SampleCodec(store, args=args), threading.RLock() |
| occ-000755 | slime/data/queue_data_source.py:153 RolloutQueueController.__init__ | self._training_lock = threading.RLock() |
| occ-000799 | slime/data/queue_data_source.py:1034 QueueReader.__init__ | self._lock = threading.RLock() |
| occ-000830 | slime/data/transport.py:105 rollout_store | _writers[key] = (store, threading.RLock()) |
| occ-000912 | slime/rollout/fully_async_distributed.py:60 RolloutScheduler.__init__ | self.condition = threading.Condition() |
| occ-000913 有接收者线索 | slime/rollout/fully_async_distributed.py:91 RolloutScheduler._retire_worker | self.condition.notify_all() |
| occ-000915 有接收者线索 | slime/rollout/fully_async_distributed.py:126 RolloutScheduler._run | self.condition.notify_all() |
| occ-000916 有接收者线索 | slime/rollout/fully_async_distributed.py:127 RolloutScheduler._run | self.condition.wait(timeout=0.05) |
| occ-000917 有接收者线索 | slime/rollout/fully_async_distributed.py:147 RolloutScheduler._run | self.condition.notify_all() |
| occ-000918 有接收者线索 | slime/rollout/fully_async_distributed.py:151 RolloutScheduler._run | self.condition.notify_all() |
| occ-000922 有接收者线索 | slime/rollout/fully_async_distributed.py:159 RolloutScheduler._run | self.condition.notify_all() |
| occ-000926 有接收者线索 | slime/rollout/fully_async_distributed.py:179 RolloutScheduler.generate | self.condition.notify_all() |
| occ-000927 有接收者线索 | slime/rollout/fully_async_distributed.py:184 RolloutScheduler.generate | self.condition.wait() |
| occ-000928 有接收者线索 | slime/rollout/fully_async_distributed.py:198 RolloutScheduler.generate | self.condition.notify_all() |
| occ-000929 有接收者线索 | slime/rollout/fully_async_distributed.py:211 RolloutScheduler.pause | self.condition.notify_all() |
| occ-000931 有接收者线索 | slime/rollout/fully_async_distributed.py:220 RolloutScheduler.pause | self.condition.wait() |
| occ-000932 有接收者线索 | slime/rollout/fully_async_distributed.py:228 RolloutScheduler.resume | self.condition.notify_all() |
| occ-000933 有接收者线索 | slime/rollout/fully_async_distributed.py:248 RolloutScheduler.close | self.condition.notify_all() |
| occ-001027 | slime/rollout/fully_async_rollout.py:88 AsyncRolloutWorker.__init__ | self.condition = threading.Condition() |
| occ-001034 有接收者线索 | slime/rollout/fully_async_rollout.py:114 AsyncRolloutWorker.pause | self.condition.wait() |
| occ-001042 有接收者线索 | slime/rollout/fully_async_rollout.py:171 AsyncRolloutWorker._thread_main | self.condition.notify_all() |
| occ-001052 有接收者线索 | slime/rollout/fully_async_rollout.py:223 AsyncRolloutWorker._loop | self.condition.notify_all() |
| occ-001192 | slime/utils/accelerator/__init__.py:39 <module> | _SELECTION_LOCK = threading.RLock() |
| occ-001314 | slime_plugins/rollout_buffer/buffer.py:232 RolloutBuffer.__init__ | self.lock = threading.RLock() |
| occ-001315 | slime_plugins/rollout_buffer/buffer.py:233 RolloutBuffer.__init__ | self.not_empty = threading.Condition(self.lock) |
| occ-001316 有接收者线索 | slime_plugins/rollout_buffer/buffer.py:242 RolloutBuffer.write | self.not_empty.notify_all() |
| occ-001407 | tests/pipeline_rl_test_helpers.py:46 GenerationProbe.__init__ | self.condition = threading.Condition() |
| occ-001410 有接收者线索 | tests/pipeline_rl_test_helpers.py:80 GenerationProbe.run | self.condition.notify_all() |
| occ-001411 有接收者线索 | tests/pipeline_rl_test_helpers.py:86 GenerationProbe.run | self.condition.notify_all() |
| occ-001412 有接收者线索 | tests/pipeline_rl_test_helpers.py:90 GenerationProbe.wait | ready = self.condition.wait_for(lambda: self.error or self.done or predicate(self.events), timeout=180) |
| occ-001867 | tests/test_published_payload_retry.py:36 test_result_and_shelve_payloads_recover_at_the_storage_layer | timer = threading.Timer(0.05, restore) |
| occ-001868 有接收者线索 | tests/test_published_payload_retry.py:37 test_result_and_shelve_payloads_recover_at_the_storage_layer | timer.start() |
| occ-001870 有接收者线索 | tests/test_published_payload_retry.py:47 test_result_and_shelve_payloads_recover_at_the_storage_layer | timer.join() |
专题首页 · 上一节 · 下一节:线程池 Future 的结果与退出 · 源码与教程对照 · 全部用法