工作移到线程后,多了什么责任

线程共享同一个进程中的对象。一个操作由“读、计算、写”几步组成时,多个线程交错执行可能破坏它的整体含义。锁保护的是这段需要一起完成的操作。

import threading

lock = threading.Lock()
ready = threading.Event()
state = {"count": 0}

def worker():
    ready.wait()
    for _ in range(1000):
        with lock:
            state["count"] += 1

if __name__ == "__main__":
    threads = [threading.Thread(target=worker) for _ in range(2)]
    for thread in threads:
        thread.start()
    ready.set()
    for thread in threads:
        thread.join()
    print("count", state["count"])

下载并运行本节 example.py。只使用 Python 标准库;推荐 Python 3.12,最低 Python 3.11。

运行结果:

count 2000

实验把累加操作放在 with lock 中;Event 负责通知两个线程开始;join() 负责等它们结束。三者分别解决互斥、通知、等待退出,不能互相替代。

回到 Slime 的健康检查线程

RolloutHealthMonitor 使用 Event 表示停止、暂停,使用 Lock 协调检查中的工作。stop() 发出停止信号,调用带超时的 join(),再用 is_alive() 检查是否真的退出。

展开 Slime 源码:RolloutHealthMonitor.stop

RolloutHealthMonitor.stop
slime/utils/health_monitor.py · L63–L78
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

        """Stop the health monitor thread completely. Called during dispose."""
        if not self._thread:
            return

        logger.info("Stopping RolloutHealthMonitor...")
        self._stop_event.set()
        if timeout is None:
            timeout = self._check_timeout + 5
        self._thread.join(timeout=timeout)
        if self._thread.is_alive():
            raise TimeoutError(f"Rollout health monitor did not terminate within {timeout:.1f}s")
        else:
            logger.info("RolloutHealthMonitor stopped.")

        self._thread = None
展开 Slime 源码:QueueReader._heartbeat_loop

QueueReader._heartbeat_loop
slime/data/queue_data_source.py · L1047–L1054
快照 2f2318653f6f794dddd321eff7c9d4b7b174643f· 查看固定版本源码

    def _heartbeat_loop(self):
        interval = max(0.1, getattr(self.args, "rollout_queue_lease_seconds", 300) / 3)
        while not self._stop.wait(interval):
            with self._lock:
                leases = list(self._leases.values())
            try:
                for offset in range(0, len(leases), 128):
                    batch = leases[offset : offset + 128]

单独执行 join(timeout=...),只能说明等待时间结束;线程可能仍然活着。把线程设成 daemon,也不能替代正常关闭流程。

停止通知怎样保存在 Event 中

方法这里表达的意思
event.set()把标记设为真,通知等待这个标记的线程
event.clear()把标记恢复为假,供下一轮使用
event.is_set()只查看标记,不等待,也不消耗通知
event.wait(timeout)等标记变真;返回值说明标记是否在等待结束时为真

例如 _stop.wait(interval) 既可等待检查间隔,也可在停止标记设置后提前返回。Event 保存一个布尔标记,多次 set() 不会积累成多份消息;需要逐项保存结果时仍用 Queue。

threading.Thread(target=...).start() 启动线程,直接调用目标函数仍在当前线程执行。threading.get_ident() 取得当前线程标识,可用于 Slime 判断是否在同一线程重复进入某个区域;它不是进程编号,线程退出后标识也可能被复用。

要给共享状态加锁,可使用 with lock 自动配对获取和释放;显式 acquire(timeout=...) 则要检查返回值,只有获取成功后才可 release()。后面的第 16 节再展开 RLock、Condition 与 Timer。

两种 Lock 分别属于哪里

  • threading.Lock 协调 OS 线程;等待会阻塞调用它的线程。
  • asyncio.Lock 协调一个事件循环中的任务;等待通过异步方式进行,不用于跨线程互斥。
  • RLock 允许同一线程重入同一把锁,但不会自动防止多把锁互相等待。
复述检查

停止 Event 是通知;线程自身必须检查这个通知并返回。join 等待它退出,is_alive 才能检查超时后是否仍在运行。

语义核对:threading。

本节对应的全部源码位置

下面把同类用法在 Slime 中的每个扫描位置接回本节讲解;点击文件名可查看完整源码快照。重复出现的 API 共用解释,具体输入与清理由所在函数决定。

展开本节 120 条源码记录
记录 / 讲解源码位置原代码
occ-000468slime/agent/aiohttp_threaded.py:6
<module>
import threading
occ-000469slime/agent/aiohttp_threaded.py:29
AppHandle
thread: threading.Thread
occ-000477slime/agent/aiohttp_threaded.py:61
run_app_in_thread
started = threading.Event()
occ-000483
有接收者线索
slime/agent/aiohttp_threaded.py:80
run_app_in_thread._run
started.set()
occ-000485
有接收者线索
slime/agent/aiohttp_threaded.py:84
run_app_in_thread._run
started.set()
occ-000486slime/agent/aiohttp_threaded.py:87
run_app_in_thread
thread = threading.Thread(target=_run, name=thread_name, daemon=True)
occ-000487
有接收者线索
slime/agent/aiohttp_threaded.py:88
run_app_in_thread
thread.start()
occ-000488
有接收者线索
slime/agent/aiohttp_threaded.py:89
run_app_in_thread
started.wait(timeout=start_timeout_sec)
occ-000569slime/backends/megatron_utils/server/megatron_server.py:3
<module>
import threading
occ-000638slime/backends/megatron_utils/server/megatron_server.py:616
_start_http_server
stats_thread = threading.Thread(
occ-000639
有接收者线索
slime/backends/megatron_utils/server/megatron_server.py:622
_start_http_server
stats_thread.start()
occ-000734slime/data/archive.py:56
RolloutArchive.save
import threading
occ-000747slime/data/queue_data_source.py:16
<module>
import threading
occ-000754slime/data/queue_data_source.py:147
RolloutQueueController.__init__
self._producer_lock = threading.Lock()
occ-000756slime/data/queue_data_source.py:154
RolloutQueueController.__init__
self._gc_stop = threading.Event()
occ-000757slime/data/queue_data_source.py:184
RolloutQueueController._start_gc
self._gc_thread = threading.Thread(target=self._gc_loop, name="straw-gc", daemon=True)
occ-000758
有接收者线索
slime/data/queue_data_source.py:185
RolloutQueueController._start_gc
self._gc_thread.start()
occ-000759
有接收者线索
slime/data/queue_data_source.py:200
RolloutQueueController._gc_loop
while not self._gc_stop.wait(60):
occ-000760
有接收者线索
slime/data/queue_data_source.py:205
RolloutQueueController._gc_loop
self._gc_stop.set()
occ-000795
有接收者线索
slime/data/queue_data_source.py:970
RolloutQueueController.close
self._gc_stop.set()
occ-000796
有接收者线索
slime/data/queue_data_source.py:972
RolloutQueueController.close
self._gc_thread.join()
occ-000801slime/data/queue_data_source.py:1036
QueueReader.__init__
self._stop = threading.Event()
occ-000802slime/data/queue_data_source.py:1044
QueueReader.__init__
self._heartbeats = threading.Thread(target=self._heartbeat_loop, name="queue-leases", daemon=True)
occ-000803
有接收者线索
slime/data/queue_data_source.py:1045
QueueReader.__init__
self._heartbeats.start()
occ-000804
有接收者线索
slime/data/queue_data_source.py:1049
QueueReader._heartbeat_loop
while not self._stop.wait(interval):
occ-000815
有接收者线索
slime/data/queue_data_source.py:1288
QueueReader.close
self._stop.set()
occ-000816
有接收者线索
slime/data/queue_data_source.py:1289
QueueReader.close
self._heartbeats.join(timeout=5)
occ-000824slime/data/transport.py:8
<module>
import threading
occ-000825slime/data/transport.py:25
<module>
_writers_lock = threading.Lock()
occ-000911slime/rollout/fully_async_distributed.py:6
<module>
import threading
occ-000924slime/rollout/fully_async_distributed.py:177
RolloutScheduler.generate
self.thread = threading.Thread(target=self._run, name="rollout-collect", daemon=True)
occ-000925
有接收者线索
slime/rollout/fully_async_distributed.py:178
RolloutScheduler.generate
self.thread.start()
occ-000934
有接收者线索
slime/rollout/fully_async_distributed.py:250
RolloutScheduler.close
self.thread.join()
occ-001024slime/rollout/fully_async_rollout.py:34
<module>
import threading
occ-001030slime/rollout/fully_async_rollout.py:99
AsyncRolloutWorker.__init__
self.worker_thread: threading.Thread | None = None
occ-001032slime/rollout/fully_async_rollout.py:106
AsyncRolloutWorker.start
self.worker_thread = threading.Thread(target=self._thread_main, name="fully-async-rollout", daemon=True)
occ-001033
有接收者线索
slime/rollout/fully_async_rollout.py:107
AsyncRolloutWorker.start
self.worker_thread.start()
occ-001036
有接收者线索
slime/rollout/fully_async_rollout.py:138
AsyncRolloutWorker.close
self.worker_thread.join(timeout=35)
occ-001037
有接收者线索
slime/rollout/fully_async_rollout.py:139
AsyncRolloutWorker.close
if self.worker_thread.is_alive():
occ-001191slime/utils/accelerator/__init__.py:14
<module>
import threading
occ-001197slime/utils/async_utils.py:2
<module>
import threading
occ-001208slime/utils/async_utils.py:49
AsyncLoopThread.__init__
self._thread = threading.Thread(target=self._start_loop, daemon=True)
occ-001209
有接收者线索
slime/utils/async_utils.py:50
AsyncLoopThread.__init__
self._thread.start()
occ-001218slime/utils/health_monitor.py:2
<module>
import threading
occ-001219slime/utils/health_monitor.py:27
RolloutHealthMonitor.__init__
self._stop_event = threading.Event()
occ-001220slime/utils/health_monitor.py:28
RolloutHealthMonitor.__init__
self._pause_event = threading.Event()
occ-001221
有接收者线索
slime/utils/health_monitor.py:29
RolloutHealthMonitor.__init__
self._pause_event.set() # Engines may be offloaded before the first rollout.
occ-001222slime/utils/health_monitor.py:34
RolloutHealthMonitor.__init__
self._check_lock = threading.Lock()
occ-001223
有接收者线索
slime/utils/health_monitor.py:50
RolloutHealthMonitor.start
self._stop_event.clear()
occ-001224
有接收者线索
slime/utils/health_monitor.py:51
RolloutHealthMonitor.start
self._pause_event.set()
occ-001225slime/utils/health_monitor.py:53
RolloutHealthMonitor.start
self._thread = threading.Thread(
occ-001226
有接收者线索
slime/utils/health_monitor.py:58
RolloutHealthMonitor.start
self._thread.start()
occ-001227
有接收者线索
slime/utils/health_monitor.py:68
RolloutHealthMonitor.stop
self._stop_event.set()
occ-001228
有接收者线索
slime/utils/health_monitor.py:71
RolloutHealthMonitor.stop
self._thread.join(timeout=timeout)
occ-001229
有接收者线索
slime/utils/health_monitor.py:72
RolloutHealthMonitor.stop
if self._thread.is_alive():
occ-001230
有接收者线索
slime/utils/health_monitor.py:82
RolloutHealthMonitor.pause
self._pause_event.set()
occ-001231
有接收者线索
slime/utils/health_monitor.py:84
RolloutHealthMonitor.pause
if not self._check_lock.acquire(timeout=self._check_timeout if timeout is None else timeout):
occ-001232
有接收者线索
slime/utils/health_monitor.py:86
RolloutHealthMonitor.pause
self._check_lock.release()
occ-001233
有接收者线索
slime/utils/health_monitor.py:92
RolloutHealthMonitor.resume
self._pause_event.clear()
occ-001234
有接收者线索
slime/utils/health_monitor.py:95
RolloutHealthMonitor._health_monitor_loop
while not self._stop_event.is_set():
occ-001235
有接收者线索
slime/utils/health_monitor.py:97
RolloutHealthMonitor._health_monitor_loop
while self._pause_event.is_set() and not self._stop_event.is_set():
occ-001236
有接收者线索
slime/utils/health_monitor.py:97
RolloutHealthMonitor._health_monitor_loop
while self._pause_event.is_set() and not self._stop_event.is_set():
occ-001237
有接收者线索
slime/utils/health_monitor.py:98
RolloutHealthMonitor._health_monitor_loop
self._stop_event.wait(timeout=0.5)
occ-001238
有接收者线索
slime/utils/health_monitor.py:100
RolloutHealthMonitor._health_monitor_loop
if self._stop_event.is_set():
occ-001239
有接收者线索
slime/utils/health_monitor.py:106
RolloutHealthMonitor._health_monitor_loop
if self._stop_event.wait(self._check_first_wait):
occ-001240
有接收者线索
slime/utils/health_monitor.py:109
RolloutHealthMonitor._health_monitor_loop
if self._pause_event.is_set():
occ-001241
有接收者线索
slime/utils/health_monitor.py:116
RolloutHealthMonitor._health_monitor_loop
if not self._pause_event.is_set() and not self._stop_event.is_set():
occ-001242
有接收者线索
slime/utils/health_monitor.py:116
RolloutHealthMonitor._health_monitor_loop
if not self._pause_event.is_set() and not self._stop_event.is_set():
occ-001244
有接收者线索
slime/utils/health_monitor.py:123
RolloutHealthMonitor._health_monitor_loop
if self._stop_event.wait(self._check_interval):
occ-001245
有接收者线索
slime/utils/health_monitor.py:134
RolloutHealthMonitor._run_health_checks
if self._stop_event.is_set():
occ-001246
有接收者线索
slime/utils/health_monitor.py:136
RolloutHealthMonitor._run_health_checks
if not force and self._pause_event.is_set():
occ-001296slime/utils/routed_experts.py:4
<module>
import threading
occ-001305slime/utils/routed_experts.py:213
RoutedExpertsMicrobatch.__init__
self._lock = threading.Lock()
occ-001311slime_plugins/rollout_buffer/buffer.py:6
<module>
import threading
occ-001404tests/pipeline_rl_test_helpers.py:5
<module>
import threading
occ-001408tests/pipeline_rl_test_helpers.py:49
GenerationProbe.__init__
self.thread = threading.Thread(target=self.run, daemon=True)
occ-001414
有接收者线索
tests/pipeline_rl_test_helpers.py:110
GenerationProbe.verify
self.thread.join(timeout=30)
occ-001415
有接收者线索
tests/pipeline_rl_test_helpers.py:111
GenerationProbe.verify
assert not self.thread.is_alive()
occ-001417
有接收者线索
tests/pipeline_rl_test_helpers.py:116
GenerationProbe.verify_flush
self.thread.join(timeout=30)
occ-001418
有接收者线索
tests/pipeline_rl_test_helpers.py:117
GenerationProbe.verify_flush
assert not self.thread.is_alive()
occ-001632tests/test_distributed_rollout.py:9
<module>
import threading
occ-001656tests/test_distributed_rollout.py:602
test_slow_task_acquire_does_not_block_async_generation
entered, release = threading.Event(), threading.Event()
occ-001657tests/test_distributed_rollout.py:602
test_slow_task_acquire_does_not_block_async_generation
entered, release = threading.Event(), threading.Event()
occ-001676tests/test_distributed_rollout.py:672
scheduler_factory.create
release = threading.Event()
occ-001678
有接收者线索
tests/test_distributed_rollout.py:687
scheduler_factory.create.make_generator.execute
while not release.is_set():
occ-001697tests/test_distributed_rollout.py:811
test_slow_worker_control_does_not_block_collection_or_close
entered = threading.Event()
occ-001698
有接收者线索
tests/test_distributed_rollout.py:815
test_slow_worker_control_does_not_block_collection_or_close.stall
entered.set()
occ-001702
有接收者线索
tests/test_distributed_rollout.py:829
test_slow_worker_control_does_not_block_collection_or_close
assert entered.wait(5)
occ-001787tests/test_distributed_rollout.py:2010
test_background_gc_failure_stops_queue_work_and_fails_close
controller._gc_thread = threading.Thread(target=controller._gc_loop)
occ-001788
有接收者线索
tests/test_distributed_rollout.py:2011
test_background_gc_failure_stops_queue_work_and_fails_close
controller._gc_thread.start()
occ-001789
有接收者线索
tests/test_distributed_rollout.py:2012
test_background_gc_failure_stops_queue_work_and_fails_close
controller._gc_thread.join(5)
occ-001790
有接收者线索
tests/test_distributed_rollout.py:2013
test_background_gc_failure_stops_queue_work_and_fails_close
assert not controller._gc_thread.is_alive()
occ-001824tests/test_fully_async_rollout.py:23
<module>
import threading
occ-001833tests/test_fully_async_rollout.py:218
test_done_callback_never_blocks_event_loop_thread
pusher = threading.Thread(target=_push_all, daemon=True)
occ-001834
有接收者线索
tests/test_fully_async_rollout.py:219
test_done_callback_never_blocks_event_loop_thread
pusher.start()
occ-001835
有接收者线索
tests/test_fully_async_rollout.py:220
test_done_callback_never_blocks_event_loop_thread
pusher.join(timeout=30)
occ-001836
有接收者线索
tests/test_fully_async_rollout.py:222
test_done_callback_never_blocks_event_loop_thread
assert not pusher.is_alive(), "done-callback blocked on a full output queue"
occ-001863tests/test_published_payload_retry.py:5
<module>
import threading
occ-001984tests/test_score_centering_binary.py:3
<module>
import threading
occ-001985tests/test_score_centering_binary.py:96
test_http_worker_decodes_off_event_loop
main_thread = threading.get_ident()
occ-001986tests/test_score_centering_binary.py:101
test_http_worker_decodes_off_event_loop.decode
calls.append(threading.get_ident())
occ-002060tests/test_straw_fully_async_recovery.py:15
<module>
import threading
occ-002073tests/test_straw_fully_async_recovery.py:132
_run_job
thread = threading.Thread(target=generate, daemon=True)
occ-002074
有接收者线索
tests/test_straw_fully_async_recovery.py:133
_run_job
thread.start()
occ-002075tests/test_straw_fully_async_recovery.py:158
_run_job
threading.Event().wait()
occ-002076
有接收者线索
tests/test_straw_fully_async_recovery.py:158
_run_job
threading.Event().wait()
occ-002092tests/test_straw_r3.py:2
<module>
import threading
occ-002095tests/test_straw_r3.py:143
test_cancelled_straw_publication_drains_before_releasing_capacity
entered, release = threading.Event(), threading.Event()
occ-002096tests/test_straw_r3.py:143
test_cancelled_straw_publication_drains_before_releasing_capacity
entered, release = threading.Event(), threading.Event()
occ-002230tests/test_training_recovery.py:292
test_recovery_e2e_contacts_dashboard_without_proxy
import threading
occ-002233tests/test_training_recovery.py:339
test_recovery_e2e_contacts_dashboard_without_proxy
thread = threading.Thread(target=server.serve_forever, daemon=True)
occ-002234
有接收者线索
tests/test_training_recovery.py:340
test_recovery_e2e_contacts_dashboard_without_proxy
thread.start()
occ-002236
有接收者线索
tests/test_training_recovery.py:346
test_recovery_e2e_contacts_dashboard_without_proxy
thread.join(timeout=5)
occ-002249tools/convert_hf_to_fp8.py:22
<module>
import threading
occ-002251tools/convert_hf_to_fp8.py:105
ConversionResult.__init__
self.lock = threading.Lock()
occ-002255tools/convert_hf_to_int4_direct.py:16
<module>
import threading
occ-002257tools/convert_hf_to_int4_direct.py:158
ConversionResult.__init__
self.lock = threading.Lock()
occ-002262tools/convert_torch_dist_to_hf_parallel.py:8
<module>
import threading
occ-002265tools/convert_torch_dist_to_hf_parallel.py:43
WrappedStorageReader.__init__
self._cache_lock = threading.Lock() if max_workers and max_workers > 1 else None
occ-002272tools/convert_torch_dist_to_hf_parallel.py:284
save_tensors
lock = threading.Lock()

专题首页 · 上一节 · 下一节:10 · async with、async for 与异步生成器 · 源码与教程对照 · 全部用法