Python中的并发 - 进程篇Ⅱ

多进程共享数据

先来看一段错误的代码

import multiprocessing global_num = 0 mutex = multiprocessing.Lock() def func(): global global_num, mutex for i in xrange(10000): with mutex: global_num += 1 process = [] for i in range(10): process.append(multiprocessing.Process(target=func)) process[i].start() for p in process: p.join() print global_num

Output

这是因为每个子进程会复制一份父进程的地址空间，之后的修改都在子进程的地址空间中操作，并不会影响到父进程。但是我们可以通过共享内存的方法

import multiprocessing global_num = multiprocessing.Value('d', 0) mutex = multiprocessing.Lock() def func(): for i in xrange(10000): with mutex: global_num.value += 1 process = [] for i in range(10): process.append(multiprocessing.Process(target=func)) process[i].start() for p in process: p.join() print global_num.value

不过并发编程时通常最好尽量避免使用共享的状态。当使用多进程编程时，这显得特别正确。

multiprocessing.Pipe()

Pipe()返回的两个连接对象表示管道的两端。每个连接对象都有send()和recv()方法

import multiprocessing import random def sender(pipe): while True: value = random.randrange(10) pipe.send(value) print 'Value sent {}'.format(value) def receiver(pipe): while True: value = pipe.recv() print 'Value received {}'.format(value) pipe1, pipe2 = multiprocessing.Pipe() p1 = multiprocessing.Process(target=sender, args=(pipe1,)) p2 = multiprocessing.Process(target=receiver, args=(pipe2,)) p1.start() p2.start() multiprocessing.Queue() import multiprocessing import random from Queue import Empty def sender(q): count = 0 while count < 5: value = random.randrange(10) q.put(value) print 'Value sent {}'.format(value) count += 1 def receiver(q): while True: try: value = q.get(timeout=2) except Empty: break print 'Value received {}'.format(value) queue = multiprocessing.Queue() p1 = multiprocessing.Process(target=sender, args=(queue,)) p2 = multiprocessing.Process(target=receiver, args=(queue,)) p1.start() p2.start() print 'MainProcess exit'

Output

MainProcess exit Value sent 4 Value sent 5 Value received 4 Value received 5 Value sent 7 Value sent 4 Value sent 5 Value received 7 Value received 4 Value received 5 multiprocessing.Event()

多进程中也有事件 Event ,工作方式与多线程类似。但是事件对象不能被传递到子进程函数中。

from multiprocessing import Event, Pool import time event = Event() event.set() def worker(i, e): if e.is_set(): time.sleep(0.1) print "A - %s" % (time.time()) e.clear() else: time.sleep(0.1) print "B - %s" % (time.time()) e.set() pool = Pool(3) pool.map(worker, [(x, event) for x in range(9)])

Output

RuntimeError: Semaphore objects should only be shared between processes through inheritance

信号对象只能通过继承机制在进程间共享,所以我们应该做如下改动

from multiprocessing import Event, Pool import time event = Event() event.set() def worker(i): if event.is_set(): time.sleep(0.1) print "A - %s" % (time.time()) event.clear() else: time.sleep(0.1) print "B - %s" % (time.time()) event.set() pool = Pool(3) pool.map(worker, range(9))

Output

A - 1473072277.97 A - 1473072277.97 A - 1473072277.97 B - 1473072278.07 B - 1473072278.07 B - 1473072278.07 A - 1473072278.17 A - 1473072278.17 A - 1473072278.17