Python Thread Pool

Last Updated On Monday 15th Nov 2021

python threadpool

  • multiprocessing is a package that supports spawning processes using an API similar to the threading module.
  • The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.
  • Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine.
  • It runs on both Unix and Windows.
	from multiprocessing.pool import ThreadPool
	
	from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))
	
	[1, 4, 9]
	

The Process class

	from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
	

python thread pool

Queues

The Queue class is a near clone of queue.Queue.

	from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()
	

Pipes

The Pipe() function returns a pair of connection objects connected by a pipe which by default is duplex (two-way).

	from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print(parent_conn.recv())   # prints "[42, None, 'hello']"
    p.join()
	

References