Multiprocessing Interview Questions & Answers

Threading runs multiple threads inside one process that share one interpreter — and therefore one GIL (Global Interpreter Lock), which lets only one thread execute Python bytecode at a time. Multiprocessing spawns separate OS processes, each with its own interpreter and own GIL, so they can run Python code in true parallel on multiple CPU cores.

from multiprocessing import Process
import os

def work():
    print(f"running in pid {os.getpid()}")  # a distinct process each time

if __name__ == "__main__":            # required guard on Windows/spawn
    ps = [Process(target=work) for _ in range(4)]
    for p in ps: p.start()
    for p in ps: p.join()

The tradeoff: processes don't share memory, so passing data costs serialization (pickling) and IPC overhead, and each process has higher startup cost than a thread. Rule of thumb: reach for multiprocessing when you need real CPU parallelism, not just concurrency.

A Process represents a single child process you start and join manually — good when you have a fixed, small number of distinct tasks. A Pool manages a reusable group of worker processes and hands out work to them, which is far more convenient for many homogeneous tasks over a dataset.

from multiprocessing import Pool

def square(n):
    return n * n

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, range(10))   # distributed across 4 workers
    print(results)   # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Pool reuses workers (amortizing startup cost) and offers map, imap, apply_async, etc. Use Process for a few long-lived distinct jobs; use Pool when you're fanning the same function over many inputs.

Because processes don't share memory, they communicate through IPC primitives. A Queue is a multi-producer/multi-consumer, thread- and process-safe FIFO — the general-purpose choice. A Pipe is a faster but lower-level two-endpoint connection, best for communication between exactly two processes.

from multiprocessing import Process, Queue

def producer(q):
    q.put("result")          # values are pickled across the boundary

if __name__ == "__main__":
    q = Queue()
    p = Process(target=producer, args=(q,))
    p.start()
    print(q.get())           # "result"
    p.join()

Both serialize objects under the hood, so only picklable data flows through them. Use Queue for fan-in/fan-out among many workers; use Pipe for a tight one-to-one channel where you want the lower overhead.

Every argument and return value crossing a process boundary must be serialized with pickle, sent, then deserialized on the other side. For large objects this copying cost can dwarf the parallelism gains, and some objects simply can't be pickled.

import pickle

pickle.dumps(lambda x: x)   # PicklingError — lambdas aren't picklable
# Also unpicklable: open file handles, sockets, locks, db connections,
# local/nested functions, and generators.

Picklable things include module-level functions and classes, and basic containers of picklable values. The practical implications: pass small, picklable payloads, define worker functions at module top level, and avoid shipping huge data structures between processes. Minimizing what crosses the boundary is the key to multiprocessing performance.

Since each process has its own memory, you need explicit shared-state tools. Value and Array put simple data in shared memory (fast, but limited types and you must guard with a lock). A Manager hosts richer shared objects (dict, list, etc.) via a server process — more flexible but slower because access is proxied.

from multiprocessing import Process, Value, Lock

def inc(counter, lock):
    for _ in range(1000):
        with lock:               # protect the shared value
            counter.value += 1

if __name__ == "__main__":
    counter = Value("i", 0)      # shared int in shared memory
    lock = Lock()
    ps = [Process(target=inc, args=(counter, lock)) for _ in range(4)]
    for p in ps: p.start()
    for p in ps: p.join()
    print(counter.value)         # 4000

Prefer message passing (Queue/Pipe) over shared state when you can — it's easier to reason about. Reach for Value/Array for hot, simple counters and a Manager only when you genuinely need shared complex objects.

Use multiprocessing for CPU-bound work — number crunching, image processing, data transforms — where you need to saturate multiple cores with Python code. Threads and asyncio can't do that because the GIL serializes bytecode execution; only separate processes get separate GILs.

from multiprocessing import Pool

def heavy(n):                      # pure-Python CPU work
    return sum(i * i for i in range(n))

if __name__ == "__main__":
    with Pool() as pool:           # defaults to os.cpu_count() workers
        print(pool.map(heavy, [10_000_000] * 8))   # runs in parallel

For I/O-bound work (network, disk, DB), threads or asyncio are usually better — they're cheaper and the GIL is released during I/O anyway. Rule of thumb: CPU-bound -> multiprocessing; I/O-bound -> threads/asyncio.

On spawn start (default on Windows and macOS), each child process re-imports your module to rebuild the target function. Without the guard, the child would re-run the code that starts more processes, causing infinite recursion / a crash. The guard ensures process-launching code runs only in the parent.

from multiprocessing import Process

def work(): ...

if __name__ == "__main__":     # children import the module but skip this
    p = Process(target=work)
    p.start(); p.join()

Rule of thumb: always put process creation behind the __main__ guard, or spawn-based multiprocessing will misbehave.

They control how child processes are created. fork (Linux default historically) copies the parent via os.fork — fast, inherits state, but unsafe with threads. spawn starts a fresh interpreter and re-imports the module — slower but safe and consistent (default on Win/macOS, and Linux from 3.14). forkserver forks from a clean server process — a middle ground.

import multiprocessing as mp
mp.set_start_method("spawn")     # set once, at program start

Rule of thumb: prefer spawn for predictable cross-platform behavior, especially in programs that also use threads.

map blocks and returns all results as a list (order preserved). imap returns a lazy iterator that yields results as they're ready in order — better for large inputs. apply_async submits a single call and returns an AsyncResult you .get() later, optionally with a callback.

with Pool() as p:
    p.map(f, data)                       # eager, full list
    for r in p.imap(f, data): ...        # lazy, streamed
    res = p.apply_async(f, (x,))         # one call, non-blocking
    res.get(timeout=10)

Rule of thumb: map for simple batches, imap/imap_unordered to stream large inputs, apply_async for individual fire-and-forget calls.

Queue/Pipe copy data by pickling it across processes. shared_memory (3.8+) exposes a raw memory block multiple processes map directly — no copy — ideal for large NumPy arrays. You trade convenience for speed and must manage lifetime (close()/unlink()) and synchronization yourself.

from multiprocessing import shared_memory
shm = shared_memory.SharedMemory(create=True, size=1024)
shm.buf[:4] = b"data"            # other procs attach by name
shm.close(); shm.unlink()

Rule of thumb: use shared_memory for big buffers where copying dominates; stick with Queue/Pipe for ordinary message passing.

They overlap heavily — both run work across a pool of processes. Pool (older) has a richer API: map, imap, imap_unordered, starmap, apply_async with callbacks. ProcessPoolExecutor (from concurrent.futures) has a smaller, unified submit/map Future-based API shared with the thread executor, so it's easy to swap.

from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor() as ex:
    list(ex.map(f, data))        # same code works with ThreadPoolExecutor

Rule of thumb: use ProcessPoolExecutor for a clean, swappable API; reach for Pool when you need its extra methods like imap_unordered/starmap.

A Manager runs a server process that hosts shared Python objects — list, dict, Namespace, etc. — and hands other processes proxies to them. Mutations go through the server (via IPC), so all processes see updates. It's convenient but slower than Value/Array because every access is a message.

from multiprocessing import Manager, Process

with Manager() as m:
    shared = m.dict()
    shared["count"] = 0
    # processes given `shared` all see the same dict

Rule of thumb: use a Manager for shared high-level objects when convenience matters; use Value/Array/shared_memory when speed matters.

Value and Array allocate data in shared memory holding C types, so children created via fork/spawn can read and write the same underlying bytes — much faster than a Manager. Because access isn't atomic for compound updates, they come with a lock you should use (with val.get_lock()).

from multiprocessing import Value, Process

counter = Value("i", 0)          # shared C int
def inc():
    with counter.get_lock():     # guard the read-modify-write
        counter.value += 1

Rule of thumb: use Value/Array for small fixed shared C data, and always guard read-modify-write with the built-in lock.

Finished children become zombie/defunct entries until the parent reaps them, and the parent may exit while children still run, leaving orphans. join() waits for completion and cleans up; for pools, the with block (or pool.close(); pool.join()) does this for you.

p = Process(target=work)
p.start()
p.join()                         # reap it; without this -> zombie risk
print(p.exitcode)                # 0 = clean exit

Rule of thumb: every start() should be paired with a join() (or use a pool's context manager) so processes are cleaned up properly.

Each process has startup cost (spawning an interpreter, re-importing), its own memory (no sharing by default), and IPC cost to pickle arguments and results. For small or short tasks this overhead can exceed the parallelism gain, making it slower than a simple loop.

# bad: tiny tasks, huge per-task pickling/startup overhead
with Pool() as p:
    p.map(lambda x: x + 1, range(1_000_000))   # also: lambdas aren't picklable!

Rule of thumb: multiprocessing pays off only when each task is CPU-heavy enough to dwarf process startup and data-transfer overhead.

KeyboardInterrupt is delivered to all processes, and a pool.map() blocked in the parent may not surface it cleanly, leaving workers hanging. A common fix is to use map_async(...).get(timeout) so the main thread stays interruptible, and to ignore SIGINT in workers via an initializer.

import signal
from multiprocessing import Pool

def init():
    signal.signal(signal.SIGINT, signal.SIG_IGN)   # workers ignore Ctrl-C

with Pool(initializer=init) as p:
    p.map_async(work, data).get(timeout=999999)    # parent handles Ctrl-C

Rule of thumb: for interruptible pools, let workers ignore SIGINT and use the async .get(timeout=...) pattern in the parent.