[{"data":1,"prerenderedAt":806},["ShallowReactive",2],{"blog-\u002Fblog\u002Fpython-concurrent-futures-explained":3},{"id":4,"title":5,"body":6,"description":793,"difficulty":794,"extension":795,"framework":796,"frameworkSlug":54,"meta":797,"navigation":82,"order":99,"path":798,"qaPath":799,"seo":800,"stem":801,"subtopic":20,"topic":802,"topicSlug":803,"updated":804,"__hash__":805},"blog\u002Fblog\u002Fpython-concurrent-futures-explained.md","Python concurrent.futures Explained — ThreadPoolExecutor, ProcessPoolExecutor, and Futures",{"type":7,"value":8,"toc":783},"minimark",[9,14,35,39,49,179,188,192,195,277,284,288,302,416,424,428,442,556,559,563,573,643,653,657,660,747,751,779],[10,11,13],"h2",{"id":12},"python-concurrentfutures-explained","Python concurrent.futures, explained",[15,16,17,21,22,26,27,30,31,34],"p",{},[18,19,20],"code",{},"concurrent.futures"," is the ",[23,24,25],"strong",{},"high-level"," way to run code in parallel. Instead of managing\nthreads or processes by hand, you hand work to an ",[23,28,29],{},"executor"," and get back ",[18,32,33],{},"Future","\nobjects. The same tiny API works for both threads and processes, so switching between\nI\u002FO-bound and CPU-bound strategies is often a one-line change.",[10,36,38],{"id":37},"the-executor-pattern","The Executor pattern",[15,40,41,42,45,46,48],{},"An executor manages a pool of workers. You ",[18,43,44],{},"submit"," callables; it runs them on the pool and\nreturns a ",[18,47,33],{}," representing the eventual result. Use it as a context manager so the\npool is cleaned up automatically.",[50,51,56],"pre",{"className":52,"code":53,"language":54,"meta":55,"style":55},"language-python shiki shiki-themes github-light github-dark","from concurrent.futures import ThreadPoolExecutor\n\ndef square(n):\n    return n * n\n\nwith ThreadPoolExecutor(max_workers=4) as pool:\n    future = pool.submit(square, 5)   # schedules the call, returns immediately\n    print(future.result())            # 25 — blocks until done\n","python","",[18,57,58,77,84,97,112,117,146,167],{"__ignoreMap":55},[59,60,63,67,71,74],"span",{"class":61,"line":62},"line",1,[59,64,66],{"class":65},"szBVR","from",[59,68,70],{"class":69},"sVt8B"," concurrent.futures ",[59,72,73],{"class":65},"import",[59,75,76],{"class":69}," ThreadPoolExecutor\n",[59,78,80],{"class":61,"line":79},2,[59,81,83],{"emptyLinePlaceholder":82},true,"\n",[59,85,87,90,94],{"class":61,"line":86},3,[59,88,89],{"class":65},"def",[59,91,93],{"class":92},"sScJk"," square",[59,95,96],{"class":69},"(n):\n",[59,98,100,103,106,109],{"class":61,"line":99},4,[59,101,102],{"class":65},"    return",[59,104,105],{"class":69}," n ",[59,107,108],{"class":65},"*",[59,110,111],{"class":69}," n\n",[59,113,115],{"class":61,"line":114},5,[59,116,83],{"emptyLinePlaceholder":82},[59,118,120,123,126,130,133,137,140,143],{"class":61,"line":119},6,[59,121,122],{"class":65},"with",[59,124,125],{"class":69}," ThreadPoolExecutor(",[59,127,129],{"class":128},"s4XuR","max_workers",[59,131,132],{"class":65},"=",[59,134,136],{"class":135},"sj4cs","4",[59,138,139],{"class":69},") ",[59,141,142],{"class":65},"as",[59,144,145],{"class":69}," pool:\n",[59,147,149,152,154,157,160,163],{"class":61,"line":148},7,[59,150,151],{"class":69},"    future ",[59,153,132],{"class":65},[59,155,156],{"class":69}," pool.submit(square, ",[59,158,159],{"class":135},"5",[59,161,162],{"class":69},")   ",[59,164,166],{"class":165},"sJ8bj","# schedules the call, returns immediately\n",[59,168,170,173,176],{"class":61,"line":169},8,[59,171,172],{"class":135},"    print",[59,174,175],{"class":69},"(future.result())            ",[59,177,178],{"class":165},"# 25 — blocks until done\n",[15,180,181,183,184,187],{},[18,182,44],{}," returns instantly; the work happens on a worker thread. Calling ",[18,185,186],{},".result()"," blocks\nuntil that work finishes (or re-raises any exception it threw).",[10,189,191],{"id":190},"threads-vs-processes-pick-by-workload","Threads vs processes — pick by workload",[15,193,194],{},"The two executors share an identical interface but solve opposite problems:",[50,196,198],{"className":52,"code":197,"language":54,"meta":55,"style":55},"from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor\n\n# I\u002FO-bound (network, disk): threads — GIL is released during I\u002FO waits\nwith ThreadPoolExecutor(max_workers=20) as pool:\n    ...\n\n# CPU-bound (math, parsing): processes — each has its own GIL, true parallelism\nwith ProcessPoolExecutor(max_workers=4) as pool:\n    ...\n",[18,199,200,211,215,220,239,244,248,253,272],{"__ignoreMap":55},[59,201,202,204,206,208],{"class":61,"line":62},[59,203,66],{"class":65},[59,205,70],{"class":69},[59,207,73],{"class":65},[59,209,210],{"class":69}," ThreadPoolExecutor, ProcessPoolExecutor\n",[59,212,213],{"class":61,"line":79},[59,214,83],{"emptyLinePlaceholder":82},[59,216,217],{"class":61,"line":86},[59,218,219],{"class":165},"# I\u002FO-bound (network, disk): threads — GIL is released during I\u002FO waits\n",[59,221,222,224,226,228,230,233,235,237],{"class":61,"line":99},[59,223,122],{"class":65},[59,225,125],{"class":69},[59,227,129],{"class":128},[59,229,132],{"class":65},[59,231,232],{"class":135},"20",[59,234,139],{"class":69},[59,236,142],{"class":65},[59,238,145],{"class":69},[59,240,241],{"class":61,"line":114},[59,242,243],{"class":135},"    ...\n",[59,245,246],{"class":61,"line":119},[59,247,83],{"emptyLinePlaceholder":82},[59,249,250],{"class":61,"line":148},[59,251,252],{"class":165},"# CPU-bound (math, parsing): processes — each has its own GIL, true parallelism\n",[59,254,255,257,260,262,264,266,268,270],{"class":61,"line":169},[59,256,122],{"class":65},[59,258,259],{"class":69}," ProcessPoolExecutor(",[59,261,129],{"class":128},[59,263,132],{"class":65},[59,265,136],{"class":135},[59,267,139],{"class":69},[59,269,142],{"class":65},[59,271,145],{"class":69},[59,273,275],{"class":61,"line":274},9,[59,276,243],{"class":135},[15,278,279,280,283],{},"Threads share memory and are cheap but the GIL serialises CPU work. Processes get real\nparallelism but pay for ",[23,281,282],{},"pickling"," arguments\u002Fresults across the process boundary.",[10,285,287],{"id":286},"map-runs-a-function-over-an-iterable","map runs a function over an iterable",[15,289,290,293,294,297,298,301],{},[18,291,292],{},"executor.map"," mirrors the built-in ",[18,295,296],{},"map",", but parallel. It returns results in ",[23,299,300],{},"input\norder"," and is the cleanest option when you apply one function to many inputs.",[50,303,305],{"className":52,"code":304,"language":54,"meta":55,"style":55},"from concurrent.futures import ProcessPoolExecutor\n\ndef heavy(n):\n    return sum(i * i for i in range(n))\n\nwith ProcessPoolExecutor() as pool:\n    for result in pool.map(heavy, [100_000, 200_000, 300_000]):\n        print(result)        # yielded in the order of the inputs\n",[18,306,307,318,322,331,360,364,375,405],{"__ignoreMap":55},[59,308,309,311,313,315],{"class":61,"line":62},[59,310,66],{"class":65},[59,312,70],{"class":69},[59,314,73],{"class":65},[59,316,317],{"class":69}," ProcessPoolExecutor\n",[59,319,320],{"class":61,"line":79},[59,321,83],{"emptyLinePlaceholder":82},[59,323,324,326,329],{"class":61,"line":86},[59,325,89],{"class":65},[59,327,328],{"class":92}," heavy",[59,330,96],{"class":69},[59,332,333,335,338,341,343,346,349,351,354,357],{"class":61,"line":99},[59,334,102],{"class":65},[59,336,337],{"class":135}," sum",[59,339,340],{"class":69},"(i ",[59,342,108],{"class":65},[59,344,345],{"class":69}," i ",[59,347,348],{"class":65},"for",[59,350,345],{"class":69},[59,352,353],{"class":65},"in",[59,355,356],{"class":135}," range",[59,358,359],{"class":69},"(n))\n",[59,361,362],{"class":61,"line":114},[59,363,83],{"emptyLinePlaceholder":82},[59,365,366,368,371,373],{"class":61,"line":119},[59,367,122],{"class":65},[59,369,370],{"class":69}," ProcessPoolExecutor() ",[59,372,142],{"class":65},[59,374,145],{"class":69},[59,376,377,380,383,385,388,391,394,397,399,402],{"class":61,"line":148},[59,378,379],{"class":65},"    for",[59,381,382],{"class":69}," result ",[59,384,353],{"class":65},[59,386,387],{"class":69}," pool.map(heavy, [",[59,389,390],{"class":135},"100_000",[59,392,393],{"class":69},", ",[59,395,396],{"class":135},"200_000",[59,398,393],{"class":69},[59,400,401],{"class":135},"300_000",[59,403,404],{"class":69},"]):\n",[59,406,407,410,413],{"class":61,"line":169},[59,408,409],{"class":135},"        print",[59,411,412],{"class":69},"(result)        ",[59,414,415],{"class":165},"# yielded in the order of the inputs\n",[15,417,418,420,421,423],{},[18,419,296],{}," re-raises the first exception when you reach that result during iteration. For more\ncontrol over completion order or errors, use ",[18,422,44],{}," instead.",[10,425,427],{"id":426},"collecting-results-as-they-finish","Collecting results as they finish",[15,429,430,431,433,434,437,438,441],{},"With ",[18,432,44],{}," you hold the futures and can react the moment each completes — useful when\nsome tasks are much slower than others. ",[18,435,436],{},"as_completed"," yields futures in ",[23,439,440],{},"finish"," order.",[50,443,445],{"className":52,"code":444,"language":54,"meta":55,"style":55},"from concurrent.futures import ThreadPoolExecutor, as_completed\n\nurls = [\"a\", \"b\", \"c\"]\n\nwith ThreadPoolExecutor() as pool:\n    futures = {pool.submit(fetch, url): url for url in urls}\n    for fut in as_completed(futures):\n        url = futures[fut]\n        print(url, fut.result())     # printed in whatever order finishes first\n",[18,446,447,458,462,489,493,504,524,536,546],{"__ignoreMap":55},[59,448,449,451,453,455],{"class":61,"line":62},[59,450,66],{"class":65},[59,452,70],{"class":69},[59,454,73],{"class":65},[59,456,457],{"class":69}," ThreadPoolExecutor, as_completed\n",[59,459,460],{"class":61,"line":79},[59,461,83],{"emptyLinePlaceholder":82},[59,463,464,467,469,472,476,478,481,483,486],{"class":61,"line":86},[59,465,466],{"class":69},"urls ",[59,468,132],{"class":65},[59,470,471],{"class":69}," [",[59,473,475],{"class":474},"sZZnC","\"a\"",[59,477,393],{"class":69},[59,479,480],{"class":474},"\"b\"",[59,482,393],{"class":69},[59,484,485],{"class":474},"\"c\"",[59,487,488],{"class":69},"]\n",[59,490,491],{"class":61,"line":99},[59,492,83],{"emptyLinePlaceholder":82},[59,494,495,497,500,502],{"class":61,"line":114},[59,496,122],{"class":65},[59,498,499],{"class":69}," ThreadPoolExecutor() ",[59,501,142],{"class":65},[59,503,145],{"class":69},[59,505,506,509,511,514,516,519,521],{"class":61,"line":119},[59,507,508],{"class":69},"    futures ",[59,510,132],{"class":65},[59,512,513],{"class":69}," {pool.submit(fetch, url): url ",[59,515,348],{"class":65},[59,517,518],{"class":69}," url ",[59,520,353],{"class":65},[59,522,523],{"class":69}," urls}\n",[59,525,526,528,531,533],{"class":61,"line":148},[59,527,379],{"class":65},[59,529,530],{"class":69}," fut ",[59,532,353],{"class":65},[59,534,535],{"class":69}," as_completed(futures):\n",[59,537,538,541,543],{"class":61,"line":169},[59,539,540],{"class":69},"        url ",[59,542,132],{"class":65},[59,544,545],{"class":69}," futures[fut]\n",[59,547,548,550,553],{"class":61,"line":274},[59,549,409],{"class":135},[59,551,552],{"class":69},"(url, fut.result())     ",[59,554,555],{"class":165},"# printed in whatever order finishes first\n",[15,557,558],{},"The dict maps each future back to its input so you know which result is which.",[10,560,562],{"id":561},"what-a-future-gives-you","What a Future gives you",[15,564,565,566,568,569,572],{},"A ",[18,567,33],{}," is a handle to a result that may not exist yet. Beyond ",[18,570,571],{},".result(timeout=...)","\nyou can inspect state and attach callbacks:",[50,574,576],{"className":52,"code":575,"language":54,"meta":55,"style":55},"fut = pool.submit(heavy, 1_000_000)\nfut.done()                       # False while running\nfut.add_done_callback(lambda f: print(\"finished:\", f.result()))\nfut.result(timeout=5)            # raises TimeoutError if not ready in 5s\n",[18,577,578,594,602,625],{"__ignoreMap":55},[59,579,580,583,585,588,591],{"class":61,"line":62},[59,581,582],{"class":69},"fut ",[59,584,132],{"class":65},[59,586,587],{"class":69}," pool.submit(heavy, ",[59,589,590],{"class":135},"1_000_000",[59,592,593],{"class":69},")\n",[59,595,596,599],{"class":61,"line":79},[59,597,598],{"class":69},"fut.done()                       ",[59,600,601],{"class":165},"# False while running\n",[59,603,604,607,610,613,616,619,622],{"class":61,"line":86},[59,605,606],{"class":69},"fut.add_done_callback(",[59,608,609],{"class":65},"lambda",[59,611,612],{"class":69}," f: ",[59,614,615],{"class":135},"print",[59,617,618],{"class":69},"(",[59,620,621],{"class":474},"\"finished:\"",[59,623,624],{"class":69},", f.result()))\n",[59,626,627,630,633,635,637,640],{"class":61,"line":99},[59,628,629],{"class":69},"fut.result(",[59,631,632],{"class":128},"timeout",[59,634,132],{"class":65},[59,636,159],{"class":135},[59,638,639],{"class":69},")            ",[59,641,642],{"class":165},"# raises TimeoutError if not ready in 5s\n",[15,644,645,646,648,649,652],{},"If the callable raised, ",[18,647,186],{}," re-raises that exception in the calling thread, and\n",[18,650,651],{},".exception()"," returns it without raising — so errors are never silently lost.",[10,654,656],{"id":655},"handling-exceptions-cleanly","Handling exceptions cleanly",[15,658,659],{},"Exceptions from workers surface only when you touch the result, so always read it:",[50,661,663],{"className":52,"code":662,"language":54,"meta":55,"style":55},"with ThreadPoolExecutor() as pool:\n    futures = [pool.submit(risky, x) for x in data]\n    for fut in as_completed(futures):\n        try:\n            fut.result()\n        except ValueError as e:\n            print(\"task failed:\", e)   # one bad task doesn't kill the rest\n",[18,664,665,675,694,704,712,717,731],{"__ignoreMap":55},[59,666,667,669,671,673],{"class":61,"line":62},[59,668,122],{"class":65},[59,670,499],{"class":69},[59,672,142],{"class":65},[59,674,145],{"class":69},[59,676,677,679,681,684,686,689,691],{"class":61,"line":79},[59,678,508],{"class":69},[59,680,132],{"class":65},[59,682,683],{"class":69}," [pool.submit(risky, x) ",[59,685,348],{"class":65},[59,687,688],{"class":69}," x ",[59,690,353],{"class":65},[59,692,693],{"class":69}," data]\n",[59,695,696,698,700,702],{"class":61,"line":86},[59,697,379],{"class":65},[59,699,530],{"class":69},[59,701,353],{"class":65},[59,703,535],{"class":69},[59,705,706,709],{"class":61,"line":99},[59,707,708],{"class":65},"        try",[59,710,711],{"class":69},":\n",[59,713,714],{"class":61,"line":114},[59,715,716],{"class":69},"            fut.result()\n",[59,718,719,722,725,728],{"class":61,"line":119},[59,720,721],{"class":65},"        except",[59,723,724],{"class":135}," ValueError",[59,726,727],{"class":65}," as",[59,729,730],{"class":69}," e:\n",[59,732,733,736,738,741,744],{"class":61,"line":148},[59,734,735],{"class":135},"            print",[59,737,618],{"class":69},[59,739,740],{"class":474},"\"task failed:\"",[59,742,743],{"class":69},", e)   ",[59,745,746],{"class":165},"# one bad task doesn't kill the rest\n",[10,748,750],{"id":749},"recap","Recap",[15,752,753,755,756,759,760,763,764,766,767,769,770,772,773,775,776,778],{},[18,754,20],{}," gives you one high-level API over two backends: ",[23,757,758],{},"ThreadPoolExecutor","\nfor I\u002FO-bound work (cheap, shared memory, GIL released on I\u002FO) and ",[23,761,762],{},"ProcessPoolExecutor","\nfor CPU-bound work (true parallelism, but arguments are pickled). Use ",[18,765,296],{}," for ordered\nresults of one function over many inputs, and ",[18,768,44],{}," + ",[18,771,436],{}," when you want\nresults as they finish or finer error control. A ",[18,774,33],{}," is a placeholder whose\n",[18,777,186],{}," blocks until ready and re-raises any worker exception, so always consume it.",[780,781,782],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}",{"title":55,"searchDepth":79,"depth":79,"links":784},[785,786,787,788,789,790,791,792],{"id":12,"depth":79,"text":13},{"id":37,"depth":79,"text":38},{"id":190,"depth":79,"text":191},{"id":286,"depth":79,"text":287},{"id":426,"depth":79,"text":427},{"id":561,"depth":79,"text":562},{"id":655,"depth":79,"text":656},{"id":749,"depth":79,"text":750},"How to use concurrent.futures — the high-level Executor API, the difference between ThreadPoolExecutor and ProcessPoolExecutor, working with Future objects, and map vs submit.","medium","md","Python",{},"\u002Fblog\u002Fpython-concurrent-futures-explained","\u002Fpython\u002Fconcurrency\u002Fconcurrent-futures",{"title":5,"description":793},"blog\u002Fpython-concurrent-futures-explained","Concurrency & Parallelism","concurrency","2026-06-19","p63lSQ5Ji7Jm-VGj3dUT-ATvgOEPKAmriRFITD0-GO0",1782244093597]