Python线程-动态更改并行线程的数量

我正在尝试并发.futures.ThreadPoolExecutor并行运行一些作业。它使我可以在初始化时设置一次max_workers。但是出于某种原因,我的要求是即时更改活动线程的数量。我找不到任何办法。试图即时重置“ _max_workers”,但这没有影响。

这里是代码。

from concurrent.futures import ThreadPoolExecutor, as_completed
import urllib.request
from random import randint
from time import sleep

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the url and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

if __name__ == "__main__":
    pool = ThreadPoolExecutor(1)
    fs = []
    for url in URLS:
        fs.append(pool.submit(load_url, url, 60))

    r = randint(1,5)
    print ("Changing max_workers to: {}".format(r))
    pool._max_workers = r
    print ("Current queue size: {}. Number of threads in parallel: {}".format(pool._work_queue.qsize(), len(pool._threads)))

    for f in as_completed(fs):
        r = randint(1,5)
        print ("Changing max_workers to: {}".format(r))
        pool._max_workers = r
        print ("Current queue size: {}. Number of threads in parallel: {}".format(pool._work_queue.qsize(), len(pool._threads)))

        try:
            data = f.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('Page is %d bytes' % (len(data)))

    print ("done")

这里是打印内容:

Changing max_workers to: 3  
Current queue size: 4. Number of threads in parallel: 1  
Changing max_workers to: 1  
Current queue size: 3. Number of threads in parallel: 1  
Page is 233315 bytes  
Changing max_workers to: 4  
Current queue size: 2. Number of threads in parallel: 1  
Page is 1127911 bytes  
Changing max_workers to: 2  
Current queue size: 1. Number of threads in parallel: 1  
Page is 2741410 bytes  
Changing max_workers to: 4  
Current queue size: 0. Number of threads in parallel: 1  
Page is 296361 bytes  
done  

有没有一种方法可以更改max_worker,它将更改pool._threads中的活动线程?

0
投票

嗯,无法弄清楚如何使用current.futures.ThreadPoolExecutor完成此任务,但是仍然可以通过在线程周围使用一些自定义逻辑来设法在运行时更改活动线程的数量。

这是最终的工作代码-https://github.com/gireeshp/API-Hitting-Simulator