
How to Make Python Wait - jangid
https://blog.miguelgrinberg.com/post/how-to-make-python-wait
======
BiteCode_dev
Don't follow this blog post advice.

Manually dealing with threads and processes is useful if you want to build a
framework or a very complex workflow. But chances are you just want to run
stuff concurrently, in the background.

In that case (which is most people case), you really want to use one of the
stdlib pools: it takes care of sync, serialization, communication with queues,
worker life cycle, task distribution, etc. for you.

Plus in such case, waiting for results is super easy:

    
    
        import time
        import random
    
        from concurrent.futures import ThreadPoolExecutor, as_completed
    
        # the work to distribute
        def hello():
            seconds = random.randint(0, 5)
            print(f'Hi {seconds}s')
            time.sleep(seconds)
            print(f'Bye {seconds}s')
            return seconds
    
        # max concurrency is 2
        executor = ThreadPoolExecutor(max_workers=2)
       
        # submit the work
        a = executor.submit(hello)
        b = executor.submit(hello)
    
        # and here we wait for results
        for future in as_completed((a, b)):
            print(future.result())
    

Want multiple process instead ? It's the same API:

    
    
        import time
        import random
    
        from concurrent.futures import ProcessPoolExecutor, as_completed
    
        def hello():
            seconds = random.randint(0, 5)
            print(f'Hi {seconds}s')
            time.sleep(seconds)
            print(f'Bye {seconds}s')
            return seconds
    
        # Don't forget this for processes, or you'll get in trouble
        if __name__ == "__main__":
    
            executor = ProcessPoolExecutor(max_workers=2)
    
            a = executor.submit(hello)
            b = executor.submit(hello)
    
            for future in as_completed((a, b)):
                print(future.result())
    

This is Python. Don't make your life harder that it needs to be.

~~~
ZeroCool2u
This example still involves a lot of manual work. It's often times even
easier.

    
    
        from concurrent.futures import ProcessPoolExecutor
        import string
        
        def hello() -> int:
            seconds = random.randint(0, 5)
            print(f'Hi {seconds}s')
            time.sleep(seconds)
            print(f'Bye {seconds}s')
            return seconds
        
        # Don't forget this for processes, or you'll get in trouble
        if __name__ == "__main__":
        
        inputs = list(string.printable)
        results = []
        
        # You can sub out ProcessPool with ThreadPool. 
        with ProcessPoolExecutor() as executor:
            results += executor.map(hello, inputs)
        
        [print(s) for s in results]

~~~
BiteCode_dev
"with" is always a good idea indeed.

But be careful, map() and submit() + as_completed() don't have the same
effect.

The first one will give you the result in the insertion order, while the later
give you the results in the order they are completed.

------
amelius
Or just create a Queue(), give it as a parameter to the long calculation so it
can store the result in it, and in the main thread just do q.get(). There are
probably a dozen other synchronization primitives you can use, but this one is
very versatile and you only need to keep one API in your head. Also, this
approach somewhat mimics the concept of channels in Go.

~~~
hangonhn
This is among the best concurrent programming advices I've ever been given and
have given out. Use a queue and a pool of workers. It makes your
synchronization so much easier and simpler.

------
baq
automatic progress bars for console programs in Python:
[https://tqdm.github.io/](https://tqdm.github.io/)

------
pyjonista
I often find myself overwhelmed thinking about concurrency/parallelism in
Python: time.sleep, concurrent.futures, multiprocessing, threads, queues,
asyncio, uvloop, tornado, twisted, events, coroutines, curio, trio, select,
gevent, eventlet...

What is a good solution that the majority of us simple humans with limited
time and resources should pursue? I want to write software that competes with
similar solutions written in Go. Does that exist? Is there _one_ solution that
it's safe to adopt and generally recommended by the community? I used to think
that asyncio was the solution to that but I was unable to fully understand its
API. Please help!

~~~
rickycook
i’d say that multiprocessing probably covers 90% of “do it in the background”
tasks, and asyncio covers 90% of async networking tasks

futures, uvloop, tornado, twisted, coros, gevent, etc are kinda just all
related to, or do similar to asyncio

threading is kinda not as useful as you might like in python because of the
GIL (simplistically, assume that python can only do 1 thing at once regardless
of having multiple threads available or not until you know why that’s not
always the case)

------
pjc50
The general strategy of "use the OS threading primitives" like join and events
works in most languages.

------
Akababa
Nice, I learned about a couple new things. I take it these can all be
implemented with the lock/semaphore primitives?

------
williamDafoe
The idea of waiting with a progress indicator has a large bug, on an 8-bit or
16-bit machine you cannot read or write atomically from a progress variable. I
guess the code works because of the interpreter lock that cripples python but
it's very bad hygeine in all languages.

~~~
beering
Can you elaborate 1) why you can't r/w atomatically from a progress variable
(even if it's a byte?), 2) why are we writing Python code for 8-bit or 16-bit
machines, and 3) why is it bad to take advantage of our language's guarantees
on execution, e.g. with Python and JavaScript?

~~~
saagarjha
Not every processor supports atomic memory accesses.

------
2rsf
Good article on an important topic for novices, but why is it so long ?

~~~
PostOnce
Because it helps you learn.

You can tell someone a brief answer to memorize, but to teach them anything
takes a little more explaining.

For example, Question #1 is often "how do I do this in some particular way",
but 2 and 3, "why" and "when" should I do it that particular way -- they take
more explaining than #1 -- and through that explanation, you start to get good
at your craft.

------
ngcc_hk
Is javascript a better mainstream language to do this?

