Parallelism
Thread-based Parallelism
The 4.x version of the Python SDK significantly improves how the SDK handles the Global Interpreter Lock (GIL). As part of the improvements, options available in previous series of the SDK (2.x and 3.x) are no longer needed.
Notable options that are no longer utilized (or available) in the 4.x SDK:
lockmodeClusterOption has been deprecated as it is a no-op (i.e. has no functionality) and will be removed in a future version of the SDKunlock_giloption is no longer available
Due to the underlying architecture of the 4.x SDK it is recommended to share a cluster instance across multiple threads. However, creating a cluster instance per thread can work as well. As should be the standard for all applications, we recommend sufficient testing to determine the path that best fits the needs of your application.
See the Python threading docs for details on thread-based parallelism.
Process-based Parallelism
Due to the underlying architecture of the 4.x SDK a cluster instance cannot be shared across across multiple processes; therefore, a cluster instance must be created for each process used in a multiprocessing scenario. As should be the standard for all applications, we recommend sufficient testing to determine the path that best fits the needs of your application.
See the Python multiprocessing docs for details on process-based parallelism.