get_async(submit, num_workers, dsk, result, cache=None, get_id=<function default_get_id at 0x0000000>, rerun_exceptions_locally=None, pack_exception=<function default_pack_exception at 0x0000000>, raise_exception=<function reraise at 0x0000000>, callbacks=None, dumps=<function identity at 0x0000000>, loads=<function identity at 0x0000000>, chunksize=None, **kwargs)

This is a general version of various asynchronous schedulers for dask. It takes a concurrent.futures.Executor.submit function to form a more specific get method that walks through the dask array with parallel workers, avoiding repeat computation and minimizing memory use.

Parameters

submit : function

A concurrent.futures.Executor.submit function

num_workers : int

The number of workers that task submissions can be spread over

dsk : dict

A dask dictionary specifying a workflow

result : key or list of keys

Keys corresponding to desired data

cache : dict-like, optional

Temporary storage of results

get_id : callable, optional

Function to return the worker id, takes no arguments. Examples are threading.current_thread and multiprocessing.current_process .

rerun_exceptions_locally : bool, optional

Whether to rerun failing tasks in local process to enable debugging (False by default)

pack_exception : callable, optional

Function to take an exception and dumps method, and return a serialized tuple of (exception, traceback) to send back to the scheduler. Default is to just raise the exception.

raise_exception : callable, optional

Function that takes an exception and a traceback, and raises an error.

callbacks : tuple or list of tuples, optional

Callbacks are passed in as tuples of length 5. Multiple sets of callbacks may be passed in as a list of tuples. For more information, see the dask.diagnostics documentation.

dumps: callable, optional :

Function to serialize task data and results to communicate between worker and parent. Defaults to identity.

loads: callable, optional :

Inverse function of :None:None:`dumps`. Defaults to identity.

chunksize: int, optional :

Size of chunks to use when dispatching work. Defaults to 1. If -1, will be computed to evenly divide ready work across workers.

Asynchronous get function

See Also

threaded.get

Examples

See :

Local connectivity graph

Hover to see nodes names; edges to Self not shown, Caped at 50 nodes.

Using a canvas is more power efficient and can get hundred of nodes ; but does not allow hyperlinks; , arrows or text (beyond on hover)

SVG is more flexible but power hungry; and does not scale well to 50 + nodes.

All aboves nodes referred to, (or are referred from) current nodes; Edges from Self to other have been omitted (or all nodes would be connected to the central node "self" which is not useful). Nodes are colored by the library they belong to, and scaled with the number of references pointing them

---

File: /dask/local.py#346
type: <class 'function'>
Commit: