dew.data.sources.urls
URL catalogs and volatile fetches, executed by the ordinary Grain worker path.
| Name | Summary |
|---|---|
Fetch | Fetch the bytes at a record’s url into output, in one attempt: a url that fails now is dropped as transient, for a later epoch or a cache refill to fetch again; one its server reports gone (HTTP 404 or 410) is dropped for good. |
class Fetch( field: str = 'url', output: str = 'bytes', timeout: float = 10.0, max_bytes: int = 64 * 2 ** 20, threads: int | None = None,)Fetch the bytes at a record’s url into output, in one attempt: a url
that fails now is dropped as transient, for a later epoch or a cache
refill to fetch again; one its server reports gone (HTTP 404 or 410) is
dropped for good. An HTTP(S) url is read through fsspec’s aiohttp session
as urllib would read it, its redirects and environment’s proxies
included, so a reader waits at most timeout on it from connecting to its
last byte, however slowly a server sends; any other url (a file’s) by
urllib. Resolving a host is the system resolver’s, which takes no timeout.
A cache never freezes a fetch’s outcome; only snapshot(url) seals
outcomes. threads fetches that many records’ urls at once on a pool of
the stage’s own, whatever the read’s threads, and delivers them in the
order they were read; however the step is read (the shards an interleave
opens, a cache’s reads of its inputs), each loading process runs at most
that many of its fetches at once. It changes no record, so no plan or key
records it.