Skip to content

dew.data.sources.urls

URL catalogs and volatile fetches, executed by the ordinary Grain worker path.

NameSummary
FetchFetch the bytes at a record’s url into output, in one attempt: a url that fails now is dropped as transient, for a later epoch or a cache refill to fetch again; one its server reports gone (HTTP 404 or 410) is dropped for good.

dataclass source

class Fetch(
field: str = 'url',
output: str = 'bytes',
timeout: float = 10.0,
max_bytes: int = 64 * 2 ** 20,
threads: int | None = None,
)

Fetch the bytes at a record’s url into output, in one attempt: a url that fails now is dropped as transient, for a later epoch or a cache refill to fetch again; one its server reports gone (HTTP 404 or 410) is dropped for good. An HTTP(S) url is read through fsspec’s aiohttp session as urllib would read it, its redirects and environment’s proxies included, so a reader waits at most timeout on it from connecting to its last byte, however slowly a server sends; any other url (a file’s) by urllib. Resolving a host is the system resolver’s, which takes no timeout. A cache never freezes a fetch’s outcome; only snapshot(url) seals outcomes. threads fetches that many records’ urls at once on a pool of the stage’s own, whatever the read’s threads, and delivers them in the order they were read; however the step is read (the shards an interleave opens, a cache’s reads of its inputs), each loading process runs at most that many of its fetches at once. It changes no record, so no plan or key records it.