dew.data.sinks
Immutable exports to filesystem URLs: ArrayRecord shards Dew writes, and Parquet or any other
format a native writer writes from one Arrow stream (Writer), all committed by one publisher.
| Name | Summary |
|---|---|
write_array_record | payloads written to a local ArrayRecord file at path, one record each in their order, the file closed whether or not they all were: the shard an export and a cache fill each upload. |
Writer | A native writer an export hands its rows to as one Arrow stream (pyarrow.RecordBatchReader, the stream Arrow’s dataset writer, Delta’s, Lance’s and DuckDB’s take, and a Spark session reads as a table): it writes them as files of format, one Files reads, under directory of filesystem, Arrow’s over the export’s storage. |
passed | One pass over a store’s or a chain’s records, in their one order, as export writes them: a store as a chain of it, by a read it supports, physical shards or none. |
export | Write immutable shards, then conditionally commit their manifest and _SUCCESS. |
write_array_record
Section titled “write_array_record”def write_array_record(path: Path, payloads: Iterable[bytes]) -> Nonepayloads written to a local ArrayRecord file at path, one record each in their order, the
file closed whether or not they all were: the shard an export and a cache fill each upload.
Writer
Section titled “Writer”class Writer(Protocol)A native writer an export hands its rows to as one Arrow stream (pyarrow.RecordBatchReader,
the stream Arrow’s dataset writer, Delta’s, Lance’s and DuckDB’s take, and a Spark session reads
as a table): it writes them as files of format, one Files reads, under directory of
filesystem, Arrow’s over the export’s storage. The export then lists what it wrote and commits
its manifest and _SUCCESS after them (commit_export), the one publisher every export shares,
so a reader finds the whole export or, where the writer stopped, none. Files reads the files
in the order of their paths.
Writer.write
Section titled “Writer.write”def write( rows: pa.RecordBatchReader, directory: str, filesystem: pafs.FileSystem,) -> Nonepassed
Section titled “passed”def passed(values: Source | Dataset) -> Iterator[Batch]One pass over a store’s or a chain’s records, in their one order, as export writes them: a
store as a chain of it, by a read it supports, physical shards or none. Close it when done.
export
Section titled “export”def export( values: Source | grain.IterDataset | Iterable[Batch | bytes | None] | Pipeline, url: str, *, format: Literal['parquet', 'array_record'] | Writer = 'parquet', rows_per_shard: int = 100000, partitioning: tuple[str, ...] | None = None, storage_options: Mapping[str, object] | None = None, metadata: Mapping[str, object] | None = None, length_field: str | None = None,) -> Mapping[str, object]Write immutable shards, then conditionally commit their manifest and _SUCCESS.
ArrayRecord shards are Dew’s own, one of rows_per_shard records each. Parquet is Arrow’s
writer (_Parquet), each partition column a Hive directory; any other Writer, a native one
of the caller’s, writes the rows as one Arrow stream. Each export owns a UUID directory, so
racing exporters cannot share shards, and commits only once its writer has written them all.