dew.data.sources.memory
Sources whose records are already resident in the caller’s memory.
| Name | Summary |
|---|---|
DataFrame | A table that hands itself over through the dataframe interchange protocol: a pandas or polars DataFrame, or an Arrow table. |
SparkFrame | |
Memory | Records, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable positional IDs. |
DataFrame
Section titled “DataFrame”class DataFrame(Protocol)A table that hands itself over through the dataframe interchange protocol: a pandas or polars DataFrame, or an Arrow table.
SparkFrame
Section titled “SparkFrame”class SparkFrame(Protocol)SparkFrame.toArrow
Section titled “SparkFrame.toArrow”def toArrow() -> DataFrameMemory
Section titled “Memory”class Memory( values: Records | Sequence[Batch] | Mapping[str, Sequence] | DataFrame, identity: str | None = None, shard_rows: int = 1024,)Records, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable
positional IDs. A table’s rows become records of its columns, its index none of them.
Memory.from_spark
Section titled “Memory.from_spark”def from_spark(frame: SparkFrame) -> MemoryGather a small pyspark 4.0+ frame through its Arrow handoff.
Memory.label
Section titled “Memory.label”def label() -> strMemory.describe
Section titled “Memory.describe”def describe() -> strMemory.fingerprint
Section titled “Memory.fingerprint”def fingerprint() -> Mapping[str, object]Its content, hashed once a compilation (dew.data.plan.once), which a plan, a cache key and
a description each read, its records the caller’s to change between two.
Memory.supports
Section titled “Memory.supports”def supports(read: Read) -> boolMemory.records
Section titled “Memory.records”def records(read: Read) -> RecordsMemory.manifest
Section titled “Memory.manifest”def manifest() -> ShardManifestMemory.metadata
Section titled “Memory.metadata”def metadata() -> Mapping[str, object]Memory.lengths
Section titled “Memory.lengths”def lengths() -> np.ndarray | None