Basaltic-Red

High-performance Data Lake acceleration & data-quality toolkit in Rust + Apache Arrow (Python via PyO3)

Overview

basaltic-red is a Rust compute core for file-based data lakes with Python bindings. It provides a memory-mapped catalog (.br_map.bazan), format-aware row and column slicing, parallel quality filtering with per-row audit codes, and DataFusion SQL execution over Arrow batches. Results return as pyarrow.Table or RecordBatch objects for use with Polars, DuckDB, or pandas.

Demo in demo.ipynb: NYC TLC Yellow Taxi 2009 to 2025, with 204 Parquet files, 29.66 GB, and 1,826,960,642 rows by 20 columns.

graph LR
    A["Raw files (204 Parquet / 29.66 GB)"] --> B["Rust core"]
    B --> C["Catalog (.br_map.bazan / memmap2)"]
    B --> D["Parallel filter (Rayon, 1.82B rows)"]
    B --> E["DataFusion SQL stream"]
    D --> F["Clean (1.78B)"]
    D --> G["Trash (46.7M + audit codes)"]
    E --> H["DuckDB / Polars"]

What it does

• Catalog (.br_map.bazan): one Arrow IPC payload at the lake root; warm reads via memmap2 avoid traversing directory trees. Doctor reports HEALTHY, DRIFT_DETECTED, or HEALED.
• Slicing: slice_rows and slice_cols return the requested range. A healthy map lets supported readers skip unrelated blocks; some formats still decode extra data, and XLSX materializes the worksheet through Calamine.
• Filtering: dynamic rules evaluated per batch; invalid rows carry an audit_error_code (u64 bitmask, chunked when rules exceed 64).
• SQL: DataFusion session; execute_sql_stream returns a PyBatchIterator whose to_pyarrow() method hands Arrow batches to Polars or DuckDB. Compatible buffers may be shared at the Arrow interop boundary; readers and downstream conversions can still decode or materialize data.
• Formats: custom delimited formats via br.formats.register_delimited; direct single-file reads can identify some extension-less files by magic-byte sniffing. Directory discovery for Lake Map, Lake Doctor, and filter_files_parallel is extension-based and requires an active handler.

Demo measurements

Single run of demo.ipynb on Apple Silicon / macOS. Hardware dependent, provided for reference.

Filtering loops are plain Rust over Arrow arrays, auto-vectorized by LLVM. No hand-written SIMD intrinsics.

Getting started

uv add "git+https://github.com/VanDung-dev/basaltic-red.git"

import basaltic_red as br

report = br.lake.doctor("data/", auto_heal=True)
print(report["status"], report["total_files"])

clean, trash = br.filter.filter_matrix("data/sample.parquet", rules=[
    "passenger_count >= 1",
    "trip_distance > 0.0",
    "fare_amount > 0.0",
])

Full walkthrough: Quickstart · Lake Doctor · Benchmarks