Format Registry & Magic-Byte Sniffing

Every file access in basaltic-red resolves to a FormatHandler, the pluggable abstraction in src/engine/formats/mod.rs.

The FormatHandler Trait

pub struct OpenedSource {
    pub schema: SchemaRef,
    pub batches: Box<dyn Iterator<Item = Result<RecordBatch, BazanError>> + Send>,
}

pub trait FormatHandler: Send + Sync {
    fn open(&self, file_path: &str, batch_size: usize) -> Result<OpenedSource, BazanError>;
    fn process_file(&self, engine: &MatrixEngine, file_path: &str, batch_size: usize)
        -> Result<(usize, usize, usize), BazanError>;
    fn read_range(&self, file_path: &str, offset: usize, limit: usize, batch_size: usize)
        -> Result<RecordBatch, BazanError>;
    fn open_with_columns(&self, file_path: &str, batch_size: usize, columns: &[String])
        -> Result<OpenedSource, BazanError>;
    fn read_range_columns(&self, /* ... */) -> Result<RecordBatch, BazanError>;
}

Handlers return a lazy OpenedSource, an Arrow schema plus a streaming batch iterator. The CSV/JSON readers and columnar Parquet/IPC readers yield Arrow batches directly; the Avro, MessagePack, and XLSX adapters use plugins/base_templates/row_chunker.rs to convert decoded rows into batches.

Built-in Formats

HANDLERS in formats/mod.rs currently registers 16 extensions. Rows below group aliases that share one handler. Locator inspection and slice routing live in map.rs and slice.rs.

All slice_cols calls preserve requested order and error if a field is missing. Parquet, Arrow IPC/Feather, and ORC push projection into their readers; mapped CSV/TSV/PSV/TXT parses selected columns while reading complete selected record bytes. JSON, Avro, MsgPack, and XLSX map paths decode the selected row range then project. Map entries require matching file size and modification time. Missing/stale entries or entries for another file fall back; an unreadable sidecar returns an error. Dynamic handlers for built-in extensions bypass built-in locators. Dynamic delimited handlers use plugins/base_templates/delimited.rs with registration-selected delimiter/header.

Handler Resolution Order

resolve_handler_for_file() tries, in order:

1. Extension lookup (handler_for), O(1), checks the dynamic registry first, then the static table. Extension case is normalized to lowercase.
2. Magic-byte sniffing (sniff_format_from_file), reads the first 512 bytes and inspects them via sniff_format_from_bytes:

Routes that use resolve_handler_for_file() can also open extensionless files through magic-byte sniffing. filter_files_parallel currently looks up handlers by extension, so its input files need a registered extension.

Dynamic Registration from Python

The dynamic registry lives behind a RwLock<HashMap<String, Arc<dyn FormatHandler>>> and is manipulated via br.formats.*:

import basaltic_red as br

br.formats.register_delimited(ext="dat", delimiter="|", has_header=True)
print(br.formats.list_formats())     # includes "dat" among the built-ins
table = br.read.slice_rows("data/custom.dat", offset=0, limit=50)
br.formats.unregister_format("dat")  # returns True if it existed

handler_for() prefers a registered handler over a built-in handler for the same extension. Slice APIs honor that override even when a Lake Map exists; the map records no built-in locator for the overridden extension and slicing falls back to the registered handler. SQL's native DataFusion ListingTable readers use their own format readers and do not apply dynamic overrides; see DataFusion SQL.