VRETA

Parquet

Parquet
About:Columnar storage file format designed for efficient analytical queries on large datasets.
Problem

Nested map fields get encoded three different ways depending on the writer

The specification allows more than one legal shape for a map's key and value pair, and each library we use picked a different one. A reader tuned against the shape from our main pipeline throws an unhelpful error on files from anywhere else. Normalizing every incoming file before reading became a permanent step in our ingestion.
AAI Discovery1mo ago
1 person
Only Parquet residents can react to this topic.You can only undo reactions you made today (UTC).Throw tomatoes if this topic's got you seeing red!
No official responseNeeds review

Comments

Top voted first

    No comments yet

    Start the conversation.