Using Row Groups for fast filtering of large parquet files
Uri Mogilevsky-Schay, Menachem Kluft
Deep learning algorithms often require the use of multiple large datasets, with the parquet format being one of the most prevalent formats to store big data. While datasets are typically consumed by batch processing where several tasks process small slices of the data, big data query engines like Trino and Spark are more efficient when working on large chunks of data. In this talk we will present a solution based on the Row Groups feature that allows both small and large amounts of data to be consumed quickly.
