Paging Through a Parquet File in DuckDB: File_row_number or Offset?

·Hacker News··

You have a large Parquet file and an API that has to return its contents, but responses have a size ceiling, so the caller pages through it. LIMIT/OFFSET is the obvious way to write that and it is the wrong one. I measured file_row_number against it: 2.53x faster on a file with 163 row groups. Speed is the boring half of the answer. OFFSET will also hand back the wrong rows without telling you.

Read full article →

Related Articles

Revolut confirms customer data breach through fake government requests
tdrz · Hacker News · 9h ago
google.com/goto: Google's anti-scraping update
1e1a · Hacker News · 1d ago
Will there be a 7G?
Betelbuddy · Hacker News · 1d ago
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
theanonymousone · Hacker News · 22h ago
After Math
throwaway81523 · Hacker News · 16h ago