Paging Through a Parquet File in DuckDB: File_row_number or Offset?

·Hacker News··

You have a large Parquet file and an API that has to return its contents, but responses have a size ceiling, so the caller pages through it. LIMIT/OFFSET is the obvious way to write that and it is the wrong one. I measured file_row_number against it: 2.53x faster on a file with 163 row groups. Speed is the boring half of the answer. OFFSET will also hand back the wrong rows without telling you.

Read full article →

Related Articles

GCC steering committee announces AI policy
arto · Hacker News · 5h ago
Why Is Everyone Trying to Build a Solid-State Battery?
crescit_eundo · Hacker News · 4h ago
AI's top startups are barely publishing their research
YeGoblynQueenne · Hacker News · 19h ago
Why Don't People Use Formal Methods? (2019)
Thom2503 · Hacker News · 4h ago
Document-borne AI worms can self-propagate through Copilot for Word
Canopy9560 · Hacker News · 1d ago