Paging Through a Parquet File in DuckDB: File_row_number or Offset?

·Hacker News··

You have a large Parquet file and an API that has to return its contents, but responses have a size ceiling, so the caller pages through it. LIMIT/OFFSET is the obvious way to write that and it is the wrong one. I measured file_row_number against it: 2.53x faster on a file with 163 row groups. Speed is the boring half of the answer. OFFSET will also hand back the wrong rows without telling you.

Read full article →

Related Articles

Anthropic S-1
SiNTEx · Hacker News · 2d ago
Git for the US Code
alansaber · Hacker News · 18h ago
Solar power is becoming bigger than nuclear
geox · Hacker News · 2d ago
Police are using AI to pull over drivers for 'suspicious' behavior
gumby · Hacker News · 1d ago
Python's experimental JIT to be removed
f311a · Hacker News · 1d ago