Tables with small write.target-file-size-bytes fail to ingest very large records #2795
Description
Apache Iceberg version
0.10.0 (latest release)
Please describe the bug 🐞
When using a small value for write.target-file-size-bytes, appending any records larger than this value result in the following exception:
File "/var/lang/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 485, in append
data_files = list(
^^^^^
File "/var/lang/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2774, in _dataframe_to_data_files
for batches in bin_pack_arrow_table(partition.arrow_table_partition, target_file_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2588, in bin_pack_arrow_table
batches = tbl.to_batches(max_chunksize=target_rows_per_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 5099, in pyarrow.lib.Table.to_batches
ValueError: 'max_chunksize' should be strictly positive
The offending lines appear to be here:
iceberg-python/pyiceberg/io/pyarrow.py
Lines 2680 to 2693 in abae20f
target_rows_per_file = target_file_size // avg_row_size_bytes could probably be changed to target_rows_per_file = max(1, target_file_size // avg_row_size_bytes) to ensure at least 1 row always gets written per chunk.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time