Skip to content

fix: read parquet metadata after writing#3668

Open
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/parquet-metadata-read-metadata
Open

fix: read parquet metadata after writing#3668
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/parquet-metadata-read-metadata

Conversation

@GayathriSrividya

Copy link
Copy Markdown
Contributor

closes #3108

Summary

  • reopen newly written parquet files and read metadata via pyarrow.parquet.read_metadata
  • use the read metadata consistently when computing DataFile statistics
  • add a regression test covering the write_file metadata path

Testing

  • /opt/homebrew/bin/ruff format pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
  • /opt/homebrew/bin/ruff check --fix pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
  • PYTHONPATH=. uv run --project . pytest tests/io/test_pyarrow.py -k write_file_reads_parquet_metadata -xvs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistency in arguments to parquet_metadata parameter of data_file_statistics_from_parquet_metadata() in iceberg-python/pyiceberg/io/pyarrow.py

1 participant