[rahulsmahadev]

Rationale for this change

add_files looks up each Parquet column path in a mapping built from the table's current column names, so a file that is valid for the table fails when its paths differ from those names:

catalog.create_table("db.t", schema=Schema(NestedField(1, "a", LongType()), NestedField(2, "b", StringType())))
with catalog.load_table("db.t").update_schema() as update:
    update.rename_column("a", "c")
# The file has field IDs 1 and 2, and column 1 is still named "a" in it.
catalog.load_table("db.t").add_files(["a.parquet"])  # KeyError: 'a'

The same happens for a nested field renamed after the file was written (KeyError: 'address.zip') and for a list whose element is named item, the layout pyarrow writes with use_compliant_nested_type=False (KeyError: 'tags.list.item'). #716 reported the list case; the stale bot closed it without a fix.

The name-based mapping can also record statistics under the wrong column. If the table drops a (field ID 1) and adds a new a (field ID 3), an old file's a is recorded under field ID 3, so a scan for a IS NULL prunes the file and misses its rows.

This change builds the mapping from the file's own column paths and field IDs (with name mapping for files without IDs), so statistics follow field IDs as they do for files PyIceberg writes.

Are these changes tested?

Yes, in tests/io/test_pyarrow.py. test_add_files_matches_columns_by_field_id covers the nested rename and the item list element with and without field IDs. test_add_files_rejects_columns_missing_from_table covers a file column whose field ID the table lacks and a column whose name matches but whose field ID differs. Both fail without this change.

Are there any user-facing changes?

Yes:

• add_files accepts files whose columns were renamed in the table and files whose list element is not named element.
• A file column whose field ID is not in the table now gets the existing "PyArrow table contains more columns" ValueError instead of a KeyError.
• A file column whose name matches a table column but whose field ID differs is rejected with the same error. Before, such a file was added with its statistics under the wrong column.

This pull request and its description were written by Isaac.

[rahulsmahadev]

@kevinjqliu @jeroko could you take a look when you have a chance? It builds on #2662 and keys add_files statistics by the file's own field IDs (the list case is #716).