[rahulsmahadev]

Rationale for this change

Table.append fails for a table identity-partitioned on a uuid column:

schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "u", UUIDType()))
tbl = catalog.create_table("db.t", schema=schema, partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "u")))
tbl.append(pa.table({"id": pa.array([1], pa.int64()), "u": pa.array([uuid.UUID(int=1).bytes], pa.uuid())}))
# pyarrow.lib.ArrowNotImplementedError: Keys of type extension<arrow.uuid>

The identity transform returns the arrow.uuid column unchanged, and PyArrow has no group by or equality kernel for extension types, so the group_by in _determine_partitions fails before any file is written. Bucketing on the uuid column works because the bucket transform returns an int.

This change groups and filters on the storage type of extension-typed partition columns (fixed_size_binary(16) for uuid). PyArrow does not plan compute kernels for extension types (apache/arrow#39024 was closed as not planned), so this stays needed. The data columns are unchanged, so the Parquet files keep the uuid type, and the partition values are the 16 bytes the manifests store for a uuid.

Not changed here: an equality filter on a uuid identity partition still plans no files, because partition pruning compares the UUID read from the manifest with a bytes literal. That also affects tables written by other engines, and I'll send it as a separate fix.

Are these changes tested?

Yes. Two new tests in tests/io/test_pyarrow.py fail with ArrowNotImplementedError: Keys of type extension<arrow.uuid> without this change and pass with it, on pyarrow 25 and 18:

• test_determine_partitions_with_uuid_column: the partition keys (including a null partition), the rows of each partition, and that the data keeps the uuid type.
• test_append_uuid_to_table_partitioned_on_uuid: an append through the catalog fixture, the rows read back, and the partition values of the data files.

Are there any user-facing changes?

Bug fix only: appending to a table identity-partitioned on a uuid column no longer raises. No API change.

This pull request and its description were written by Isaac.

[rahulsmahadev]

@Fokko @DG47 could you take a look when you have a chance? It is a small change in _determine_partitions (next to #4121 and #4344) so appends work for identity partitions on uuid columns.

[Fokko]

Thanks @rahulsmaydevv! Verified locally: both tests fail on main with the ArrowNotImplementedError and pass with the change, and the manifest partition values read back correctly. LGTM! 🚀