Skip to content

Fix inspect file and partition tables for UUID columns - #4358

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/inspect-uuid-columns
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/inspect-uuid-columns

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

The file, entry and partition metadata tables raise on any table with a uuid column:

schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "u", UUIDType()))
tbl = catalog.create_table("default.t", schema=schema)
tbl.append(pa.table({"id": pa.array([1], pa.int64()), "u": pa.array([uuid.UUID(int=1).bytes], pa.uuid())}))
tbl.inspect.files()
# pyarrow.lib.ArrowNotImplementedError: extension

readable_metrics holds the uuid bounds as arrow.uuid values inside a struct, and pyarrow cannot build an extension type nested in a struct from Python values, so pa.Table.from_pylist raises. This affects entries(), files(), data_files(), delete_files(), all_files(), all_data_files(), all_delete_files(), and partitions() for a table partitioned by identity on a uuid column, even when the bounds are null.

This change builds these tables with the storage types (fixed_size_binary(16) for a uuid) and casts the result to the unchanged schema. UUID partition values are passed as their 16 bytes, which pyarrow 18 (the minimum supported version) needs. UUID row filters (#2372) are a separate issue and are not touched.

Are these changes tested?

Yes. Three new tests in tests/table/test_inspect.py fail with ArrowNotImplementedError: extension without the fix and pass with it, on pyarrow 25 and 18:

  • test_inspect_files_and_entries_render_uuid_bounds: the bounds and the arrow.uuid output type for the file and entry tables.
  • test_inspect_files_of_empty_table_with_uuid_column: files() and all_files() on a table without snapshots.
  • test_inspect_partitions_and_files_render_uuid_partition: partitions(), files() and entries() for a table partitioned by identity on a uuid column.

Are there any user-facing changes?

Bug fix only: these metadata tables no longer raise on tables with a uuid column. The output schema is unchanged.

This pull request and its description were written by Isaac.

inspect.files(), entries() and partitions() raised ArrowNotImplementedError
on tables with a uuid column: pa.Table.from_pylist cannot build the
arrow.uuid type nested in readable_metrics or a partition struct. Build
these tables from the storage types and cast them to the schema, so the
output types are unchanged. UUID partition values are passed as their 16
bytes, which pyarrow 18 requires.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@geruh @soumya-ghosh could you take a look when you have a chance? It fixes the file and partition metadata tables from #1626 for tables with uuid columns.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant