[Bug]: AuthenticationFailed when querying multiple Unity catalog tables in same Datafusion context
binding/rustunity-catalogawaiting-feedback
### What happened?
When using the catalog-unity crate to register a catalog in Datafusion, a query with multiple tables, or consecutive queries in the same context, fails.
### Expected behavior
A single Datafusion context can query multiple tables from Unity.
### Operating System
macOS
Linux
### Binding
Rust
### Bindings Version
0.16.0
### Steps to reproduce
1. Have a Unity catalog catalog in Databricks (enable external access)
2. Register the catalog in Datafusion
3. Query table1 (stored in `abfss://cont@acc.dfs.core.windows.net/path1`)
4. Query table2 (stored in `abfss://cont@acc.dfs.core.windows.net/path2`)
(OBS: same storage account and container)
```rust
let ctx = SessionContext::new();
ctx.register_catalog("new_catalog", Arc::new(new_catalog));
let df = ctx
.sql("SELECT * FROM new_catalog.schema.table1 LIMIT 100")
.await
.expect("run query");
df.show().await.expect("show results");
let df = ctx
.sql("SELECT * FROM new_catalog.schema.table2 LIMIT 100")
.await
.expect("run query");
df.show().await.expect("show results");
```
### Relevant logs
```bash
ParquetError(General("Failed to fetch metadata for file xxxx: Parquet error: External: The operation lacked the necessary privileges to complete for path xxxx: Error performing GET REDACTED in 64.81975ms - Server returned non-2xx status code: 403 Forbidden: \u{feff}<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.Message><AuthenticationErrorDetail>Signature did not match. String to sign used was xxxx </AuthenticationErrorDetail></Error>"))
```
### Investigations (and speculation)
The temporary storage credentials created at:
https://github.com/delta-io/delta-rs/blob/54f43b6fc68d5db7030832b379591d9486105c61/crates/catalog-unity/src/lib.rs#L838
Gives a vended credential for Azure storage including the path to the table root. (A SAS for a path)
The credential is then used to create a table, including storage connection here:
https://github.com/delta-io/delta-rs/blob/54f43b6fc68d5db7030832b379591d9486105c61/crates/catalog-unity/src/datafusion.rs#L248
The registration is likely to be on a storage root level, mentioned here:
https://github.com/delta-io/delta-rs/blob/54f43b6fc68d5db7030832b379591d9486105c61/crates/core/src/delta_datafusion/table_provider.rs#L804
So, when registering a second table, with same storage root, the old credential is reused.
### What does work
- Restart SessionContext between the queries
- Queries on tables stored in different storage accounts/containers
2 条评论