ITADN

Error reading from an Arrow ArrayStream on a GeoParquet layer with an attribute filter defined

#918Closedctoney 创建于 2026-03-03
C
ctoneycommented
Error attempting to reproduce in R the examples for Source Coop DEM-Terrain at https://source.coop/walkthru-earth/dem-terrain (using the V1 files that have `geometry` column): * reading from the ArrayStream is done by `nanoarrow` * a schema validation error occurs when reading from the Parquet layer if it has an attribute filter defined * same result if the Parquet file is opened as a SQL layer instead (with same WHERE filter) * no error and the data are read as expected if no attribute filter is defined (attributes look correct and geometries returned as WKB) Also, setting an attribute filter on a GeoPackage layer does not generate the error when reading via ArrayStream (tested against the NZ Building Outlines GPKG used for benchmarks at https://firelab.github.io/gdalraster/articles/vector-read-benchmarks.html). The code below was run in a container from `ubuntu-full-latest` GDAL 3.13dev. The same result was seen with a local build of GDAL 3.12.1. ### Error with attribute filter set: ```r library(gdalraster) #> GDAL 3.13.0dev-12f6982db49c570208274f5ad9aa5aa2bf86b5eb (released 2026-03-08), GEOS 3.12.1, PROJ 9.9.0 packageVersion("nanoarrow") #> [1] ‘0.8.0’ set_config_option("AWS_NO_SIGN_REQUEST", "YES") set_config_option("AWS_REGION", "us-west-2") options(nanoarrow.warn_unregistered_extension = FALSE) f <- "/vsis3/us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v1/h3/h3_res=5/data.parquet" (lyr <- new(GDALVector, f)) #> C++ object of class GDALVector #> Driver : (Geo)Parquet (Parquet) #> DSN : /vsis3/us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/h3_res=5/data.parquet #> Layer : data #> CRS : WGS 84 (EPSG:4326) #> Geom : POINT lyr$getFieldNames() #> [1] "h3_index" "lat" "lon" "elev" "slope" "aspect" "tri" #> [8] "tpi" "geometry" lyr$setAttributeFilter("lat BETWEEN 35 AND 45 AND lon BETWEEN -10 AND 5") (stream <- lyr$getArrowStream()) #> <nanoarrow_array_stream struct<h3_index: string, geometry: geoarrow.wkb{binary}, lat: float, lon: float, elev: float, slope: float, aspect: float, tri: float, tpi: float>> #> $ get_schema:function () #> $ get_next :function (schema = x$get_schema(), validate = TRUE) #> $ release :function () arr <- stream$get_next() #> Error in nanoarrow_array_set_schema(array, schema, validate = validate) : #> Expected string array buffer 0 to have size >= 27 bytes but found buffer with 0 bytes lyr$close() ``` ### No error without the attribute filter: ```r library(gdalraster) #> GDAL 3.13.0dev-12f6982db49c570208274f5ad9aa5aa2bf86b5eb (released 2026-03-08), GEOS 3.12.1, PROJ 9.9.0 set_config_option("AWS_NO_SIGN_REQUEST", "YES") set_config_option("AWS_REGION", "us-west-2") options(nanoarrow.warn_unregistered_extension = FALSE) f <- "/vsis3/us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v1/h3/h3_res=5/data.parquet" (lyr <- new(GDALVector, f)) #> C++ object of class GDALVector #> Driver : (Geo)Parquet (Parquet) #> DSN : /vsis3/us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/h3_res=5/data.parquet #> Layer : data #> CRS : WGS 84 (EPSG:4326) #> Geom : POINT (stream <- lyr$getArrowStream()) #> <nanoarrow_array_stream struct<h3_index: string, geometry: geoarrow.wkb{binary}, lat: float, lon: float, elev: float, slope: float, aspect: float, tri: float, tpi: float>> #> $ get_schema:function () #> $ get_next :function (schema = x$get_schema(), validate = TRUE) #> $ release :function () # without filter arr <- stream$get_next() typeof(arr) #> [1] "externalptr" class(arr) #> [1] "nanoarrow_array" d <- as.data.frame(arr) nrow(d) #> [1] 65536 head(d) #> h3_index #> 1 85000133fffffff #> 2 850001a3fffffff #> 3 850001abfffffff #> 4 850001b3fffffff #> 5 85001803fffffff #> 6 85001813fffffff #> geometry #> 1 01, 01, 00, 00, 00, 00, 00, 00, 40, 5b, 89, 40, 40, 00, 00, 00, 00, f0, 16, 54, 40 #> 2 01, 01, 00, 00, 00, 00, 00, 00, e0, f0, ea, 3f, 40, 00, 00, 00, c0, 5c, 0c, 54, 40 #> 3 01, 01, 00, 00, 00, 00, 00, 00, 80, d2, 65, 40, 40, 00, 00, 00, 20, df, 0d, 54, 40 #> 4 01, 01, 00, 00, 00, 00, 00, 00, 60, 98, a9, 3f, 40, 00, 00, 00, a0, 43, 03, 54, 40 #> 5 01, 01, 00, 00, 00, 00, 00, 00, e0, 97, 40, 49, 40, 00, 00, 00, a0, df, 41, 54, 40 #> 6 01, 01, 00, 00, 00, 00, 00, 00, 20, 0d, f7, 48, 40, 00, 00, 00, 80, f2, 39, 54, 40 #> lat lon elev slope aspect tri tpi #> 1 80.35840 33.07310 0.5000000 NA NA NA NA #> 2 80.19316 31.91774 194.3317871 2.870463 324.89664 17.608427 -0.2735468 #> 3 80.21674 32.79549 348.1593933 1.951790 66.51669 6.001579 -1.4144241 #> 4 80.05100 31.66248 21.9455624 4.065749 225.69536 20.571117 -6.9174838 #> 5 81.02927 50.50463 0.7928278 NA NA NA NA #> 6 80.90543 49.93009 18.2626114 1.200685 215.51103 8.230590 0.6501796 stream$release() lyr$close() ```
关闭于 2026-03-28 8 条评论