pandas_udf 给出与 pyarrow 相关的错误

Question

我有数据框，我想使用 pysaprk 中的折线库获取给定地理位置的 lat_long+-----------------+--------------------+----------+                             |              vid|        geolocations| trip_date|+-----------------+--------------------+----------+|58AC21B17LU006754|eurnE||yqU???????...|2020-02-22||2T3EWRFV0LW060632|uocbGfjniOK[Fs@rC...|2020-02-25||JTDP4RCE0LJ014008|w}wtFpdxtM????Q_@...|2020-02-25||4T1BZ1HK8KU029845|}rz_Dp~hhN?@?@???...|2020-03-03|我正在使用 pandas_udf 并且 apache arrow 已启用from pyspark.sql.functions import col, pandas_udfspark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "true")lat_long_udf = pandas_udf(lambda geoloc:  polyline.decode(geoloc)[0],ArrayType(StringType()))df1=df.withColumn('lat_long',lat_long_udf(df.geolocations))当调用 df.count() 给出结果时，但在执行 df.show() 时，我收到如下错误： 248, in init_stream_yield_batches    for series in iterator:  File "/Users/prantik.pariksha/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 450, in mapper    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)  File "/Users/prantik.pariksha/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 450, in <genexpr>    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)  File "/Users/prantik.pariksha/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 110, in <lambda>    verify_result_type(f(*a)), len(a[0])), arrow_return_type)  File "/Users/prantik.pariksha/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/util.py", line 107, in wrapper    return f(*args, **kwargs)

Answer 1

您很可能会收到此错误，因为 apandas_udf采用 pandas Series 作为输入，并且您将该decode函数直接应用于该系列，而不是将其应用于 pandas Series 中的值。

例如，在下面的示例中，我稍微扩展了您的 lambda 函数，以便您可以看到它。我采用 pandas 系列，将polyline.decode函数应用于该系列，然后再次返回结果系列。请注意，我还将返回类型更改为ArrayType(DoubleType())而不是ArrayType(StringType()).

import pandas as pd

from pyspark.sql.types import ArrayType, DoubleType

....

df = spark.createDataFrame([["~sqU__pR_jpv@_pR"], ["_~t[__pR~qy@_pR"]], ["geolocations"])

@pandas_udf(ArrayType(DoubleType()))

def lat_long_udf(s: pd.Series) -> pd.Series:

  return s.apply(lambda x: polyline.decode(x)[0])

df1=df.withColumn('decoded', lat_long_udf(df.geolocations))

df1.collect()