Pyspark-用不同的列值替换列中的空值

Question

如何用category其中的不同值填充这一列空值？+---++--------+----------+| id||category|      Date|+---+---------+----------+| A1|     Null|2010-01-02|| A1|     Null|2010-01-03|| A1|    Nixon|2010-01-04|| A1|     Null|2010-01-05|| A9|     Null|2010-05-02|| A9|  Leonard|2010-05-03|| A9|     Null|2010-05-04|| A9|     Null|2010-05-05|+---+---------+----------+所需的数据框：+---++--------+----------+| id||category|      Date|+---+---------+----------+| A1|    Nixon|2010-01-02|| A1|    Nixon|2010-01-03|| A1|    Nixon|2010-01-04|| A1|    Nixon|2010-01-05|| A9|  Leonard|2010-05-02|| A9|  Leonard|2010-05-03|| A9|  Leonard|2010-05-04|| A9|  Leonard|2010-05-05|+---+---------+----------+我试过：w = Window().partitionBy("ID").orderBy("Date")df = df.withColumn("category", F.when(col("category").isNull(), col("category")\.distinct().over(w))\.otherwise(col("category")))我也尝试过：df = df.fillna({'category': col('category').distinct()})我也尝试过：df = df.withColumn('category', when(df.category.isNull(), df.category.distinct()).otherwise(df.category))

Answer 1

您可以使用first()参数ignorenullsas True。

另外，请rowsBetween(-sys.maxsize, sys.maxsize)在窗户上使用。

from pyspark.sql import functions as F

from pyspark.sql.functions import *

from pyspark.sql.window import Window

import sys

w = Window().partitionBy("id").orderBy("Date")

df.withColumn("new", F.first('category', True).over(w.rowsBetween(-sys.maxsize, sys.maxsize)))\

        .orderBy("id", "Date").show()

+---+--------+----------+

| id|category|      Date|

+---+--------+----------+

| A1|   Nixon|2010-01-02|

| A1|   Nixon|2010-01-03|

| A1|   Nixon|2010-01-04|

| A1|   Nixon|2010-01-05|

| A9| Leonard|2010-05-02|

| A9| Leonard|2010-05-03|

| A9| Leonard|2010-05-04|

| A9| Leonard|2010-05-05|

+---+--------+----------+