pyspark.sql.functions.spark_partition_id

pyspark.sql.functions.spark_partition_id()

A column for partition ID.

New in version 1.6.0.

Notes

This is non deterministic because it depends on data partitioning and task scheduling.

Examples

>>> df.repartition(1).select(spark_partition_id().alias("pid")).collect()
[Row(pid=0), Row(pid=0)]

pyspark.sql.functions.soundex pyspark.sql.functions.split