repartitionByRange
repartitionByRange(numPartitions, *cols) method of pyspark.sql.dataframe.DataFrame instance Returns a new :class:`DataFrame` partitioned by the given partitioning expressions. The resulting DataFrame is range partitioned. :param numPartitions: can be an int to specify the target number of partitions or a Column. If it is a Column, it will be used as the first partitioning column. If not specified, the default number of partitions is used. At least one partition-by expression must be specified. When no explicit sort order is specified, "ascending nulls first" is assumed.
begin = time.time() df = merge_data df.repartitionByRange(10,"probeset_id").write.format("delta").mode("append").save(f) print(time.time()-begin)