PySpark – unionByName()
In Spark or PySpark let’s see how to merge/union two DataFrames with a different number of columns (different schema). In Spark 3.1, you can easily …
PySpark Tutorial
PySpark – distinct() and dropDuplicates()
PySpark distinct() function is used to drop/remove the duplicate rows (all columns) from DataFrame and dropDuplicates() is used to drop rows based on selected (one or multiple) columns. …
PySpark – withColumnRenamed()
Use PySpark withColumnRenamed() to rename a DataFrame column, we often need to rename one column or multiple (or all) columns on PySpark DataFrame, you can do this …
PySpark – collect()
PySpark RDD/DataFrame collect() is an action operation that is used to retrieve all the elements of the dataset (from all nodes) to the driver node. We should …
PySpark – show()
PySpark DataFrame show() is used to display the contents of the DataFrame in a Table Row and Column Format. By default, it shows only 20 …
PySpark – Convert RDD to DataFrame
In PySpark, toDF() the function of the RDD is used to convert RDD to DataFrame. We would need to convert RDD to DataFrame as DataFrame provides …
PySpark – Create an Empty DataFrame
In this article, we will learn about How to Create an Empty PySpark DataFrame/RDD manually with or without schema (column names) in different ways. While …
