PySpark – date_format()
In PySpark use date_format() function to convert the DataFrame column from Date to String format. In this tutorial, we will show you a Spark SQL example of …
PySpark – to_date()
PySpark functions provide to_date() function to convert timestamp to date (DateType), this is ideally achieved by just truncating the time part from the Timestamp column. …
PySpark – to_timestamp()
Use to_timestamp() function to convert String to Timestamp (TimestampType) in PySpark. The converted time would be in a default format of MM-dd-yyyy HH:mm:ss.SSS, I will explain how …
PySpark – regexp_replace(), translate() and overlay()
We can replace column values of PySpark DataFrame by using SQL string functions regexp_replace(), translate(), and overlay() with Python examples. In this article, we will cover …
PySpark – concat_ws()
In this PySpark article, We will learn how to convert an array of String column on DataFrame to a String column (separated or concatenated with …
PySpark – split()
PySpark SQL provides split() function to convert delimiter separated String to an Array (StringType to ArrayType) column on DataFrame. This can be done by splitting a string column based …
PySpark – lit()
PySpark SQL functions lit() and typedLit() are used to add a new column to DataFrame by assigning a literal or constant value. Both these functions return Column type as return type. …
