Exploring Date Functions in PySpark: current_date(), date_format(), and to_date()

In this blog post, we will explore three important date functions in PySpark: current_date(), date_format(), and to_date(). Understanding these functions is crucial for effectively working with date data in PySpark DataFrames. We will provide practical examples to illustrate how these functions can be utilized.

Date functions in PySpark allow users to manipulate and format date data efficiently. In this post, we will focus on:

• current_date(): Retrieves the current system date.
• date_format(): Formats a date column into a specified string format.
• to_date(): Converts a date string into a date type based on a specified format.

Creating a DataFrame

To demonstrate these functions, we first need to create a DataFrame. Here’s how to do it:

1. Create a New Notebook: Start by creating a new notebook in your PySpark environment.
2. Generate a DataFrame: Use the range function to create a DataFrame with a single column called ID containing incremental numbers.

from pyspark.sql import SparkSession


spark = SparkSession.builder.appName("Date Functions Notebook").getOrCreate()


DF = spark.range(2)
DF.show()

Using current_date() Function

Once we have our DataFrame, we can add a new column that contains the current system date. The current_date() function retrieves the current date in the default format (YYYY-MM-DD).

1. Import the Function: Import current_date from the PySpark SQL functions module.
2. Add Current Date Column: Use the withColumn function to add a new column named current_date.

from pyspark.sql.functions import current_date


DF1 = DF.withColumn("current_date", current_date())
DF1.show()
DF1.printSchema()

The output will show the current date in the DataFrame with the data type as DateType.

Using date_format() Function

Next, we will demonstrate how to format the date using the date_format() function. This function allows you to convert a date column into a specified string format.

1. Add a Date Format Column: We will create a new column that formats the current date into a different format (e.g., YY.MM.DD).

from pyspark.sql.functions import date_format


DF2 = DF1.withColumn("date_format", date_format(DF1.current_date, "yy.MM.dd"))
DF2.show()
DF2.printSchema()

In this example, the date_format function converts the date into a string format, and the data type of the new column will be StringType.

Using to_date() Function

Finally, we will explore the to_date() function, which converts a date string into a date type based on a specified format.

1. Convert Back to Date Type: We will take the formatted date string and convert it back to a date type.

from pyspark.sql.functions import to_date


DF3 = DF2.withColumn("current_date", to_date(DF2.date_format, "yy.MM.dd"))
DF3.show()
DF3.printSchema()

The to_date function will convert the string back into a date type, maintaining the default format (YYYY-MM-DD).

Conclusion

In this blog post, we have explored three essential date functions in PySpark: current_date(), date_format(), and to_date(). These functions are invaluable for manipulating and formatting date data within PySpark DataFrames. By understanding how to use these functions, you can enhance your data processing capabilities in PySpark.

If you have any questions or need further clarification, feel free to revisit the examples or watch the video for a more in-depth understanding. Thank you for reading!