Objective

You are working with a dataset of social media posts that contain some incorrect references that need to be corrected before publication.

Task

Replace every occurrence of the word "Python" with "PySpark" in the text column. Return the entire DataFrame with the same schema. Only the text column should be modified; all other columns must remain unchanged. Posts that do not contain the word "Python" should appear in the output exactly as they are in the input. Save your result as result_df.

File Path

• Dataset: /home/interview/posts.csv
• Starter script: /home/interview/correct_posts.py

Schema

Example

Input:

Expected Output:

The word "Python" in posts 1 and 3 has been replaced with "PySpark." Post 2 did not contain the word "Python," so it remains unchanged.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("PrepareshSpark").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

df = spark.read.csv("/home/interview/posts.csv", header=True, inferSchema=True)

result_df = df.withColumn("text", F.regexp_replace("text", "Python", "PySpark"))

result_df.coalesce(1).write.csv("/home/interview/output", header=True, mode="overwrite")

spark.stop()

Explanation

Step 1: Read the dataset

We start by reading the CSV file into a DataFrame with inferSchema=True so that numeric columns like likes, comments, and shares are properly typed as integers rather than strings.

df = spark.read.csv("/home/interview/posts.csv", header=True, inferSchema=True)

Step 2: Replace "Python" with "PySpark" in the text column

The regexp_replace function from pyspark.sql.functions allows us to search for a pattern in a string column and replace it with a new value. Here we target the text column and swap every occurrence of "Python" for "PySpark." Because withColumn replaces a column in place when the name matches an existing column, the rest of the schema stays exactly the same.

result_df = df.withColumn("text", F.regexp_replace("text", "Python", "PySpark"))

You could also use the simpler F.col("text") approach with a UDF, but regexp_replace is the idiomatic and most performant way to do string substitution in PySpark.

Step 3: Write the result

Finally, we coalesce the output into a single partition and write it out as a CSV.

result_df.coalesce(1).write.csv("/home/interview/output", header=True, mode="overwrite")