Scenario

A Parquet file contains customer data with many columns, but only a subset of columns is needed for analysis.

Task

Write a Python script at /home/interview/select_columns.py using pandas that reads /home/interview/customers.parquet, selects only the columns id, first_name, last_name, email, and total_purchases, and writes the result to /home/interview/selected_data.parquet.

Note: pandas and pyarrow are already installed.

Step 1: Examine the Parquet file structure

python3 -c "import pandas as pd; df = pd.read_parquet('/home/interview/customers.parquet'); print(df.columns.tolist()); print(df.shape)"

Shows all available columns and the file dimensions.

Step 2: Create the Python script

nano /home/interview/select_columns.py

Write a script to select specific columns and save to a new Parquet file:

import pandas as pd

# Read the Parquet file
df = pd.read_parquet('/home/interview/customers.parquet')

# Select only the required columns
selected_df = df[['id', 'first_name', 'last_name', 'email', 'total_purchases']]

# Write to new Parquet file
selected_df.to_parquet('/home/interview/selected_data.parquet', index=False)

print(f"Selected {len(selected_df.columns)} columns from {len(selected_df)} records")

Step 3: Run the script

python3 /home/interview/select_columns.py

Step 4: Verify the output

python3 -c "import pandas as pd; df = pd.read_parquet('/home/interview/selected_data.parquet'); print(df.columns.tolist()); print(df.head())"

Should show only the 5 selected columns with 1000 records.