Scenario

A large dataset needs memory analysis to understand which columns consume the most memory and identify optimization opportunities.

Task

Write a Python script at /home/interview/analyze_memory.py using pandas that reads /home/interview/sales_data.csv, calculates the memory usage of each column in megabytes (rounded to 2 decimal places), and saves the report to /home/interview/memory_report.csv.

Note: pandas is already installed.

Example

Expected output format in /home/interview/memory_report.csv:

column_name,memory_mb
product_name,2.15
customer_email,1.83
quantity,0.08
order_date,0.08

Step 1: Examine the dataset

head /home/interview/sales_data.csv

Review the column types and data to understand the dataset structure.

Step 2: Create the Python script

nano /home/interview/analyze_memory.py

Write a script to calculate memory usage per column:

import pandas as pd

# Read the CSV file
df = pd.read_csv('/home/interview/sales_data.csv')

# Get memory usage for each column in bytes (deep=True for accurate object memory)
memory_usage = df.memory_usage(deep=True)

# Convert to megabytes and create report
memory_report = pd.DataFrame({
    'column_name': memory_usage.index,
    'memory_mb': (memory_usage.values / (1024**2)).round(2)
})

# Remove the 'Index' row (first row from memory_usage)
memory_report = memory_report[memory_report['column_name'] != 'Index']

# Save to CSV
memory_report.to_csv('/home/interview/memory_report.csv', index=False)

print(f"Memory report generated for {len(memory_report)} columns")
print(f"Total memory: {memory_report['memory_mb'].sum():.2f} MB")

The memory_usage(deep=True) parameter ensures accurate memory calculation for object types like strings.

Step 3: Run the script

python3 /home/interview/analyze_memory.py

Step 4: Verify the output

cat /home/interview/memory_report.csv

Should show memory usage in megabytes for each column in the dataset.