Scenario

A Parquet file contains structured data and you need to extract and document its schema information for analysis purposes.

Task

Write a Python script at /home/interview/extract_schema.py using pyarrow that reads /home/interview/data.parquet, extracts the schema information (column names, data types, compression codec, row count, and file size), and saves the output as JSON to /home/interview/schema_info.json.

Note: The pyarrow module is already installed.

Example

Expected output format in /home/interview/schema_info.json:

{
  "file": "/home/interview/data.parquet",
  "row_count": 390,
  "file_size_bytes": 45632,
  "file_size_kb": 44.56,
  "compression_codec": "SNAPPY",
  "columns": [
    {
      "name": "id",
      "type": "int64"
    },
    {
      "name": "name",
      "type": "string"
    },
    ...
  ]
}

Step 1: Create the Python script

nano /home/interview/extract_schema.py

Write a script using pyarrow to extract schema information:

import pyarrow.parquet as pq
import os
import json

# Read Parquet file metadata
parquet_file = pq.ParquetFile('/home/interview/data.parquet')

# Get schema
schema = parquet_file.schema_arrow

# Get metadata
metadata = parquet_file.metadata
row_count = metadata.num_rows
file_size = os.path.getsize('/home/interview/data.parquet')

# Get compression codec
compression = metadata.row_group(0).column(0).compression

# Build schema info dictionary
schema_info = {
    "file": "/home/interview/data.parquet",
    "row_count": row_count,
    "file_size_bytes": file_size,
    "file_size_kb": round(file_size / 1024, 2),
    "compression_codec": str(compression),
    "columns": []
}

for field in schema:
    schema_info["columns"].append({
        "name": field.name,
        "type": str(field.type)
    })

# Write to JSON file
with open('/home/interview/schema_info.json', 'w') as f:
    json.dump(schema_info, f, indent=2)

print("Schema information extracted successfully")

Step 2: Run the script

python3 /home/interview/extract_schema.py

Step 3: Verify the output

cat /home/interview/schema_info.json

Should display schema information in JSON format with column names, data types, compression codec, row count, and file size.