Python - Working with different file formats (text, CSV, JSON)

Working with Different File Formats (Text, CSV, JSON) in Python

In modern software development, working with different file formats is a routine task. Python offers extensive and easy-to-use libraries to handle common file formats such as text files, CSV files, and JSON files. Understanding how to read, write, and manipulate these files is essential for any Python developer. This guide provides a comprehensive walkthrough of how to work with each of these formats effectively.

Introduction

File formats determine how data is stored in a file. Text files store plain text, CSV files are used for tabular data, and JSON files store structured data in a readable format. Each format has its own characteristics and best use cases. Python's built-in modules like open, csv, and json make it easy to interact with these file types.

Working with Text Files

What is a Text File?

A text file (.txt) contains sequences of characters and is one of the simplest forms of storing data. It can be used for storing notes, logs, configuration files, or any other human-readable content.

Opening and Reading a Text File

To read a text file in Python, you can use the open() function with mode 'r':

with open('example.txt', 'r') as file:
    content = file.read()
    print(content)

Reading Line by Line

Sometimes it's useful to read a file line by line:

with open('example.txt', 'r') as file:
    for line in file:
        print(line.strip())

Writing to a Text File

Use mode 'w' to write to a file. This will overwrite existing content:

with open('example.txt', 'w') as file:
    file.write("Hello, World!")

Appending to a Text File

Use mode 'a' to add new content at the end of an existing file:

with open('example.txt', 'a') as file:
    file.write("\nAnother line.")

Reading and Writing

Use mode 'r+' or 'w+' to both read and write:

with open('example.txt', 'r+') as file:
    content = file.read()
    file.write("\nAppended after reading.")

Working with CSV Files

What is a CSV File?

CSV (Comma Separated Values) is a popular file format used for storing tabular data. Each row corresponds to a record, and each value is separated by a comma. Python provides a built-in csv module to handle CSV files easily.

Reading CSV Files

You can read a CSV file using csv.reader():

import csv

with open('data.csv', newline='') as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

This reads each line as a list of strings.

Reading CSV as Dictionary

Use csv.DictReader() to read each row as a dictionary:

import csv

with open('data.csv', newline='') as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(row['name'], row['age'])

Writing to CSV Files

Use csv.writer() to write rows to a CSV file:

import csv

data = [['name', 'age'], ['Alice', 30], ['Bob', 25]]

with open('data.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerows(data)

Writing CSV Using Dictionaries

import csv

data = [{'name': 'Alice', 'age': 30}, {'name': 'Bob', 'age': 25}]

with open('data.csv', 'w', newline='') as file:
    fieldnames = ['name', 'age']
    writer = csv.DictWriter(file, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerows(data)

Handling Custom Delimiters

CSV files are not always comma-separated. You can specify a different delimiter:

import csv

with open('data.tsv', newline='') as file:
    reader = csv.reader(file, delimiter='\t')
    for row in reader:
        print(row)

Error Handling in CSV

Use try-except blocks to catch csv.Error when dealing with malformed CSV files.

try:
    with open('corrupt.csv') as file:
        reader = csv.reader(file)
        for row in reader:
            print(row)
except csv.Error as e:
    print("CSV Error:", e)

Working with JSON Files

What is a JSON File?

JSON (JavaScript Object Notation) is a format used for storing structured data. It is widely used in web APIs and configuration files. Python provides a built-in json module for working with JSON data.

Reading JSON Files

Use json.load() to read a JSON file:

import json

with open('data.json', 'r') as file:
    data = json.load(file)
    print(data)

Writing to JSON Files

import json

data = {'name': 'Alice', 'age': 30, 'skills': ['Python', 'Data Science']}

with open('data.json', 'w') as file:
    json.dump(data, file)

Pretty Printing JSON

You can use the indent parameter to format the output:

json.dump(data, file, indent=4)

Converting JSON Strings

Use json.loads() and json.dumps() to convert between strings and Python objects:

json_str = '{"name": "Bob", "age": 25}'
data = json.loads(json_str)
print(data)

string = json.dumps(data)
print(string)

Error Handling in JSON

Use try-except to catch json.JSONDecodeError:

try:
    with open('bad.json') as file:
        data = json.load(file)
except json.JSONDecodeError:
    print("Invalid JSON format")

Comparison of File Formats

Practical Use Cases

Log File Storage

Use text files to store logs. Easy to write and read using write() and readlines().

Data Science

CSV files are commonly used for dataset exchange between Excel and data analysis tools.

APIs and Configurations

JSON files are widely used in RESTful APIs and for storing application configuration settings.

Advanced Topics

Handling Encoding

Always specify encoding explicitly for non-ASCII text:

with open('unicode.txt', 'r', encoding='utf-8') as file:
    data = file.read()

File Existence and Creation

import os

if not os.path.exists("file.txt"):
    with open("file.txt", "w") as file:
        file.write("Created!")

Using Pandas for CSV and JSON

import pandas as pd

# CSV
df = pd.read_csv("data.csv")
df.to_csv("out.csv", index=False)

# JSON
df = pd.read_json("data.json")
df.to_json("out.json")

Python provides rich capabilities for working with different file formats such as Text, CSV, and JSON. Each of these formats serves a unique purpose and is suited to particular types of data storage and manipulation. Text files are great for logs and notes; CSV is ideal for spreadsheets and tabular data; JSON is perfect for structured, hierarchical data commonly used in APIs and configs.

By mastering the file handling techniques in Python for these formats, you enhance your ability to build data-driven applications, automate data processing workflows, and interact with external data sources efficiently.