Loading data into databases is the final step of the ETL process (Extract \u2192 Transform \u2192 Load). After cleaning and transforming data, we store it in a database for reporting, analytics, or application use.<\/p>\n\n\n\n
This is a core skill in Data Engineering and backend development.<\/p>\n\n\n\n
Databases provide:<\/p>\n\n\n\n
Best for structured data with relationships.<\/p>\n\n\n\n
Best for flexible or semi-structured data.<\/p>\n\n\n\n
We commonly use:<\/p>\n\n\n\n
Step 1: Install Required Libraries<\/p>\n\n\n\n
pip install pandas sqlalchemy pymysql<\/p>\n\n\n\n
Step 2: Connect to MySQL<\/p>\n\n\n\n
from sqlalchemy import create_engine
import pandas as pd<\/p>\n\n\n\n
engine = create_engine(“mysql+pymysql:\/\/username:password@localhost:3306\/database_name”)<\/p>\n\n\n\n
Step 3: Load DataFrame into Database<\/p>\n\n\n\n
df = pd.read_csv(“cleaned_sales.csv”)<\/p>\n\n\n\n
df.to_sql(
name=”sales”,
con=engine,
if_exists=”replace”,
index=False
)<\/p>\n\n\n\n
Now the data is stored inside MySQL.<\/p>\n\n\n\n
Install:<\/p>\n\n\n\n
pip install psycopg2-binary<\/p>\n\n\n\n
Connect and load:<\/p>\n\n\n\n
engine = create_engine(“postgresql:\/\/username:password@localhost:5432\/database_name”)<\/p>\n\n\n\n
df.to_sql(“sales”, engine, if_exists=”append”, index=False)<\/p>\n\n\n\n
import mysql.connector<\/p>\n\n\n\n
conn = mysql.connector.connect(
host=”localhost”,
user=”username”,
password=”password”,
database=”database_name”
)<\/p>\n\n\n\n
cursor = conn.cursor()<\/p>\n\n\n\n
for _, row in df.iterrows():
cursor.execute(
“INSERT INTO sales (product, price, quantity) VALUES (%s, %s, %s)”,
(row[“product”], row[“price”], row[“quantity”])
)<\/p>\n\n\n\n
conn.commit()
conn.close()<\/p>\n\n\n\n
Note: This method is slower for large datasets.<\/p>\n\n\n\n
For large files, use database bulk import tools:<\/p>\n\n\n\n
These are much faster than row-by-row insertion.<\/p>\n\n\n\n
Before loading:<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
df[“date”] = pd.to_datetime(df[“date”])
df[“quantity”] = df[“quantity”].astype(int)<\/p>\n\n\n\n
Always use try-except:<\/p>\n\n\n\n
try:
df.to_sql(“sales”, engine, if_exists=”append”, index=False)
print(“Data loaded successfully.”)
except Exception as e:
print(“Error:”, e)<\/p>\n\n\n\n
Extract \u2192 API \/ CSV \/ Database<\/p>\n\n\n\n
Transform \u2192 Clean using Pandas<\/p>\n\n\n\n
Load \u2192 Insert into PostgreSQL Data Warehouse<\/p>\n\n\n\n
Use \u2192 Power BI \/ Tableau \/ Dashboards<\/p>\n\n\n\n
Loading data into databases ensures structured, reliable, and scalable storage of transformed datasets.<\/p>\n\n\n\n
In Data Engineering, mastering this step is critical for building efficient ETL pipelines and production-ready data systems.<\/p>\n\n\n