Data transformation is the process of cleaning, modifying, and preparing raw data for analysis.<\/p>\n\n\n\n
Pandas<\/strong> is one of the most powerful Python libraries for data transformation and manipulation.<\/p>\n\n\n\n It is widely used in:<\/p>\n\n\n\n Data Engineering Before transforming, we load the data.<\/p>\n\n\n\n Now the data is ready for transformation.<\/p>\n\n\n\n Check missing values:<\/p>\n\n\n\n Remove missing values:<\/p>\n\n\n\n Fill missing values:<\/p>\n\n\n\n This ensures clean and accurate data.<\/p>\n\n\n\n Filter rows:<\/p>\n\n\n\n Filter multiple conditions:<\/p>\n\n\n\n Selecting only required columns improves performance.<\/p>\n\n\n\n This creates a calculated column.<\/p>\n\n\n\n Correct data types improve accuracy and memory usage.<\/p>\n\n\n\n Group data and calculate totals:<\/p>\n\n\n\n Multiple aggregations:<\/p>\n\n\n\n Combine two datasets:<\/p>\n\n\n\n Join types:<\/p>\n\n\n\n inner Apply custom logic to columns.<\/p>\n\n\n\n Create summary table:<\/p>\n\n\n\n Useful for reporting and dashboards.<\/p>\n\n\n\n Or save to database for analytics.<\/p>\n\n\n\n Extract:<\/p>\n\n\n\n Load raw sales data<\/p>\n\n\n\n Transform:<\/p>\n\n\n\n Remove duplicates Load:<\/p>\n\n\n\n Save cleaned data into data warehouse<\/p>\n\n\n\n Avoid loops, use vectorized operations Modifying original data without backup Transforming data using Pandas involves cleaning, filtering, aggregating, and restructuring datasets to make them analysis-ready.<\/p>\n\n\n\n Pandas provides powerful and efficient tools that are essential for ETL processes and modern data workflows.<\/p>\n\n\n
Data Analysis
Machine Learning
ETL Pipelines<\/p>\n\n\n\n1. Loading Data<\/h1>\n\n\n\n
import pandas as pddf = pd.read_csv(\"sales.csv\")
print(df.head())<\/pre>\n\n\n\n2. Handling Missing Values<\/h1>\n\n\n\n
print(df.isnull().sum())<\/pre>\n\n\n\n
df = df.dropna()<\/pre>\n\n\n\n
df[\"price\"] = df[\"price\"].fillna(0)<\/pre>\n\n\n\n
3. Removing Duplicates<\/h1>\n\n\n\n
df = df.drop_duplicates()<\/pre>\n\n\n\n
4. Filtering Data<\/h1>\n\n\n\n
df = df[df[\"price\"] > 100]<\/pre>\n\n\n\n
df = df[(df[\"price\"] > 100) & (df[\"quantity\"] >= 2)]<\/pre>\n\n\n\n
5. Selecting Columns<\/h1>\n\n\n\n
df = df[[\"product\", \"price\", \"quantity\"]]<\/pre>\n\n\n\n
6. Creating New Columns<\/h1>\n\n\n\n
df[\"total\"] = df[\"price\"] * df[\"quantity\"]<\/pre>\n\n\n\n
7. Changing Data Types<\/h1>\n\n\n\n
df[\"quantity\"] = df[\"quantity\"].astype(\"int32\")
df[\"date\"] = pd.to_datetime(df[\"date\"])<\/pre>\n\n\n\n8. Renaming Columns<\/h1>\n\n\n\n
df = df.rename(columns={\"product_name\": \"product\"})<\/pre>\n\n\n\n9. Grouping and Aggregation<\/h1>\n\n\n\n
summary = df.groupby(\"product\")[\"total\"].sum()
print(summary)<\/pre>\n\n\n\nsummary = df.groupby(\"product\").agg({
\"total\": \"sum\",
\"quantity\": \"mean\"
})<\/pre>\n\n\n\n10. Sorting Data<\/h1>\n\n\n\n
df = df.sort_values(by=\"total\", ascending=False)<\/pre>\n\n\n\n
11. Merging DataFrames<\/h1>\n\n\n\n
df1 = pd.read_csv(\"customers.csv\")
df2 = pd.read_csv(\"orders.csv\")merged = pd.merge(df1, df2, on=\"customer_id\", how=\"inner\")<\/pre>\n\n\n\n
left
right
outer<\/p>\n\n\n\n12. Applying Custom Functions<\/h1>\n\n\n\n
df[\"discounted_price\"] = df[\"price\"].apply(lambda x: x * 0.9)<\/pre>\n\n\n\n
13. Pivot Tables<\/h1>\n\n\n\n
pivot = df.pivot_table(
values=\"total\",
index=\"product\",
columns=\"region\",
aggfunc=\"sum\"
)<\/pre>\n\n\n\n14. Export Transformed Data<\/h1>\n\n\n\n
df.to_csv(\"cleaned_sales.csv\", index=False)<\/pre>\n\n\n\n
Real-World ETL Example<\/h1>\n\n\n\n
Handle missing values
Calculate total revenue
Aggregate by product<\/p>\n\n\n\nBest Practices<\/h1>\n\n\n\n
Handle missing values carefully
Use proper data types
Document transformations
Validate output data
Optimize memory usage<\/p>\n\n\n\nCommon Mistakes<\/h1>\n\n\n\n
Ignoring data types
Using loops instead of vectorization
Not checking missing values
Not validating final dataset<\/p>\n\n\n\nKey Takeaway<\/h1>\n\n\n\n