PySpark is the Python API for Apache Spark. It allows you to process large-scale data using Python while leveraging Spark\u2019s distributed computing power.<\/p>\n\n\n\n
PySpark is widely used in Data Engineering, Big Data processing, and Machine Learning workflows.<\/p>\n\n\n\n
PySpark is useful when:<\/p>\n\n\n\n
You can install PySpark using pip:<\/p>\n\n\n\n
pip install pyspark<\/pre>\n\n\n\nOr use it in environments like:<\/p>\n\n\n\n
The first step in PySpark is creating a Spark session.<\/p>\n\n\n\n
from pyspark.sql import SparkSessionspark = SparkSession.builder \\
.appName(\"PySpark Basics\") \\
.getOrCreate()<\/pre>\n\n\n\nSparkSession is the entry point for working with data in Spark.<\/p>\n\n\n\n
Reading Data in PySpark<\/h2>\n\n\n\n
Read CSV File<\/h3>\n\n\n\n
df = spark.read.csv(\"data.csv\", header=True, inferSchema=True)
df.show()<\/pre>\n\n\n\nRead JSON File<\/h3>\n\n\n\n
df = spark.read.json(\"data.json\")
df.show()<\/pre>\n\n\n\nUnderstanding DataFrames<\/h2>\n\n\n\n
In PySpark, the main data structure is a DataFrame.<\/p>\n\n\n\n
It is similar to a Pandas DataFrame but distributed across multiple machines.<\/p>\n\n\n\n
Display Schema<\/h3>\n\n\n\n
df.printSchema()<\/pre>\n\n\n\nShow Data<\/h3>\n\n\n\n
df.show()<\/pre>\n\n\n\nBasic DataFrame Operations<\/h2>\n\n\n\n
Select Columns<\/h3>\n\n\n\n
df.select(\"name\", \"salary\").show()<\/pre>\n\n\n\nFilter Data<\/h3>\n\n\n\n
df.filter(df.salary > 50000).show()<\/pre>\n\n\n\nGroup By<\/h3>\n\n\n\n
df.groupBy(\"department\").count().show()<\/pre>\n\n\n\nAdd New Column<\/h3>\n\n\n\n
from pyspark.sql.functions import coldf = df.withColumn(\"bonus\", col(\"salary\") * 0.10)
df.show()<\/pre>\n\n\n\nTransformations vs Actions<\/h2>\n\n\n\n
Transformations<\/h3>\n\n\n\n
Operations that create a new DataFrame (lazy execution).<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
\n
- select()<\/li>\n\n\n\n
- filter()<\/li>\n\n\n\n
- groupBy()<\/li>\n<\/ul>\n\n\n\n
Actions<\/h3>\n\n\n\n
Operations that trigger execution.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
\n
- show()<\/li>\n\n\n\n
- count()<\/li>\n\n\n\n
- collect()<\/li>\n<\/ul>\n\n\n\n
Spark follows lazy evaluation, meaning it waits until an action is called before executing transformations.<\/p>\n\n\n\n
Writing Data<\/h2>\n\n\n\n
Save as CSV<\/h3>\n\n\n\n
df.write.csv(\"output_folder\", header=True)<\/pre>\n\n\n\nSave as Parquet<\/h3>\n\n\n\n
df.write.parquet(\"output_folder\")<\/pre>\n\n\n\nParquet is optimized for Big Data processing.<\/p>\n\n\n\n
PySpark vs Pandas<\/h2>\n\n\n\n
Pandas:<\/p>\n\n\n\n
\n
- Works on a single machine<\/li>\n\n\n\n
- Best for small to medium datasets<\/li>\n<\/ul>\n\n\n\n
PySpark:<\/p>\n\n\n\n
\n
- Distributed processing<\/li>\n\n\n\n
- Handles massive datasets<\/li>\n\n\n\n
- Scalable<\/li>\n<\/ul>\n\n\n\n
Real-World Use Case<\/h2>\n\n\n\n
Example workflow:<\/p>\n\n\n\n
Raw Sales Data \u2192 Clean with PySpark \u2192 Aggregate Revenue \u2192 Store in Data Warehouse \u2192 Visualize in Power BI<\/p>\n\n\n\n
Final Takeaway<\/h2>\n\n\n\n
PySpark allows Python developers to work with Big Data efficiently using distributed computing.<\/p>\n\n\n\n
Mastering PySpark is essential for building scalable data pipelines and becoming a Data Engineer.<\/p>\n\n\n