{"id":207,"date":"2026-03-03T13:29:00","date_gmt":"2026-03-03T08:29:00","guid":{"rendered":"https:\/\/gigz.pk\/python\/?post_type=lesson&p=207"},"modified":"2026-03-22T19:15:33","modified_gmt":"2026-03-22T14:15:33","slug":"extracting-data-from-apis","status":"publish","type":"lesson","link":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/","title":{"rendered":"Extracting Data from APIs"},"content":{"rendered":"\n

Extracting data from APIs is a common step in ETL and data pipelines.
APIs (Application Programming Interfaces) allow applications to communicate and share data over the internet.<\/p>\n\n\n\n

In simple terms:<\/p>\n\n\n\n

API \u2192 Sends request
Server \u2192 Returns data (usually in JSON format)<\/p>\n\n\n\n

This data can then be stored, transformed, and analyzed.<\/p>\n\n\n\n

What is an API?<\/h1>\n\n\n\n

An API is a service that provides access to data through endpoints.<\/p>\n\n\n\n

Example endpoint: <\/p>\n\n\n\n

https:\/\/api.example.com\/users<\/a><\/p>\n\n\n\n

When you send a request, the server returns structured data.<\/p>\n\n\n\n

Most APIs return:<\/p>\n\n\n\n

JSON
XML<\/p>\n\n\n\n

JSON is the most common format.<\/p>\n\n\n\n

Common HTTP Methods<\/h1>\n\n\n\n

GET \u2192 Retrieve data
POST \u2192 Send data
PUT\/PATCH \u2192 Update data
DELETE \u2192 Remove data<\/p>\n\n\n\n

For extracting data, we usually use GET.<\/p>\n\n\n\n

Extracting Data Using Python<\/h1>\n\n\n\n

The most popular library is requests.<\/p>\n\n\n\n

Install:<\/p>\n\n\n\n

pip install requests<\/pre>\n\n\n\n
Basic example:<\/p>\n\n\n\n
import requestsurl = \"https:\/\/jsonplaceholder.typicode.com\/posts\"response = requests.get(url)if response.status_code == 200:
    data = response.json()
    print(data[:2])
else:
    print(\"Error:\", response.status_code)<\/pre>\n\n\n\n
Explanation:<\/p>\n\n\n\n
requests.get() sends request
response.status_code checks success
response.json() converts JSON into Python dictionary<\/p>\n\n\n\n
Working with JSON Data<\/h1>\n\n\n\n
API responses are usually dictionaries or lists.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
for post in data:
    print(post[\"title\"])<\/pre>\n\n\n\n
You can convert JSON to Pandas DataFrame:<\/p>\n\n\n\n
import pandas as pddf = pd.DataFrame(data)
print(df.head())<\/pre>\n\n\n\n
This is useful for analysis.<\/p>\n\n\n\n
Handling API Authentication<\/h1>\n\n\n\n
Some APIs require authentication.<\/p>\n\n\n\n
Common methods:<\/p>\n\n\n\n
API Key
Bearer Token
OAuth<\/p>\n\n\n\n
Example with API key:<\/p>\n\n\n\n
headers = {
    \"Authorization\": \"Bearer YOUR_API_TOKEN\"
}response = requests.get(url, headers=headers)<\/pre>\n\n\n\n
Always keep API keys secure.<\/p>\n\n\n\n
Handling Query Parameters<\/h1>\n\n\n\n
APIs often accept parameters.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
params = {
    \"userId\": 1
}response = requests.get(url, params=params)
print(response.json())<\/pre>\n\n\n\n
This filters data from the server.<\/p>\n\n\n\n
Handling Errors and Exceptions<\/h1>\n\n\n\n
Use try-except to prevent crashes:<\/p>\n\n\n\n
try:
    response = requests.get(url)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.RequestException as e:
    print(\"API Error:\", e)<\/pre>\n\n\n\n
Handling Pagination<\/h1>\n\n\n\n
Large APIs return data in pages.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
page = 1while True:
    response = requests.get(url, params={\"page\": page})
    data = response.json()    if not data:
        break    print(\"Processing page:\", page)
    page += 1<\/pre>\n\n\n\n
Pagination ensures complete data extraction.<\/p>\n\n\n\n
Saving API Data<\/h1>\n\n\n\n
Save to CSV:<\/p>\n\n\n\n
df.to_csv(\"api_data.csv\", index=False)<\/pre>\n\n\n\n
Save to database:<\/p>\n\n\n\n
Use INSERT queries inside loop or bulk insert.<\/p>\n\n\n\n
Real-World Example<\/h1>\n\n\n\n
E-commerce analytics:<\/p>\n\n\n\n
Extract product data from API
Transform data
Load into data warehouse
Build dashboard<\/p>\n\n\n\n
This process is automated in data pipelines.<\/p>\n\n\n\n
Best Practices<\/h1>\n\n\n\n
Check status codes
Handle exceptions
Manage pagination
Secure API keys
Respect rate limits
Log errors
Automate extraction<\/p>\n\n\n\n
Common Mistakes<\/h1>\n\n\n\n
Ignoring API rate limits
Hardcoding credentials
Not handling errors
Loading all data without pagination
Not validating JSON structure<\/p>\n\n\n\n
Key Takeaway<\/h1>\n\n\n\n
Extracting data from APIs involves sending HTTP requests, receiving JSON responses, and converting them into usable formats.<\/p>\n\n\n\n
APIs are a powerful source of real-time and external data in modern ETL and data engineering workflows.<\/p>\n\n\n
Home<\/a><\/span> \u00bb PYTHON FOR DATA ENGINEERING (PYDE) > ETL and Data Pipelines > Extracting Data from APIs<\/span><\/span><\/div>\n\n\n
<\/strong> 
<\/p> <\/div> <\/div>\n","protected":false},"menu_order":122,"template":"","class_list":["post-207","lesson","type-lesson","status-publish","hentry"],"yoast_head":"\nExtracting Data from APIs - One Language. Endless Possibilities<\/title>\n<meta name=\"description\" content=\"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Extracting Data from APIs - One Language. Endless Possibilities\" \/>\n<meta property=\"og:description\" content=\"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/\" \/>\n<meta property=\"og:site_name\" content=\"One Language. Endless Possibilities\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-22T14:15:33+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":[\"WebPage\",\"FAQPage\"],\"@id\":\"https:\\\/\\\/gigz.pk\\\/python\\\/lesson\\\/extracting-data-from-apis\\\/\",\"url\":\"https:\\\/\\\/gigz.pk\\\/python\\\/lesson\\\/extracting-data-from-apis\\\/\",\"name\":\"Extracting Data from APIs - One Language. Endless Possibilities\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/gigz.pk\\\/python\\\/#website\"},\"datePublished\":\"2026-03-03T08:29:00+00:00\",\"dateModified\":\"2026-03-22T14:15:33+00:00\",\"description\":\"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/gigz.pk\\\/python\\\/lesson\\\/extracting-data-from-apis\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/gigz.pk\\\/python\\\/lesson\\\/extracting-data-from-apis\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/gigz.pk\\\/python\\\/lesson\\\/extracting-data-from-apis\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/gigz.pk\\\/python\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"PYTHON FOR DATA ENGINEERING (PYDE) > ETL and Data Pipelines > Extracting Data from APIs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/gigz.pk\\\/python\\\/#website\",\"url\":\"https:\\\/\\\/gigz.pk\\\/python\\\/\",\"name\":\"One Language. Endless Possibilities\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/gigz.pk\\\/python\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Extracting Data from APIs - One Language. Endless Possibilities","description":"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/","og_locale":"en_US","og_type":"article","og_title":"Extracting Data from APIs - One Language. Endless Possibilities","og_description":"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.","og_url":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/","og_site_name":"One Language. Endless Possibilities","article_modified_time":"2026-03-22T14:15:33+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["WebPage","FAQPage"],"@id":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/","url":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/","name":"Extracting Data from APIs - One Language. Endless Possibilities","isPartOf":{"@id":"https:\/\/gigz.pk\/python\/#website"},"datePublished":"2026-03-03T08:29:00+00:00","dateModified":"2026-03-22T14:15:33+00:00","description":"Learn how to extract data from APIs using Python, handle JSON responses, pagination, and integrate into ETL pipelines efficiently.","breadcrumb":{"@id":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/gigz.pk\/python\/lesson\/extracting-data-from-apis\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/gigz.pk\/python\/"},{"@type":"ListItem","position":2,"name":"PYTHON FOR DATA ENGINEERING (PYDE) > ETL and Data Pipelines > Extracting Data from APIs"}]},{"@type":"WebSite","@id":"https:\/\/gigz.pk\/python\/#website","url":"https:\/\/gigz.pk\/python\/","name":"One Language. Endless Possibilities","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/gigz.pk\/python\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/gigz.pk\/python\/wp-json\/wp\/v2\/lesson\/207","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gigz.pk\/python\/wp-json\/wp\/v2\/lesson"}],"about":[{"href":"https:\/\/gigz.pk\/python\/wp-json\/wp\/v2\/types\/lesson"}],"wp:attachment":[{"href":"https:\/\/gigz.pk\/python\/wp-json\/wp\/v2\/media?parent=207"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}