Draft article not currently submitted for review. This is a draft Articles for creation (AfC) submission. It is not currently pending review. While there are no deadlines, abandoned drafts may be deleted after six months. To edit or make changes to this draft, simply click on the "Edit" tab at the top of the window. To be accepted, a draft should: Show the subject qualifies for a Wikipedia article by using multiple sources that meet four criteria. The sources should be (1) reliable (2) secondary (3) independent of the subject (4) talk about the subject in some depth. For some topics, there are alternative criteria. Be written from a neutral point of view Respect copyright and do not plagiarize. Do not copy-paste. It is strongly discouraged to write about either yourself or your business or employer. If you do so, you must declare it. Where to get help If you need help editing or submitting your draft, please ask us a question at the AfC Help Desk or get live help from experienced editors. These venues are only for help with editing and the submission process, not to get reviews. If you need feedback on your draft, or if the review is taking a lot of time, you can try asking for help on the talk page of a relevant WikiProject. Some WikiProjects are more active than others so a speedy reply is not guaranteed. How to improve a draft Wikipedia:Contributing to Wikipedia – a basic overview on how to edit Wikipedia. Help:Wikitext – how to use the markup Help:Referencing for beginners – how to include references Wikipedia:Article development – how to develop your article Wikipedia:Writing better articles – how to improve your article Wikipedia:Verifiability – make sure your article includes reliable third-party sources You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Add tags to your draft Editor resources Easy tools: Citation bot (help) | Advanced: Fix bare URLs Last edited by Bearcat (talk | contribs) 2 months ago. (Update) Submit the draft for review!

urllib is a package in the Python standard library for working with URLs and making HTTP requests. It provides a collection of submodules for fetching data from the web, parsing URLs, and handling network-related errors. urllib ships with every standard Python 3 installation and requires no additional dependencies.

Python's built-in URL handling has evolved considerably across major versions of the language.

The original urllib module was introduced in Python 1.2 (later accounts place it around version 1.4), providing a simple interface for fetching URLs via the urlopen() and urlretrieve() functions. It was a significant step forward from lower-level socket-based programming and made network requests more accessible.

In Python 1.6, a second module, urllib2, was added alongside the original urllib. Despite the name suggesting an incremental upgrade, urllib2 was effectively a brand-new library with a fundamentally different design. It introduced a flexible handler-based architecture, support for HTTP authentication (basic and digest), cookie handling, custom request headers, and redirect processing. urllib2 was the standard choice for more complex HTTP use cases throughout the Python 2.x era.

When Python 3 was released in 2008, the original urllib module was deprecated and removed. The functionality of urllib2 was merged and refactored into the new urllib package, split across several submodules. The transition tool 2to3 can assist in converting Python 2 urllib2 code to the Python 3 urllib equivalent, owing to their relative similarity.

In Python 3, urllib is organized into four submodules:

urllib.request is the primary submodule for opening and reading URLs. It defines the urlopen() function and the Request class, and supports:

• HTTP and HTTPS requests (GET, POST, etc.)
• HTTP authentication (basic and digest)
• HTTP cookies via http.cookiejar integration
• Proxy handling via ProxyHandler
• Redirect following
• Custom request headers and user-agent strings

Basic usage example:

import urllib.request

with urllib.request.urlopen("https://www.example.com") as response:
    html = response.read()

To send a POST request, a data argument is passed:

import urllib.request
import urllib.parse

data = urllib.parse.urlencode({"key": "value"}).encode()
req = urllib.request.Request("https://www.example.com/post", data=data)
with urllib.request.urlopen(req) as response:
    result = response.read()

urllib.parse provides functions for parsing and constructing URLs. It conforms to RFC 3986, the current standard for URI syntax.

Key functions include:

Example:

from urllib.parse import urlparse, urlencode

parsed = urlparse("https://www.example.com/index.html?query=arg#frag")
print(parsed.scheme)   # https
print(parsed.netloc)   # www.example.com

params = urlencode({"search": "python", "page": 1})
print(params)          # search=python&page=1

urllib.error defines the exceptions that can be raised by urllib.request:

Example of error handling:

import urllib.request
import urllib.error

try:
    urllib.request.urlopen("https://www.example.com/missing")
except urllib.error.HTTPError as e:
    print(f"HTTP Error: {e.code}")
except urllib.error.URLError as e:
    print(f"URL Error: {e.reason}")

urllib.robotparser provides the RobotFileParser class for reading and interpreting robots.txt files, which specify which parts of a website may be accessed by automated crawlers. It is commonly used in web scraping and crawler applications to respect the crawling policies of websites.

Example:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://www.example.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://www.example.com/page"))

Despite superficial naming similarity, urllib3 is an entirely independent third-party library unrelated to the Python standard library urllib. It was created in 2008 to address limitations of the built-in library, adding features such as:

• Connection pooling
• Thread safety
• Multipart form data
• Automatic decoding of compressed responses
• Retry logic and redirect control

The popular requests library is built on top of urllib3 and provides a higher-level, more user-friendly API for making HTTP requests. Both urllib3 and requests must be installed separately (e.g., via pip).

By default, urllib.request.urlopen() verifies TLS/SSL certificates when making HTTPS requests. Disabling this via an unverified context is strongly discouraged in production environments as it leaves connections vulnerable to man-in-the-middle attacks.

Because urllib can be used to open arbitrary URLs, applications that pass user-supplied input to urlopen() without validation may be vulnerable to Server-side request forgery (SSRF) attacks.

• Python (programming language)
• Requests (software) — popular third-party HTTP library
• HTTP
• Uniform Resource Locator
• Web scraping
• robots.txt

• Official urllib documentation (Python 3)
• urllib.request documentation
• urllib.parse documentation
• urllib.error documentation
• urllib.robotparser documentation

Category:Python (programming language) libraries Category:Internet protocols Category:Free software programmed in Python Category:Application programming interfaces