Project Structure & Packaging

0/3 in this phase0/54 across the roadmap

📖 Concept

A well-organized Python project structure makes code easier to navigate, test, import, and distribute. The modern Python ecosystem has converged on the src layout and pyproject.toml as the standard approach for professional projects.

The src layout:

my-project/
├── src/
│   └── my_package/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── test_core.py
│   └── test_utils.py
├── pyproject.toml
├── README.md
└── LICENSE

The src layout prevents accidental imports from the project root — tests always import the installed version of your package, catching packaging errors early.

pyproject.toml (PEP 621) is the modern unified configuration file for Python projects, replacing setup.py, setup.cfg, MANIFEST.in, and scattered tool configs:

Section	Purpose
`[project]`	Name, version, description, dependencies
`[build-system]`	Build backend (setuptools, hatchling, flit, maturin)
`[tool.pytest]`	Pytest configuration
`[tool.ruff]`	Linter/formatter settings
`[tool.mypy]`	Type checking configuration

Modern dependency management tools:

Tool	Strengths	Lock File
Poetry	Mature, deterministic builds, virtual env management	`poetry.lock`
uv	Extremely fast (Rust-based), pip-compatible, growing rapidly	`uv.lock`
PDM	PEP 582 support, flexible backends	`pdm.lock`
pip-tools	Minimal, compiles requirements	`requirements.txt`

Publishing to PyPI: Build with python -m build (creates wheel + sdist), then upload with twine upload dist/*. Modern tools like Poetry (poetry publish) and uv simplify this to one command. Always use TestPyPI first for validation.

Key files: __init__.py marks directories as packages and controls the public API via __all__. py.typed marker file signals that your package ships type stubs. .python-version pins the Python version for tools like pyenv and uv.

💻 Code Example

codeTap to expand ⛶

1# ============================================================
2# Modern Python Project Structure & Configuration
3# ============================================================
4
5# --- pyproject.toml (the single source of truth) ---
6# File: pyproject.toml
7
8"""
9[build-system]
10requires = ["hatchling"]
11build-backend = "hatchling.build"
12
13[project]
14name = "my-awesome-lib"
15version = "1.0.0"
16description = "A well-structured Python library"
17readme = "README.md"
18license = {text = "MIT"}
19requires-python = ">=3.11"
20authors = [
21    {name = "Your Name", email = "you@example.com"},
22]
23dependencies = [
24    "httpx>=0.27",
25    "pydantic>=2.0",
26]
27
28[project.optional-dependencies]
29dev = [
30    "pytest>=8.0",
31    "ruff>=0.5",
32    "mypy>=1.10",
33    "pre-commit>=3.7",
34]
35docs = [
36    "mkdocs>=1.6",
37    "mkdocs-material>=9.5",
38]
39
40[project.scripts]
41my-cli = "my_awesome_lib.cli:main"
42
43[tool.pytest.ini_options]
44testpaths = ["tests"]
45pythonpath = ["src"]
46addopts = "-v --tb=short"
47
48[tool.ruff]
49target-version = "py311"
50line-length = 88
51src = ["src"]
52
53[tool.ruff.lint]
54select = ["E", "F", "I", "N", "UP", "B", "SIM"]
55
56[tool.mypy]
57python_version = "3.11"
58strict = true
59warn_return_any = true
60"""
61
62
63# --- src/my_awesome_lib/__init__.py ---
64# Controls the public API of your package
65
66"""My Awesome Library — a well-structured example."""
67
68__version__ = "1.0.0"
69__all__ = ["Client", "Config", "process_data"]
70
71from my_awesome_lib.client import Client
72from my_awesome_lib.config import Config
73from my_awesome_lib.core import process_data
74
75
76# --- src/my_awesome_lib/config.py ---
77from dataclasses import dataclass, field
78from pathlib import Path
79import tomllib  # Python 3.11+ built-in TOML parser
80
81
82@dataclass(frozen=True)
83class Config:
84    """Immutable application configuration."""
85    base_url: str = "https://api.example.com"
86    timeout: int = 30
87    retries: int = 3
88    debug: bool = False
89    cache_dir: Path = field(default_factory=lambda: Path.home() / ".cache" / "mylib")
90
91    @classmethod
92    def from_toml(cls, path: Path) -> "Config":
93        """Load configuration from a TOML file."""
94        with open(path, "rb") as f:
95            data = tomllib.load(f)
96        return cls(**{k: v for k, v in data.items() if k in cls.__dataclass_fields__})
97
98    def __post_init__(self):
99        self.cache_dir.mkdir(parents=True, exist_ok=True)
100
101
102# --- src/my_awesome_lib/core.py ---
103from typing import Any
104import logging
105
106logger = logging.getLogger(__name__)
107
108
109def process_data(raw: list[dict[str, Any]]) -> list[dict[str, Any]]:
110    """Process and validate raw data entries."""
111    results = []
112    for item in raw:
113        if not item.get("id"):
114            logger.warning("Skipping item without ID: %s", item)
115            continue
116        results.append({
117            "id": item["id"],
118            "name": str(item.get("name", "")).strip(),
119            "active": bool(item.get("active", True)),
120        })
121    logger.info("Processed %d/%d items", len(results), len(raw))
122    return results
123
124
125# --- src/my_awesome_lib/client.py ---
126from my_awesome_lib.config import Config
127
128
129class Client:
130    """HTTP client for the API."""
131    def __init__(self, config: Config | None = None):
132        self.config = config or Config()
133
134    def get(self, endpoint: str) -> dict:
135        """Make a GET request (simplified example)."""
136        url = f"{self.config.base_url}/{endpoint.lstrip('/')}"
137        print(f"GET {url} (timeout={self.config.timeout}s)")
138        return {"status": "ok"}
139
140
141# --- Common CLI Commands ---
142
143# Initialize a new project with uv
144# uv init my-project && cd my-project
145
146# Add dependencies
147# uv add httpx pydantic
148# uv add --dev pytest ruff mypy
149
150# Build the package
151# uv build  # or: python -m build
152
153# Publish to TestPyPI first
154# uv publish --publish-url https://test.pypi.org/legacy/
155
156# Publish to PyPI
157# uv publish

🏋️ Practice Exercise

Exercises:

Create a complete project from scratch using uv init (or poetry new). Add httpx and pydantic as dependencies, pytest and ruff as dev dependencies. Write a pyproject.toml with all tool configurations.
Convert a flat Python script into a proper src-layout package with __init__.py, separate modules for config, core logic, and CLI entry point. Add a [project.scripts] entry so it installs as a CLI command.
Write a comprehensive __init__.py that uses __all__ to define the public API. Import key classes and functions to make them available at the package level (e.g., from mylib import Client instead of from mylib.client import Client).
Set up a pre-commit configuration (.pre-commit-config.yaml) that runs ruff (linting + formatting), mypy (type checking), and pytest on every commit. Test it by making a commit with a linting error.
Create a minimal Python package and publish it to TestPyPI. Verify you can install it with pip install --index-url https://test.pypi.org/simple/ your-package.

⚠️ Common Mistakes

Using the flat layout (package at project root) instead of src layout. The flat layout allows tests to accidentally import from the source directory instead of the installed package, masking packaging bugs.
Still using setup.py for new projects. pyproject.toml (PEP 621) is the modern standard — it consolidates project metadata, build configuration, and tool settings in one file.
Forgetting init.py files in Python packages, causing import failures. Even with implicit namespace packages (PEP 420), explicit init.py is recommended for clarity.
Not pinning dependency versions in production. Use a lock file (poetry.lock, uv.lock) for reproducible builds. Specify version ranges in pyproject.toml but always commit the lock file.
Publishing to PyPI without testing on TestPyPI first. Always validate your package metadata, dependencies, and installation flow on test.pypi.org before going live.

💼 Interview Questions

🎤 Mock Interview

Practice a live interview for Project Structure & Packaging

Was this topic helpful?

← PreviousSOLID Principles & Clean Code Next →Profiling & Benchmarking