python-project-structure

Python Project Structure & Module Architecture

Design well-organized Python projects with clear module boundaries, explicit public interfaces, and maintainable directory structures. Good organization makes code discoverable and changes predictable.

When to Use This Skill

• Starting a new Python project from scratch
• Reorganizing an existing codebase for clarity
• Defining module public APIs with __all__
• Deciding between flat and nested directory structures
• Determining test file placement strategies
• Creating reusable library packages

When reorganizing an existing codebase, first propose the target layout and the list of file moves and renames. Get the user's confirmation before moving or renaming anything. Then restructure in small, reviewable steps, update imports and packaging configuration, and run the project's tests after each step.

Core Concepts

1. Module Cohesion

Group related code that changes together. A module should have a single, clear purpose.

2. Explicit Interfaces

Define what's public with __all__. Everything not listed is an internal implementation detail.

3. Flat Hierarchies

Prefer shallow directory structures. Add depth only for genuine sub-domains.

4. Consistent Conventions

Apply naming and organization patterns uniformly across the project.

Quick Start

myproject/
├── src/
│   └── myproject/
│       ├── __init__.py
│       ├── services/
│       ├── models/
│       └── api/
├── tests/
├── pyproject.toml
└── README.md

Fundamental Patterns

Pattern 1: One Concept Per File

Each file should focus on a single concept or closely related set of functions. Consider splitting when a file:

• Handles multiple unrelated responsibilities
• Grows beyond 300-500 lines (varies by complexity)
• Contains classes that change for different reasons

# Good: Focused files
# user_service.py - User business logic
# user_repository.py - User data access
# user_models.py - User data structures

# Avoid: Kitchen sink files
# user.py - Contains service, repository, models, utilities...

Pattern 2: Explicit Public APIs with __all__

Define the public interface for every module. Unlisted members are internal implementation details.

# mypackage/services/__init__.py
from .user_service import UserService
from .order_service import OrderService
from .exceptions import ServiceError, ValidationError

__all__ = [
    "UserService",
    "OrderService",
    "ServiceError",
    "ValidationError",
]

# Internal helpers remain private by omission
# from .internal_helpers import _validate_input  # Not exported

Pattern 3: Flat Directory Structure

Prefer minimal nesting. Deep hierarchies make imports verbose and navigation difficult.

# Preferred: Flat structure
project/
├── api/
│   ├── routes.py
│   └── middleware.py
├── services/
│   ├── user_service.py
│   └── order_service.py
├── models/
│   ├── user.py
│   └── order.py
└── utils/
    └── validation.py

# Avoid: Deep nesting
project/core/internal/services/impl/user/

Add sub-packages only when there's a genuine sub-domain requiring isolation.

Pattern 4: Test File Organization

Choose one approach and apply it consistently throughout the project.

Option A: Colocated Tests

src/
├── user_service.py
├── test_user_service.py
├── order_service.py
└── test_order_service.py

Benefits: Tests live next to the code they verify. Easy to see coverage gaps.

Caution: colocated test files and their fixtures can end up in built and distributed packages. Exclude them in the build configuration in pyproject.toml. Keep real credentials and secrets out of test files. Prefer Option B for published libraries.

Option B: Parallel Test Directory

src/
├── services/
│   ├── user_service.py
│   └── order_service.py
tests/
├── services/
│   ├── test_user_service.py
│   └── test_order_service.py

Benefits: Clean separation between production and test code. Standard for larger projects.

Advanced Patterns

Pattern 5: Package Initialization

Use __init__.py to provide a clean public interface for package consumers.

# mypackage/__init__.py
"""MyPackage - A library for doing useful things."""

from .core import MainClass, HelperClass
from .exceptions import PackageError, ConfigError
from .config import Settings

__all__ = [
    "MainClass",
    "HelperClass",
    "PackageError",
    "ConfigError",
    "Settings",
]

__version__ = "1.0.0"

Keep the version in a single place, usually pyproject.toml, or read it from the installed package metadata. That way the version cannot drift from a value hardcoded in the source, like the one shown here.

Consumers can then import directly from the package:

from mypackage import MainClass, Settings

Pattern 6: Layered Architecture

Organize code by architectural layer for clear separation of concerns.

myapp/
├── api/           # HTTP handlers, request/response
│   ├── routes/
│   └── middleware/
├── services/      # Business logic
├── repositories/  # Data access
├── models/        # Domain entities
├── schemas/       # API schemas (Pydantic)
└── config/        # Configuration

Each layer should only depend on layers below it, never above.

Pattern 7: Domain-Driven Structure

For complex applications, organize by business domain rather than technical layer.

ecommerce/
├── users/
│   ├── models.py
│   ├── services.py
│   ├── repository.py
│   └── api.py
├── orders/
│   ├── models.py
│   ├── services.py
│   ├── repository.py
│   └── api.py
└── shared/
    ├── database.py
    └── exceptions.py

File and Module Naming

Conventions

• Use snake_case for all file and module names: user_repository.py
• Avoid abbreviations that obscure meaning: user_repository.py not usr_repo.py
• Match class names to file names: UserService in user_service.py

Import Style

Use absolute imports for clarity and reliability:

# Preferred: Absolute imports
from myproject.services import UserService
from myproject.models import User

# Avoid: Relative imports
from ..services import UserService
from . import models

Relative imports can break when modules are moved or reorganized.

One exception applies: relative imports are acceptable inside the __init__.py of a package when it re-exports its own submodules, as shown in Patterns 2 and 5. Imports across packages should stay absolute.

Best Practices Summary

1. Keep files focused - One concept per file, consider splitting at 300-500 lines (varies by complexity)
2. Define __all__ explicitly - Make public interfaces clear
3. Prefer flat structures - Add depth only for genuine sub-domains
4. Use absolute imports - More reliable and clearer, except for re-exports inside the __init__.py of a package
5. Be consistent - Apply patterns uniformly across the project
6. Match names to content - File names should describe their purpose
7. Separate concerns - Keep layers distinct and dependencies flowing one direction
8. Document your structure - Include a README explaining the organization