Data model, generators, decorators, asyncio, GIL and packaging. Open a question to read the answer.
Both, in practice. CPython, the reference implementation, first compiles source into bytecode and then interprets that bytecode on a stack-based virtual machine. The compiled bytecode is cached as .pyc files in __pycache__, so unchanged modules skip the compile step on the next import.
The phrase “interpreted language” is therefore a statement about the usual implementation, not about the language itself. Other implementations compile differently: PyPy uses a tracing JIT, Cython and Nuitka translate to C, and CPython 3.13 ships an experimental copy-and-patch JIT that is disabled unless the interpreter is built with --enable-experimental-jit.
import dis
def area(r):
return 3.14159 * r * r
dis.dis(area) # shows LOAD_FAST, BINARY_OP, RETURN_VALUE...Two consequences matter in interviews: there is no separate link step and no static type checking at compile time, so type errors surface at run time, and performance depends mainly on how much work is pushed into C-level built-ins and extension modules rather than on the bytecode loop itself. Python 3.11 and later also specialise bytecode at run time (the “adaptive interpreter”), which is why upgrading alone often speeds code up noticeably.
A lambda is an anonymous function defined in a single expression: lambda arguments: expression. It creates a normal function object; the only differences from def are that it has no name (its __name__ is <lambda>), it cannot contain statements, and its body must be one expression whose value is returned implicitly.
orders = [("A-1", 250.0), ("A-2", 80.5), ("A-3", 120.0)]
by_amount = sorted(orders, key=lambda o: o[1], reverse=True)
print(by_amount[0]) # ('A-1', 250.0)
square = lambda x: x * x # legal but discouraged: use def insteadLambdas belong where a short callable is passed inline: key= arguments to sorted, min and max, callbacks, or small transformations in map. They support default values, *args and **kwargs, and they close over enclosing variables like any function, which means the usual late-binding caveat applies inside loops.
PEP 8 advises against assigning a lambda to a name, because a def gives a real name in tracebacks and allows a docstring. When the logic needs more than one line, a conditional chain, or a comment, a named function is the right tool.
These are the four built-in collection types, and each one trades off ordering, mutability and lookup cost differently.
| Type | Ordered | Mutable | Duplicates | Typical lookup |
|---|---|---|---|---|
list | Yes | Yes | Yes | O(1) by index, O(n) by value |
tuple | Yes | No | Yes | O(1) by index, O(n) by value |
set | No | Yes | No | O(1) average membership |
dict | Insertion order (3.7+) | Yes | Unique keys | O(1) average by key |
Lists are dynamic arrays: appending at the end is amortised O(1), but inserting or deleting at the front is O(n), which is why collections.deque exists for queues. Tuples are fixed-size and hashable when their elements are, so they can be dictionary keys or set members and double as lightweight records. Sets and dictionaries are hash tables; their elements and keys must be hashable, and set operations (&, |, -, ^) make them the natural tool for deduplication and membership testing.
seen = set()
unique = [x for x in [3, 1, 3, 2, 1] if not (x in seen or seen.add(x))]
print(unique) # [3, 1, 2]
point = (10, 20) # hashable, usable as a dict key
distances = {point: 22.4}Dictionaries are the backbone of the language itself: object attributes, keyword arguments and module namespaces are all dictionaries. Since Python 3.7 they are guaranteed to preserve insertion order, and dict | other merges two of them (3.9+).
Apply. In an order-processing service I pick the collection by access pattern, not by habit. Incoming line items arrive as a list because order matters for the invoice. Lookups of product metadata by SKU go through a dictionary built once at startup, so each enrichment is O(1) instead of scanning a list per item. Deduplicating customer e-mail addresses for a campaign is a set operation, and set(new_ids) -set(existing_ids) gives me the delta in one line. Immutable configuration such as (host, port) pairs becomes tuples so they can serve as dictionary keys and cannot be mutated by accident.
Test. Unit tests assert behaviour that depends on the type’s guarantees: a parametrised pytest case checks that duplicates are collapsed, that ordering is preserved where the contract promises it, and that a lookup for a missing key raises KeyError or returns the documented default. For large datasets I add a benchmark test with pytest-benchmark to catch a regression from O(1) to O(n), which typically appears when someone replaces a set membership check with x in some_list.
Troubleshoot. The classic symptom is a request that gets slower as data grows. cProfile shows time concentrated in list.__contains__ or in list.insert(0, ...); the fix is a set, a dictionary or a deque. The second common bug is a mutable default argument (def f(items=[])) that accumulates state across calls, visible as data from one request leaking into the next; the fix is items=None and creating the list inside the function. “dictionary changed size during iteration” means a loop mutated the dictionary it was iterating; iterate over list(d.items()) instead.
An important consideration is that a tuple is only as immutable as its contents: ([1], 2) cannot be a dictionary key because the inner list is unhashable.
A list comprehension builds a list from an iterable in one expression: [expression for item in iterable if condition]. It replaces the explicit loop-append pattern, runs faster than the equivalent loop because the append is executed in C rather than as a method call, and states intent in a single line.
prices = {"pen": 1.5, "book": 12.0, "bag": 35.0}
expensive = [name.upper() for name, p in prices.items() if p > 10]
print(expensive) # ['BOOK', 'BAG']
matrix = [[1, 2], [3, 4]]
flat = [x for row in matrix for x in row] # nested: outer loop first
print(flat) # [1, 2, 3, 4]Key points:
for loop. Python 3.12 inlines comprehensions into the surrounding function (PEP 709) for speed, but preserves that isolation.{x for x in ...}), dictionaries ({k: v for ...}) and, with parentheses, a lazy generator expression.for clauses nest left to right, and if filters can appear after any of them.[x if x > 0 else 0 for x in data].A comprehension becomes the wrong tool once it needs side effects, more than two for clauses, or logic that no longer fits on one or two lines; at that point an explicit loop or a helper function reads better.
Apply. In a reporting pipeline I use comprehensions for the pure transform-and-filter steps: converting a list of database rows into DTOs, selecting the subset of records that failed validation, or building a lookup dictionary with {row.id: row for row in rows}. When the input is a large file or a streaming query result, the same expression written with parentheses becomes a generator expression, so I can feed sum(...), any(...) or csv.writer.writerows(...) without materialising a list. Anything that calls out to a database or an API inside the loop stays an explicit for statement so the side effects and error handling are visible.
Test. Comprehensions are usually wrapped in small pure functions, which makes them straightforward to test with parametrised inputs: empty input, a single element, and an input where every element is filtered out. For data-heavy code I compare the result against a naive loop implementation in a test so a later “optimisation” cannot silently change semantics. ruff rules such as C4 flag unnecessary comprehensions (list(x for x in y)) and PERF rules flag loops that should be comprehensions.
Troubleshoot. The two production problems I have actually seen are memory and readability. A comprehension over a multi-million-row cursor materialised everything at once and the worker was OOM-killed; tracemalloc pointed to the line, and the fix was a generator expression plus chunked processing. The second is a triple-nested comprehension with two conditionals that nobody could debug; rewriting it as a loop with named intermediate variables fixed the logic error in minutes. Also watch for calling an expensive function twice ([f(x) for x in d if f(x)]); the walrus operator ([y for x in d if (y := f(x))]) evaluates it once.
The key consideration is that a comprehension signals “pure transformation” to readers, so hiding side effects inside one misleads the next maintainer.
An iterator is any object that implements __iter__() returning itself and __next__() returning the next value or raising StopIteration. An iterable is anything that iter() can turn into an iterator: lists, dictionaries, files, and user classes with __iter__. The for loop, unpacking, sum() and comprehensions all drive this protocol.
A generator is the simplest way to write an iterator. Any function containing yield returns a generator object when called; its body runs lazily, pausing at each yield and resuming on the next request, with local state preserved between steps.
def read_batches(path, size):
batch = []
with open(path, encoding="utf-8") as f:
for line in f:
batch.append(line.rstrip("\n"))
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
for rows in read_batches("events.log", 1000):
process(rows) # memory stays flat regardless of file sizeWhy they matter:
itertools.count()).yield from delegates to a sub-generator.map or zip object twice.send(), throw() and close() turn a generator into a coroutine-like object, which is the mechanism asyncio was originally built on.Generator expressions (x * 2 for x in data) give the same laziness without a function definition.
Apply. In an ETL job that ingests multi-gigabyte CSV exports, every stage is a generator: one reads lines, one parses them into records, one validates, and one yields insert batches of 500 rows. The stages compose with plain function calls, so adding a filter is a one-line change and peak memory stays at a few megabytes. The same pattern paginates a REST API: a generator hides the next_page_token loop and the caller just iterates. For resource cleanup I wrap generators that hold files or connections in contextlib.closing or make the pipeline a context manager, because a generator abandoned mid-way only releases its resources when garbage-collected.
Test. I test each stage with small in-memory iterables rather than files, asserting on list(stage(input)). Laziness itself is tested by passing an infinite source like itertools.count() and using itertools.islice to take the first few items; if the stage secretly calls list() on its input, the test hangs or is killed by a pytest-timeout. Exhaustion behaviour is covered by asserting that a second iteration yields nothing where that is the contract.
Troubleshoot. The most frequent bug report is “the function returns nothing”: someone added yield to a function and the caller still expects a list, so the generator object is never consumed. The opposite is passing a generator to two consumers; the second sees an empty sequence. Memory profiles that still grow usually reveal a hidden sorted() or list() in one stage that forces materialisation. GeneratorExit tracebacks at interpreter shutdown mean a finally block in a generator did something non-trivial while being closed.
One consideration: a generator with try/finally must not yield inside the finally, and errors raised inside a generator surface at the consumer’s next() call, which can be far from the faulty code.
A closure is a function that captures variables from the enclosing scope and keeps them alive after that scope has returned. Python stores the captured variables in cell objects reachable through func.__closure__. A decorator is a callable that takes a function (or class) and returns a replacement; the @name syntax is sugar for target = name(target). Most decorators are closures, because the wrapper they return must remember the original function.
import functools, time
def retry(times): # decorator factory
def decorator(func):
@functools.wraps(func) # keeps name, docstring, signature
def wrapper(*args, **kwargs):
for attempt in range(times):
try:
return func(*args, **kwargs)
except ConnectionError:
time.sleep(0.1 * 2 ** attempt)
return func(*args, **kwargs) # last attempt raises normally
return wrapper
return decorator
@retry(times=3)
def fetch_rate(currency): ...Key points:
functools.wraps copies metadata and sets __wrapped__, so inspect and debuggers can see through the wrapper.nonlocal. This late binding is why [lambda: i for i in range(3)] all return 2.__call__, and decorators stack bottom-up: the one nearest the function is applied first.The standard library uses the pattern everywhere: functools.lru_cache, functools.cache, property, contextlib.contextmanager and dataclasses.dataclass.
Apply. In a payments API decorators carry the cross-cutting concerns that must be identical on every endpoint: @require_scope("payments:write") for authorisation, @timed("payments.capture") to emit a latency histogram, and @idempotent(key=lambda req: req.headers["Idempotency-Key"]) to short-circuit retries. Each one is a closure over its configuration, so the business handler stays free of plumbing. Closures also replace tiny single-method classes: a make_rate_limiter(limit) returning a function that remembers its token bucket is clearer than a class with one method.
Test. I test the decorator in isolation by applying it to a trivial function inside the test and asserting on behaviour: the wrapped function is called the expected number of times (a Mock as the target), the metric emitter received the right name, and wrapped.__name__ and __doc__ survive, which guards against someone deleting functools.wraps. Because decorators run at import, a syntax or configuration error in one breaks every module that imports it, so a smoke test that imports every package catches that early.
Troubleshoot. Symptoms I have debugged: FastAPI endpoints losing their request-parameter signature because a decorator dropped functools.wraps, which made the framework see *args, **kwargs; inspect.signature immediately showed the problem. Tracebacks ending in wrapper() with no useful frame are solved by __wrapped__ and by keeping wrappers thin. A cache decorator holding references to large request objects produced a slow memory leak; tracemalloc snapshots pointed at the cache dictionary, and bounding it with lru_cache(maxsize=...) fixed it. Finally, UnboundLocalError inside a closure means an assignment turned a captured variable into a local; nonlocal is the fix.
The important consideration is that decorators hide control flow, so each one should do exactly one thing and be documented where it is defined.
*args and **kwargs are conventions, not keywords: a single star in a parameter list collects surplus positional arguments into a tuple, and a double star collects surplus keyword arguments into a dictionary. The names args and kwargs are only a habit; the stars do the work. At call sites the same syntax does the reverse, unpacking an iterable into positional arguments and a mapping into keyword arguments.
def log_event(event, *tags, level="info", **fields):
print(event, tags, level, fields)
log_event("login", "auth", "web", user="ana", ip="10.0.0.1")
#=> login ('auth', 'web') info {'user': 'ana', 'ip': '10.0.0.1'}
params = {"level": "warn", "user": "bob"}
log_event("logout", *["auth"], **params)
#=> logout ('auth',) warn {'user': 'bob'}The full parameter order is: positional-only parameters, then /, regular parameters, *args (or a bare *), keyword-only parameters, and finally **kwargs. A bare * forces everything after it to be passed by keyword, which is how sorted(data, key=..., reverse=True) refuses sorted(data, f, True). The / marker (Python 3.8+) makes parameters positional-only, protecting their names from becoming part of the public API.
Typical uses are forwarding arguments unchanged in decorators and subclasses (super().__init__(*args, **kwargs)), building flexible factory functions, and merging configuration dictionaries. The cost is lost discoverability: an IDE cannot tell what **kwargs accepts, so explicit parameters are preferable in public interfaces, with TypedDict plus Unpack (3.12 typing) available when the dictionary shape must be typed.
Apply. The legitimate production uses are narrow. In a service layer I use *args, **kwargs in decorators and in thin adapter classes that forward to a vendor SDK, so that a new SDK parameter needs no change in my wrapper. Builders for database clients accept **options and pass them through to the driver. Everywhere else I prefer explicit, keyword-only parameters: def create_order(*, customer_id: int, lines: list[Line], currency: str = "EUR") cannot be called with arguments in the wrong order, which has saved us from real bugs where two integer IDs were swapped.
Test. For forwarding code the test patches the downstream callable with Mock(autospec=True) and asserts mock.assert_called_once_with(expected_args, key=value), which checks that both the tuple and the dictionary arrive intact. For keyword-only APIs a test asserts that positional misuse raises TypeError. Static typing does a lot of the work: mypy --strict rejects a call that passes an unexpected keyword to a function with explicit parameters, and ParamSpec lets a decorator preserve the wrapped signature so those checks still apply through the wrapper.
Troubleshoot. TypeError: got multiple values for argument 'x' means a value was passed both positionally and via **kwargs, usually from a caller merging dictionaries. TypeError: got an unexpected keyword argument after a library upgrade means a parameter was renamed and the pass-through hid it until run time. Silently ignored options are the worst case: a function accepting **kwargs and never reading them swallows typos like timout=5; I add an explicit check that kwargs is empty after known keys are popped. Mutating the kwargs dictionary before forwarding is fine because it is a fresh dictionary per call, but mutating objects inside it is not.
The key consideration is that **kwargs trades compile-time clarity for flexibility; the trade is worth it only at genuine pass-through boundaries.
Assignment never copies in Python; b = a binds a second name to the same object. A shallow copy creates a new outer container whose elements are the same objects as the original, while a deep copy recursively copies every nested object so the two structures share nothing mutable.
| Aspect | Shallow copy | Deep copy |
|---|---|---|
| How | copy.copy(x), list(x), x[:], dict(x), x.copy() | copy.deepcopy(x) |
| Nested objects | Shared with the original | Independent clones |
| Cost | O(n) over the top level | Walks the whole graph; handles cycles via memo |
| Safe when | Elements are immutable (ints, str, tuples) | Elements are mutable and will be changed |
import copy
order = {"id": 7, "lines": [{"sku": "A", "qty": 1}]}
shallow = copy.copy(order)
deep = copy.deepcopy(order)
shallow["lines"][0]["qty"] = 99 # also changes order["lines"]
deep["lines"][0]["qty"] = 5 # order unaffected
print(order["lines"][0]["qty"]) # 99deepcopy keeps a memo dictionary so shared references and cycles are copied once and preserved, and classes can customise either operation with __copy__ and __deepcopy__. Immutable objects such as strings and tuples of immutables are returned as-is by both functions because copying them would be pointless. Python 3.13 adds copy.replace(), which creates a modified copy of objects that define __replace__, including dataclasses and named tuples, covering the common “copy with one field changed” case without a full deep copy.
Apply. In a pricing engine the base configuration is loaded once and then specialised per tenant. Each tenant gets copy.deepcopy(base_config) before overrides are applied, because the configuration is a nested dictionary of lists and a shallow copy would let one tenant’s override leak into every other tenant. Shallow copies are deliberately used where they are enough and cheaper: returning list(self._items) from a getter protects the internal list from external append calls without duplicating the items themselves. For dataclass-based domain objects I prefer dataclasses.replace() or copy.replace() (3.13) over deep copying, and I make value objects frozen=True so the question disappears entirely.
Test. Tests for anything that copies state assert isolation explicitly: copy the object, mutate a nested field in the copy, and assert the original is unchanged. A second test asserts the reverse direction. For custom __deepcopy__ implementations I also test that cycles do not recurse forever and that resources which must not be duplicated, such as a database connection held by the object, are shared rather than cloned.
Troubleshoot. The symptom is “data changes by itself”: a request handler sees fields modified by a previous request, or a default fixture in tests is polluted by an earlier test. I confirm it with id() on the nested objects, which match between the supposedly independent copies, then fix with deepcopy or by restructuring so the mutable state is created fresh. The opposite problem is performance: deepcopy on large object graphs per request showed up at the top of a cProfile listing, and the fix was to copy only the branch being modified or to switch to immutable structures. deepcopy failing with “cannot pickle” errors means the graph contains a lock, socket or generator; implementing __deepcopy__ to skip that attribute resolves it.
The key consideration is that deep copying is a sign of shared mutable state; immutability usually removes the need for it.
Identity asks whether two names refer to the same object in memory; equality asks whether two objects have the same value. is tests identity by comparing id() values and cannot be overridden. == calls __eq__, which classes define to express value semantics; the default implementation inherited from object falls back to identity, so instances of a plain class compare equal only to themselves.
a = [1, 2, 3]
b = [1, 2, 3]
c = a
print(a == b, a is b) # True False (same value, different objects)
print(a == c, a is c) # True True (same object)
x = 1000; y = 1000
print(x == y, x is y) # True, and 'is' is implementation-dependentRules that follow from this:
is: if value is None, is True, is NotImplemented. There is exactly one None, so identity is both correct and fastest.==. Code like if count is 0 works only because CPython caches small integers from -5 to 256, and Python 3.8+ emits a SyntaxWarning for is with a literal.__eq__ sets __hash__ to None unless you define it too, because equal objects must hash equally for sets and dictionaries to work.== but first check identity as a shortcut, which is why nan in [nan] is True even though nan == nan is False.String interning and constant folding make "abc" is "abc" appear to work in the REPL; relying on it is a bug waiting for a different input source.
Apply. Domain objects in a billing system get explicit equality semantics. Value objects such as Money(amount, currency) are frozen dataclasses, so __eq__ and __hash__ are generated from their fields and two equal amounts behave identically in sets and as dictionary keys. Entities such as Invoice compare by business identifier only, so I write __eq__ to compare self.id and keep __hash__ consistent with it, which lets a changed invoice still match its cached version. Sentinel objects are compared with is: _MISSING = object() as a default distinguishes “argument omitted” from “argument was None”, and if result is _MISSING is unambiguous.
Test. Each class with custom equality gets tests for reflexivity, symmetry, inequality with a different type (must return NotImplemented, not raise), and the hash contract: hash(a) == hash(b) whenever a == b. A set-based test, inserting two equal objects and asserting the length is one, catches the forgotten __hash__. For ORMs I test that an object loaded twice in different sessions compares equal, since identity will differ.
Troubleshoot. is comparisons that break after a refactor are the classic incident: a status is "active" check passed for years because the literal was interned, then failed once the value came from a database driver as a fresh string. ruff rule F632 and the interpreter’s SyntaxWarning flag this. TypeError: unhashable type after adding __eq__ to a class means __hash__ was implicitly removed. Duplicate entries in a set of objects point at an __eq__ and __hash__ pair that disagree, often because the hash includes a mutable field that changed after insertion. For NumPy arrays, == returns an element-wise array and if arr == other raises; np.array_equal is the correct comparison.
An important consideration is that mutable objects should not define __hash__ based on mutable fields, because changing them silently corrupts any set or dictionary containing the object.
Python resolves a name by searching four scopes in order: Local (the current function), Enclosing (any outer functions, innermost first), Global (the module), and Built-in (len, print, Exception). The first match wins; if none is found, NameError is raised. This lookup diagram shows the search order for a name used inside a nested function.

Scopes are decided at compile time, per function, not per block: if and for bodies do not create scopes, but comprehensions and lambdas do. Assignment makes a name local to the whole function, which produces the well-known UnboundLocalError when a function reads a global and later assigns to the same name.
def make_counter():
count = 0
def increment():
nonlocal count # rebind the enclosing variable
count += 1
return count
return increment
counter = make_counter()
counter(); print(counter()) # 2nonlocal declares that a name belongs to the nearest enclosing function scope, allowing rebinding; without it, count += 1 would create a new local and fail. global does the same for module scope. Neither is needed to mutate an object in place (items.append(x)), only to rebind the name. Class bodies are a special case: their names are not visible to methods, which must use self. or ClassName..
Apply. In production code I treat scope rules as a reason to keep state explicit. Module-level globals are limited to constants and lazily initialised singletons such as a connection pool, accessed through a function rather than rebound with global. nonlocal appears in exactly one pattern I consider idiomatic: small stateful closures created by factory functions, for example a debounce helper that remembers the last call time, or an accumulator passed as a callback to a streaming parser. Anything with more than one or two pieces of state becomes a class, because self.count is more discoverable than a nonlocal three levels up.
Test. Closures created by factories are tested through their public behaviour: create two counters and verify they are independent, which proves the state lives in the closure and not in a module global. To catch accidental global mutation, a pytest fixture snapshots module attributes before a test and compares after it, failing when a test leaks state. Linters do the structural checking: ruff ’s PLW0603 warns on global statements and F823 / F841 catch unused or shadowed locals.
Troubleshoot. UnboundLocalError: cannot access local variable 'x' where it is not associated with a value (the 3.11+ wording) means the function assigns to x somewhere, usually at the bottom, turning every earlier read into a local read. The fix is to rename, pass the value in, or declare global / nonlocal deliberately. A shadowed built-in (list = fetch_rows()) produces TypeError: 'list' object is not callable later in the module; ruff rule A001 flags it. Lambdas created in a loop all seeing the final value is the late-binding problem, solved with a default argument lambda x, i=i: ... or functools.partial. Config values that “do not update” are typically modules that imported a global with from settings import DEBUG, which copied the value at import time; accessing settings.DEBUG reads the live binding.
The important consideration is that global and nonlocal are rebinding declarations, not sharing mechanisms; shared mutable objects already work without them.
The Global Interpreter Lock is a mutex in CPython that allows only one thread to execute Python bytecode at a time. It exists because CPython’s memory management (reference counting) is not thread-safe, and a single lock is simpler and faster for single-threaded code than fine-grained locking on every object. Threads still take turns: the interpreter forces a switch roughly every 5 ms (sys.setswitchinterval), and a thread releases the GIL voluntarily whenever it blocks on I/O or enters C code that explicitly drops it, such as NumPy, hashlib and zlib. The diagram shows the practical consequence.

Consequences for concurrency design:
multiprocessing, a C extension that releases the GIL, or the free-threaded build.python3.13t, in which the GIL is disabled by default and sys._is_gil_enabled() reports False. It costs some single-threaded performance and many C extensions were still being ported in 3.13, so the default build keeps the GIL.The GIL does not make Python code thread-safe: counter += 1 is several bytecodes and can still interleave, so shared mutable state needs a threading.Lock regardless.
Apply. In a document-ingestion service the work splits along the GIL boundary. Downloading files and calling the OCR vendor’s API are I/O-bound, so a ThreadPoolExecutor with 32 workers handles them and throughput scales almost linearly. Text normalisation in pure Python is CPU-bound, so it runs in a ProcessPoolExecutor sized to os.cpu_count(). Image resizing stays on threads because Pillow releases the GIL inside its C code. Shared counters and caches are protected with threading.Lock, and queues between stages are queue.Queue, which is thread-safe by design.
Test. Concurrency tests target invariants, not timing. A stress test runs 50 threads incrementing a shared counter 10,000 times each and asserts the exact final total, which fails deterministically often enough to catch a missing lock. Throughput assumptions are checked with a benchmark that runs the CPU stage with 1, 2 and 4 threads and asserts no speed-up is expected, documenting why processes are used. Deadlock-prone code runs under pytest-timeout so a hang fails instead of blocking CI.
Troubleshoot. A CPU-bound thread pool that shows 100% of one core and idle others is the GIL at work; py-spy top shows all threads but only one with Python frames running, and py-spy dump confirms the rest are waiting for the GIL. The remedy is processes or a vectorised library. The opposite surprise is an I/O pool that stops scaling: usually one C library that does not release the GIL, or a requests.Session shared without a lock. Lost updates in a counter under load are the missing-lock bug. When evaluating the 3.13 free-threaded build I check sys._is_gil_enabled() at startup, because importing an extension that is not free-threading-safe re-enables the GIL silently unless PYTHON_GIL=0 is set.
The important consideration is that the GIL protects the interpreter, not your data; every shared mutable structure still needs explicit synchronisation.
Both run work concurrently, but threads share one interpreter and its GIL, while processes each get their own interpreter, memory space and GIL.
| Aspect | threading | multiprocessing |
|---|---|---|
| Parallel CPU execution | No (GIL), except in C code | Yes, one core per process |
| Memory | Shared; cheap to pass objects | Separate; data is pickled across |
| Start cost | Microseconds | Milliseconds and a new interpreter |
| Failure isolation | A crash kills the process | A crashed worker is restartable |
| Best for | I/O-bound: network, disk, databases | CPU-bound: parsing, numeric, compression |
concurrent.futures gives both the same interface, so the choice is one line. |
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
def checksum(path): # CPU-bound
import hashlib
with open(path, "rb") as f:
return hashlib.file_digest(f, "sha256").hexdigest()
with ProcessPoolExecutor() as pool: # swap to ThreadPoolExecutor
digests = list(pool.map(checksum, paths)) # for I/O-bound workProcesses come with constraints: arguments and results must be picklable, so lambdas, open connections and locks cannot cross the boundary, and the start method matters. On Linux the default in 3.12 and 3.13 is still fork, which copies the parent including any held locks, so 3.12 warns when forking a multi-threaded process; macOS and Windows default to spawn, which re-imports the main module and requires the if __name__ == "__main__": guard. Sharing large arrays efficiently uses multiprocessing.shared_memory instead of pickling. The 3.13 free-threaded build narrows the gap by letting threads run bytecode in parallel, but in the default build this table still holds.
Apply. In a nightly report generator the fetch phase pulls from a dozen APIs and databases, so it uses a ThreadPoolExecutor; the render phase turns each dataset into PDFs with a pure-Python library and is CPU-heavy, so it runs in a ProcessPoolExecutor with max_workers=os.cpu_count() - 1 to leave a core for the scheduler. Work units are small, picklable dataclasses rather than ORM objects, and each worker process opens its own database connection in an initializer (ProcessPoolExecutor(initializer=connect)) because connections cannot be forked safely. On Linux I set the start method to spawn or forkserver explicitly to avoid inheriting the parent’s threads and locks.
Test. Unit tests run the worker function directly, single-threaded, with the pool replaced by a fake executor that calls fn synchronously; this keeps tests fast and deterministic. A separate integration test runs the real pool with two workers to verify that tasks are picklable, that the __main__ guard is present, and that results come back in the expected order via executor.map. Resource tests assert that the pool is shut down (with block) and that no worker processes survive, using multiprocessing.active_children().
Troubleshoot. PicklingError: Can't pickle <function <lambda>> or local object means a closure or nested function was submitted to a process pool; move it to module level. A process pool that hangs on startup under macOS or in a Docker container is almost always a missing if __name__ == "__main__": guard with the spawn method, visible as recursive re-imports in the log. Deadlocks after fork show up as children stuck inside the logging module; the fix is forkserver / spawn or os.register_at_fork. For threads, py-spy dump reveals a pool where every worker waits on the same lock, and a thread pool with no speed-up on a CPU task is the GIL rather than a bug.
The important consideration is that processes multiply memory usage: a 2 GB in-memory model times eight workers will not fit, so load it in the initializer once per worker or share it via shared_memory.
asyncio provides single-threaded cooperative concurrency: one event loop runs many coroutines, and each coroutine gives up control at an await whenever it would otherwise block on I/O. async def defines a coroutine function; calling it returns a coroutine object that does nothing until the loop drives it. await suspends the current coroutine until the awaited object completes, letting the loop run others meanwhile. The diagram shows that cycle.

import asyncio, httpx
async def fetch(client, url):
r = await client.get(url, timeout=5)
return r.status_code
async def main():
async with httpx.AsyncClient() as client:
async with asyncio.TaskGroup() as tg: # 3.11+
tasks = [tg.create_task(fetch(client, u)) for u in urls]
print([t.result() for t in tasks])
asyncio.run(main())Essentials:
time.sleep or requests.get inside a coroutine freezes every other task; use await asyncio.sleep, an async client, or await asyncio.to_thread(blocking_fn).asyncio.run() creates the loop, runs the coroutine and closes it. TaskGroup (3.11) and asyncio.timeout() (3.11) replace gather and wait_for for structured cancellation and error handling.await produces a RuntimeWarning: coroutine ... was never awaited.Apply. A notification gateway I worked on fans each event out to e-mail, SMS and push providers. With asyncio the handler awaits all three in a TaskGroup, so latency is the slowest provider rather than the sum, and a single worker holds thousands of in-flight requests. Database access goes through an async driver (asyncpg), HTTP through httpx.AsyncClient with a shared connection pool, and the one legacy SDK that is synchronous is isolated behind asyncio.to_thread. Every outbound call sits inside async with asyncio.timeout(3):, and a Semaphore caps concurrency per provider to respect their rate limits.
Test. Tests are written with pytest-asyncio (or anyio), marking coroutine tests and using asyncio.Event and fake clients instead of sleeping for timing. respx or httpx.MockTransport stubs the HTTP layer so the full fan-out path runs without the network. Cancellation and timeout paths get explicit tests: a provider that never responds must raise TimeoutError within the budget and must not leave tasks running, which I assert with asyncio.all_tasks() after the test. Running the suite with PYTHONASYNCIODEBUG=1 turns forgotten awaits into logged errors.
Troubleshoot. Rising latency across unrelated endpoints is the signature of a blocking call in the loop; debug mode logs Executing <Task ...> took 2.3 seconds, and py-spy dump shows the loop thread inside a synchronous driver or json.loads on a huge payload. The fix is an async library or to_thread. “Task was destroyed but it is pending” at shutdown means fire-and-forget create_task calls without keeping references; TaskGroup fixes the lifecycle. RuntimeError: attached to a different loop comes from creating a client or lock at import time with a different loop than asyncio.run later uses; create them inside main(). Memory growth often turns out to be an unbounded gather over a large list, replaced with a semaphore or asyncio.Queue with workers.
The important consideration is that one blocking call anywhere stalls the whole process; asyncio is only as good as the least asynchronous library in the dependency tree.
A context manager is an object that defines __enter__ and __exit__, and the with statement guarantees that __exit__ runs when the block ends, whether it ends normally, by return, or by an exception. It replaces try/finally boilerplate for acquiring and releasing resources: files, locks, database transactions, temporary directories, and timing or logging around a block.
import contextlib, time
@contextlib.contextmanager
def timed(label):
start = time.perf_counter()
try:
yield # the with-block runs here
finally:
print(f"{label}: {time.perf_counter() - start:.3f}s")
with timed("load"), open("data.csv", encoding="utf-8") as f:
rows = f.read().splitlines()
#=> prints "load: 0.012s" even if read() raisedHow the protocol works:
__enter__ runs first and its return value is bound by as. __exit__(exc_type, exc, tb) receives the exception, if any; returning True suppresses it, anything falsy re-raises it after cleanup.contextlib.contextmanager turns a generator into a context manager: code before yield is the enter phase, code after it is the exit phase, and try/finally around the yield is what makes cleanup reliable.with, and since Python 3.10 they may be wrapped in parentheses across lines. contextlib.ExitStack handles a variable number of managers.async with uses __aenter__ / __aexit__ for asynchronous resources like database sessions and HTTP clients.contextlib also provides suppress, redirect_stdout, closing, nullcontext and chdir (3.11) for common cases. A lock used with with lock: is the smallest example and the one most people meet first.
Apply. In a service backed by PostgreSQL, the unit of work is a context manager: with uow() as session: opens a transaction, yields the session, commits on normal exit and rolls back when an exception propagates. Request handlers cannot forget to commit or roll back because the structure of the code enforces it. The same pattern wraps temporary files (tempfile.TemporaryDirectory), feature-flag overrides in tests, metric timers, and a contextlib.ExitStack that opens a dynamic number of output files for a multi-partition export and closes all of them in reverse order even if the fifth one fails to open.
Test. A context manager is tested through its three exits: normal completion, return inside the block, and an exception. For the transaction manager a fake session records calls and the test asserts commit was called exactly once on success and rollback exactly once when the block raises, and that the original exception still propagates. I also test that __exit__ does not swallow exceptions by accident, since returning a truthy value there is a common mistake. pytest.raises is itself a context manager, which makes the exception-path test read naturally.
Troubleshoot. Connection-pool exhaustion after a few hours is almost always a resource acquired without with on an error path; ResourceWarning: unclosed <socket> appears when running with python -X dev or -W error::ResourceWarning, and tracebacks name the allocation site. A generator-based manager that fails to clean up points at a missing try/finally around the yield. “generator didn’t yield” or “didn’t stop” errors mean the generator has a conditional yield or yields twice. Exceptions that vanish silently trace back to an __exit__ returning True, or a bare except inside the cleanup. For async code, calling an async with manager with a plain with raises AttributeError: __enter__, which is the fastest way to spot a sync/async mismatch.
The important consideration is that cleanup in __exit__ must itself be exception-safe; if it raises, it replaces the original exception and hides the real cause.
Exceptions are objects derived from BaseException that unwind the call stack until a matching except clause catches them. A try statement has four optional parts: except handles a matching exception, else runs only when the try block raised nothing, and finally runs on every exit path: success, handled exception, unhandled exception, return, break or continue. That guarantee is what makes finally the right place for cleanup when no context manager exists.
def load_config(path):
f = None
try:
f = open(path, encoding="utf-8")
data = parse(f.read())
except FileNotFoundError:
return {} # finally still runs before returning
except ValueError as exc:
raise ConfigError(f"bad config in {path}") from exc
else:
return data
finally:
if f is not None:
f.close()Points that interviewers probe:
except: also catches KeyboardInterrupt and SystemExit; except Exception: is the widest sensible net.raise ... from exc sets __cause__ for explicit chaining; an exception raised inside a handler without from gets an implicit __context__, shown as “During handling of the above exception, another exception occurred”.return inside finally overrides the pending return value and silently discards any in-flight exception. It is legal, almost always a bug, and Python 3.14 starts warning about it.ExceptionGroup and except* for handling several concurrent failures, plus exc.add_note() to attach context without wrapping.try/except costs almost nothing when no exception is raised (3.11 made it zero-cost), so the “easier to ask forgiveness than permission” style is idiomatic for dictionary access and type conversion.Custom exceptions should form a small hierarchy rooted at one application base class so callers can catch broadly or narrowly.
Apply. A payment-capture service defines a hierarchy PaymentError with DeclinedError, GatewayTimeout and DuplicateCapture subclasses. The HTTP layer maps each to a status code in one place; the domain layer raises them with raise DeclinedError(code) from vendor_exc so the vendor traceback is preserved in logs without leaking into the API response. Retries happen only for the subclass that is safe to retry. finally blocks are rare because context managers handle most cleanup, but one appears in the job runner to mark a job as finished in the database whatever happened. Unexpected exceptions bubble to a single middleware that logs with logger.exception(), attaches a correlation ID with add_note, and returns a 500.
Test. Every custom exception has a test that it is raised for the intended input, using pytest.raises(DeclinedError) as info and asserting on info.value.code and on info.value.__cause__. The middleware test feeds it each exception class and asserts the status code and that the response body carries no stack trace. Cleanup paths are tested by making the body raise and asserting the finally effect, such as the job row being marked failed. A linter configuration (ruff rules BLE001, E722, TRY) rejects bare excepts and except Exception: pass.
Troubleshoot. Silent failures almost always trace back to an over-broad except Exception that logs at debug level or swallows the error; the fix is to narrow it and re-raise. “During handling of the above exception” chains with an unrelated second error usually come from cleanup code in a handler that itself fails, hiding the root cause; __context__ in the traceback shows both. A finally with return produced a function that “never fails” and masked a TypeError for weeks. ExceptionGroup tracebacks from TaskGroup confused on-call engineers until the handler used except* to split retryable from fatal sub-exceptions.
The important consideration is that an exception handler should either handle the error fully or add information and re-raise; doing neither destroys the evidence needed to fix the problem.
A class is a namespace that produces instances; inheritance lets a class reuse and specialise another by listing bases in its definition. Python supports multiple inheritance, so it needs a deterministic rule for which implementation of a method is found first: the Method Resolution Order, computed with the C3 linearisation algorithm and exposed as Cls.__mro__. C3 guarantees that a class precedes its bases and that the order in the base list is respected, and it raises TypeError if those constraints cannot be satisfied. This hierarchy shows the classic diamond.

class Base:
def send(self, req): return f"sent {req}"
class LoggingMixin(Base):
def send(self, req):
print("log:", req)
return super().send(req)
class RetryMixin(Base):
def send(self, req):
return super().send(req) or self.send(req)
class ApiClient(LoggingMixin, RetryMixin): pass
print([c.__name__ for c in ApiClient.__mro__])
#=> ['ApiClient', 'LoggingMixin', 'RetryMixin', 'Base', 'object']super() does not mean “my parent”; it means “the next class after me in the MRO of the instance’s type”. That is what lets each mixin call super().send() and still reach RetryMixin before Base, a pattern known as cooperative multiple inheritance. For it to work, every class in the chain must call super() and accept compatible arguments.
Attribute lookup follows the same order for data attributes, and isinstance / issubclass consult it too. Abstract base classes (abc.ABC) and Protocol classes shape hierarchies without forcing shared implementation.
Apply. In an integrations platform each external provider is a subclass of an abstract Connector with fetch(), map() and push() declared as abstractmethods, so a missing method fails at instantiation instead of at 3 a.m. Cross-cutting behaviour is added by small mixins (RateLimitedMixin, CachedMixin) that each override one method and call super(), so a concrete connector is class Shopify(CachedMixin, RateLimitedMixin, Connector) and the MRO is the execution order of those layers. Hierarchies stay shallow, two or three levels, and composition replaces inheritance when the relationship is “has a” rather than “is a”.
Test. The abstract base gets a test that instantiating it, or a subclass missing a method, raises TypeError. For mixin stacks I assert on Cls.__mro__ directly in a test, because the order is part of the design: if someone reorders bases, the test explains why that matters. Behaviour tests stub Connector.push and assert that the cache layer is consulted before the rate limiter. typing.override (3.12) on every overriding method lets mypy or pyright fail the build when a base-class method is renamed and an override silently becomes a dead method.
Troubleshoot. TypeError: Cannot create a consistent method resolution order (MRO) appears when base classes are listed in an order that contradicts their own hierarchy, typically class X(Base, Mixin) where Mixin already subclasses Base; swapping the order fixes it. A mixin whose code never runs is usually placed after a class that does not call super(), breaking the chain; Cls.__mro__ and a breakpoint confirm it. TypeError: __init__() got an unexpected keyword argument in a cooperative hierarchy means one __init__ does not accept and forward **kwargs. Mutable class attributes (items = []) shared by every instance produce cross-instance state leaks; the fix is to assign them in __init__.
The important consideration is that multiple inheritance only stays manageable when every participating class is written cooperatively and does one thing.
All three are built-in descriptors that change how a function defined in a class body is bound when accessed through the class or an instance.
| Decorator | First argument | Typical use |
|---|---|---|
| plain method | self (the instance) | Behaviour that needs instance state |
@classmethod | cls (the class) | Alternative constructors; factories that respect subclasses |
@staticmethod | none | Utility grouped in the class for namespacing |
@property | self, called on attribute access | Computed or validated attributes with field-like syntax |
from datetime import date
class Invoice:
def __init__(self, number: str, issued: date):
self.number, self.issued = number, issued
@classmethod
def from_row(cls, row: dict) -> "Invoice":
return cls(row["number"], date.fromisoformat(row["issued"]))
@staticmethod
def is_valid_number(value: str) -> bool:
return value.startswith("INV-")
@property
def age_days(self) -> int:
return (date.today() - self.issued).daysA classmethod receives the class it was called on, so SubInvoice.from_row(...) builds a SubInvoice without any change. A staticmethod is just a function living in the class namespace; it signals that the logic needs neither instance nor class. A property turns inv.age_days into a method call, with optional @age_days.setter and .deleter for validation on assignment, which lets a plain attribute become computed later without breaking callers. functools.cached_property computes once per instance and stores the result.
One version detail: stacking @classmethod on top of @property to make a “class property” worked from 3.9, was deprecated in 3.11 and no longer works in 3.13, where Cls.attr returns a bound method instead of the computed value; a metaclass property or a plain classmethod is the alternative.
Apply. Domain models in an inventory service use classmethods as named constructors: StockLevel.from_event(evt), StockLevel.from_db_row(row) and StockLevel.empty(sku), which keeps __init__ simple and makes every creation path searchable. Properties expose derived data, such as available = on_hand - reserved, and validate writes: the reorder_point setter rejects negatives, so bad data cannot enter through attribute assignment. Expensive derived values that do not change over the object’s life, like a parsed configuration tree, use functools.cached_property. Staticmethods are limited to pure helpers that read better next to the class, for example Money.parse("12.50 EUR").
Test. Alternative constructors get a test per input shape, including a subclass test asserting type(Sub.from_row(r)) is Sub, which is the whole point of cls. Property setters get boundary tests (pytest.raises(ValueError) for -1). For cached_property I test that the underlying computation runs once by patching it with a Mock and asserting call_count == 1 after two accesses, and I test that cache invalidation (del obj.attr) works where the design relies on it.
Troubleshoot. TypeError: 'int' object is not callable after accessing obj.total() means someone converted total into a property and a caller still has parentheses; the reverse produces a bound method where a value was expected, visible as <bound method ...> in logs. A property that hides an expensive database call becomes a performance incident when a template loops over it; cProfile shows the property at the top, and the fix is cached_property or an explicit method name that signals cost. cached_property fails with TypeError: No '__dict__' attribute on classes using __slots__, so those need a manual cache slot. Mock-based tests that try to patch a property with patch.object(obj, "prop") need new_callable=PropertyMock or patching on the class.
The important consideration is that a property should behave like an attribute: cheap, side-effect free and unlikely to raise, otherwise a method is the honest interface.
Dunder (“double underscore”) methods such as __init__, __len__ and __eq__ are the hooks through which Python’s syntax and built-in functions talk to objects. Writing len(x) calls x.__len__(), a + b calls a.__add__(b) (falling back to b.__radd__(a)), for item in x calls x.__iter__(), and with x: calls __enter__ / __exit__. Implementing the right ones makes a user-defined class feel like a built-in type.
from dataclasses import dataclass
@dataclass(frozen=True)
class Money:
amount: int # minor units
currency: str
def __add__(self, other):
if not isinstance(other, Money) or other.currency != self.currency:
return NotImplemented # lets Python try __radd__ or raise
return Money(self.amount + other.amount, self.currency)
def __str__(self):
return f"{self.amount / 100:.2f} {self.currency}"
print(Money(1050, "EUR") + Money(250, "EUR")) # 13.00 EURGroups worth knowing:
__new__, __init__, __repr__ (unambiguous, for developers), __str__ (for users), __format__.__eq__, __lt__ and friends (functools.total_ordering fills in the rest), __hash__, __bool__.__len__, __getitem__, __setitem__, __contains__, __iter__, __next__.__call__, __getattr__ (fallback only), __getattribute__ (every access), __setattr__, __slots__.__enter__ / __exit__, __aenter__ / __aexit__, __await__, __del__ (unreliable, avoid), __init_subclass__, __class_getitem__.Returning NotImplemented from binary operators, rather than raising, is what allows mixed-type arithmetic to work. Dunder names are reserved for the language; inventing new ones is legal but conflicts with future versions.
Apply. In a scheduling system TimeRange implements __contains__ so if ts in shift: reads naturally, __and__ to compute overlaps, __lt__ so ranges sort by start time, and __repr__ that prints the ISO bounds, which makes log lines and failed-test output self-explanatory. A FeatureFlags object implements __getattr__ to look up flags lazily and __dir__ so tab completion works. A Pipeline class implements __call__ so it can be passed wherever a plain function is expected, and __iter__ so its stages can be inspected. __slots__ on the millions of Event instances in the ingestion path cut memory by roughly half and blocked typo’d attribute assignments.
Test. Each operator gets tests for the happy path, the reflected form (other + money when other is a foreign type returning NotImplemented), and the mixed-currency failure which must raise TypeError. __repr__ is tested with eval(repr(obj)) == obj where that round-trip is a design goal. For __eq__ / __hash__ pairs, a set-insertion test guards the contract. Protocol conformance is checked structurally: isinstance(obj, collections.abc.Sequence) or a Protocol type checked by mypy.
Troubleshoot. __getattr__ that raises anything other than AttributeError breaks hasattr, getattr(obj, name, default) and copy.deepcopy, producing bizarre errors far from the class; the fix is to raise AttributeError for unknown names. An infinite recursion inside __getattribute__ or __setattr__ comes from accessing self.x instead of object.__getattribute__(self, "x"). Objects that are “truthy when empty” are missing __bool__ or __len__. Pickling failures after adding __slots__ need __getstate__ / __setstate__. A __del__ that touches a database connection fails randomly at interpreter shutdown; replacing it with a context manager or weakref.finalize resolves it.
The important consideration is that dunder methods make syntax do work invisibly, so each should be cheap, predictable, and consistent with its built-in counterpart’s contract.
dataclasses generates boilerplate for classes that mainly hold data, and typing provides the annotations that static checkers such as mypy and pyright use to find bugs before run time. They fit together because @dataclass reads the class’s annotations to decide which fields exist.
from dataclasses import dataclass, field
from typing import Literal, Self
@dataclass(slots=True, kw_only=True)
class Order:
id: int
status: Literal["new", "paid", "shipped"] = "new"
lines: list["Line"] = field(default_factory=list)
total_cents: int = field(default=0, compare=False)
def with_status(self, status: Literal["paid", "shipped"]) -> Self:
return dataclasses.replace(self, status=status)What @dataclass generates and controls:
__init__, __repr__ and __eq__ from the fields; order=True adds ordering methods, frozen=True makes instances immutable and hashable, slots=True (3.10) removes the per-instance dictionary, and kw_only=True (3.10) forces keyword construction.field() handles mutable defaults via default_factory, excludes fields from comparison or repr, and __post_init__ runs validation.dataclasses.asdict, astuple and replace cover serialisation and copy-with-change.Typing essentials in 3.12 and 3.13: built-in generics (list[int], dict[str, Order]), unions with |, Optional written as X | None, TypedDict for dictionary shapes, Protocol for structural typing, Self, Literal, and the 3.12 type Alias = ... statement and class Stack[T]: generic syntax from PEP 695. Python 3.13 added TypeIs, ReadOnly for TypedDict fields and defaults for type parameters. Annotations are not enforced at run time; a dataclass happily accepts a string for an int field, which is why pydantic or attrs with validators exist for untrusted input.
Apply. Internal domain objects in an order service are frozen, slotted dataclasses: they are cheap, hashable, and impossible to mutate by accident, and replace() gives the functional update style. Boundary data (HTTP bodies, queue messages) goes through pydantic models instead, because there the types must be enforced, then converted to dataclasses for the domain layer. Type hints are complete across the codebase, mypy --strict runs in CI, and Protocol classes define what a repository or a payment gateway must offer, so test doubles need no inheritance. Literal types for status fields and TypedDict for the few legacy dictionary payloads catch most of the “wrong string” bugs at check time.
Test. The type checker is the first test; CI fails on new errors and warn_unused_ignores keeps the # type: ignore count from growing. Dataclass behaviour tests cover __post_init__ validation (pytest.raises(ValueError)), equality ignoring compare=False fields, and that frozen instances raise FrozenInstanceError on assignment. For serialisation, a round-trip test asserts from_dict(asdict(obj)) == obj. Where a generic function must accept a Protocol, a small test instantiates a minimal conforming class and passes it, confirming the protocol is not over-specified.
Troubleshoot. ValueError: mutable default <class 'list'> for field means a list or dict default without default_factory. TypeError: non-default argument follows default argument is solved with kw_only=True or reordering. Inheritance with slots=True fails if the base class is not slotted too, and cached_property needs a __dict__, so slotted dataclasses must not use it. mypy reporting Incompatible types on a dataclass field usually points at Optional fields initialised to None without X | None in the annotation. Circular references between typed modules are resolved with from __future__ import annotations or TYPE_CHECKING imports. Run-time NameError for a forward-referenced type means code called typing.get_type_hints() on a module where the name is only imported under TYPE_CHECKING.
The important consideration is that annotations are documentation checked by tools, not run-time guarantees; validation at system boundaries is still required.
CPython manages memory with two cooperating mechanisms: reference counting as the primary collector and a cyclic garbage collector for what reference counting cannot handle. Every object carries a count of references to it; when the count drops to zero the object is freed immediately and deterministically. Cycles (a.ref = b; b.ref = a) keep counts above zero forever, so the gc module periodically scans container objects to find unreachable cycles. The diagram shows where each piece of the system sits.

Key behaviour in 3.12 and 3.13:
gc.get_threshold() returns (700, 10, 10) in 3.12 and (2000, 10, 10) in 3.13). It can be tuned, disabled with gc.disable(), or run on demand with gc.collect().__del__ in cycles are collected since 3.4 (PEP 442), but finaliser order is undefined.pymalloc pools inside 256 KiB arenas (1 MiB since 3.10 on 64-bit builds); memory is returned to the OS only when a whole arena is empty, so a process’s RSS rarely shrinks after a spike.sys.getrefcount(x) reports the count (one higher because of the argument), tracemalloc tracks allocations by source line, and weakref lets caches reference objects without keeping them alive.Memory is therefore freed promptly in the common case, and “leaks” in Python are almost always live references held by caches, globals or closures rather than collector failures.
Apply. In a long-running worker that processes millions of small events, memory design is mostly about not keeping references: results are streamed with generators, caches are bounded (functools.lru_cache(maxsize=10_000) or cachetools.TTLCache), and observer registrations use weakref.WeakSet so an unsubscribed listener does not survive. Hot-path objects use __slots__ or slotted dataclasses, which roughly halves per-instance memory. On a web server that forks workers after loading a large model, gc.freeze() before forking stops the collector from dirtying shared pages, and batch jobs that build huge object graphs disable the collector during the build and call gc.collect() once at the end.
Test. Memory regressions are tested with tracemalloc: take a snapshot, run the operation 1,000 times, take another, and assert the growth of the top statistics is below a budget. A simpler guard asserts that a weakref.ref to a request object is dead after the handler returns. These tests run in CI on a representative fixture, not the full dataset.
Troubleshoot. RSS growing linearly in production is the common page. First I confirm whether Python objects are growing (tracemalloc snapshot diff or objgraph.show_most_common_types()) or whether it is native memory from a C extension, which tracemalloc cannot see. Typical culprits: an unbounded dictionary cache keyed by request ID, logging handlers appended on every request, exception objects stored in a list (each one holds its traceback and frames), and lru_cache on a method, which keeps every self alive. A “leak” that is really fragmentation shows RSS staying high while tracemalloc is flat; the remedy is recycling workers (max_requests in gunicorn). Long GC pauses in a latency-sensitive service appear as periodic spikes; gc.set_debug(gc.DEBUG_STATS) measures them and reducing live container objects fixes them.
The important consideration is that gc.collect() is not a fix for a leak caused by a live reference; it only frees what is already unreachable.
A virtual environment is an isolated directory with its own site-packages and a python launcher that points to a base interpreter. Packages installed inside it do not affect the system Python or other projects, which is what makes two services with conflicting dependency versions coexist on one machine. python -m venv .venv creates one; activating it only adjusts PATH, and .venv/bin/python -m pip ... works without activation. Tools such as uv, poetry, hatch and pipenv create and manage the same kind of environment with extra conveniences like lock files.
Packaging is the process of describing a project so it can be built and installed. Modern projects declare everything in pyproject.toml: metadata and dependencies in the [project] table (PEP 621), and the build backend in [build-system]. The diagram shows the path from source to an installed package.

[project]
name = "order-service"
version = "1.4.0"
requires-python = ">=3.12"
dependencies = ["httpx>=0.27", "pydantic>=2,<3"]
[project.optional-dependencies]
dev = ["pytest", "mypy", "ruff"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"Wheels are pre-built zip archives installed by copying; sdists must be built on the target machine. pip install -e . installs a project in editable mode for development. Version pins for applications belong in a lock file (requirements.txt from pip-compile, uv.lock, or the standardised pylock.toml from PEP 751); libraries declare ranges instead. distutils was removed in Python 3.12, so legacy setup.py projects must depend on setuptools explicitly.
Apply. Every service repository has a pyproject.toml with a src/ layout, a pinned requires-python, and ranges for direct dependencies only. A lock file from uv lock (or pip-compile) pins the full transitive tree with hashes, and that lock, not the ranges, is what CI and the Dockerfile install, so a build today and a build next month are identical. Developers create the environment with uv sync or python -m venv .venv && pip install -e ".[dev]". Shared internal libraries are built as wheels with python -m build, published to a private index and consumed through the same lock. Docker images use multi-stage builds: dependencies go into a venv in the builder stage and are copied into a slim runtime image.
Test. CI runs the suite in a fresh environment created from the lock file, which proves the lock is complete, across a matrix of the supported interpreters (3.12 and 3.13) because requires-python is a promise. For libraries, a job installs the built wheel into an empty venv and imports the package, catching missing package data that editable installs hide. pip check verifies the resolved set has no conflicts, and a scheduled job attempts a dependency upgrade and runs the tests to surface breakage early.
Troubleshoot. “Works on my machine” almost always means something was installed outside the lock; pip freeze diffed against the lock reveals it. ModuleNotFoundError inside Docker while tests pass locally points to the src/ package missing from the wheel or to the system interpreter being used instead of the venv; python -c "import sys; print(sys.prefix)" settles it. ResolutionImpossible is diagnosed by reading the two requirements it names and loosening the stricter one. A compiled dependency failing to install on a new interpreter means no wheel exists yet for that version; staying on the previous interpreter until upstream ships one is the pragmatic answer.
The important consideration is that lock files belong to applications; a library that pins exact versions makes itself uninstallable alongside anything else.
A module is a .py file (or a C extension) whose top-level code runs once, on first import, to populate a module object that is then cached in sys.modules. A package is a directory of modules, with or without __init__.py (namespace packages, PEP 420). import a.b finds the module through sys.path, executes it top to bottom, and binds the name; from a.b import name additionally looks up name in the finished module. Subsequent imports anywhere in the process return the cached object, so module state is effectively a process-wide singleton.
A circular import occurs when module a imports b while b is itself in the middle of importing a. Because a is already registered in sys.modules but only partially executed, import a inside b succeeds, yet from a import something fails if something has not been defined yet. The diagram shows the failing sequence.
Ways to break the cycle, in order of preference:
import models and then models.Order at call time), so the attribute is resolved after both modules finish loading.if TYPE_CHECKING: and use string annotations or from __future__ import annotations.Python 3.8+ adds “(most likely due to a circular import)” to the ImportError message, and python -X importtime prints the import tree with timings, which also shows the order in which the cycle was entered.
Apply. Package layout follows dependency direction: domain imports nothing from the application, services imports domain, api imports services, and nothing imports upward. Enforcing this with a tool (import-linter contracts in CI) is cheaper than untangling cycles later. Module-level code is kept to definitions; anything that connects to a database or reads configuration happens inside functions, so importing a module is always side-effect free and fast. Public API of a package is re-exported in __init__.py with __all__, while heavy optional dependencies (pandas, boto3) are imported lazily inside the functions that need them to keep CLI start-up quick.
Test. A smoke test imports every module in the package (pkgutil.walk_packages) in a fresh interpreter, which catches circular imports and import-time side effects regardless of which module the test suite happened to import first. The layering contracts run as a CI step. Start-up time is tracked by running python -X importtime -c "import myservice" in CI and failing if the total exceeds a budget, which has caught a stray import torch at module level more than once.
Troubleshoot. ImportError: cannot import name 'X' from partially initialized module is the circular import itself; -X importtime or the traceback shows which module was mid-execution. The quick fix is a local import; the real fix is moving the shared type to a leaf module. AttributeError: module 'x' has no attribute 'y' with a correct-looking module usually means a file in the project shadows a standard-library or third-party name (logging.py, test.py); print(x.__file__) exposes it. Tests that pass alone but fail together are import-order dependent, typically because one module monkeypatches another at import. Behaviour that differs between python script.py and python -m pkg.script comes from sys.path[0] being the script’s directory in the first case, so relative imports break; running as a module is the fix.
The important consideration is that a circular import is a design signal, two modules that know too much about each other, and a local import only hides the symptom.
These are the functional-programming primitives: lambda creates a small anonymous function, map applies a function to every element, filter keeps elements for which a predicate is true, and reduce folds a sequence into a single value by repeatedly combining an accumulator with the next element. In Python 3, map and filter return lazy iterators, and reduce lives in functools.
from functools import reduce
prices = [19.99, 5.0, 120.0, 42.5]
with_tax = map(lambda p: round(p * 1.2, 2), prices) # lazy
large = filter(lambda p: p > 20, prices) # lazy
total = reduce(lambda acc, p: acc + p, prices, 0.0) # eager
print(list(with_tax)) # [23.99, 6.0, 144.0, 51.0]
print(list(large)) # [120.0, 42.5]
print(total) # 187.49reduce(f, items, initial) computes f(f(f(initial, items[0]), items[1]), ...). Without initial the first element seeds the accumulator and an empty input raises TypeError, so the initial value is worth supplying.
Idiomatic Python usually prefers comprehensions and built-ins to these functions: [p * 1.2 for p in prices] reads more clearly than map with a lambda, [p for p in prices if p > 20] replaces filter, and sum, min, max, any, all, math.prod and "".join cover almost every real use of reduce. map still earns its place when the function already exists (map(str, numbers), map(int, line.split())) or when applying a function in parallel through executor.map. operator.add, operator.itemgetter and operator.attrgetter replace many lambdas with faster, named callables.
Apply. In a data-cleaning pipeline these primitives appear at the edges rather than in the core logic. map(str.strip, raw_lines) and map(int, ids) convert input cheaply and lazily. executor.map(process_file, paths) fans work out to a thread or process pool and pool.imap_unordered streams results back in completion order. reduce is reserved for genuinely associative folds: merging a list of dictionaries with reduce(operator.or_, dicts, {}) or composing a list of transformation functions into one callable. Lambdas stay limited to key= arguments; anything with branching gets a named function, and operator.itemgetter("sku") replaces lambda r: r["sku"].
Test. Named functions passed to map and filter are unit-tested directly with table-driven cases, which is the main reason to prefer them over inline lambdas. Pipeline functions returning iterators are tested with list(...) on small inputs, plus a test with an empty input to confirm reduce has a correct initial value. For executor.map paths a test verifies that results preserve input order (map) or that all results arrive (imap_unordered), and that an exception in one item surfaces when its result is consumed, not silently.
Troubleshoot. The most common production bug is consuming a map or filter object twice: the second pass is empty, so a report shows zero rows after the first validation step iterated it. Converting to a list at the point where reuse is needed fixes it. TypeError: reduce() of empty iterable with no initial value appears on the first empty batch in production after months of non-empty test data. Lambdas in a multiprocessing pool raise PicklingError; a module-level function or functools.partial resolves it. A lambda defined in a loop capturing the loop variable produces every callback seeing the last value, fixed with a default argument.
The important consideration is that map, filter and reduce are tools for applying existing functions, not an excuse to write logic inside lambdas that no test can target.
open(path, mode, encoding=...) returns a file object, and the mode decides whether it is a text file that yields str or a binary file that yields bytes. Text mode decodes on read and encodes on write using the encoding argument; binary mode ("rb", "wb") hands over raw bytes untouched. The with statement closes the file deterministically, including on exceptions.
from pathlib import Path
log = Path("app.log")
with log.open("a", encoding="utf-8") as f: # text, append
f.write("started\n")
for line in log.open(encoding="utf-8"): # streams line by line
print(line.rstrip("\n"))
raw: bytes = Path("image.png").read_bytes() # binary helper
text = Path("legacy.csv").read_text(encoding="cp1252", errors="replace")Encoding rules that matter:
encoding explicitly for text files. The default is locale.getencoding(), which is UTF-8 on most Linux and macOS systems but a legacy code page on many Windows machines, so the same code reads a file correctly on one platform and raises UnicodeDecodeError on another. Running with -X warn_default_encoding (3.10+) emits an EncodingWarning wherever the default is relied on, and UTF-8 mode (-X utf8 or PYTHONUTF8=1) forces UTF-8 everywhere.errors= controls undecodable bytes: "strict" (default) raises, "replace" inserts U+FFFD, "ignore" drops them, and "surrogateescape" round-trips them, which is what the OS file-system APIs use."utf-8-sig" consumes a byte-order mark on read and writes one, which Excel-generated CSVs often need.newline="" must be passed when writing CSV with the csv module, otherwise Windows gets doubled line endings.pathlib.Path is the modern API for paths and offers read_text, write_text, read_bytes, iterdir and glob. For large files, iterate line by line or read fixed-size chunks rather than calling read(), and use shutil.copyfileobj for copying streams.
Apply. A bank-statement import service receives CSV and fixed-width files from dozens of institutions, each with its own encoding. The ingest layer never opens files in text mode blindly: it reads the first few kilobytes as bytes, checks for a BOM, applies the encoding declared in the institution’s profile, and falls back to charset-normalizer detection only as a last resort, logging what it chose. Files are streamed line by line, and output is written to a temporary file and renamed with os.replace so a crash never leaves a half-written result. All internal text is UTF-8, and every open() call in the codebase passes encoding= explicitly, enforced by a lint rule.
Test. Fixtures include byte-exact sample files: UTF-8 with and without BOM, cp1252 with accented names, UTF-16 exports from a mainframe, a file with Windows line endings, and one with an invalid byte in the middle. Tests use pytest’s tmp_path fixture for writes and assert the exact bytes produced, not just the decoded string, because round-trip bugs hide in the encoder. The suite runs with -X warn_default_encoding promoted to an error so an implicit default encoding fails CI, and a Windows CI runner catches locale-dependent assumptions that Linux developers never see.
Troubleshoot. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 is the classic: a Latin-1 file opened as UTF-8; the fix is the correct encoding, never errors="ignore" in the import path, since that silently destroys customer names. A CSV with blank lines between rows on Windows is the missing newline="". “Too many open files” (OSError: [Errno 24]) is a file opened without with in a loop; -X dev turns the unclosed files into ResourceWarning with the allocation site. Partial output after a crash means buffered writes; os.fsync before the rename makes them durable.
The important consideration is that bytes have no encoding; something must declare it, and guessing is a last resort that must be logged.
pytest is the de facto testing framework for Python. Tests are plain functions named test_* that use the assert statement; pytest rewrites those asserts so a failure prints the values involved. Its distinctive features are fixtures, which provide reusable setup and teardown through dependency injection, and parametrisation, which runs one test body over many inputs. Plugins such as pytest-cov, pytest-asyncio and pytest-xdist extend it.
Mocking replaces a real dependency with a controllable stand-in so a test can run without a network, database or clock, and can assert how the dependency was called. The standard tool is unittest.mock: Mock and MagicMock record calls, patch swaps an attribute for the duration of a test, and autospec=True makes the mock reject calls that the real object would not accept. pytest’s monkeypatch fixture does the same swap with automatic undo, and pytest-mock exposes patch as the mocker fixture.
#--- payments.py ---
import requests
def capture(order_id, amount):
r = requests.post(GATEWAY_URL, json={"id": order_id, "amount": amount})
r.raise_for_status()
return r.json()["reference"]
#--- test_payments.py ---
import pytest
from unittest.mock import patch
import payments
@pytest.mark.parametrize("amount", [1, 999_99])
def test_capture_posts_amount(amount):
with patch("payments.requests.post", autospec=True) as post:
post.return_value.json.return_value = {"reference": "ref-1"}
assert payments.capture("o-1", amount) == "ref-1"
post.assert_called_once()
assert post.call_args.kwargs["json"]["amount"] == amountThe patch target is where the name is looked up (payments.requests.post), not where it is defined, because patch replaces an attribute on a module object. Mocks are best kept at the system boundary: HTTP clients, clocks, message brokers; mocking your own classes couples tests to implementation and makes refactors painful.
Apply. The test suite of an order service is layered. Fast unit tests mock only the boundaries, through a fake_gateway fixture built with Mock(spec=PaymentGateway) and a frozen_clock fixture that patches the time source, so domain logic runs in milliseconds. Integration tests use a real PostgreSQL started by a session-scoped fixture (testcontainers), with each test inside a transaction that is rolled back. conftest.py holds the shared fixtures, markers (@pytest.mark.slow) separate the tiers, and pytest -n auto from pytest-xdist runs them in parallel in CI. HTTP calls are stubbed with responses or respx rather than hand-built mocks so request matching and serialisation are realistic.
Test. Tests of the tests matter too: pytest-cov reports branch coverage with a floor the build must meet, and mutation testing (mutmut) on the pricing module occasionally confirms that assertions actually bite. For mock-heavy tests I use autospec=True so that a renamed method on the real class fails the test instead of letting a stale mock pass. Fixture scoping is reviewed deliberately: a module-scoped fixture that one test mutates is a flaky-test factory.
Troubleshoot. A mock that “does not take effect” is patched at the wrong place, typically patch("requests.post") when the module under test did from requests import post; patching mymodule.post fixes it. AttributeError: Mock object has no attribute with autospec means the production code is calling something the real object lacks, which is a real bug found. Tests that pass alone and fail in the suite share state through a module-scoped fixture or a module-level cache; pytest -p no:randomly versus pytest-randomly ordering, and --lf / -x to bisect, locate it. Flaky asyncio tests usually mix event loops across fixtures; a single asyncio_mode = "auto" setting and function-scoped loops stabilise them. pytest --durations=10 finds the slow fixtures that make developers skip running tests locally.
The important consideration is that every mock encodes an assumption about the real dependency; a contract test against the real service catches the day that assumption stops being true.
Profiling means measuring where a program actually spends time or memory before changing it, because intuition about Python hot spots is usually wrong. The standard toolkit has three layers, and the workflow below moves from measuring to fixing.

timeit and time.perf_counter() measure a snippet or a block; pytest-benchmark turns that into repeatable tests.cProfile is a deterministic function-level profiler built in: python -m cProfile -o out.prof app.py, then pstats or snakeviz to sort by cumulative time. It adds overhead and attributes time to functions, not lines. line_profiler gives per-line detail for a chosen function.py-spy is a sampling profiler that attaches to a running process without code changes and produces flame graphs, which makes it the tool for production. Python 3.12 added -X perf so Linux perf can see Python function names, and scalene combines CPU, memory and native-versus-Python attribution.tracemalloc tracks allocations by source line and compares snapshots; memray adds native allocations and flame graphs.Typical fixes, in order of pay-off: a better algorithm or data structure (O(n²) list membership to a set), moving work out of a loop, caching with functools.cache, batching I/O and database queries, pushing numeric work into NumPy or another C-backed library, and only then concurrency or a faster interpreter. Upgrading to 3.11+ alone typically gives 10–60% because of the specialising interpreter, and 3.13’s experimental JIT is opt-in and not yet a general speed-up.
Apply. Performance work on a reporting API starts with observability, not profiling: request latency histograms and database query timings per endpoint show which 5% of requests cost 60% of the time. For a suspect endpoint I reproduce it locally with realistic data volume and run cProfile through snakeviz, or attach py-spy record --pid to a staging worker under load to get a flame graph without redeploying. Fixes are made one at a time, with a benchmark before and after. Typical wins: an N+1 query replaced with a join, a per-row datetime.strptime replaced with one parse per distinct value, json.loads on a 20 MB payload moved to orjson, and a dict keyed lookup replacing a nested loop.
Test. Performance regressions are caught by pytest-benchmark tests on the pure-Python hot paths, comparing against a stored baseline with a tolerance so noise does not fail the build, and by a nightly load test (locust or k6) against staging with p95 latency and error-rate thresholds.
Troubleshoot. A latency jump after a deploy is bisected by diffing the dependency lock as well as the code; a minor upgrade once swapped a C accelerator for a pure-Python fallback, visible in py-spy as time in the library’s Python frames. CPU at 100% on one core while others idle is the GIL with a thread pool on CPU-bound work. High latency with low CPU points at blocking I/O or lock contention, and py-spy dump shows the waiting stacks. Growing memory is approached with tracemalloc diffs; a flat profile with no hot spot usually means data volume, so the fix is pagination or a different query. cProfile inflates the cost of many tiny calls, so a sampling profiler confirms the ranking before anyone rewrites code.
The important consideration is that optimising without a measurement is guesswork; a profile, a baseline and a re-measurement are the minimum evidence for any performance change.
Add your resume and the job description in CrackItX's Settings. When the interviewer asks a Python question, Claude suggests an answer using your real projects and the role's requirements, in real time.
See pricing →