New submission from Germán Méndez Bravo <german...@gmail.com>: As the size of a Python project increases, the number of modules and the complexity of its dependencies increases too, producing two problems in large codebases: increased risk of import cycles and slow start times due to the number of modules that tend to need getting loaded.
We propose implementing a robust and transparent lazy loader in CPython, based on an existing implementation we have in Cinder that proved to be extremely valuable to thousands of developers across Meta, and battle-tested in Instagram Server in production (https://docs.google.com/document/d/1l8I-FDE1xrIShm9eSNJqsGmY_VanMDX5-aK_gujhYBI/edit#heading=h.pu7ja6wu0ib). Our internal implementation is based on introducing deferred objects and converting top-level imported names to these deferred objects that are evaluated on first use. Enabling this across multiple types of workloads in Meta consistently shown improvements in startup times (up to 70%) and memory footprint (up to 40%), while virtually eliminating occurrences of import cycles. Converting imported names to deferred objects is a semantic change, and the benefits are usually meaningful only at very large scale, so we propose not changing the default behavior, and making it possible for users to opt-in to this lazy loading mechanism via -x flags or environment variables. This change would require related documentation changes that would cover the gotchas and edge cases, such as: some packages rely and import-time side-effects for correct operation; nested modules can’t be accessed unless imported explicitly; deferred loading may also defer exceptions from import time and first-access time which could be confusing. The import-time side effects gotcha can be mitigated using eager-import lists and directives, though it could be desirable to use this as a forcing function to discourage package maintainers from relying on these side-effects. ---------- components: Interpreter Core messages: 414770 nosy: Kronuz, carljm, dino.viehland, itamaro priority: normal severity: normal status: open title: Deep Lazy Imports - Interpreter-level deferred module loading type: enhancement versions: Python 3.11 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue46963> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
