python - ¿Por qué se crean archivos.pyc en la importación?

bytecode python-internals

Question

He visto varios recursos que describen qué son los archivos .pyc y cuándo se crearon. Pero ahora me pregunto por qué se crean cuando se importan archivos .py .

Además, ¿por qué no crear un archivo .pyc para el archivo principal de Python que realiza la importación?

Supongo que tiene que ver con la optimización del rendimiento y aprender esto me ha animado a separar mis archivos, ya que la compilación incorporada parece agradable de aprovechar. Pero no estoy seguro de si este es el caso, y también tengo curiosidad por saber si alguien tiene estadísticas sobre la diferencia entre ejecutar programas con y sin los archivos .pyc , si es realmente por velocidad.

Los ejecuté yo mismo, pero no tengo un buen código base de Python para probarlo. :(

Answer 1

El código fuente de Python se compila a bytecode, y es el bytecode que se ejecuta. Un archivo .pyc contiene una copia de ese código de bytes, y al almacenar en caché Python no tiene que volver a compilar el código de Python cada vez que necesita cargar el módulo.

Puede tener una idea de cuánto tiempo se ahorra al sincronizar la función compile() :

>>> import urllib2
>>> import timeit
>>> urllib2_source = open(urllib2.__file__.rstrip(''c'')).read()
>>> timeit.timeit("compile(source, '''', ''exec'')", ''from __main__ import urllib2_source as source'', number=1000)
6.977046966552734
>>> _ / 1000.0
0.006977046966552734

Por lo tanto, se necesitan 7 milisegundos para compilar el código fuente de urllib2.py . Eso no suena como mucho, pero se agrega rápidamente a medida que Python carga muchos módulos en su vida útil. Simplemente ejecute un script promedio con el -v de línea de comandos -v ; Aquí ejecuto la salida de ayuda para la herramienta pydoc :

$ bin/python -v -m pydoc -h
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/site.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/site.py
import site # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/site.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/os.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/os.py
import os # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/posixpath.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/posixpath.py
import posixpath # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/posixpath.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/stat.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/stat.py
import stat # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/stat.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/genericpath.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/genericpath.py
import genericpath # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/genericpath.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/warnings.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/warnings.py
import warnings # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/warnings.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/linecache.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/linecache.py
import linecache # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/linecache.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/types.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/types.py
import types # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/types.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/UserDict.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/UserDict.py
import UserDict # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/UserDict.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_abcoll.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_abcoll.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/abc.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/abc.py
import abc # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/abc.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_weakrefset.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/copy_reg.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/copy_reg.pyc
import encodings # directory /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/__init__.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/__init__.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/codecs.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/codecs.py
import codecs # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/codecs.pyc
import _codecs # builtin
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/aliases.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/aliases.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/utf_8.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/encodings/utf_8.pyc
Python 2.7.8 (default, Sep  9 2014, 11:33:29) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.py
import runpy # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/runpy.pyc
import imp # builtin
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.py
import pkgutil # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/pkgutil.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/re.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/re.py
import re # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/re.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_compile.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_parse.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_parse.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_constants.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/sre_constants.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_locale.so", 2);
import _locale # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_locale.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.py
import inspect # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/inspect.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.py
import string # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/string.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/strop.so", 2);
import strop # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/strop.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.py
import dis # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/dis.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.py
import opcode # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/opcode.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.py
import tokenize # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/tokenize.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/itertools.so", 2);
import itertools # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/itertools.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.py
import token # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/token.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/operator.so", 2);
import operator # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/operator.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.py
import collections # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/collections.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_collections.so", 2);
import _collections # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_collections.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.py
import keyword # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/keyword.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.py
import heapq # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/heapq.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_heapq.so", 2);
import _heapq # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_heapq.so
import thread # builtin
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.py
import repr # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/repr.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.py
import traceback # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/traceback.pyc
# /Users/mpieters/Development/venvs/-2.7/lib/python2.7/locale.pyc matches /Users/mpieters/Development/venvs/-2.7/lib/python2.7/locale.py
import locale # precompiled from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/locale.pyc
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.py
import functools # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/functools.pyc
dlopen("/Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_functools.so", 2);
import _functools # dynamically loaded from /Users/mpieters/Development/venvs/-2.7/lib/python2.7/lib-dynload/_functools.so
# /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.pyc matches /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.py
import getopt # precompiled from /Users/mpietre/Development/Library/buildout.python/parts/opt/lib/python2.7/getopt.pyc
pydoc - the Python documentation tool

pydoc.py <name> ...
    Show text documentation on something.  <name> may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If <name> contains a ''/'', it is used as the path to a
    Python source file to document. If name is ''keywords'', ''topics'',
    or ''modules'', a listing of these things is displayed.

pydoc.py -k <keyword>
    Search for a keyword in the synopsis lines of all available modules.

pydoc.py -p <port>
    Start an HTTP server on the given port on the local machine.

pydoc.py -g
    Pop up a graphical interface for finding and serving documentation.

pydoc.py -w <name> ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If <name> contains a ''/'', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] _collections
# cleanup[1] locale
# cleanup[1] functools
# cleanup[1] encodings
# cleanup[1] site
# cleanup[1] runpy
# cleanup[1] operator
# cleanup[1] supervisor
# cleanup[1] _heapq
# cleanup[1] abc
# cleanup[1] _weakrefset
# cleanup[1] sre_constants
# cleanup[1] collections
# cleanup[1] _codecs
# cleanup[1] opcode
# cleanup[1] _warnings
# cleanup[1] mpl_toolkits
# cleanup[1] inspect
# cleanup[1] encodings.utf_8
# cleanup[1] repr
# cleanup[1] codecs
# cleanup[1] getopt
# cleanup[1] pkgutil
# cleanup[1] _functools
# cleanup[1] thread
# cleanup[1] keyword
# cleanup[1] strop
# cleanup[1] signal
# cleanup[1] traceback
# cleanup[1] itertools
# cleanup[1] posix
# cleanup[1] encodings.aliases
# cleanup[1] exceptions
# cleanup[1] _weakref
# cleanup[1] token
# cleanup[1] dis
# cleanup[1] tokenize
# cleanup[1] heapq
# cleanup[1] string
# cleanup[1] imp
# cleanup[1] zipimport
# cleanup[1] re
# cleanup[1] _locale
# cleanup[1] sre_compile
# cleanup[1] _sre
# cleanup[1] sre_parse
# cleanup[2] copy_reg
# cleanup[2] posixpath
# cleanup[2] errno
# cleanup[2] _abcoll
# cleanup[2] types
# cleanup[2] genericpath
# cleanup[2] stat
# cleanup[2] warnings
# cleanup[2] UserDict
# cleanup[2] os.path
# cleanup[2] linecache
# cleanup[2] os
# cleanup sys
# cleanup __builtin__
# cleanup ints: 21 unfreed ints
# cleanup floats

Eso es 53 importaciones:

$ bin/python -v -m pydoc -h 2>&1 | egrep ^import | wc -l
      53

En lugar de cargar el archivo fuente (más grande) cada vez y compilarlo, se puede leer y usar un archivo de código de bytes más pequeño de inmediato. Eso agrega fácilmente hasta 1 / 3rd o 1/2 por segundo solo para imprimir información de ayuda para una herramienta de línea de comandos.

Python no crea un archivo de caché para el script principal; eso se debe a que abarrotaría el directorio de scripts con archivos que no se cargarán con tanta frecuencia como se cargan los módulos.

Si ejecuta un script que a menudo le afecta el tiempo de compilación de ese archivo, siempre puede mover la mayoría del código a un módulo (y evitar tener que compilar un script grande) o puede usar la herramienta compileall para crear un archivo de caché .pyc para el script, luego ejecute ese archivo .pyc directamente . ¡Tenga en cuenta que Python no volverá a compilar ese archivo si cambió la secuencia de comandos!