[python] m1 with python 3.7.9 issue("ModuleNotFoundError: No module named '_ctypes'")

0. 상황

    - M1맥북에서 pyenv로 Python 3.7.9를 다운로드 받은 후 새로 생성한 가상환경에서, "coreschema==0.0.4"를 포함한 requirements.txt를 pip install 시도하던 중 마주한 문제.

1. 에러코드 전문

    - 상황 발생 전 입력한 쉘

>> pyenv install 3.7.9                            //3.7.9 설치(by pyenv)
>> pyenv versions                                 //3.7.9 설치 확인
>> pyenv virtualenv 3.7.9 yammi-api-django-virt   //3.7.9버전으로 가상환경 생성
>> pyenv versions                                 //가상환경 생성 확인
>> pyenv local yammi-api-django-virt              //위치한 디렉토리에 가상환경 설정
>> pip install -r requirements.txt                //requirements.txt를 통한 패키지 다운

 

    - 마주한 에러코드

Collecting coreschema==0.0.4 (from -r requirements.txt (line 14))
  Using cached coreschema-0.0.4.tar.gz (10 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [61 lines of output]
      running dist_info
      creating /private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info
      writing /private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/PKG-INFO
      writing dependency_links to /private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/dependency_links.txt
      writing requirements to /private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/requires.txt
      writing top-level names to /private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/top_level.txt
      writing manifest file '/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/SOURCES.txt'
      reading manifest file '/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/SOURCES.txt'
      writing manifest file '/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema.egg-info/SOURCES.txt'
      creating '/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-modern-metadata-1yngb3k3/coreschema-0.0.4.dist-info'
      Traceback (most recent call last):
        File "/Users/yanako/.pyenv/versions/3.7.9/envs/yammi-api-django-virt/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/yanako/.pyenv/versions/3.7.9/envs/yammi-api-django-virt/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/yanako/.pyenv/versions/3.7.9/envs/yammi-api-django-virt/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 75, in <module>
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/command/dist_info.py", line 104, in run
          bdist_wheel = self.get_finalized_command('bdist_wheel')
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
          cmd_obj = self.distribution.get_command_obj(command, create)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
          klass = self.get_command_class(command)
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 979, in get_command_class
          self.cmdclass[command] = cmdclass = ep.load()
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/overlay/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 208, in load
          module = import_module(match.group('module'))
        File "/Users/yanako/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py", line 127, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
        File "<frozen importlib._bootstrap>", line 983, in _find_and_load
        File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 728, in exec_module
        File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/normal/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 27, in <module>
          from .macosx_libfile import calculate_macosx_platform_tag
        File "/private/var/folders/nq/cjn7n4ls1n94nqsn4ysr51zm0000gn/T/pip-build-env-kiomphtr/normal/lib/python3.7/site-packages/wheel/macosx_libfile.py", line 43, in <module>
          import ctypes
        File "/Users/yanako/.pyenv/versions/3.7.9/lib/python3.7/ctypes/__init__.py", line 7, in <module>
          from _ctypes import Union, Structure, Array
      ModuleNotFoundError: No module named '_ctypes'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

---

2. 원인

    - Python@3.7.9와 리눅스 특정 버전에서 발생하는 이슈인듯 하다.

---

3. 해결방안

0.결론적으로는 1,2번 방법을 시도했다가 c언어 연관 library에서 연쇄적인 문제가 발생해 다른 방법을 모색했고, miner 버전을 하나씩 올려보다가 python 3.7의 miner 버전 중 12버전부터 _ctypes 관련 patching이 들어간다는 것을 확인했다. 따라서 major 버전의 변경이 아니기 때문에 다른 라이브러리에 영향을 주지 않을 것으로 판단해 3.7.12까지 python의 버전을 끌어올리기러 결정해 사용중이다.

 

1. mac에서 brew혹은 다른 툴을 사용해 python을 다운로드 받을 시 libff-dev 라이브러리를 건너뛴채로 다운로드 받아져 발생했을거라는 이슈를 발견해, 해당 라이브러리를 설치한 뒤 python@3.7.9를 다시 다운로드 시도했다.

>> pyenv uninstall 3.7.9
>> brew install libffi libffi-dev
>> pyenv install 3.7.9
>> pyenv virtualenv 3.7.9 yammi-api-django-virt
>> pyenv local yammi-api-django-virt

 

2. "Python 3.7 needs libffi headers to build on Linux" 글참조하여, libffi와 python 3.7.9설치시 플래그를 설정하고, 설치된 python 버전 함수의 특정 구문(CFUNCTYPE(c_int)(lambda: None))을 주석처리 했다.

def _reset_cache():
    _pointer_type_cache.clear()
    _c_functype_cache.clear()
    if _os.name == "nt":
        _win_functype_cache.clear()
    # _SimpleCData.c_wchar_p_from_param
    POINTER(c_wchar).from_param = c_wchar_p.from_param
    # _SimpleCData.c_char_p_from_param
    POINTER(c_char).from_param = c_char_p.from_param
    _pointer_type_cache[None] = c_void_p
    # XXX for whatever reasons, creating the first instance of a callback
    # function is needed for the unittests on Win64 to succeed.  This MAY
    # be a compiler bug, since the problem occurs only when _ctypes is
    # compiled with the MS SDK compiler.  Or an uninitialized variable?
    # CFUNCTYPE(c_int)(lambda: None)

 

 

---

4. 결과

[3.7.12버전으로 miner 버전업 한 후]

    - rosetta로 설정한 intel 기반 터미널에서는 python 3.7버전이 문제없이 다운로드 받아지며, c언어와 관련된 파이썬 라이브러리들도 가상환경에 문제없이 pip install 할 수 있었다.

 

[3.7.12버전으로 miner 버전업 하기 이전, 1,2번 해결방법]

    - ctypes 관련 에러가 나던것이 사라졌다.

    - 하지만 python 자체 init 파일을 손본 만큼, 추후 영향이 있을지 없을지는 확인이 추가로 필요하다.