View Issue Details

IDProjectCategoryView StatusLast Update
0000378fileGeneralpublic2022-09-03 16:25
Reporterbandieramonte Assigned Tochristos  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionfixed 
OSWin 11 64 BitOS Version10.0 Build 22000 
Summary0000378: errorcheck_negative_one throwing error on magic load
DescriptionHi,
I hope this is the right place to report this issue, otherwise, kindly point me to the right place.
I'm trying to use libmagic with Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]. It is failing to load and I have tried everything I could find in Google to solve this. The trace of the error is shown in the "Additional Information" box. By using 'pip install', I have installed:
python-magic-bin 0.4.14
python-magic 0.4.27

I have also the following DLLs within PATH reach (in the System32 folder):

libgnurx-0.dll
magic.mgc
magic1.dll
regex2.dll
zlib1.dll
magic.py

The aforementioned magic files are of the 64 bit version, since I'm using a 64 bit version of Python. These files have been attached herein in case it helps. I tried to reinstall python-magic and python-magic-bin, up to no avail.
The funny thing is that this was working before, and somehow it started to show this error. I can't recall any change I did in the system that affected this.
Any help would be appreciated, since I'm pretty stuck here.
Thank you.
Additional InformationTrace of error:

ERROR saleor.product.product_images Thumbnail fetch failed [PID:12096:Thread-8 (process_request_thread)]
Traceback (most recent call last):
  File "C:\Users\gpb\saleorTest\saleor\product\product_images.py", line 101, in get_thumbnail
    thumbnail = getattr(image_file, method)[used_size]
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\versatileimagefield\datastructures\sizedimage.py", line 141, in __getitem__
    self.create_resized_image(
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\versatileimagefield\datastructures\sizedimage.py", line 196, in create_resized_image
    image, file_ext, image_format, mime_type = self.retrieve_image(
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\versatileimagefield\datastructures\base.py", line 162, in retrieve_image
    image_format, mime_type = get_image_metadata_from_file(image)
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\versatileimagefield\utils.py", line 152, in get_image_metadata_from_file
    mime_type = magic.from_buffer(file_like.read(1024), mime=True)
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\magic\__init__.py", line 191, in from_buffer
    m = _get_magic_type(mime)
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\magic\__init__.py", line 165, in _get_magic_type
    i = _instances[mime] = Magic(mime=mime)
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\magic\__init__.py", line 73, in __init__
    magic_load(self.cookie, magic_file)
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\magic\__init__.py", line 332, in magic_load
    return _magic_load(cookie, coerce_filename(filename))
  File "c:\Users\gpb\saleorTest\saleor-venv\lib\site-packages\magic\__init__.py", line 225, in errorcheck_negative_one
    raise MagicException(err)
magic.MagicException: None
TagsNo tags attached.

Activities

bandieramonte

2022-08-30 10:35

reporter  

magic1.dll (717,504 bytes)
regex2.dll (79,360 bytes)
zlib1.dll (88,888 bytes)
libgnurx-0.dll (67,720 bytes)
magic.py (9,840 bytes)   
"""
magic is a wrapper around the libmagic file identification library.

See README for more information.

Usage:

>>> import magic
>>> magic.from_file("testdata/test.pdf")
'PDF document, version 1.2'
>>> magic.from_file("testdata/test.pdf", mime=True)
'application/pdf'
>>> magic.from_buffer(open("testdata/test.pdf").read(1024))
'PDF document, version 1.2'
>>>


"""

import sys
import glob
import os.path
import ctypes
import ctypes.util
import threading

from ctypes import c_char_p, c_int, c_size_t, c_void_p

default_magic_file=None

class MagicException(Exception):
    def __init__(self, message):
        super(MagicException, self).__init__(message)
        self.message = message


class Magic:
    """
    Magic is a wrapper around the libmagic C library.

    """

    def __init__(self, mime=False, magic_file=None, mime_encoding=False,
                 keep_going=False, uncompress=False):
        """
        Create a new libmagic wrapper.

        mime - if True, mimetypes are returned instead of textual descriptions
        mime_encoding - if True, codec is returned
        magic_file - use a mime database other than the system default
        keep_going - don't stop at the first match, keep going
        uncompress - Try to look inside compressed files.
        """
        self.flags = MAGIC_NONE
        if mime:
            self.flags |= MAGIC_MIME
        if mime_encoding:
            self.flags |= MAGIC_MIME_ENCODING
        if keep_going:
            self.flags |= MAGIC_CONTINUE

        if uncompress:
            self.flags |= MAGIC_COMPRESS

        if magic_file is None:
            magic_file = default_magic_file

        self.cookie = magic_open(self.flags)
        self.lock = threading.Lock()
        
        magic_load(self.cookie, magic_file)

    def from_buffer(self, buf):
        """
        Identify the contents of `buf`
        """
        with self.lock:
            try:
                return maybe_decode(magic_buffer(self.cookie, buf))
            except MagicException as e:
                return self._handle509Bug(e)

    def from_file(self, filename):
        # raise FileNotFoundException or IOError if the file does not exist
        with open(filename):
            pass
        with self.lock:
            try:
                return maybe_decode(magic_file(self.cookie, filename))
            except MagicException as e:
                return self._handle509Bug(e)

    def _handle509Bug(self, e):
        # libmagic 5.09 has a bug where it might fail to identify the
        # mimetype of a file and returns null from magic_file (and
        # likely _buffer), but also does not return an error message.
        if e.message is None and (self.flags & MAGIC_MIME):
            return "application/octet-stream"
        else:
            raise e
        
    def __del__(self):
        # no _thread_check here because there can be no other
        # references to this object at this point.

        # during shutdown magic_close may have been cleared already so
        # make sure it exists before using it.

        # the self.cookie check should be unnecessary and was an
        # incorrect fix for a threading problem, however I'm leaving
        # it in because it's harmless and I'm slightly afraid to
        # remove it.
        if self.cookie and magic_close:
            magic_close(self.cookie)
            self.cookie = None

_instances = {}

def _get_magic_type(mime):
    i = _instances.get(mime)
    if i is None:
        i = _instances[mime] = Magic(mime=mime)
    return i

def from_file(filename, mime=False):
    """"
    Accepts a filename and returns the detected filetype.  Return
    value is the mimetype if mime=True, otherwise a human readable
    name.

    >>> magic.from_file("testdata/test.pdf", mime=True)
    'application/pdf'
    """
    m = _get_magic_type(mime)
    return m.from_file(filename)

def from_buffer(buffer, mime=False):
    """
    Accepts a binary string and returns the detected filetype.  Return
    value is the mimetype if mime=True, otherwise a human readable
    name.

    >>> magic.from_buffer(open("testdata/test.pdf").read(1024))
    'PDF document, version 1.2'
    """
    m = _get_magic_type(mime)
    return m.from_buffer(buffer)




libmagic = None
# Let's try to find magic or magic1
dll = ctypes.util.find_library('magic') or ctypes.util.find_library('magic1') or ctypes.util.find_library('cygmagic-1')

bin_dist_path = os.path.join(os.path.dirname(__file__), 'libmagic')
if os.path.isdir(bin_dist_path):
    if sys.platform == 'darwin':
        dll = os.path.abspath(os.path.join(bin_dist_path, 'libmagic.dylib'))
    elif sys.platform == 'win32':
        dll = os.path.abspath(os.path.join(bin_dist_path, 'libmagic.dll'))
    default_magic_file = os.path.join(bin_dist_path, 'magic.mgc')


# This is necessary because find_library returns None if it doesn't find the library
if dll:
    libmagic = ctypes.CDLL(dll)

if not libmagic or not libmagic._name:
    windows_dlls = ['magic1.dll','cygmagic-1.dll']
    platform_to_lib = {'darwin': ['/opt/local/lib/libmagic.dylib',
                                  '/usr/local/lib/libmagic.dylib'] +
                         # Assumes there will only be one version installed
                         glob.glob('/usr/local/Cellar/libmagic/*/lib/libmagic.dylib'),
                       'win32': windows_dlls,
                       'cygwin': windows_dlls,
                       'linux': ['libmagic.so.1'],    # fallback for some Linuxes (e.g. Alpine) where library search does not work
                      }
    platform = 'linux' if sys.platform.startswith('linux') else sys.platform
    for dll in platform_to_lib.get(platform, []):
        try:
            libmagic = ctypes.CDLL(dll)
            break
        except OSError:
            pass

if not libmagic or not libmagic._name:
    # It is better to raise an ImportError since we are importing magic module
    raise ImportError('failed to find libmagic.  Check your installation')

magic_t = ctypes.c_void_p

def errorcheck_null(result, func, args):
    if result is None:
        err = magic_error(args[0])
        raise MagicException(err)
    else:
        return result

def errorcheck_negative_one(result, func, args):
    if result is -1:
        err = magic_error(args[0])
        raise MagicException(err)
    else:
        return result


# return str on python3.  Don't want to unconditionally
# decode because that results in unicode on python2
def maybe_decode(s):
    if str == bytes:
        return s
    else:
        return s.decode('utf-8')
    
def coerce_filename(filename):
    if filename is None:
        return None

    # ctypes will implicitly convert unicode strings to bytes with
    # .encode('ascii').  If you use the filesystem encoding 
    # then you'll get inconsistent behavior (crashes) depending on the user's
    # LANG environment variable
    is_unicode = (sys.version_info[0] <= 2 and
                  isinstance(filename, unicode)) or \
                  (sys.version_info[0] >= 3 and
                   isinstance(filename, str))
    if is_unicode:
        return filename.encode('utf-8')
    else:
        return filename

magic_open = libmagic.magic_open
magic_open.restype = magic_t
magic_open.argtypes = [c_int]

magic_close = libmagic.magic_close
magic_close.restype = None
magic_close.argtypes = [magic_t]

magic_error = libmagic.magic_error
magic_error.restype = c_char_p
magic_error.argtypes = [magic_t]

magic_errno = libmagic.magic_errno
magic_errno.restype = c_int
magic_errno.argtypes = [magic_t]

_magic_file = libmagic.magic_file
_magic_file.restype = c_char_p
_magic_file.argtypes = [magic_t, c_char_p]
_magic_file.errcheck = errorcheck_null

def magic_file(cookie, filename):
    return _magic_file(cookie, coerce_filename(filename))

_magic_buffer = libmagic.magic_buffer
_magic_buffer.restype = c_char_p
_magic_buffer.argtypes = [magic_t, c_void_p, c_size_t]
_magic_buffer.errcheck = errorcheck_null

def magic_buffer(cookie, buf):
    return _magic_buffer(cookie, buf, len(buf))


_magic_load = libmagic.magic_load
_magic_load.restype = c_int
_magic_load.argtypes = [magic_t, c_char_p]
_magic_load.errcheck = errorcheck_negative_one

def magic_load(cookie, filename):
    return _magic_load(cookie, coerce_filename(filename))

magic_setflags = libmagic.magic_setflags
magic_setflags.restype = c_int
magic_setflags.argtypes = [magic_t, c_int]

magic_check = libmagic.magic_check
magic_check.restype = c_int
magic_check.argtypes = [magic_t, c_char_p]

magic_compile = libmagic.magic_compile
magic_compile.restype = c_int
magic_compile.argtypes = [magic_t, c_char_p]



MAGIC_NONE = 0x000000 # No flags
MAGIC_DEBUG = 0x000001 # Turn on debugging
MAGIC_SYMLINK = 0x000002 # Follow symlinks
MAGIC_COMPRESS = 0x000004 # Check inside compressed files
MAGIC_DEVICES = 0x000008 # Look at the contents of devices
MAGIC_MIME = 0x000010 # Return a mime string
MAGIC_MIME_ENCODING = 0x000400 # Return the MIME encoding
MAGIC_CONTINUE = 0x000020 # Return all matches
MAGIC_CHECK = 0x000040 # Print warnings to stderr
MAGIC_PRESERVE_ATIME = 0x000080 # Restore access time on exit
MAGIC_RAW = 0x000100 # Don't translate unprintable chars
MAGIC_ERROR = 0x000200 # Handle ENOENT etc as real errors

MAGIC_NO_CHECK_COMPRESS = 0x001000 # Don't check for compressed files
MAGIC_NO_CHECK_TAR = 0x002000 # Don't check for tar files
MAGIC_NO_CHECK_SOFT = 0x004000 # Don't check magic entries
MAGIC_NO_CHECK_APPTYPE = 0x008000 # Don't check application type
MAGIC_NO_CHECK_ELF = 0x010000 # Don't check for elf details
MAGIC_NO_CHECK_ASCII = 0x020000 # Don't check for ascii files
MAGIC_NO_CHECK_TROFF = 0x040000 # Don't check ascii/troff
MAGIC_NO_CHECK_FORTRAN = 0x080000 # Don't check ascii/fortran
MAGIC_NO_CHECK_TOKENS = 0x100000 # Don't check ascii/tokens
magic.py (9,840 bytes)   

bandieramonte

2022-08-31 17:09

reporter   ~0003809

I'm happy to report that upon downgrading python-magic to version 0.4.26, down from version 0.4.27, this issue was solved for me.
On the one hand, there is a solution to this. On the other hand, this seems to reveal an issue with version 0.4.27.
Maybe this information will be helpful.

christos

2022-09-01 10:00

manager   ~0003810

the package name we distribute is file_magic.

christos

2022-09-01 12:28

manager   ~0003812

We don't maintain this package; we maintain file-magic only.

bandieramonte

2022-09-01 19:24

reporter   ~0003814

I see, I guess we can close this issue then.
I will try to find the right place to report this.
Thank you!

christos

2022-09-03 16:25

manager   ~0003815

We don't maintain that python package

Issue History

Date Modified Username Field Change
2022-08-30 10:35 bandieramonte New Issue
2022-08-30 10:35 bandieramonte File Added: magic1.dll
2022-08-30 10:35 bandieramonte File Added: regex2.dll
2022-08-30 10:35 bandieramonte File Added: zlib1.dll
2022-08-30 10:35 bandieramonte File Added: libgnurx-0.dll
2022-08-30 10:35 bandieramonte File Added: magic.py
2022-08-31 17:09 bandieramonte Note Added: 0003809
2022-09-01 10:00 christos Note Added: 0003810
2022-09-01 12:27 christos Assigned To => christos
2022-09-01 12:27 christos Status new => assigned
2022-09-01 12:28 christos Status assigned => feedback
2022-09-01 12:28 christos Note Added: 0003812
2022-09-01 19:24 bandieramonte Note Added: 0003814
2022-09-01 19:24 bandieramonte Status feedback => assigned
2022-09-03 16:25 christos Status assigned => closed
2022-09-03 16:25 christos Resolution open => fixed
2022-09-03 16:25 christos Note Added: 0003815