View Issue Details

IDProjectCategoryView StatusLast Update
0000285fileGeneralpublic2021-09-09 17:49
ReporterBenjamin Assigned Tochristos  
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSUbuntuOS VersionUbuntu 18.04.5
Fixed in Version5.41 
Summary0000285: Python biding of libmagic crash when trying to detect encoding in multithread
DescriptionHello,

When i use the following code in a multithreaded environment (some django workers using Huey or Dramatiq)

magic.detect_from_content(
            bytes_file.read(MAX_BYTES)
        ).encoding

I got some:

  File "/home/benjamin/.cache/pypoetry/virtualenvs/qvNem8AN-py3.9/lib/python3.9/site-packages/magic.py", line 284, in detect_from_content
    return _create_filemagic(mime_magic.buffer(byte_content),
  File "/home/benjamin/.cache/pypoetry/virtualenvs/qvNem8AN-py3.9/lib/python3.9/site-packages/magic.py", line 251, in _create_filemagic
    mime_type, mime_encoding = mime_detected.split('; ')
ValueError: too many values to unpack (expected 2)

With the reproductible code given in this ticket errors can be differents ("munmap_chunk(): invalid pointer", "free(): invalid size", "double free or corruption (fasttop)")


It works correctly on a single thread.

Steps To Reproduce

import threading

import magic

MAX_BYTES = 4096


def thread_function():
    with open("/tmp/foo.csv", "rb") as bytes_file:
        print(magic.detect_from_content(
            bytes_file.read(MAX_BYTES)
        ).encoding)


if __name__ == '__main__':
    threads = list()
    for index in range(3):
        thread = threading.Thread(target=thread_function)
        threads.append(thread)
        thread.start()

    for index, thread in enumerate(threads):
        thread.join()
Additional Information
My ubuntu libmagic1 version is: 1:5.32-2ubuntu0.4
My file-magic python packaque is 0.4.0
Tagsmagic

Activities

christos

2021-09-09 17:49

manager   ~0003641

Thanks, fixed in HEAD. Will release as 0.4.1 when I release the next version of file(1).

Issue History

Date Modified Username Field Change
2021-08-26 09:25 Benjamin New Issue
2021-08-26 09:25 Benjamin Tag Attached: magic
2021-09-09 17:49 christos Assigned To => christos
2021-09-09 17:49 christos Status new => assigned
2021-09-09 17:49 christos Status assigned => resolved
2021-09-09 17:49 christos Resolution open => fixed
2021-09-09 17:49 christos Fixed in Version => 5.41
2021-09-09 17:49 christos Note Added: 0003641