Memory problems with MozJPEG and Pillow

2018-08-19T15:08:00-04:00

We implemented MozJPEG to be used with Pillow 4.x to create smaller thumbnails of files uploaded by users, when we noticed that sometimes this process did not work. We looked into our logs and noticed the following error message I/O suspension not supported in scan optimization. Time to enter the GSO workflow, GSO stands for Google Stack Overflow, in other words search the Internet. The error message results in links to the source code of MozJPEG, not very helpful at first.

Time to brush up on my C knowledge, OK I never programmed in C but that doesn’t stop me from going through the source.

The error message is defined in jerror.h

#endif
JMESSAGE(JERR_BAD_PARAM, "Bogus parameter")
JMESSAGE(JERR_BAD_PARAM_VALUE, "Bogus parameter value")
JMESSAGE(JERR_UNSUPPORTED_SUSPEND, "I/O suspension not supported in scan optimization")

#ifdef JMAKE_ENUM_LIST

So now we have to find the JERR_UNSUPPORTED_SUSPEND constant. Luckily it appears only in one file, jcmaster.c

while (size >= cinfo->dest->free_in_buffer)
  {
    MEMCOPY(cinfo->dest->next_output_byte, src, cinfo->dest->free_in_buffer);
    src += cinfo->dest->free_in_buffer;
    size -= cinfo->dest->free_in_buffer;
    cinfo->dest->next_output_byte += cinfo->dest->free_in_buffer;
    cinfo->dest->free_in_buffer = 0;

    if (!(*cinfo->dest->empty_output_buffer)(cinfo))
      ERREXIT(cinfo, JERR_UNSUPPORTED_SUSPEND);
  }

Cool, it seems to be related to memory cleanup, just my guess because of the empty_output_buffer line. Now we have to find out where Pillow sets the buffersize for saving an JPEG image.

The file PIL/JpegImagePlugin.py is used for all functions related to a JPEG image, and this includes saving.

The whole save method is a bit large to post here, but the part below determines the buffer size and it’s used to save the image. The buffer size is set to be holding the entire image file.

bufsize = 0
if optimize or progressive:
    # CMYK can be bigger
    if im.mode == 'CMYK':
        bufsize = 4 * im.size[0] * im.size[1]
    # keep sets quality to 0, but the actual value may be high.
    elif quality >= 95 or quality == 0:
        bufsize = 2 * im.size[0] * im.size[1]
    else:
        bufsize = im.size[0] * im.size[1]

# The exif info needs to be written as one block, + APP1, + one spare byte.
# Ensure that our buffer is big enough. Same with the icc_profile block.
bufsize = max(ImageFile.MAXBLOCK, bufsize, len(info.get("exif", b"")) + 5,
            len(extra) + 1)

ImageFile._save(im, fp, [("jpeg", (0, 0)+im.size, 0, rawmode)], bufsize)

I don’t want to change the Pillow source itself cause of potential issues whenever we upgrade Pillow in the future. So the best thing I can do is modify the ImageFile.MAXBLOCK, not that big of deal I think.

I came up with the following solution

new_maxblock = 3 * image.size[0] * image.size[1],  # ...3 bytes per every pixel in the image
old_maxblock = ImageFile.MAXBLOCK
if new_maxblock > ImageFile.MAXBLOCK:
    ImageFile.MAXBLOCK = new_maxblock
requested_size = (int(width), int(height))
image.thumbnail(requested_size, Image.ANTIALIAS)
image.save(thumb_file, "JPEG", progressive=True,)
ImageFile.MAXBLOCK = old_maxblock

We determine a new max block size, as most JPEG files are 24bits color (RGB), we need 3 bytes per pixel. This might be overkill in certain situations but at times I prefer overkill over not having having the thumbnail.

After implementing the above solution the I/O suspension not supported in scan optimization error message has not been seen in the logs.

Move from WordPress to Pelican

2018-03-11T22:15:24-04:00

I have been playing with the thought of moving my blog from WordPress to a static website generator for years.

I never really had the energy to start this projects, for several reasons. I know I could never find a theme that is completely to my liking so I would have to change it, possibly altering some code. There were some generator written in PHP, my language of choice for a long time, but they seemed not very mature. The best known static web site generator was Jekyll, written in Ruby and I really couldn’t get the energy to start learning Ruby on the side. The other reason that was keeping me from moving was how to move all my articles from WordPress to the static generator.

I started working for ONeil Interactive in 2017. We build websites for the home building industry and the language of choice is Python. We started looking into building an internal site for documenting our projects. We develop in Python and it was quickly decided the static generator should be written in Python as it would make it easier to extend the generator if needed.

As a result of this project I decided to bite the bullet for my personal site as well. After a quick look at the site StaticGen and some quick research I decided to go with Pelican. Of course I needed a theme, and there is never a theme that completely satisfies my needs but my starter theme is Plumage by Kevin Deldycke. I’ve modified it to work with Bootstrap 4, and used the Slate theme by Bootswatch.

I only moved two articles from my old blog as the rest of the articles were not visited that often. I still have them in the database so if it’s ever needed I can pull them up.

Oh in case you were wondering, for the project at work we decided to go with MkDocs

A Virtual Home - python

Memory problems with MozJPEG and Pillow

Move from WordPress to Pelican