Raspberry Pi Image Cruncher Part 3: Scripts and Automation
August 6, 2014 11:22 pm Leave your thoughts
Continuing on from part 2, here is the full script for the image resizer. Let’s call the script crunch2.py for now, so create a file called crunch2.py under /home/pi/image_cruncher. You can of course use any other directory but we will refer to this one in this article. We will go through the script bits by bits and hopefully I will be able to explain why things are coded as such.
The script itself is quite simple, it’s just a top-down procedural programming and not object oriented at all. However the script does the job crunching images in the Raspberry Pi. Of course, it’s also possible to run this script on another machines too.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 |
# Import os module so that we can traverse directories import os import time from wand.image import Image baseDirectory = "/home/pi/owncloud/Resize" targetDirectory = "resized" allowedExtensions = [".jpg", ".png"] # set the maximum image dimension to 4500 maximumDimension = 4500 #set maximum filesize to process maximumFilesize = 7340032 holdingImageFilename = "/home/pi/image_cruncher/too_large.gif" print("Running image cruncher 2") # some handy functions so that we don't repeat ourselves # creates the passed in directory if needed otherwise do nothing or return false if the directory existed # is a file def createDir(fullPath): if not os.path.exists(fullPath): #create dir os.makedirs(fullPath) elif os.path.isfile(fullPath): print ("Path is file... not good") return False return True def createHolderImage(fullTargetFilePath, fileExtension): print ("copying holder image") with Image(filename=holdingImageFilename) as holderImg: with holderImg.convert(fileExtension.lower().replace(".", "")) as holderImgConverted: holderImgConverted.save(filename=fullTargetFilePath) for currentDir, subDirList, fileList in os.walk(baseDirectory): # only process if the current directory is not the same with the base directory # also we only process if we are at the absolute sub directory of the current dir if currentDir != baseDirectory: for fileName in fileList: fileBaseName, fileExtension = os.path.splitext(fileName) # get the directory name without the full path directoryBaseName = os.path.basename(currentDir) # if the current directory is the processedDirectoryName (directory where we saved the files already resized) # we do not continue, obviously we don't want to reprocess them if directoryBaseName == targetDirectory: print("Directory base name is the same with resized directory name, continuing") continue # we will only allow certain extensions just to be careful if fileExtension.lower() in allowedExtensions: # load the image to wand object for further processing fullFilePath = currentDir + "/" + fileName #this is the full path of the converted file #note that we include the image directory name so that we keep things organised fullTargetDirectoryPath = currentDir + "/" + targetDirectory fullTargetFilePath = fullTargetDirectoryPath + "/" + fileBaseName + "-resized" + fileExtension.lower() print ("Target file destination" + fullTargetFilePath) # we then check if the target directory exist, if not we will create it # also if the target directory is actually a file, we will simply skip the rest of the directory by breaking if createDir(fullTargetDirectoryPath) == False: break # exit from the for each, skipping to the next directory #make sure the file does not exist yet in the resized path if it does we skip it if os.path.isfile(fullTargetFilePath): print ("File existed, continuing - " + fullTargetFilePath) continue print ("Cooling down before processing"); # sleep for 5 seconds time.sleep(5) print ("Processing: " + fullFilePath + "...") #check first if the file size is too big... if os.path.getsize(fullFilePath) > maximumFilesize: print ("File too large to process") createHolderImage(fullTargetFilePath, fileExtension) continue # now we continue processing our image try: with Image(filename=fullFilePath) as currentImg: if currentImg.width < maximumDimension and currentImg.height < maximumDimension: # we now resize the image to the maximum bounding square of 1280px currentImg.transform(resize='1280x1280>') print ("Resize completed, now saving to disk") # lastly write to the directory currentImg.save(filename=fullTargetFilePath) print ("Image save completed\n") else: # copy the holder image as the target file path # we would need the correct format of course createHolderImage(fullTargetFilePath, fileExtension) except: print ("Image file is no good, not doing anything") |
Cruncher Script Walkthrough
|
1 2 3 4 |
# Import os module so that we can traverse directories import os import time from wand.image import Image |
One of the first thing to do in this script is importing os, time and wand image libraries. We need the os library in order to traverse the directory structures and find the files that we want to resize. Time is also required should we wish to print the execution time or giving the resized file timestamp. In this example we are not using it yet.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
baseDirectory = "/home/pi/owncloud/Resize" targetDirectory = "resized" allowedExtensions = [".jpg", ".png"] # set the maximum image dimension to 4500 maximumDimension = 4500 #set maximum filesize to process maximumFilesize = 7340032 holdingImageFilename = "/home/pi/image_cruncher/too_large.gif" |
We then setup the variables that will be used throughout the script. The baseDirectory is the starting point of the directory traversal later on. Remember how we setup Owncloud using WebDav in the previous part, this is where it comes in handy. We are able to access the owncloud directory just like normal physical directory. Also note the Resize directory, this is where you should be uploading the files from mobile phones to process. It’s possible to have sub-directories within this Resize directory though. TargetDirectory is the name of the subdirectory that will house the resized image. After that we setup the allowedExtensions array variable. It’s important to specify which file extension is to be processed to reduce the amount of files we have to open.
In addition to the previous variables, we also setup the maximumDimension and maximumFilesize these 2 variables are important so that we can skip the files that are too big for the Raspberry Pi to process. Please tweak the numbers yourself, these settings are quite conservative as I’m running the 256MB RAM Raspberry Pi. Simply said, if an image exceeds 45oo pixels or exceeds 7340032 bytes, we will not process the image at all as the Raspberry Pi won’t be powerful enough.
Which brings us to the last bit, holdingImageFilename; if an image file is too big and we don’t process it there should be something to tell us. I have resort to creating a holding image called too_large.gif or something so we know straight away when looking at the folder that a file is not processed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# creates the passed in directory if needed otherwise do nothing or return false if the directory existed # is a file def createDir(fullPath): if not os.path.exists(fullPath): #create dir os.makedirs(fullPath) elif os.path.isfile(fullPath): print ("Path is file... not good") return False return True def createHolderImage(fullTargetFilePath, fileExtension): print ("copying holder image") with Image(filename=holdingImageFilename) as holderImg: with holderImg.convert(fileExtension.lower().replace(".", "")) as holderImgConverted: holderImgConverted.save(filename=fullTargetFilePath) |
Continuing on, we define 2 functions for our Raspberry Pi image cruncher/resizer. CreateDir as the name suggests is there to simplify checking if the destination directory exist or not, if not then we create the directory. CreateHolderImage on the other hand is a function that copies the image defined in the holdingImageFilename variable into specified path. The idea is if we try to resize test.png and test.png appears to be un-resizeable, then we substitute it in the resized folder using the holding image instead.
|
1 2 3 4 5 |
for currentDir, subDirList, fileList in os.walk(baseDirectory): # only process if the current directory is not the same with the base directory # also we only process if we are at the absolute sub directory of the current dir if currentDir != baseDirectory: ..... |
The for loop above basically walks through the directories and loads all the files within the directory. This will automatically traverse the directory structure till the last directory. You should be able to follow the rest of the script since it’s pretty linear and there are a lot of comments on it. However I will explain now a couple validations and tricks that we have to do to prevent the script from crashing altogether.
|
1 2 3 |
print ("Cooling down before processing"); # sleep for 5 seconds time.sleep(5) |
The code above as noted, delays the continuation of the script for 5 seconds. This is just an estimated number that I took to give the Raspberry Pi enough I/O time before continuing to the next image to process. Not allowing this delay seems to create bottleneck and in many cases (throughout the usage of this script) can cause input output error.
|
1 2 3 4 5 |
#check first if the file size is too big... if os.path.getsize(fullFilePath) > maximumFilesize: print ("File too large to process") createHolderImage(fullTargetFilePath, fileExtension) continue |
This bit of code above checks the file size in case it’s too big. It resizing means we need to load the image in memory and do the processing. The limited amount of RAM that my Raspberry Pi means we need to limit the filesize to prevent potential errors. Perhaps if you have the Raspberry Pi model B or model B+ then you’ll be able to get away with bigger sizes.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
if currentImg.width < maximumDimension and currentImg.height < maximumDimension: # we now resize the image to the maximum bounding square of 1280px currentImg.transform(resize='1280x1280>') print ("Resize completed, now saving to disk") # lastly write to the directory currentImg.save(filename=fullTargetFilePath) print ("Image save completed\n") else: # copy the holder image as the target file path # we would need the correct format of course createHolderImage(fullTargetFilePath, fileExtension) |
Lastly we also check for the image dimension. Even if the file size is not that big, resizing an image with huge dimension can cause out of memory crash as well. So we limit the image dimensions. Again, the limit is based on my 256mb Raspberry Pi.
Launcher Script
So now that we have gone through the script codes, you should have /home/pi/image_cruncher/crunch2.py. The next thing that we have to do is create a launcher script that can be executed by the cron. Why do we need a launcher when we can just run crunch2.py from cron directly? This is because we need to ensure that owncloud webdav is mounted before we run the script.
The easiest way to do it is by using a bash script. It’s pretty safe to keep trying to mount the directory everytime we execute as mount will simply tell you the directory is already mounted. So create a file called startImageCruncher and insert the codes below. Don’t forget to make the file executable by running chmod 755 startImageCruncher.
|
1 2 3 4 5 6 7 |
#!/bin/bash # Attempt to mount owncloud directory just to be sure mount /home/pi/owncloud # run the python script to crunch our images flock -n /home/pi/image_cruncher/cruncher2.lockfile -c "python crunch2.py" |
In the script above you will notice that we also use flock command to run the cruncher. This is to prevent the crunch2.py process from running more than one instance at a time. If we run more than one resizing process on the Raspberry Pi, things will be very slow and even crashing.
Automating with Cron
Lastly we automate our crunching process! Create a file called cruncher inside the /etc/cron.d/ directory.
|
1 |
vi /etc/cron.d/cruncher |
Then add the following cron task inside the file. The system will pick up this task schedule automatically. In this example, we set the script to run every 35 minutes.
|
1 |
*/35 * * * * pi cd /home/pi/image_cruncher && ./startImageCruncher |
Closing
This concludes the 3 part article of our Raspberry Pi Image Cruncher. I hope these instructions work well on your Raspberry Pi. As usual, if you have any questions or suggestions, feel free to leave comments, I will try to answer as much as I could. Also share this article to your friends if you like it, who knows it may be a good weekend project :) Have fun.
Categorised in: Fun computing, Raspberry Pi