It’s increasingly appreciated that AI can play a crucial role in supporting the work that we do, and a good example of how this will play out comes via a recent study from MIT CSAIL that describes how AI can help support image and multimedia editing.
The editing of images is now not only a major part of publishing and advertising, but increasingly of film making, and a big part of the process is ‘compositing’, which is the merging of foreground and background images so that actors are placed on top of items, such as planes or planets. Making this look realistic is very hard however.
“The tricky thing about these images is that not every pixel solely belongs to one object,” the researchers say. “In many cases it can be hard to determine which pixels are part of the background and which are part of a specific person.”
Autonomous editing
Getting this right can be extremely time-consuming, so the researchers set out to try and automate large parts of the editing process. The new approach allows them to take any image and decompose it automatically into different layers that are then separated by a number of so called ‘soft transitions’ between each layer.
The process, which is called ‘semantic soft segmentation’ (SSS), analyzes the texture and color of the original image, before combining it with information captured by a neural network that describes what the objects within the image are.
“Once these soft segments are computed, the user doesn’t have to manually change transitions or make individual modifications to the appearance of a specific layer of an image,” the researchers say. “Manual editing tasks like replacing backgrounds and adjusting colors would be made much easier.”
Whilst at the moment the application is primarily focusing on static images, the team are confident that it could work equally well for moving images, and therefore could have a number of applications in filmmaking.
“Instead of needing an expert editor to spend several minutes tweaking an image frame-by-frame and pixel-by-pixel, we’d like to make the process simpler and faster so that image-editing can be more accessible to casual users,” they explain. “The vision is to get to a point where it just takes a single click for editors to combine images to create these full-blown, realistic fantasy worlds.”
At the moment however, they believe it could be used on social media platforms like Instagram to make their filters more realistic. They will continue to work on the software, with the immediate next steps being to shorten the time required to compute an image down to a few seconds, whilst also making the images more realistic by improving the ability of the system to match colors and manage things like shadows.
It’s a fascinating example of the progress being made, and how it can help us do our jobs more effectively. Check out the video below to see the system in action.
