Camera Takes Pictures by Describing What it Sees to AI
An artist has created a novel AI camera that describes what it sees to an AI image generator which then synthesizes the picture and instantly prints them out for the photographer.
Jasper van Loenen invented the Black Box camera, the first camera in the world that operates this way. He had to reverse engineer a Fujifilm Instax portable printer, which sits inside the square instrument.
“The Black Box camera is a camera you can use to take a photo, which then gets turned into a caption, which then is used to generate an image,” the Dutchman tells PetaPixel.
The twin-lens reflex camera uses a Raspberry Pi to take the photo which sends it to OpenAI’s DALL-E for image generation. An example video can be watched below.
“The image recognition returns a very simple caption,” explains Van Loenen. “For example, in the video when I took a photo of a plant on the table the caption was something like: ‘a room filled with plants.’
“I know what these captions are because I looked them up in the logs, but a regular user wouldn’t be able to see the caption used: they only get the final, generated image.”
Van Loenen has released a few example images. One of a blue and yellow train where the camera produced an image that looked similar to what the camera was seeing in real life.
However, he tells PetaPixel that sometimes the camera has a mind of its own.
“[I was] taking a photo of a wall in a climbing gym and it generated something like ‘colorful kites hanging from the ceiling,’ which I thought was pretty funny,” he says.
Van Loenen already has plans for a second prototype camera that will add more controls.
“They will be more abstract than regular camera settings,” he explains. “I’m not sure yet what these will be, but I’m thinking of things like ‘confusing’, ‘mesmerizing’, ‘deep’, ‘popular’, et cetera.
“These settings will then be used to alter the generated caption before using it to create the final image.”
Van Loenen’s reverse engineering of the Instax printer is available via open source here. More of his work can be found by visiting his website.
Image credits: All photos by Jasper van Loenen.
