The Japan Times - HoloCoder technology promises brighter smartphone images and 3D depth data

One of the most essential pieces of technology that makes your phone’s camera work was created in 1974, and it hasn’t changed much since. When you snap a photo with your smartphone, the camera captures light and converts it into electrical signals to build a digital image. But the Bayer filter—a red, green, and blue layer that sits above the sensor and adds color to your images—absorbs and wastes around 60 percent of that light before it can reach the sensor. To make a red pixel, the filter blocks and throws away blue and green light.

Now, the Boston company Metalenz has acquired an Israeli startup called PxE Holographic Imaging, which has developed a new optical layer called a HoloCoder. This microscopic structure sits directly on top of the image sensor and diffracts light, bending the light waves rather than absorbing them. With the HoloCoder redirecting the light waves toward the sensor’s pixels instead of blocking them, the camera sensor can collect up to a claimed 2.5 times more light. Such a leap in light-gathering ability could dramatically improve a smartphone's low-light photography and dynamic range.

There's a surprising secondary benefit too: HoloCoder technology can capture 3D depth information at the same time. Most smartphones use secondary cameras, dedicated depth sensors, or AI guesswork to figure out the distance between the camera and the subject to judge how much depth should be shown in features like portrait mode. By merging light sensing and depth sensing into one layer, the HoloCoder promises to simplify the electronics on the back of your smartphone, saving space and processing power. The acquisition of PxE—details of which were not disclosed—makes sense for Metalenz. The company launched publicly in 2021 and pioneered the commercialization of metasurfaces, a flat-lens system with nanostructures that bend light, reducing the need for multiple lens elements over the camera sensor while also capturing more light. The company has shipped more than 300 million of these metasurface optics to consumer devices today.

The technology can also capture polarization data, and Metalenz is working with Qualcomm to create a more effective, secure facial biometric authentication system for Android called Polar ID to rival Apple's Face ID, expected to arrive on consumer devices at the end of 2027. Rob Devlin, Metalenz CEO and cofounder, tells published reports that PxE's approach mirrors his own company's by patterning microscopic structures directly onto an image sensor to unlock light data that conventional camera systems throw away.

"Apple, Samsung, Google—everyone is fighting to improve the visible camera every year, right?" Devlin says. "A lot of what they're doing is investing hundreds of millions of dollars into software to try to somehow improve it, but the downside of that is that once information is lost when you're absorbing all of that light, you can't really buy it back, even if you're throwing all of the software into the world at it." Low-light photography and night-time video capture could see dramatic improvements with the HoloCoder system. In high-contrast scenes—like taking a selfie in front of a bright window—the sensor can capture more light and better re-create the scene without blowing out highlights or drowning shadows in noise.

"PxE's technology isn't on the market yet, but as Metalenz has already developed a supply chain, processed an image pipeline, and set up machine learning algorithms on how to take advantage of these new data streams, Devlin thinks PxE's HoloCoder will be able to fit neatly into its platform and "completely disrupt" the visible sensing market. While the HoloCoder functions as a diffractive optic rather than a true metasurface, Devlin says Metalenz has already replicated the technology in the lab using its own metasurface optics." "It's really then bringing all of that information together through a single approach," Devlin says. "We would do a metasurface design where now from a single image sensor, you're getting all of the information that is in light: You're getting polarization, you're getting color without filtering, and you're getting this depth channel. It's like the one sensor to rule them all."

For now, Devlin says the two technologies will be separate products for different markets, but in the long term they could fuse into one Metalenz system. He expects we could see PxE's HoloCoder used in consumer electronics as early as late 2027 or early 2028. Simplifying and shrinking the hardware with metasurface optics was just the first phase for the company—the real goal has always been to unlock that new information channel, which Devlin thinks can be transformational for other categories, like autonomous robots and autonomous vehicles. Instead of relying on large modules with multiple sensors and computationally expensive chips to align all that spatial data in real time, a single-sensor system could do the job in a smaller, cheaper design while using less computing power.

"It is a real problem in the sense that, in the color filter, you lose a lot of light," says Arka Majumdar, who researches low-energy opto-electronics at the University of Washington. He points out that other industry heavyweights, including Samsung and a startup called Eyeo, have been actively researching their own alternatives to Bayer filters, called "color routers." While he agrees that ditching the filter can meaningfully improve smartphone photography in low light, he is skeptical that it can revolutionize categories like robotics and self-driving cars. Unlike phones, vehicles have plenty of space to mount stereo cameras or lidar. The bottleneck isn't low light or image clarity, but the computational power needed to make split-second decisions.

"In fact, arguably, if you get better data, it's actually worse," he says. Airbnb hosts are furious they can no longer make guests do their chores

S.Ogawa--JT