Open CSDN APP
Copyright © 1999-2020, CSDN.NET, All Rights Reserved

Open APP

LightGen: All-optical synthesis chip for large-scale intelligent semantic vision generation Original

2025-12-30 20:54:42
Reads
539

Favorites
20

10 Likes

Xy-unu

6 years of coding experience

Follow
Paper Basic Information
Title: All-optical synthesis chip for large-scale intelligent semantic vision generation
Address: https://www.science.org/doi/abs/10.1126/science.adv7434
Journal/Time: Science 2025
Author: Shanghai Jiao Tong University (School of Electronic Information and Electrical Engineering) and Tsinghua University (Department of Automation/Department of Electronic Engineering)
1. Core Idea
Exploring the application of all-optical computing chips in the field of generative artificial intelligence.

It addresses the contradiction between computational power and energy consumption in generative AI, and designs and manufactures an all-optical computing chip called LightGen for large-scale intelligent semantic vision generation.

2. Research Background and Motivation
Traditional all-optical computing chips are mainly limited to small-scale classification tasks, and optoelectronic cascading or multiplexing severely weakens the speed of optical computing.

Advantages of optical computing: high speed, low power consumption.
Disadvantages:

Too small scale: Generative tasks require millions of neurons, while previous optical chips (such as MZI, micro-rings) usually only have tens to hundreds.
Fixed dimensions: Analog optical signals are difficult to change dimensions during propagation, while generative models usually require a process of "compressing features and then decompressing" (i.e., VAE architecture).
Training relies on ground truth: Previous optical chips relied on a one-to-one correspondence between output and standard answers for training, but generative AI aims to create "non-existent data," so there is no standard answer.
3. Methodology
Input: High-resolution images or semantic information (e.g., 512×512 pixels), without the need to divide them into small blocks like previous optical chips. Output: Images or even videos after semantic generation or manipulation. Specific tasks include semantic generation (drawing images from scratch), style transfer (such as transforming a photo into a Van Gogh painting), denoising (repairing blurry images), and 3D visual generation (such as NeRF).
Implementation: A physical photonic chip integrating millions of optical neurons connected through a fiber optic array.

3.1 Core Innovations
Increased Scale (3D Packaging):  Utilizes 3D packaging technology, integrating over 2 million optical neurons in a space of only 136.5 mm². This increases the scale of the optical chip by several orders of magnitude compared to previous chips, enabling it to process high-resolution 512×512 images.
All-Optical Dimensional Transformation (Optical Latent Space, OLS): Achieves all-optical dimensional compression and transformation using the physical properties of single-mode optical fibers.
Unsupervised Training Algorithm (BOGT): Proposes a Bayesian-based optical generative model training algorithm (BOGT). It trains the model to learn the probability distribution Q(Z|X) of the data, making it approximate the prior distribution P(Z).
[Image Description]
Figure 1B: Previous MZI or microring chip structures were simple with few neurons. Figure 1D: LightGen is a dense stack of diffractive layers connected by OLS (fiber bundles). Figure 1E (OLS principle): Dimensional compression at the physical level.

[Image Description]
Extracting and visualizing the data of the optical signal (t-SNE).

3.1 Core Process:
Spatial Light Modulator (SLM): Digital signal to analog signal conversion. The digital end receives digital images, and the physical end shines a parallel, uniform laser beam onto the SLM. Each liquid crystal cell on the SLM changes the amplitude (brightness) or phase (delay) of the light according to the pixel value of the image. This results in a light field carrying image information.
Optical Encoder: Light passes through an integrated diffractive metasurface, extracting high-dimensional image features. Similar to encoders in computer vision, it extracts features.
Optical Latent Space (OLS): The optical signal passes through a single-mode fiber array, using physical properties to complete the "compression" and "sampling" of dimensions. This is crucial for generation. See Figure 1E and Figure 2.  Generally, in electronic computers, we need to transform a high-dimensional image into a low-dimensional vector (Latent Vector), and then reconstruct the image from this vector. However, in the optical path, light propagates in all directions, with infinitely many dimensions. We need to reduce the dimensionality through physical mechanisms.
Optical Generator: Reads features from the latent space and passes through the diffractive layer again to reconstruct a high-resolution semantic image. Similar to a decoder, it receives low-dimensional feature signals from the OLS (i.e., the filtered light spots), and then uses multiple layers of diffractive metasurfaces to "reconstruct" and "amplify" these features into a high-resolution semantic image. It can generate images that did not exist before, or generate images with altered style or perspective based on the values ​​modified in the OLS.
3D Packaging: This entire system (encoder, fiber array, generator) is stacked and packaged, containing over 2 million optical neurons.
3.2. OLS Optical Latent Space
A **single-mode fiber array** is used to connect the encoder and the generator.
Filtering Mechanism: The free-space light field contains countless modes. However, single-mode fibers only allow the fundamental mode (LP01) to pass through; other modes are filtered out. This achieves sampling and conversion from "high-dimensional" to "low-dimensional".

After passing through the fiber array, light is output at the other end. The light spots are the "features". Looking at the output end of the fiber with a microscope (as shown in Figure 1H and Figure 2B) reveals a Gaussian speckle array. The amplitude and phase of each light spot represent a **"latent feature"** of the image. Coupling Efficiency Formula:
Whether light can enter the optical fiber depends on the degree of overlap between the input light field Ein and the fiber's fundamental mode Efiber.
Overlap Integral Formula:
η = |∬ Ein(x,y)Efiber*(x,y)dxdy|²
This formula demonstrates that only light with a specific spatial distribution (i.e., a clearly defined fundamental mode) can pass through; chaotic higher-order light is physically integrated out. This is the physical-level **"nonlinear activation" and "dimensionality reduction"**.
Ultimately, a 10×10 (100-dimensional) fiber array was chosen. Figure S11 shows the ablation experiment regarding dimension selection.

Introducing Fluctuations: Light coupling into the optical fiber is extremely sensitive to complex field distributions, providing the necessary fluctuations for the generative model. To generate a new image, the system must have a certain degree of "randomness" or "probability distribution." In electronic computers (such as VAEs), this randomness is simulated using mathematical formulas (Gaussian sampling). In LightGen, the optical fiber coupling process itself provides this fluctuation.
Manifold Learning (Figures 2D, 2E) demonstrates that in this optical latent space, images with similar semantics (e.g., dogs with white backgrounds, dogs outdoors) automatically cluster together. This indicates that the optical signal understands semantics rather than just pixel arrangements.
3.3. Training Algorithm (BOGT)
For generative AI, there is no standard answer, and physics is not differentiable.
A Bayesian-based training algorithm for Optical Generative model Training (BOGT) is proposed.
It learns probability distributions, not correspondences.
It calculates the feature distribution Q(Z|X) and constrains it using Kullback-Leibler divergence. The feature distribution of the light output from OLS is forced to approximate a desired standard distribution.

The loss function L consists of three parts:
L = αl<sub>KLD</sub> + βl<sub>mse</sub> + γl<sub>op</sub>

l<sub>KLD</sub> (KL divergence): This is the core. It forces the light intensity distribution in OLS to approximate a "Truncated Biased Gaussian distribution". Ordinary AI uses a standard normal distribution, but in optics, light intensity is always positive and cannot be negative, so we modified the mathematical model to adapt to physical reality.
l<sub>mse</sub> (Mean Squared Error): Ensures that the generated image and the target image are similar at the pixel level.
l<sub>op</sub> (Optical Penalty Term): This is an engineering technique. It prevents the light from becoming too strong during propagation (burning out the device) or too weak (being overwhelmed by noise), ensuring that the light energy at each layer is within the optimal operating range of the detector.

3.4. Training Strategy
Modeling: An accurate physical model of LightGen (simulating how light diffracts and couples into optical fibers) was built on a computer.

Training: The BOGT algorithm was run on this digital model, continuously adjusting the phase parameters of the virtual diffraction layer.

Manufacturing: Once the computer model learned to generate high-quality images, we "froze" this set of optimal parameters and etched them onto a real chip using photolithography.

This "digital training, physical inference" approach circumvents the weakness of optical chips in terms of difficulty in real-time parameter updates.

4. Experimental Results
Generation tasks demonstrated: Figure 3A (generation), Figure 3C (denoising), Figure 3E (style transfer).

Figures 3A and 3B. High-resolution direct output, LightGen's most intuitive capability is generating 512 × 512 pixel animal images. Previous optical computing was limited by input and output ports, usually only able to process very small images, or requiring large images to be cut into 7×7 or even smaller patches for separate processing, resulting in noticeable stitching seams in the generated images. LightGen uses 3D packaging and can directly generate a complete large image without stitching seams. Magnified details clearly show the fur texture of cats and dogs and the reflections in their eyes. This demonstrates that the optical signal not only preserves the contours during diffraction but also restores high-frequency details. Using an additional electronic neural network (CNN) to identify these generated images, the classification accuracy is almost as high as that of the real animal dataset (AFHQ), indicating that the generated features are reliable for computer processing.

Figures 3C and 3D show images with significant noise added. LightGen utilizes the characteristic of single-mode optical fibers to filter out clutter, physically denoising and restoring clear handwritten characters. Noise in images usually manifests as high-frequency clutter, which often contains a large number of higher-order modes in the optical field. Because of this characteristic, only the core semantic information (fundamental mode) is transmitted through OLS. The generator receives a clean signal, naturally restoring a clear image.

Figures 3E, 3F, and 3G: Inputting a simple drawing of an apple allows it to be transformed into a "Van Gogh style" or "metallic style." Compared to the "patched" method, which results in broken strokes, LightGen produces images with smooth lines and complete structures. A general encoder is designed to extract content features, and multiple different generators are designed, each corresponding to a different artistic style. The optical path can be controlled, allowing the optical signal carrying image features to enter different physical regions (different diffraction channels), thus undergoing different "stylization treatments." This is somewhat similar to current MoE (Mixture of Experts) or Adapter models. The backbone remains unchanged, and the data is routed to different Heads or Adapters based on task requirements (Prompt).
[Image description here]
Figure 4E (semantic manipulation) uses KL divergence to constrain the distribution, so in the latent space, slightly changing the values ​​results in a gradual change in the generated image. Decoupling is possible, and the algorithm learns the distribution and corresponding light spots itself.