Researchers from Stanford University have developed a new photonic chip capable of training artificial neural networks at the speed of light, marking a significant advancement in the field of AI. This is the first photonic chip capable of training neural network models. Neural networks are currently used in a variety of applications, from GPT to AlphaFold, and are trained on supercomputers like Alpha Tensor. However, these supercomputers are enormous, consume a lot of power, and are estimated to consume 10 to 20 percent of the world's total power by the end of this decade.
Photonic chips, unlike conventional electronic chips, leverage light as their primary physical principle. They use components like waveguides, filters, and light detectors to perform computations. Light, being the fastest carrier of information in the universe, can transmit information while dissipating less heat and energy than electrical signals. In a photonic chip, computations are carried out by light passing through tiny optical waveguides. These waveguides are passive and drain almost no power, meaning the main power in the system is spent on generating light and processing the output.
Photonic chips have several advantages. They use light-based connections, similar to the principles used for long-distance data transmission using fiber-optical cables. This allows for low losses and high bandwidth, which practically means high data rates. Photonic chips can practically double or even triple the throughput of the chip by performing parallel computations at different wavelengths of light. This is possible because light has many different colors that do not interfere with each other.
Photonic chips have been particularly promising for running artificial neural networks because photonics is really good at certain mathematical operations, such as matrix multiplication, which is performed three million times during a neural network training. Photonics can perform these operations at least one thousand times more efficiently than traditional methods. Companies like Light Intelligence and Light Matter are already working on commercial photonic chips.
So far, photonic chips have only been used for inference applications. Inference is the phase where a trained neural network is used in the real world to make predictions based on new inputs. Training, on the other hand, is the phase where the neural network learns patterns and relationships by iteratively adjusting weights and biases to minimize the error between the predicted and desired output. The training process typically requires a large amount of labeled data and a lot of computational resources.
In 2018, a paper proposed an algorithm that theoretically could carry out back propagation (a fundamental algorithm used for updating the weights during the training process) on a photonic chip. This was the first time that back propagation and neural network training were realized on a photonic chip. The Stanford team built a hybrid chip that combines the best of both worlds: electronic and photonic. The most computationally expensive part, matrix multiplication, was carried out optically with light, while the remaining calculations were done digitally on the electronic part of the chip.
This new chip managed to enable light propagation in both directions, forward and backwards. It added light sources and light detectors at both sides of the chip so they can send light forward and backwards through the chip. The training data is encoded in a light signal, then they slide pass through the photonic neural network, and the error is calculated at the output. The calculated error is then encoded in the light and sent backwards so the photonic neural network can measure the inference with the input photonic signal. They adjusted the connections of the network to improve predictions.
This work demonstrated that back propagation can effectively train photonic neural networks, marking a huge milestone. This shows that AI training could fundamentally change in the future with most of the computations taking place optically. However, there are still some technical challenges to overcome, such as the size of the optical components and the need for a special fabrication process. Despite these challenges, the future of photonics looks bright.
1. Researchers from Stanford have built a new chip capable of training artificial neural networks at the speed of light. This is the first ever photonic chip for this purpose .
2. Neural networks are used in various applications, from ChatGPT to AlphaFault, and are trained on large supercomputers like Alpha Tensor .
3. These supercomputers consume a significant amount of power and energy. Estimates suggest that 10 to 20 percent of the world's total power could be consumed by such systems by the end of the decade .
4. Photonic chips are different from conventional electronic chips. They leverage light and components like waveguides, filters, and light detectors to perform computations .
5. Light is the fastest carrier of information in the universe and can transmit information while dissipating less heat and energy than electrical signals .
6. In a photonic chip, computations are carried out by light passing through tiny optical waveguides. These waveguides are passive and drain almost no power .
7. The main power in the system is spent on generating light and processing the output .
8. Photonic chips can practically double or even triple the throughput of a chip by performing parallel computations at different wavelengths of light .
9. Photonic chips are particularly promising for running artificial neural networks because photonics is good at certain math operations, particularly matrix multiplication .
10. Companies like Light Intelligence from MIT Science Labs and Light Matter have started working on commercial photonic chips .
11. Photonic chips were previously only used for inference applications, not for training. However, a new paper from Stanford University has demonstrated back propagation and neural network training on a photonic chip .
12. The new photonic chip from Stanford allows light propagation in both directions, forward and backward. It has added light sources and detectors at both sides of the chip .
13. The chip has demonstrated that back propagation can effectively train photonic neural networks .
14. The Netherlands has invested 1.1 billion euros in the photonic chip industry, with a focus on the photon Delta ecosystem, which includes companies involved in design, fabrication, and packaging of photonic chips .
15. The future of photonics looks bright, with significant investments in recent years and indications that photonics will deliver on its promise .