The Peking University Team Initiates the Reproduction of the Sora Project ‘Open Sora’
The Peking University team, in collaboration with RabbitPre, has launched the project to reproduce Sora, named ‘Open Sora’. Behind this initiative is the limited resources situation where the team hopes to gather the strength of the open-source community to complete the reproduction work as much as possible.
The core team of the project is led by Assistant Professor Yuan Li from the School of Information Engineering at Peking University and Professor Tian Yonghong from the School of Computer Science at Peking University, with a total of 13 members. In addition to team members, alumni from Peking University and contributors to AnimateDiff have also actively responded, with some expressing willingness to provide high-quality datasets.
Technical details about Sora’s reproduction have been announced, and the project framework consists of three main components:Video VQ-VAE, Denoising Diffusion Transformer and Condition Encoder.
In order to solve the variable aspect ratio issue of Sora videos, the team adopted the FiT technology proposed by Shanghai AI Lab and implemented a dynamic masking strategy. At the same time, the team also used position interpolation to handle variable resolutions and employed Video VQ-VAE to support processing of videos with varying durations. The team has initially achieved three functions and provided two demos on the project homepage, demonstrating the reconstruction effects of 10-second and 18-second videos.
Although some progress has been made, the team is facing many challenges. The current training is being conducted on 8 A100-80G GPUs, but it is still far from enough and more data and GPUs are needed for training. The team has completed some tasks, including sampling scripts and adding class conditions to embeddings, but there are still some pending items that need to be completed.
One of the leaders of the project is Yuan Li, an assistant professor at the School of Information Engineering at Peking University. He has been honored on Forbes’ list of outstanding individuals under 30 in Asia. Another leader is Tian Yonghong, a distinguished professor at Peking University with extensive research experience and honors.
The question of who will be the first to release a Chinese version of Sora is still an unsolved mystery. In the industry, there are rumors that ByteDance might become the first company to release a Chinese version of Sora. Although there are some AI models similar to Sora, such as Boximator and Dreamina which have recently emerged, they still have certain gaps compared to Sora. However, the answer to this question remains to be revealed with time.
Sign in with google
