What is Sora artificial intelligence and how does it create interesting videos?

As it can be seen, the speed of developme of artificial ielligence is moving towards a poi beyond human understanding, and in this coext, the text-to-video conversion system of OpenAI is the latest artificial ielligence technology that has affected the world of technology and shows that what happens in The field of artificial ielligence is happening sooner than expected.

OpenAI, better known for its AI chatbot tool “Chat GPT” (Chat GPT), has unveiled its new AI tool called “Sora” for converting text to video in rece weeks. This tool is not yet publicly available and only limited testers have the opportunity to use it, but as the published videos show, it is a significa improveme over other tools for making videos from text, and the final videos produced with it look completely realistic. they arrive A topic that can be exciting and worrying at the same time.

What is OpenAI Sora?

Like other artificial ielligence generating tools such as DALL-E and MidJourney, Sora's AI-based tool takes text messages from you and turns them io a visual image. But unlike the other meioned AI-based image generators, Sora is a complete video clip with motion, differe camera angles, direction, and everything else you'd expect from a traditionally produced video.

Looking at the samples on Sora's website, the end results are often indistinguishable from real, professionally produced video. The comparison includes everything from expensive drone footage to multi-million dollar productions complete with AI-generated actors and special effects.

Of course, Sora is not the first technology to produce video from text, and so far the most promine example in this field was RunwayML, which offers its services to the public for a fee. However, even at their best, runway videos look more like the early generations of midceury stills. There's no image stabilization, the physics in it don't make sense, and currely the longest clip it offers is 16 seconds.

Lumiere, released a few weeks ago, claimed to produce better videos than its predecessors.But Sora seems to be more powerful than Lumiere at least in some cases. This technology can produce videos with a maximum resolution of 1920 x 1080 pixels and various aspect ratios.

The best output that Sora offers is quite stable, the physics preseed in it appear true to the human mind, and the clips can be up to a minute long. The videos generated by Sora are sile, but there are other AI systems that can generate music, sound effects, and speech for you to put over your AI-generated videos.

As such, the huge leap Sora has made over previous generations of AI-generated videos cannot be ignored. It was only a year ago that artificial ielligence was producing completely unrealistic videos, but now with the preview of Sora, a big shock has been created for the activists in the field of visual arts. Sora is likely to affect the eire video industry, from single-person video makers to the level of big-budget projects like Disney and Marvel. This may be the true beginning of the syhetic film industry.

How does Sora work?

As much as possible, we will review the process of producing images by Sora, but we cannot go io details. First, because OpenAI doesn't talk about the inner workings of its technology. All of this is proprietary, so the details of the secret technology that sets Sora apart from the competition are not known. Second, the details of the subject may not be ieresting and understandable to the general public and only a computer scieist would understand, so we can only understand how this technology works in general.

Fortunately, “Mike Young” (Mike Young) has provided a complete explanation of Sora's technology on Medium, based on an OpenAI technical report, and here we review the most importa pois.

Sora is built on the lessons that companies like OpenAI learned when creating technologies like ChatGPT or DALL-E. Sora generates videos by dividing those videos io segmes that are similar to the tokens used in the GPT chat training model. Because these tokens are all the same size, things like clip length, aspect ratio, and resolution don't matter to Sora.

In fact, this text-to-video generator uses the same extensive transformation approach that is used in other AI language transformation models such as Chat GPT and Google Gemini. These transformers were first iroduced by Google in 2017. While Transformers was originally designed to find patterns in the symbols that represe text, Sora now uses symbols that represe small chunks of space and time.

During training, Sora looks at noisy and partially sparse patch tokens of a video and tries to predict a clean, noise-free token. By comparing it with a ground truth, the linguistic model learns the video and puts these images together to form a complete video. It is based on this practice and training that the examples on the Sora website look very autheic and real.

Aside from this remarkable ability, Sora also has very detailed annotations for the video frames it's trained on, which is a big part of why the tool can modify generated videos based on text requests.

Sora's ability to accurately simulate the physics in videos appears to be an emerging feature, stemming solely from training on millions of videos that coain motion based on real-world physics. Object persistence is very good in Sora, and even when objects move out of the frame or are temporarily obscured by something else in the frame, they stay in the background and reappear without cluttering the camera angle.

However, sometimes when things in the video are connected, it has problems understanding the reason for the connection to generate the next image, including reproducing the objects. Also, Sora seems to sometimes confuse left and right. However, what has been shown so far in the iroduction of Sora's power is not only usable right now, it's quite advanced.

When will you have access to Sora?

We're all very excited to use Sora, and we'll definitely be writing more about how useful this technology can be in the future. But when will this happen?

It's not yet clear exactly how long it will take for Sora to become available to the public, or how much it will cost. According to OpenAI, the technology is in the hands of the Red Team, a group of people tasked with trying to get Sora to do all the wrong things it shouldn't, and then helping to put in place protections against such things. which may be requested by real customers. These include the poteial to create false information, create offensive or viole coe, and many other conceivable abuses.

In addition, some select coe creators already have access to it, which appears to be both for testing purposes and to receive various third-party commes and approvals that could eveually lead to its final release. As a result, the public release time of Sora is not yet known. That's because if it's in the hands of safety testers now, problems may be discovered that take longer to fix than expected, thus delaying its public release.

The fact that OpenAI feels ready to show off Sora's capabilities, and has even received public requests through X for AI video generation, means the company thinks the quality of the final product is almost ready, but uil the image As long as there is better public opinion on this and safety issues are raised and discovered, no one can say for sure when it will be released. At the same time, we can expect this technology to be released on Sora's website in the coming mohs, not years, but probably not next week!

Poteial applications of text to video conversion

Currely, video coe is produced either by shooting real-world footage or using special effects, both of which can be costly and time-consuming. But if Sora becomes available to the public at a reasonable price, people can use it as a prototyping software to visualize ideas at a much lower cost. Based on what we know of Sora's capabilities, it can even be used to create short videos for some applications in eertainme, advertising, and education.

OpenAI's technical paper on Sora is published under the title “Video Production Models as World Simulators”. The article argues that larger versions of video generators like Sora may be “capable simulators of the physical and digital worlds, and the objects and animals and people that inhabit them.”

If this is true, future versions may have scieific applications for physical, chemical, and even social experimes. For example, it may be possible to investigate and test the impact of tsunamis of differe dimensions on various types of infrastructure and the physical and meal health of people near the affected areas.

However, achieving this level of simulation is extremely challenging, and some experts argue that a system like Sora is fundameally incapable of doing so. A complete simulator must accou for physical and chemical reactions at the most precise levels of the universe. However, approximating the world and making videos realistic to the human eye may be readily available in the coming years.

Ethical risks and concerns

The main concerns about tools like Sora revolve around their social and ethical impact. In a world already plagued by misinformation, tools like Sora may make things even worse.

It's easy to see how the ability to produce real video of any conceivable scene could be used to spread convincing fake news or cast doubt on real footage. It may endanger public health measures, be used to influence society, or even challenge judicial systems with poteially false evidence.

Video producers may also pose direct threats to target individuals by producing deepfakes, especially immoral ones. Such activities may have dire consequences on the lives of the affected individuals and their families.

Beyond these concerns, there are issues related to copyright and iellectual property. AI generators require large amous of data for training, and OpenAI has not disclosed where Sora's training data came from.

“Large language models” (LLM) and image generators have also been criticized for this reason. In America, a group of famous authors have sued OpenAI for possible misuse of their coe. The case argues that the big language models and the companies that use them are stealing the work of writers to create new coe.

This is not the first time in social memory that technology moves ahead of the law. For example, the issue of social media platforms' obligations to moderate coe has generated heated debates in rece years, most of which revolve around Section 230 of the United States Code.

While these concerns are real and worth investigating, based on past experience, they are not expected to stop the developme of video production technology. As noted, OpenAI has taken several importa safety steps before making Sora publicly available, including working with experts to “preve misinformation, hateful coe, and bias” and “building tools to help ideify misleading coe.” “doer” cooperates.

Conclusion

Sora, the new OpenAI product, represes a significa advance in artificial ielligence technology and once again reminds us that the speed of progress in this field is far beyond what we imagine. This AI model can now create videos from textual descriptions that are very difficult and sometimes impossible to distinguish from real footage.

Like it or not, we are inevitably standing on the edge of a new era of innovation in the world of technology, and now is the mome when we should try to take corol of artificial ielligence and use it in a positive way instead of confroing and fearing technology. . Converting text to video, which can be much easier in the future, opens couless creative opportunities for filmmakers, coe producers, digital artists and everyone to visualize creative ideas in the easiest way possible.

If human society takes up this challenge with wisdom and grace, powerful simulators like Sora can open up unimaginable vistas for visual storytelling, shaping couless diverse voices to tell stories previously unimaginable by humans or machines.

The real exciteme of this new technology lies in its ability to empower everyone to share their unique views of the world. By iertwining the disciplines of artificial ielligence io the traditional filmmaking process, the ultimate message of art can be shared more widely than ever that despite our differences in what makes us laugh or cry, despite our dreams and anxieties, we are all still human. .

Sources: The Conversation, How to Geek, Christian Martinez, Light Works