Salesforce unveiled a new language model with the ability to process 1.5 trillion words

Salesforce has released a new suite of AI-powered tools that can handle massive amounts of textual data, up to 1.5 trillion words or tokens. Known as the XGen-7B family of models, these tools can be used specifically to handle unstructured data (data that does not fit neatly into rows and columns, such as text and images) and are much better analyzed and organized than LLAMA meta-models. Have.

As more people start using AI tools like ChatGPT, the data fed into these systems will become more complex and structured. This complexity makes it more difficult to use tools like ChatGPT, which are designed to analyze language and text, when the input or data being analyzed does not follow a clear structure. Therefore, there is a growing need for advanced systems that can handle unstructured data and do more to meet the growing demand for artificial intelligence tools.

Businesses can take advantage of chat systems like ChatGPT or BARD that can provide summaries of long documents or analyze customer data to gain insights. However, for these chat systems to be effective, they need to be trained on huge amounts of data. Many businesses opt for smaller, cheaper models of these chat systems, which aren’t always capable of complex tasks like summarizing long documents or scrutinizing customer data. Therefore, since these models cannot handle such complex tasks well, these businesses cannot fully benefit from the benefits of this technology.

Source game language models such as LLAMA, Falcon-7B, and meta MPT-7B are not ideal in managing texts or long documents, because they are not able to manage a large amount of texts and can only control the maximum sequence length of about 2000 tokens or text units. However, the XGen-7B family of language models developed by Salesforce are trained using a technique called “standard dense attention” and are therefore capable of processing much larger input data, up to 1.5 trillion tokens. . This has made the mentioned language models an effective tool for managing and analyzing long documents.

Salesforce researchers selected a set of linguistic models with seven billion parameters and trained them using a combination of Salesforce and JAXFORMER data, as well as publicly available training data. This model has achieved better results compared to open source models such as LLAMA, Falcon and Redpajama. The researchers also found that it costs only $150,000 to train a model with 1 trillion tokens using the Google Cloud Computing Platform TPU-V4, which is a more cost-effective and efficient way to train large language models. Thus, researchers have been able to create an advanced AI model that can analyze and process large amounts of data more accurately than other open-source alternatives, while keeping the cost of training the model relatively low.