Salesforce. has been granted a patent for a method of pre-training transformer models, which involves dividing the model into held-out and main components. The process includes performing forward and backward passes to determine and update self-attention hidden states and model parameters based on loss calculations. GlobalData’s report on Salesforce gives a 360-degree view of the company including its patenting strategy. Buy the report here.
Smarter leaders trust GlobalData
Access deeper industry intelligence
Experience unmatched clarity with a single platform that combines unique data, AI, and human expertise.
According to GlobalData’s company profile on Salesforce, Social media analytics was a key innovation area identified from patents. Salesforce's grant share as of July 2024 was 66%. Grant share is based on the ratio of number of grants to total number of patents.
Pre-training a transformer model using sophisticated patterns
The patent US12072955B2 outlines a method and system for pre-training transformer models, which are widely used in natural language processing and machine learning tasks. The method involves dividing a transformer model into two components: a held-out model and a main model. The held-out model consists of attention heads from a specified number of lower layers of the transformer, while the main model encompasses the remaining layers. The process includes performing forward passes on both models using a training dataset to determine self-attention hidden states, which are then concatenated to serve as inputs for subsequent layers in the main model. Additionally, backward passes are conducted to calculate losses for both models, followed by updating their parameters based on these losses.
The claims further specify that the held-out model typically has a smaller parameter size compared to the main model and allow for adjustments in the number of layers and attention heads within the held-out model. The method also permits the division of the held-out model into multiple configurations, each with varying parameter sizes. This flexibility in model architecture aims to enhance the efficiency and effectiveness of the pre-training process. The patent also describes a non-transitory machine-readable medium that contains instructions for executing these operations, emphasizing the systematic approach to optimizing transformer model training through structured parameter management and loss-based updates.
To know more about GlobalData’s detailed insights on Salesforce, buy the report here.
Data Insights
From
The gold standard of business intelligence.
Blending expert knowledge with cutting-edge technology, GlobalData’s unrivalled proprietary data will enable you to decode what’s happening in your market. You can make better informed decisions and gain a future-proof advantage over your competitors.
GlobalData, the leading provider of industry intelligence, provided the underlying data, research, and analysis used to produce this article.
GlobalData Patent Analytics tracks bibliographic data, legal events data, point in time patent ownerships, and backward and forward citations from global patenting offices. Textual analysis and official patent classifications are used to group patents into key thematic areas and link them to specific companies across the world’s largest industries.
