Skip to main content
Check out our comparison guides:

X API: Enterprise data dictionary

Introduction

Enterprise Posts are the basic atomic building block of all things X. All X APIs that return Posts provide that data encoded using JavaScript Object Notation (JSON). JSON is based on key-value pairs, with named attributes and associated values. Post objects retrieved from the API include a X User’s “status update” but Retweets, replies, and quote Tweets are all also Post objects.  If a Post is related to another Post, as a Retweet, reply or quote Tweet, each will be identified or embedded into the Post object.  Even the simplest Post in the native X data format, will have nested JSON objects to represent the other attributes of a Post, such as the author, mentioned users, tagged place location, hashtags, cashtag symbols, media or URL links.  When working with X data, this is an important concept to understand. The format of the Post data you will receive from the X API depends on the type of Post received, the X API you are using, and the format settings. Enterprise endpoints that return Post objects have been updated to provide the metadata needed to understand the Post’s edit history. Learn more about these metadata on the “Edit Posts” fundamentals page.
In native X format, the JSON payload will include of ‘root-level’ attributes, and nested JSON objects (which are represented here with the {} notation):

Available data formats

Please note: It is highly recommended to use the Enriched Native format for enterprise data APIs. 
  • The Enriched Native format includes all new metadata since 2017, such as poll metadata, and additional metrics such as reply_count and quote_count.
  • Activity Streams format has not been updated with new metadata or enrichments since the character update in 2017.
Enterprise data APIs deliver data in two different formats.  The enterprise format closest to the standard v1.1 native format is Native Enriched.  The legacy enterprise data format is Activity Streams, orignially implimented and used by Gnip as a normalized format across X and other social media data providers at the time. While this format is still available, X has only invested new features and developments on the native enriched format since 2017. The enriched native format is exactly how it sounds, it includes native X objects as well as additional enrichments available for enterprise data products such as URL unwinding metadata, profile geo, poll metadata and additional engagement metrics.  

Object comparison per data format

Whatever your X use case, understanding what these JSON-encoded Post objects and attributes represent is critical to successfully finding your data signals of interest. To help in that effort, there are a set of pages dedicated to each object in each data format_._ Reflecting the JSON hierarchy above, here are links to each of these objects:

Parsing best-practices

  • X JSON is encoded using UTF-8 characters.
  • Parsers should tolerate variance in the ordering of fields with ease. It should be assumed that Post JSON is served as an unordered hash of data.
  • Parsers should tolerate the addition of ‘new’ fields. 
  • JSON parsers must be tolerant of ‘missing’ fields, since not all fields appear in all contexts.
  • It is generally safe to consider a nulled field, an empty set, and the absence of a field as the same thing

Enterprise Native Enriched data objects

Native Enriched Tweet object

Interested in learning more about how the Native Enriched data format maps to the X API v2 format? Check out our comparison guide: Native Enriched compared to X API v2

Post Object

When using enterprise data products, you will notice that much of the data dictionary is similar to the native format of Post data, with some additional enriched metadata.  The base level of the native enriched format uses much of the same object names as the X API v1.1 data format.  The Post object has a long list of ‘root-level’ attributes, including fundamental attributes such as id, created_at, and text. Post objects will also have nested objects to include the user, entities, and extended_entities. Post objects will also have other nested Post objects such as retweeted_status, quoted_status and extended_tweet.  The native enriched format will additionally have a matching_rules object.
X Data Dictionary
Below you will find the data dictionary for these ‘root-level’ attributes, as well as links to child object data dictionaries.
Additional Post attributes
X APIs that provide Posts (e.g. the GET statuses/lookup endpoint) may include these additional Post attributes:
Deprecated Attributes

Nested Post objects

In several cases, a Post object will included other nested objects.  If you are working with nested objects, then that JSON payload will contain multiple Post objects, and each Post object may contain its own objects. The root-level object will contain information on the type of action taken, i.e. whether it is a Retweet or a Quote Tweet, and may also contain an object that describes the ‘original’ Post being shared. Extended Posts will include a nested extended object that extends beyond 140 characters, which was used to prevent breaking changes when the update was made in 2017. Each nested object dictionary is described below. Retweets Retweets always contain two Post objects. The ‘original’ Post being Retweeted is provided in a “retweeted_status” object. The root-level object encapsulates the Retweet itself, including a User object for the account taking the Retweet action and the time of the Retweet. Retweeting is an action to share a Post with your followers, and no other new content can be added. Also, a (new) location cannot be provided with a Retweet. While the ‘original’ Post may have geo-tagged, the Retweet “geo” and “place” objects will always be null. Even before the introduction of Extended Posts, the root-level “entities” object was in some cases truncated and incomplete due to the “RT @username ” string being appended to Post message being Retweeted.  Note that if a Retweet gets Retweeted, the “retweet_status” will still point to the original Post, meaning the intermediate Retweet is not included. Similar behavior is seen when using x.com to ‘display’ a Retweet. If you copy the unique Post ID assigned to the Retweet ‘action’, the original Post is displayed.  Below is an example structure for a Retweet. Again, when parsing Retweets, it is key to parse the “retweeted_status” object for complete (original) Post message and entity metadata.
Quote Tweets
Quote Tweets are much like Retweets except that they include a new Post message. These new messages can contain their own set of hashtags, links, and other “entities” metadata. Quote Tweets can also include location information shared by the user posting the Quote Tweet, along with media such as GIFs, videos, and photos. Quote Tweets will contain at least two Post objects, and in some cases, three. The Post being Quoted, which itself can be a Quoted Tweet, is provided in a “quoted_status” object. The root-level object encapsulates the Quote Tweet itself, including a User object for the account taking the sharing action and the time of the Quote Tweet. Note that Quote Tweets can now have photos, GIFs, or videos, added to them using the ‘Post’ user-interface. When links to externally hosted media are included in the Quote Tweet message, the root-level “entities.urls” will describe those. Media attached to Quote Tweets will appear in the root-level “extended_entities” metadata. When Quote Tweets were first launched, a shortened link (t.co URL) was appended to the ‘original’ Post message and provided in the root-level “text” field. In addition, metadata for that t.co URL was included in the root-level ‘entities.urls’ array. In May 2018, we changed this so that the shortened t.co URL to the quoted Tweet will not be included in the root-level “text” field. Second, the metadata for the quoted Tweet will not be included in the “entities.urls” metadata. Instead, URL metadata for the quoted Tweet will be in a new “quoted_status_permalink” object on the root-level (or top-level), so at the same level of the “quoted_status” object. Below is an example structure for a Quote Tweet using this original formatting. 
Extended Posts
JSON that describes Extended Posts was introduced when 280-character Posts were launched in November 2017. Post JSON was extended to encapsulate these longer messages, while not breaking the thousands of apps parsing these fundamental X objects. To provide full backward compatibility, the original 140-character ‘text’ field, and the entity objects parsed from that, were retained. In the case of Posts longer than 140 characters, this root-level ‘text’ field would become truncated and thus incomplete. Since the root-level ‘entities’ objects contain arrays of key metadata parsed from the ‘text’ message, such as included hashtags and links, these collections would be incomplete. For example, if a Post message was 200 characters long, with a hashtag included at the end, the legacy root-level ‘entities.hashtags’ array would not include it.  A new ‘extended_tweet’ field was introduced to hold the longer Post messages and complete entity metadata. The “extended_tweet” object provides the “full_text” field that contains the complete, untruncated Post message when longer than 140 characters. The “extended_tweet” object also contains an “entities” object with complete arrays of hashtags, links, mentions, etc. Extended Posts are identified with a root-level “truncated” boolean. When true (“truncated”: true), the “extended_tweet” fields should be parsed instead of the root-level fields. Note in the JSON example below that the root-level “text” field is truncated and the root-level “entities.hashtags” array is empty even though the Post message includes three hashtags. Since this is an Extended Post, the “truncated” field is set to true, and the “extended_tweet” object provides complete “full_text” and “entities” Post metadata.

Native Enriched User object

The User object contains X User account metadata that describes the X User referenced. 

User Data Dictionary

No longer supported (deprecated) attributes

Example user object:

Native Enriched Geo Objects

Posts can be associated with a location, generating a Post that has been ‘geo-tagged.’ Post locations can be assigned by using the X user-interface or when posting a Post using the API. Post locations can be an exact ‘point’ location, or a X Place with a ‘bounding box’ that describes a larger area ranging from a venue to an entire region. There are three ‘root-level’ JSON objects used to describe the location associated with a Post: place, geo and coordinates Additionally, the native enriched format includes the profile geo enrichment’s derived location within the user object. The place object is always present when a Post is geo-tagged with a place,. Places are specific, named locations with corresponding geo coordinates. When users decide to assign a location to their Post, they are presented with a list of candidate X Places. When using the API to post, a X Place can be attached by specifying a place_id when posting. Posts associated with Places are not necessarily issued from that location but could also potentially be about that location. The geo and coordinates objects only present (non-null) when the Post is assigned an exact location. If an exact location is provided, the coordinates object will provide a [long, lat] array with the geographical coordinates, and a X Place that corresponds to that location will be assigned.

Place data dictionary

Bounding box

Geo object data dictionary

Coordinates object data dictionary

Derived locations

Examples:

Data dictionary: Enterprise

X entities  

Jump to on this page Introduction Entities object   - Hashtag object   - Media object   - Media size object   - URL object   - User mention object   - Symbol object   - Poll object Retweet and Quote Tweet details Entities in user objects Entities in Direct Messages Next Steps

Introduction

Entities provide metadata and additional contextual information about content posted on X. The entities section provides arrays of common things included in Posts: hashtags, user mentions, links, stock tickers (symbols), X polls, and attached media. These arrays are convenient for developers when ingesting Posts, since X has essentially pre-processed, or pre-parsed, the text body. Instead of needing to explicitly search and find these entities in the Post body, your parser can go straight to this JSON section and there they are. Beyond providing parsing conveniences, the entities section also provides useful ‘value-add’ metadata. For example, if you are using the Enhanced URLs enrichment, URL metadata include fully-expanded URLs, as well as associated website titles and descriptions. Another example is when there are user mentions, the entities metadata include the numeric user ID, which are useful when making requests to many X APIs. Every Post JSON payload includes an entities section, with the minimum set of hashtags, urls, user_mentions, and symbols attributes, even if none of those entities are part of the Post message. For example, if you examine the JSON for a Post with a body of “Hello World!” and no attached media, the Post’s JSON will include the following content with entity arrays containing zero items:
Notes:
  • media and polls entities will only appear when that type of content is part of the Post.
  • if you are working with native media (photos, videos, or GIFs), the Extended Entities object is the way to go.

Entities object

The entities and extended_entities sections are both made up of arrays of entity objects. Below you will find descriptions for each of these entity objects, including data dictionaries that describe the object attribute names, types, and short description. We’ll also indicate which PowerTrack Operators match these attributes, and include some sample JSON payloads. A collection of common entities found in Posts, including hashtags, links, and user mentions. This entities object does include a media attribute, but its implementation in the entiites section is only completely accurate for Posts with a single photo. For all Posts with more than one photo, a video, or animated GIF, the reader is directed to the extended_entities section.

Entities data dictionary

The entities object is a holder of arrays of other entity sub-objects. After illustrating the entities structure, data dictionaries for these sub-objects, and the Operators that match them, will be provided.

Hashtag object  

The entities section will contain a hashtags array containing an object for every hashtag included in the Post body, and include an empty array if no hashtags are present. The PowerTrack # Operator is used to match on the text attribute. The has:hashtags Operator will match if there is at least one item in the array.

Media object  

The entities section will contain a media array containing a single media object if any media object has been ‘attached’ to the Post. If no native media has been attached, there will be no media array in the entities. For the following reasons the extended_entities section should be used to process Post native media: + Media type will always indicate ‘photo’ even in cases of a video and GIF being attached to Post. + Even though up to four photos can be attached, only the first one will be listed in the entities section. The has:media Operator will match if this array is populated.

Media size objects

All Posts with native media (photos, video, and GIFs) will include a set of ‘thumb’, ‘small’, ‘medium’, and ‘large’ sizes with height and width pixel sizes.  For photos and preview image media URLs, Photo Media URL formatting specifies how to construct different URLs for loading different sized photo media.

Sizes object 

Size object 

Photo Media URL Formatting

Photo media on X can be loaded in different sizes.  It is best to load the smallest size image that is larger enough to fit into a particular image viewport.  To load different sizes, the Size Object and media_url (or media_url_https) need to be combined in a particular format.  We’ll use the media entity example object already provided for our example in constructing a photo media URL. The media_url or media_url_https on their own can be loaded, which will result in the medium variant being loaded by default.  It is preferable, however, to provide a fully formatted photo media URL when possible. There are three parts of a photo media URL: We take these three parts (base URL, format and name) and combine them into the photo media URL to load.  There are 2 formats for loading images this way, legacy and modern.  All image loads should stop using the legacy format and use the modern format.  Using the modern format will result in better CDN hit rate for the caller, thus improving load latencies by being less likely to have to generate and load the media from the Data Center.

URL object 

The entities section will contain a urls array containing an object for every link included in the Post body, and include an empty array if no links are present. The has:links Operator will match if there is at least one item in the array. The url: Operator is used to match on the expanded_url attribute. If you are using the Expanded URL enrichment, the url: Operator is used to match on the unwound.url (fully unwound URL) attribute. If you are using the Exhanced URL enrichment, the url_title: and url_decription: Operators are used to match on the unwound.title and unwound.description attributes. If you are using the Expanded and/or Enhanced URL enrichments, the following metadata is available under the unwound attribute:

User mention object  

The entities section will contain a user_mentions array containing an object for every user mention included in the Post body, and include an empty array if no user mention is present. The PowerTrack @ Operator is used to match on the screen_name attribute. The has:mentions Operator will match if there is at least one item in the array.

Symbol object  

The entities section will contain a symbols array containing an object for every $cashtag included in the Post body, and include an empty array if no symbol is present. The PowerTrack $ Operator is used to match on the text attribute. The has:symbols Operator will match if there is at least one item in the array.

Poll object

The entities section will contain a polls array containing a single poll object if the Post contains a poll. If no poll is included, there will be no polls array in the entities section. Note that these Poll metadata are only available with the following Enterprise APIs:

Retweet and Quote Tweet details

From the X API perspective, Retweet and Quote Tweets are special kinds of Posts that contain the original Post as an embedded object. So Retweets and Quote Tweet objects are parents of a child ‘original’ Post (and thus double the size). Retweets have a top-level “retweeted_status” object, and Quoted Tweets have a “quoted_status” object. For consistency, these top-level Retweet and Quote Tweet objects also have a text property and associated entities. However, the entities at the top level can differ from the entities provided by the embedded ‘original’ entities. In case of Retweets, new text is prepended to the original Post body. For Quoted Posts, new text is appended to the Post body. In general, the best practice is to retrieve the text, entities, original author and date from the original Post in retweeted_status whenever this exists. An exception is getting X entities that are part of the additive Quote. See below for more details and tips.

Retweets

An important detail with Retweets is that no additional X entities can be added to the Post. Users can not add hashtags, URLs or other details when they Retweet. However, the Retweet (top-level) text attribute is composed of the original Post text with “RT @username: ” prepended.   In some cases, especially with accounts with long user names, the combination of these new characters and the original Post body can easily exceed the original Post text length limit of 140 characters. In order to preserve support for 140 character based display and storage, the top-level body truncates the end of the Post body and adds an ellipsis (“…”). Consequently, some top-level entities positioned at the end of the original Post might be incorrect or missing, for instance in the case of a truncated hashtag or URL entry. This Post,  https://x.com/FloodSocial/status/907974220298125312, has the following Post text:                Just another test Post that needs to be exactly 140 characters with trailing URL and hashtag http://wapo.st/2w8iwPQ #Testing In the above example, both the URL and hashtag were affected. Since the hashtag was completely truncated and the URL partially truncated, these are missing from the top-level entities. You will also notice the additional user_mentions top-level entity coming from the “RT @floodsocial: ” prefix on the text field. However, the Post text and entities in retweeted_status perfectly reflect the original Post with no truncation or incorrect entities, hence our recommendation to rely on the nested _retweeted_status _object for Retweets.

Quote Tweets

Quote Tweets were introduced in 2016, and differ from Retweets in that when you “quote” a Pos you are adding new content “on top” of a shared Post. This new content can include nearly anything an original Post can have, including new text, hashtags, mentions, and URLs. Quote Tweets can contain native media (photos, videos, and GIFs), and will appear under the entities object. Since X entities can be added, the Quote entities are likely different from the original entities. In this example, a new URL and hashtag were positioned at the end of the Quote Tweet. This Post, https://x.com/FloodSocial/status/907983973225160704, has the following Post text:                   strange and equally tragic when islands flood… trans-atlantic testing of quote tweets | @thisuser @thatuserhttp://bit.ly/2vMMDuu #testing In this case, the top-level entities do not reflect the Quote details.  However, the Post text and entities in extended_tweet perfectly reflect the Quote Tweet with no truncation or incorrect entities, hence our recommendation to rely on the nested _extended_tweet _object for Quote Tweets.

Entities for user object

Entities for User Objects describe URLs that appear in the user defined profile URL and description fields. They do not describe hashtags or user_mentions. Unlike Post entities, user entities can apply to multiple fields within its parent object — to disambiguate, you will find a parent nodes called url and description that indicate which field contains the entitized URL. In this example, the user url field contains a t.co link that is fully expanded within the entities/url/urls[0] node of the response. The user does not have a wrapped URL in their description.

JSON example

X extended entities 

Jump to on this page Introduction Extended Entities object Example Tweets and JSON payloads   - Tweet with four native photos   - Tweet with native video   - Tweet with an animated GIF Next Steps

Introduction

If a Post contains native media (shared with the Post user-interface as opposed via a link to elsewhere), there will also be a extended_entities section. When it comes to any native media (photo, video, or GIF), the extended_entities is the preferred metadata source for several reasons. Currently, up to four photos can be attached to a Post. The entities metadata will only contain the first photo (until 2014, only one photo could be included), while the extended_entities section will include all attached photos. With native media, another deficiency of the entities.media metadata is that the media type will always indicate ‘photo’, even in cases where the attached media is a video or animated GIF. The actual type of media is specified in the extended_entities.media[].type attribute and is set to either photovideo, or animated_gif. For these reasons, if you are working with native media, the extended_entities metadata is the way to go. All Posts with attached photos, videos and animated GIFs will include an extended_entities JSON object. The extended_entities object contains a single media array of media objects (see the entities section for its data dictionary). No other entity types, such as hashtags and links, are included in the extended_entities section. The media object in the extended_entities section is identical in structure to the one included in the entities section. Posts can only have one type of media attached to it. For photos, up to four photos can be attached. For videos and GIFs, one can be attached. Since the media type metadata in the extended_entities section correctly indicates the media type (‘photo’, ‘video’ or ‘animated_gif’), and supports up to 4 photos, it is the preferred metadata source for native media.

Example Posts and JSON payloads

Below are some example Posts and their associated entities metadata. Post with four native photos Post with hashtag, user mention, cashtag, URL, and four native photos:
Here is the entities section for this Post:
Only in this ‘extended’ payload below will you find the four (maximum) native photos. Notice that the first photo in the array is the same as the single photo included in the non-extended X entities section. The media metadata structure for photos is the same for both entities and extended_entities sections. Here is the extented_entities section for this Post:

Post with native video

Below is the extended entities metadata for this Post with a video:
When an advertiser chooses to limit video playback to just X owned and operated platforms, the video_info object will be replaced with an additional_media_info object. The additional_media_info will contain additional media info provided by the publisher, such as title, description and embeddable flag. Video content is made available only to X official clients when embeddable=false. In this case, all video URLs provided in the payload will be X-based, so the user can open the video in a X owned property by clicking the link. Here is an example of what the extended entities object will look like in this situation:
As discussed above, here is the entities section that incorrectly has the type set to ‘photo’. Again, the extended_entities section is preferred for all native media types, including ‘video’ and ‘animated_gif’.

Post with an animated GIF

Below is the extended entities metadata for this Post with an animated GIF:

Native Enriched example payloads

Post

Post reply

Extended Post

Post with extended_entities

Retweet

Quote Tweet

Retweeted Quote Tweet

Enterprise Activity Streams data objects

Interested in learning more about how the Activity Streams data format maps to the X API v2 format?
Check out our comparison guide: Activity Streams compared to X API v2
Please note: It is highly recommended to use the Enriched Native format for enterprise data APIs. 
  • The Enriched Native format includes all new metadata since 2017, such as poll metadata, and additional metrics such as reply_count and quote_count.
  • Activity Streams format has not been updated with new metadata or enrichments since the character update in 2017.

Activity Object

Activity Streams is an object schema translation of X’s original data format created by Gnip to ‘normalize the format’ of Post data and other social media data using the third party Activity Base Schema described here. Posts are normalized into the activity streams schema, including: note, person, place and service object types as nested objects.  Posts can have other nested Post activity obejcts for Retweets, or others including twitter_quoted_status, long_object. The base level object type “activity” is similar to the Post base level object of the native enriched format.  Example payloads in activity streams format can be found here.

Data Dictionary

Below you will find the data dictionary for these ‘root-level’ “activity” attributes, as well as links to child object data dictionaries.

Additional Post attributes

Deprecated Attributes

Nested Post activity obejcts

In several cases, a Post object will included other nested Posts.  If you are working with nested objects, then that JSON payload will contain multiple objects, and each Post object may contain its own objects. The root-level object will contain information on the type of action taken, i.e. whether it is a Retweet or a Quote Tweet, and may also contain an object that describes the ‘original’ Post being shared. Extended Posts will include a nested extended object that extends beyond 140 characters, which was used to prevent breaking changes when the update was made in 2017. Each nested object dictionary is described below. Retweets Activity streams format of Retweets includes a nested object with the type “activity” and the verb “note” to represent the original Post being Retweeted.
X quoted status Activity streams format embedded quote Tweets { "id": "tag:search.x.com,2005:222222222222", "objectType": "activity", "verb": "post", "body": "Quoting a Tweet: https://t.co/mxiFJ59FlB", "actor": { "displayName": "TheQuoter2" }, "object": { "objectType": "note", "id": "object:search.x.com,2005:111111111", "summary": "https://t.co/mxiFJ59FlB" }, "twitter_entities": {}, "twitter_extended_entities": {}, "gnip": {}, "twitter_quoted_status": { "id": "tag:search.x.com,2005:111111111", "objectType": "activity", "verb": "post", "body": "console.log('Happy birthday, JavaScript!');", "actor": { "displayName": "TheOriginalTweeter" }, "object": { "objectType": "note", "id": "object:search.x.com,2005:111111111" }, "twitter_entities": {} } } Retweeted Quote Tweet:

Long object

Activity streams format of the extended_tweet

Actor object

The actor object contains X User account metadata that describes the X User which created the activity.

Data Dictionary

No longer supported (deprecated) attributes

Examples:

Location Object

Location objects can exist within the actor object set on the X account level or within the profileLocations object of the gnip object. Location objects have a place object type and can have a name, address, or geo coordinates. Location objects are similar to Geo in native enriched format.

Location data dictionary

profileLocations derived obejcts

Examples

X entities object

For Activity streams format, the twitter_entities is the same format and data dictionary shown on the native enriched format entities object here.

Example:

X extended entities object

For Activity streams format, the twitter_extended_entities is the same format and data dictionary shown on the native enriched format extended_entities object here.

Example:

Gnip object

The gnip object, within Activity streams format, contains the metadata added by the active enrichments, as well as indication of the matching rules for the activity.

Data dictionary

Example:

Activity streams payload examples

Post activity
Reply Post activity
Post activity with long_object
Post activity with twitter_extended_entities
Retweet activity
Quote Tweet activity
Retweetd Quote Tweet activity

Tweet metadata timeline

Jump to on this page Introduction Key concepts X timeline Filtering tips Next steps

Introduction

At its core, X is a public, real-time, and global communication network. Since 2006, X’s evolution has been driven by both user use-patterns and conventions and new product features and enhancements. If you are using X data for historical research, understanding the timeline of this evolution is important for surfacing Posts of interest from the data archive. X was launched as a simple SMS mobile app, and has grown into a comprehensive communication platform. A platform with a complete set of APIs. APIs have always been a pillar of the X network. The first API hit the streets soon after X was launched. When geo-tagging Posts was first introduced in 2009, it was made available through a Geo API (and later the ability to ‘geo-tag’ a Post was integrated into the X.com user-interface). Today, X’s APIs drive the two-way communication network that has become the source of breaking news and sharing information. The opportunities to build on top of this global, real-time communication channel are endless. X makes available two historical APIs that provide access to every publicly available Post: Historical PowerTrack and the Full-Archive Search API. Both APIs provide a set of operators used to query and collect Posts of interest. These operators match on a variety of attributes associated with every Post, hundreds of attributes such as the Post’s text content, the author’s account name, and links shared in the Post. Posts and their attributes are encoded in JSON, a common text-based data-interchange format. So as new features were introduced, new JSON attributes appeared, and typically new API operators were introduced to match on those attributes. If your use-case includes a need to listen to what the world has said on X, the better you understand when operators started having JSON metadata to match on, the more effective your historical PowerTrack filters can be. Next, we will introduce some key concepts that set the stage for understanding how updates in Post metadata affect finding your data signal of interest.

Key concepts

From user-conventions to X first-class objects

X users organically introduced new, and now fundamental, communication patterns to the X network. A seminal example is the hashtag, now nearly universally used across all social networks. Hashtags were introduced as a way to organize conversations and topics. On a network with hundreds of millions messages a day, tools to find Posts of interest are key, and hashtags have become a fundamental method. Soon after the use of hashtags grew, they received official status and support from X. As hashtags became a ‘first-class’ object, this meant many things. It meant hashtags became clickable/searchable in the X.com user interface. It also meant hashtags became a member of the X entities family, along with @mentions, attached media, stock symbols, and shared links. These entities are conveniently encoded in a pre-parsed JSON array, making it easier for developers to process, scan, and store them. Retweets are another example of user-driven conventions becoming official objects. Retweeting emerged as a way of ‘forwarding’ content to others. It started as a manual process of copying/pasting a Post and prepending it with a “RT @” pattern. This process was eventually automated via a new Retweet button, complete with new JSON metadata. The ‘official’ Retweet was born. Other examples include ‘mentions’, sharing of media and web links, and sharing a location with your Post. Each of these use-patterns resulted in new x.com user-interface features, new supporting JSON, and thus new ways to match on Posts. All of these fundamental Post attributes have resulted in PowerTrack Operators used to match on them.

Post metadata, mutability, updates, and currency

While Post messages can be up to a fixed number of characters long, the JSON description of a Post consists of over 100 attributes. Attributes such as who posted, at what time, whether it’s an original Post or a Retweet, and an array of first-class objects such as hashtags, mentions, and shared links. For the account that posted, there is a User (or Actor) object with a variety of attributes that provide the user’s Profile and other account metadata. Profiles include a short biographical description, a home location (freeform text), preferred language, and an optional web site link. Some account metadata never change (e.g. numeric user ID and created date), some change slowly over time, while other attributes change more frequently. People change jobs and move. Companies updates their information. When you are collecting historical Posts, it is important to understand how some metadata is as it was when Posted, and other metadata is as it is when the query is submitted With all historical APIs, the user’s profile description, display name, and profile ‘home’ attributes are updated to the values at the time of query.

“Native” media

X.com and X mobile apps support adding photos and videos to Post by clicking a button and browsing your photo galleries. Now that they are integrated as first-class actions, videos and photos shared this way are referred to as ‘native’ media. Many querying Operators work with these ‘native’ resources, including has:videos, has:images, and has:media. These will match only on media content that was shared via X features. To match on other media hosted off of the X platform, you’ll want to use Operators that match on URL metadata. So, before we dig into the Historical PowerTrack and Full-Archive Search product details, let’s take a tour of how X, as a product and platform, evolved over time. X timeline Below you will find a select timeline of X. Most of these X updates in some way fundamentally affected either user behavior, Post JSON contents, query Operators, or all three. Looking at X as an API platform, the following events in some way affected the JSON payloads that are used to encode Posts. In turn, those JSON details affect how X historical API match on them. Note that this timeline list is generally precise and not exhaustive.

2006

  • October
    • @replies becomes a convention.
    • cashtagsfirstemerge,butusingforstocktickermentionsdoesnotbecomecommonuntilearly2009.cashtags first emerge, but using for stock ticker mentions does not become common until early 2009. Cashtags became a clickable/searchable link in June 2012.
  • November - Favorites introduced.

2007

  • January - @replies become a first-class object with a UI reply button with in_reply_to metadata.
  • April - Retweets become a convention.
  • August - #hashtags emerge as a primary tool for searching and organizing Posts.

2009

  • February - $cashtags become a common convention for discussing stock ticker symbols.
  • May - Retweet ‘beta’ is introduced with “Via @” prepended to Post body.
  • June - Verified accounts introduced.
  • August - Retweets a first-class object with “RT @” pattern and new retweet_status metadata.
  • October - List feature launched.
  • November - Post Geotagging API is launched, providing the first method for users to share location via third-party apps.

2010

  • June - X Places introduced for geo-tagging Posts.
  • August - Post button for websites is launched. Made sharing links easier.

2011

  • May - Follow button introduced, making it easier to follow accounts associated with websites.
  • August - Native photos introduced.

2012

  • June - $Cashtags become a clickable/searchable link.

2014

2015

  • April - A change in X’s ‘post’ user-interface design results in fewer Posts being geo-tagged.
  • October - X Polls introduced. Polls originally supported two choices with a 24-hour voting period. In November, Polls started supporting four choices with voting periods from 5 minutes to seven days. Poll metadata made available (enriched native format only) in February 2017.

2016

2017

  • February - X Poll metadata included in Post metadata (enriched native format only).
  • April - ‘Simplified Replies’ introduced with replied-to-accounts not counted towards 140 characters (“dmw140, part 2”).
2018 2022
  • September 29 - The ability to edit Posts is rolled out to a small test group. Edited Post metadata are added to the Post object where relevant. This includes edit_history and edit_controls objects. These metadata will not be returned for Posts that were created before editable functionality was added. No associated Operators for these metadata. To learn more about how Post edits work, see the Edit Posts fundamentals
Filtering tips Being familiar with the X timeline of when and how new features were added can help you create more effective queries. Here, a query means a filter or rule that is applied by the X historical APIs to the Post archive, using PowerTrack Operators to match on Post JSON. An example is the lang: Operator, which is used to match Posts in a specified language. X provides a language classification service (supporting over 50 languages), and X APIs provide this metadata in the JSON that is generated for every Post. So, if a Post is written in Spanish the “lang” JSON attribute is set to “es”. So, if you build a filter with the lang:es clause, it will only match on Post messages classified as Spanish. The timeline information can also help better interpret the Post data received. Say you were researching the sharing of content about the 2008 and 2012 Summer Olympics. If you applied only the is:retweet Operator to match on Retweets, no data would match in 2008. However, for 2012 there would likely be millions of Retweets. From this you potentially could erroneously conclude that in 2008 Retweets were not a user convention, or that simply no one Retweeted about those Olympics. Since Retweets became a first-class object in 2009, you need to add a ”RT @” rule clause to help identify them in 2008. Both Retweets and Post language classifying are examples of Post attributes with a long history and many product details. Below we will discuss more details of these and other attribute classes important to matching on and understanding X Data.

Recognizing false negatives

When it comes to writing filters, one important takeaway is that the metadata Operators match on all have “born on” dates. If you build a filter with an Operator that acts on metadata introduced after the Post was posted, you’ll have a false negative. For example, say you are interested in all Posts that mention ‘snow’ and share a video. If you build a rule with the has:videos Operator, which matches on Posts with native videos, that clause will not match any Posts before 2015. However, sharing of videos has been common on X long before 2015. Before then users shared links to videos hosted elsewhere, but in 2015, X built new ‘sharing video’ features directly into the platform. For finding these earlier Posts of interest, you would include a rule clause such as url:”youtube.com”. Note, with the Search APIs, there are some examples of metadata being ‘backfilled’ as its index was rebuilt. One good example are cashtags,whichbecamewidelyusedtodiscussstocksymbolsin2009.Afterthecashtags, which became widely used to discuss stock symbols in 2009. After the cashtag operator was introduced in 2015, the Search index was rebuilt, and in the process the symbol entity was extracted from all Post bodies, including early 2006 when $ was used mainly for slang; “I hope it nownow $oon!”.

Identifying and filtering on Post attributes important to your use-case

Some metadata, such as X account numeric IDs, have existed since day one (and are an example of account metadata that never changes). Other metadata was not introduced until well after X started in 2006. Examples of new metadata being introduced include Retweets metadata, Post locations, URL titles and descriptions, and ‘native’ media. Below are some of the most common types of Post attributes that have been fundamentally affected by these X platform updates. Filtering/matching behavior for these depends, in most cases, on which historical Post API is used. To help determine which product is the best fit for your research and use-case, the attribute details provided below include high-level product information.

X Profiles

Since at its core X is a global real-time communication channel, research with Post data commonly has an emphasis on who is communicating. Often it is helpful to know where a X user calls home. Often knowing that an account bio includes mentions of interests and hobbies can lead you to Post of interest. It is very common to want to listen for Posts from accounts of interest. Profile attributes are key to all of these use-cases. Every account on X has a Profile that includes metadata such as X @handle, display name, a short bio, home location (freeform text entered by a user), number of followers and many others. Some attributes never change, such as numeric user ID and when the account was created. Others usually change day-to-day, week-to-week, or month-to-month, such as number of Posts posted and number of accounts followed and followers. Other account attributes can also change at any time, but tend to change less frequently: display name, home location, and bio. The JSON payload for every Post includes account profile metadata for the Post’s author. If it is a Retweet, it also includes profile metadata for the account that posted the original Post. The mutability of a Post’s profile metadata depends entirely on the historical product used. The Search APIs serve up historical Posts with the profile settings as it is at the time of retrieval. For Historical PowerTrack, the profile is as it was at the time the Post was posted, except for data before 2011. For Posts older than 2011, the profile metadata reflects the profile as it was in September 2011.

Original Post and Retweets

Retweets are another example of user-driven conventions becoming official objects. Retweeting emerged as a way of ‘forwarding’ content to others. It started as a manual process of copying/pasting a Post and prepending it with a “RT @” pattern. This process was eventually automated via a new Retweet button, complete with new JSON metadata. The ‘official’ Retweet was born and the action of retweeting became a first-class Post event. Along with the new Retweet button, new metadata was introduced such as the complete payload of the original Post. Whether a Post is original or shared is a common filtering ‘switch.’ In some cases, only original content is needed. In other cases, Post engagement is of primary importance so Retweets are key. The PowerTrack is:retweet Operator enables users to either include or exclude Retweets. If pulling data from before August 2009, users need to have two strategies for Retweet matching (or not matching). Before August 2009, the Post message itself needs to be checked, using exact phrase matching, for matches on the “@RT ” pattern. For periods after August 2009, the is:retweet Operator is available.

Post language classifications

The language a Post is written in is a common interest. Post language can help infer a Post’s location and often only a specific language is needed for analysis or display. (X profiles also have a preferred language setting.) For filtering on a Post’s language classification, X’s historical products (Search API and Historical PowerTrack) are quite different. When the Search archive was built, all posts were backfilled with the X language classification. Therefore the lang: Operator is available for the entire post archive. With Historical PowerTrack, X’s language classification metadata is available in the archive beginning on March 26, 2013. 

Geo-referencing Posts

Being able to tell where a Post was posted (i.e., geo-referencing it) is important to many use-cases. There are three primary methods for geo-referencing Posts:
  • Geographical references in a Post message
  • Posts geo-tagged by the user.
  • Account profile ‘home’ location set by a user
Geographical references in a Post message
Matching on geographic references in the Post message, while often the most challenging method since it depends on local knowledge, is an option for the entire Post archive. Here is an example geo-referenced match from 2006 for the San Francisco area based on a ‘golden gate’ filter: https://x.com/biz/statuses/28311
Posts geo-tagged by the user
In November 2009 X introduced its Post Geotagging API that enabled Posts to be geo-tagged with an exact location. In June 2010 X introduced X Places that represent a geographic area on the venue, neighborhood, or town scale. Approximately 1-2% of Posts are geo-tagged using either method. The available geo-tagging history is dependent on the Historical API you are using. With the Search APIs the ability to start matching on Posts with some Geo Operators started in March 2010, and with others on February 2015. If you are using Historical PowerTrack, geo-referencing starts on September 1, 2011. When the Historical PowerTrack archive was built, all geo-tagging before this date was not included.
Account profile ‘home’ location set by a user
All X users have the opportunity to set their Profile Location, indicating their home location. Millions of X users provide this information, and it significantly increases the amount of geodata in the X Firehose. This location metadata is a non-normalized, user-generated, free-form string. Approximately 30% of accounts have Profile Geo metadata that can be resolved to the country level. As with Post geo, the methods to match and the time periods available depends on the Historical API you are using. Historical PowerTrack enables users to attempt their own custom matching on these free-form strings. To help make that process easier, X also provides a Profile Geo Enrichment that performs the geocoding where possible, providing normalized metadata and corresponding Operators. Profile Geo Operators are available in both Historical PowerTrack and the Search APIs. With Historical PowerTrack, these Profile Geo metadata is available starting in June 2014. With the Search APIs, this metadata is available starting in February 2015. Sharing web page links, photos and videos have always been a fundamental X use-case. Early in its history, all of these actions involved including a URL link in the Post message itself. In 2011 X integrated sharing photos directly into its user-interface. In 2016, native videos were added. Given this history, there are a variety of filtering Operators used for matching on this content. There are a set of Operators that match on whether Posts have shared links, photos, and videos. Also, since most URLs shared on X are shortened to use up fewer of a Post’s characters (e.g. generated by a service such as bitly or tinyurl), X provides data enrichments that generate a complete, expanded URL that can be matched on. For example, if you wanted to match on Posts that included links discussing X and Early-warning systems, a filter that references ‘severe weather communication’ would match a Post containing this http://bit.ly/1XV1tG4 URL. In March 2012, the expanded URL enrichment was introduced. Before this time, the Post payloads included only the URL as provided by the user. So, if the user included a shortened URL it can be challenging to match on (expanded) URLs of interest. With both Historical PowerTrack and the Search APIs, these metadata are available starting in March 2012. In July 2016, the enhanced URL enrichment was introduced. This enhanced version provides a web site’s HTML title and description in the Post payload, along with Operators for matching on those. With Historical PowerTrack, these metadata become available in July 2016. With the Search APIs, these metadata begin emerging in December 2014. In September 2016 X introduced ‘native attachments’ where a trailing shared link is not counted against the 140 Post character limit. Both URL enrichments still apply to these shared links. For other URL product-specific details on URL filtering, see the corresponding articles for more information.