Twitter advanced research techniques 3: tracking #hashtags and archiving tweets

By Colin Meek (@colinmeek) and Judith Townend (@JTownend)

Our third post in the Insite series on Twitter research looks at how to follow Twitter trends marked with #hashtags. If you’re reading this you have probably already clocked how useful #hashtags are, allowing you to isolate specific topics on Twitter.

This is how Twitter defines them:

The # symbol, called a hashtag, is used to mark keywords or topics in a Tweet. It was created organically by Twitter users as a way to categorise messages.

Sometimes they’re using #inajokingmanner to no real purpose, but most of the time they allow you to follow a category of Tweets more easily – about a news story, or an event, for example. The most popular #hashtagged topics will appear in your Twitter sidebar as trending topics (advertisers can pay for their ‘promoted’ topic to trend at the top).

Hashtags are effective advertising and marketing tools, as well as useful aids to follow conversation, live events and breaking news.

• You can follow a specific keyword, using Twitter search. For example, if you’re interested in what people are saying about the television programme Newsnight: http://search.twitter.com/search?q=%23newsnight

• If you click on a #hashtag in any tweet on your Twitter homepage, you will see all the tweets on that topic. For example: http://twitter.com/#!/search?q=%23libelreform

This article on Twitter media examines what makes a hashtag popular. Maybe no surprises to learn that in the main it’s repetition, repetition, repetition. “You need to see a hashtag four or five times before it really clicks,” says the report but not too much: there’s a “sweet spot” after just a few.

For a bit of Twitter history, have a look at the first ever hashtag on Twitter. The ReadWriteWeb reported that according to serial social technology innovator Chris Messina it was #barcamp. It’s logical that a tweet about a technology meetup would be the first topic on Twitter, before it got taken over by the journalists, the spammers and the celebrities.

Tools for tracking hashtags

1. TweetDeck: is a simple, intuitive and immensely powerful Twitter monitoring desktop client that you can download for free. Many, many people admit to ‘not getting’ Twitter until they use TweetDeck. The beauty with TweetDeck is that you can monitor various topics, keywords, lists, people or hashtags in columns. New posts slot in at the top of the columns to create a cascade of posts on your topic. Here TweetDeck is monitoring the Scottish election hashtag #SP11, posts on Syria using #syria and the journalism.co.uk user news feed:

2. Hashtags.org: This site allows a simple tag search, showing you trends and recent tweets on topics. Or go directly to trendistic.com, which supplies the graphs.

3. Twitseek shows you the trending topics of the day, split into different sectors: celebrities, sport, news, stories, people (although topics could fall into more than one category on that list) and includes the URL of the tweets.

Finding old trends & archiving tweets

One problem with Twitter search is that tweets drop out in time – as you’ll see if you go to look for one of your past tweets.

Tweetscan claims to query both Twitter and its own index going back to December 2007 to provide, what it says (it would!) “the most complete and convenient backup available”.

It also allows you to backup your Twitter account if you give it access to your account, a very handy tool, if you have comments and @replies you wish to use in the future, for research or commercial purposes.

“Your last 1000 messages are downloaded from Twitter and combined with results from our historical database. Tweet Scan can retain your tweets to improve later backups.”

Tweetscan also allows offline search and storage.

Alternatives to Twitter Search

Dave Larson has an excellent list of Twitter search engines here, which includes: Searchtastic;SnapBird; TwimeMachine (browse up to 3200 of a user’s old tweets); Topsy; The Archivist (Windows only or browser version); TweetBoard (private alpha); BackTweets (subscription required for full access); FriendFeed; and http://research.ly/ (subscription required, from $9-$99).

Topsy is particularly handy because it notifies bloggers when someone has tweeted their story, via a pingback. But it’s Searchtastic and SnapBird that Larson considers the best for your “Twitter arsenal”.  Searchtastic has one particularly useful feature: you can back up tweets and export to a spreadsheet.

Read Larson’s post in full for a more detailed explanation of each engine and see his chart comparing their functionality here. And if you’re still thirsty for more, Larson has listed 30 more Twitter search options at this link. Enjoy!

• In part one of our Advanced Twitter research series we looked at Twitter search: read at this link.
• In part two we examined the best ways of finding people to follow: read at this link.

** Learn more about sophisticated search techniques on our one-day Advanced Internet Research course, Tuesday 5 July in London.**

Tags: advanced twitter research, hashtags, Online social networking, Real-time web, Technology/Internet, Twitter

Twitter advanced research techniques 1: searching twitter

By Colin Meek (@colinmeek) and Judith Townend (@JTownend)

If you are engaged in serious research on topical issues you can’t ignore Twitter. And, if you can’t ignore Twitter, then you need to grasp the best ways to tap the resource. Understanding how Twitter is used is key to getting what you want without ending up knee deep in posts about coffee breaks and late trains.

Pivotal to success is understanding that the information carried within Twitter is often not the content with value. The real value for the serious researcher are the networks of people you can access, the connections that exist between them and the links people post. You can also locate those posts in time and place.

Like ordinary search engine research – you need to be precise and have a clear idea what content you are after and where it is likely to be found. For example, the screen grab below shows results from a search for posts originating in Cairo before 27 February.

Tactics like this are easy to master. In this insite series we’ll group tactics and tools into a range of research categories including: Finding the right People; #Hashtags and Trends; and Searching the Twitter Archive. This first post explores Advanced Twitter Searching.

Searching Twitter

A lot of confusion exists about advanced searching in search engines and services such as Twitter. Most, including Twitter, offer an ‘Advanced Search‘ form. This gives you clearly useful tools to help you interrogate the Twittersphere. But more useful, flexible and powerful are the ’advanced operators‘ that are typed directly into Twitter’s search box and give you the power to construct much more precise and powerful search strings. This guide explains how to get to grips with these tools. See the insite guide to Google’s advanced operators.

The Twitter guide to each advanced operator is set out below with links to the corresponding search results. The ‘operators’ are highlighted in red.

Operator	Finds tweets…
twitter search	containing both “twitter” and “search”. This is the default operator.
“happy hour“	containing the exact phrase “happy hour”.
love OR hate	containing either “love” or “hate” (or both).
beer -root	containing “beer” but not “root”.
#haiku	containing the hashtag “haiku”.
from:alexiskold	sent from person “alexiskold”.
to:techcrunch	sent to person “techcrunch”.
@mashable	referencing person “mashable”.
“happy hour” near:“san francisco”	containing the exact phrase “happy hour” and sent near “san francisco”.
near:NYC within:15mi	sent within 15 miles of “NYC”.
superhero since:2011-02-26	containing “superhero” and sent since date “2011-02-26″ (year-month-day).
ftw until:2011-02-26	containing “ftw” and sent up to date “2011-02-26″.
movie -scary :)	containing “movie”, but not “scary”, and with a positive attitude.
flight :(	containing “flight” and with a negative attitude.
traffic ?	containing “traffic” and asking a question.
hilarious filter:links	containing “hilarious” and linking to URLs.
news source:twitterfeed	containing “news” and entered via TwitterFee

[Source: Twitter.com]

Operators such as the ‘OR‘ command or double quote marks to designate phrase searching are common to many search engines and can be extremely useful when treated carefully. Turning to the Twitter-specific operators – some are more important for serious research than others. Here are real examples from the best of the bunch.

The ‘people’ operators let you trace posts ‘from‘ and ‘to‘ specific people. Here we used the search term: from:blibrahim to obtain all the recent posts from that user in Cairo.

You can use the ‘to‘ operator to trace all posts sent directly to that user and the @ operator to obtain posts that reference a specific user.

The ‘location’ operators help you identify Twitter users geographically. At the height of the uprising in Libya we used this search to find Twitter users in Tripoli: near:Tripoli

You can use the ‘within‘ operator to search for posts ‘within’ a certain distance of a specific location. For example, your search string can be: near:tripoli wthin:25mi to search within 25 miles of that city. The Twitter search page also gives you drop-down options for searching ‘within’ specific distances.

You can search for answers by searching for questions using the ? operator. Use this operator to find the questions other Twitter users are asking about a specific topic. For example, this query: Ajdabiya ? was used to find posts about that town on the day it was bombed by pro-Gaddafi forces.

Finding links. Discovering other resources and breaking news about a search term is one of the main reasons researchers and journalists turn to Twitter and the best way to do that is to use the filter:links operator. This looks for posts that contain your search term and internet links. For example, here we used this search string: Ajdabiya filter:links.

Searching within specific time periods. To search for post up to a specific date use the “until” operator like this: until:2010-02-28. Equally, you can search for posts after a specific date like this:since:2010-02-28. This is particularly effective when you want to examine the reaction to a specific event before and after it has occurred. Searching by timeframe is not always reliable however. If you use a common search term then Twitter may tell you your specified date is too old.

Combining operators. As with Google, the most powerful way to use advanced operators is to combine them in innovative ways to get to the material you want with the minimum of fuss. In this example, we searched for posts that contain the term “tahrir square” and originate in Cairo. The results contain leads that are exactly on topic.

The next post in this series will look at ways you can identify the right people who are using Twitter (tweeps)  and how you can identify networks and connections between people.

** Learn more about sophisticated search techniques on our one-day Advanced Internet Research course, Wednesday 16 March in London. Book soon though as only two places available at the time of writing. **

Tags: Advanced Techniques, advanced twitter research, Real-time web, Technology/Internet, Twitter, twitter searching