Technologoes that powers Twitter

I am often asked the questions regarding technologies and based on the same I found that people are really curious (especially the tech fellows) that which technologies are used by several remarkable companies like FaceBook, Twitter and many more. It made me to find more about the same and share with the audience of the blog.

It is easy to find out the technologies used at Twitter and get more details about the same. It is available at http://engineering.twitter.com/.

Twitter is considered as one of the top 10 sites people used to visit. It is the one bring in (and importantly made popular) the concept of MicroBlogging. A post with only limited characters, to express your current status, feelings, emotions, important notes and many more. It teaches one to say more with less words (it is an art in itself of course, but we may talk about the same later). And as the millions of people uses it on regular bases and twitting a lot, the site is flooded with traffic. So let us try to take a look at the stuff which is making it happen and keeping it functional.

Twitter (and a lot of Web Developers) is fond of open source technologies. When talking about open source technologies, the first name comes to our mind is Unix flavors of Operating system, which made revolution in the Computer and Networking field. Linux(the UniX by LINUs Torwalds) is one of the most popular and robust flavors of Unix. Twitter uses Linux OS deliberately and as they have mentioned in the tweet(original date 25 Aug 2012) that on the 21st birthday of Linux, they are proud to support the Linux Foundation.

Twitter uses MySQL database effectively (Ref: http://engineering.twitter.com/2012/04/mysql-at-twitter.html. As we know the community edition of MySQL is available for free and is feature rich too. That is the reason the most of the programmers working with open source web development usually prefer MySQL as the database management system (and hence the entire development segment is known as LAMP – Linux Apache Mysql Php on servers OR may be WAMP – Windows Apache Mysql Php – on development PCs).

To manage the files amongst the development system there are various repositories and code and subversion management systems available. Twitter likes GIT. And the MySQL work and development/changes/customization (sharable) by them is available at https://github.com/twitter/mysql.

Memcached is one of the other open source tool used by Twitter. Memcached is one of the best tools available to optimize and cache MySQL queries. It is really important for Twitter to optimize/cache frequently used queries considering the ever growing and large scale traffic it receives. You can find details about use of Memcached at Twitter, and its impact on the performance of the website. Of course Twitter needs to customize the tools for its need (as we had seen with MySQL), and that is why Twemcache came to existence.

Hadoop, an Apache project (http://hadoop.apache.org/) is also used at Twitter. http://engineering.twitter.com/2010/04/hadoop-at-twitter.html page tells you more about the same. As one of the sponsors of Apache foundation, Twitter uses other Apache tools/technologies like Cassandra (http://cassandra.apache.org/), Lucene (http://lucene.apache.org/) and Pig (http://pig.apache.org/). You can find more details use of the same at Twitter at the following links:
http://engineering.twitter.com/2010/07/cassandra-at-twitter-today.html
http://engineering.twitter.com/2010/10/twitters-new-search-architecture.html

Of course this is not the end, Twitter also develops several tools to fulfil their needs and then makes them available by making them open source. The list includes but not limited to the tools like Lago (http://engineering.twitter.com/2012/06/building-and-profiling-high-performance.html), Scalding (https://github.com/twitter/scalding) and others.

The major benefit of the open source tools are not only they are available for free but also they are customizable. If twitter is taking the advantage of such tools then why not the other developers.

We may try to come with another installment of the same article, covering more open source and/or interesting tools used at Twitter (and probably other major websites).

Share us your thoughts/remarks about the article, we love to hear from you.