Big Data is about large amounts of digital information. It is, in fact, so large in size that managing it is complex – it is difficult to store or transport and complicated to analyze. Big Data is so big that it poses major challenges to most institutions’ processing capacity as traditional data management tools are not able to cope with the enormity of the data.
Big Data is measured in terabytes, petabytes and exabytes. No surprise then that specialised technology and special storage facilities are needed to handle the huge amounts of data. Big Data storage facilities have to deal with multiple zettabytes! Such vast quantities of data cannot be managed and analyzed with standard IT tools and procedures.
Sources of Big Data
Society constantly produces a lot of data. Social media platforms alone are creating vast quantities of digital material. People upload millions of photos and videos daily, post millions of tweets, exchange billions of messages & comments and create billions of documents daily. It is not unusual for social media to generate hundreds of terabytes of new data daily.
Apart from social networks, other sources that produce vast amounts of data on a daily basis include: business transaction systems, customer databases, financial services, healthcare services & medical records, web clickstreams, scientific research & experiment-results, mobile applications, machine-generated data and real-time data sensors… all the bytes produced daily by billions of people result in Big Data!
Big Data is often characterized using a number of “V’s”. These are explained in our video:
Managing Big Data
Storing and processing Big Data provides great challenges! Clustered servers and powerful algorithms are needed to store and analyse vast quantities of data in a matter of minutes. The capacity required to quickly process huge volumes of data can overwhelm a single server. Therefore, specialized services (that provide Big Data technologies that can cope with the massive processing involved) are essential. Now, cloud computing provides an alternative approach to Big Data analysis, as Big Data systems can now be hosted in the cloud. Cloud-based Big Data is also easily accessible as data does not have to be downloaded.
Big Data provides big challenges, but also big opportunities. It provides opportunities to find insights in a vast variety of fields. Let’s consider a handful of these fields:
● Big Data benefits the business world: It has the potential to provide companies with broader perceptions relating to their customers and competitors. These insights can be used in marketing campaigns and customer services. Data collected from emails and databases, for example, can provide valuable information that companies can use to increase revenues and improve business practices in general.
● Big Data can help healthcare sectors to be more efficient: Data gained from electronic health records, social media, the web and other sources provides valuable information to health workers and medical researchers. The information can be used to study disease trends, to improve medical diagnosis, to develop medicines and to identify disease risk factors.
●Big Data benefits transport and traffic sectors: Analysis of traffic data is used in the development of cars that are almost accident free and much safer to use. Aviation also generates data that provides valuable operational information. (Thousands of flights per day all contribute to vast volumes of data!) Even public transport produces Big Data. For example, the use of GPS technology generates (and makes use of) Big Data.
● Big Data benefits financial fields: Banking and finance make use of Big Data. The New York Stock Exchange alone generates several terabytes of new trade on a weekly basis!
● Big Data assists policing: Facial recognition technologies, CCTV footage, and DNA data bases all involves Big Data and this helps with the prevention and solving of crimes.
● Big Data Benefits leisure and entertainment: It makes it possible for video streaming platforms to analyze viewer preferences and identify prevalent watching patterns. Social media stores vast amounts of data as users post, share, chat and make comments. Social media also uses Big Data to generate billions of personalized recommendations on a weekly basis.