Taming Big Data with Apache Spark and Python
Now in the USA more than 55% of companies work with big data, in Europe and Asia - about 53%. In the last five years alone, the distribution of Big Data in business has tripled.
According to NetApp, by 2003 the world had accumulated 5 Ebytes of data (1 Ebyte = 1 billion GB). In 2015 - more than 6.5 ZB, and then only 17% of companies around the world used big data. Most of the data will be generated by the companies themselves, not their customers. In this case, an ordinary user will communicate with various devices that generate data about 4,800 times a day.
Which industries are already using Big Data?
Public administration. Big data exploration and analysis helps governments make decisions in areas such as health, employment, economic regulation, crime and security, emergency response;
Industry. The implementation of Big Data tools helps to increase the transparency of industrial processes and introduce “predictive production”, which allows more accurately predicting the demand for products and, accordingly, planning the use of resources. See more examples in Taming Big Data with Apache Spark and Python book.
Medicine. The huge amount of data collected by medical institutions and various electronic devices (fitness bracelets, etc.) opens up fundamentally new opportunities for the healthcare industry. Big data helps to find new drugs, make more accurate diagnoses, select effective treatments, and fight pandemics;
Retail. The development of network and e-commerce is impossible to imagine without Big Data-based solutions - this is how stores personalize assortment and delivery;
Internet of Things. Big Data and the Internet of Things are inextricably linked. Industrial and household appliances connected to the Internet of Things collect a huge amount of data, based on the analysis of which the operation of these devices is subsequently regulated;
Real estate market. Developers use Big Data technologies to collect and analyze the entire array of information, and then present the most interesting options to the user. Already now, the future buyer can see the house he likes without the seller;
Sport. Football clubs use big data to select the most promising players and develop an effective strategy for each opponent.
IoT solutions
An IoT solution from the field of so-called precision farming is when special weather stations that are in the fields collect data (temperature, humidity) with the help of sensors and send them to the IoT platform using transmitting GSM radio modules. It uses big data algorithms to process the information collected from the sensors and build a high-precision hourly weather forecast. The client sees it in the interface on a computer, tablet or smartphone and can quickly make decisions