The business task is to increase conversion from content display. <\/h2><\/div>\n\n\n
One of the ways to make money for an online business is to manage content so that users take targeted actions as often as possible. Content in this case is a general designation. It can mean:<\/p>\n\n\n\n
- \n
- videos<\/li>\n\n\n\n
- news<\/li>\n\n\n\n
- goods or services<\/li>\n<\/ul>\n\n\n\n
The target action also depends on the business. In one case it could be a review of a certain number of videos. In another \u2013 it might be subscription to trial lessons from the mailing list.<\/p>\n\n\n\n
Content management requires data on how the user works with the content. Maybe one skips the products cards with the highest margins for the company, or studies them, but doesn\u2019t put them in the basket. All this is needed to understand how and what to show to the user.<\/p>\n\n\n\n
A block of data gives us little information by itself. It needs to be saved and analyzed. For example, you can:<\/p>\n\n\n\n
- \n
- collect statistical reports according to data segments and metrics<\/li>\n\n\n\n
- compare these metrics for different periods<\/li>\n\n\n\n
- group data by categories: countries, cities, device types, or time of day<\/li>\n<\/ul>\n\n\n\n
Without such details, it\u2019s difficult to determine why some content is not working as planned, and what to do to solve the problem.<\/p>\n\n\n\n
When there is not a lot of data, it can be processed manually, or with the help of simple programs. But with the growth of the user database, the amount of data also increases, so it becomes no longer possible to process, let\u2019s say, a billion pieces of data in this way.<\/p>\n\n\n\n
To process a larger load, data collection and analysis systems are being designed. The main requirement for them is to withstand high loads.<\/p>\n\n\n
OrbitSoft Experience: how data collection and analysis system operates <\/h2><\/div>\n\n\n
We have experience in running systems that handle such high loads. Therefore, using the example of one of our projects, we will show our approach to the development.<\/p>\n\n\n\n
About the Project.<\/strong> Every day the ad network registers events: bids that users place, ad impressions, ad clicks. In order for analysts to work with data, it needs to be brought into a readable format – processed and loaded into a program with a clear interface.<\/p>\n\n\n\n
System load.<\/strong> In April 2021 the ad network system processed:<\/p>\n\n\n\n
- \n
- 2,4 billion impressions<\/li>\n\n\n\n
- 408 million clicks<\/li>\n\n\n\n
- 1240 conversions<\/li>\n<\/ul>\n\n\n\n
Analysis. To design the system, we looked at the type and amount of data, the predictable load increase, the results to be obtained, and budget and resource limitation. After that, the development began.<\/p>\n\n\n
\n\n\n Algorithm of data collection and analysis in the ad network <\/h3>\n
- \n
- \n \n Clicks, bids, impressions <\/div>\n <\/li>\n
- \n
\n File creation and compression <\/div>\n <\/li>\n- \n
\n Statistics calculation <\/div>\n <\/li>\n- \n
\n Recording of the results <\/div>\n <\/li>\n- \n
\n Data is transferred to MySQL <\/div>\n <\/li>\n <\/ul>\n <\/div>\n <\/div><\/div>\n\n\nSystem operation scheme:<\/strong><\/p>\n\n\n\n
- \n
- The advertising platform registers events with Apache Kafka.<\/li>\n\n\n\n
- Collector Service monitors the queue of events in Apache Kafka, adds more data to event information, and saves event information to .csv files.<\/li>\n\n\n\n
- After completing this process, Collector Service compresses it and loads this file into HDFS, in a special directory for input data.<\/li>\n\n\n\n
- Every 5 minutes Stats Service checks for new raw files in the HDSF input directory.<\/li>\n\n\n\n
- Apache Hadoop calculates statistics based on input data and writes the results to files in a special directory on HDFS.<\/li>\n\n\n\n
- Stats Server picks up the results of calculating statistics from HDFS and exports the data to MySQL.<\/li>\n\n\n\n
- Stats Server moves the processed input files to an archive located also on HDFS. Files are grouped by day – this is necessary so that statistics can be recalculated for a certain period if there were any errors in the scripts.<\/li>\n<\/ul>\n\n\n\n\n
![\"\"](\"https:\/\/orbitsoft.com\/blog\/wp-content\/uploads\/2021-05-16_22-39-49.png\")
\n <\/div>\n Such a system can be used both in hardware and in the cloud.<\/figcaption>\n <\/figure><\/div>\n\n\n Results.<\/strong> All components of the architecture are horizontally scalable and provide fault tolerance.<\/p>\n\n\n\n
Component<\/strong><\/td> What it provides<\/strong><\/td><\/tr> Apache Kafka Distributed Message Broker.<\/td> Processes up to several hundred thousand events per second.
Messages get through even if one of the servers goes down
Adapts to the growth of the load: it\u2019s easy to add new servers<\/td><\/tr>Collector Service daemon in Go<\/td> Processes tens of thousands of messages per second
Adapts to the growth of the load: it can run several copies of the service in parallel and distribute work between them<\/td><\/tr>Hadoop Distributed File System<\/td> Stores data securely
Gives the data that was asked for Records what was sent to the system.
Replicated blocks across data nodes will not lose data<\/td><\/tr>Stats Server daemon in Java<\/td> Fast computation speed with Map Reduce computation model: the data array is divided into parts; each part is processed simultaneously and gathered in to one<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n
- \n
- \n
![\"\"](\"https:\/\/orbitsoft.com\/blog\/wp-content\/uploads\/friendship.png\")
\n <\/div>\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"