Case: In two weeks, we developed a service for calculating results for an educational portal

The customer is a platform for organizing distance learning. Client companies buy a subscription to the platform’s services, and save courses in the constructor for training staff, for example: upgrading professional skills, improving communication skills, and training new employees.

Employees watch lectures and webinars, practice skills on simulators, and take tests to control and improve knowledge. The test results and the course completion rating are seen by client companies in their personal account on the platform.

The platform has a client with a special request. It was convenient for him to work with the courses, but it turned out to be insufficient for him to just see the test results and the rating of employees in his personal account. He wanted to post them on his website. To do this, the data had to be sent not to the platform, but to the company’s server.

Making changes to the platform itself is a long and expensive process. It was necessary to quickly develop a separate data processing service that stores user statistics, calculates the results for all tasks, and sends them to the client’s server. The platform turned to OrbitSoft for a solution.

We discussed the requirements for the service with the customer and formulated the terms of reference. We had to implement:

• HTTP API endpoint — for receiving and processing webhooks
• Data base — for storing results
• Task queue handler — to send results after processing webhooks
• Integration with Telegram — to send notifications

We suggested developing a service in Django, a Python framework. It’s the perfect fit for several reasons:

• The service needs to be developed quickly, otherwise the platform client will go to competitors. The Django framework allows the use of ready-made solutions without having to develop extra code. The development of the service, including approval and testing, eventually took us just two weeks.
• The service does not require a web interface that the user interacts with. Its task is to fill in the missing functions between the platform and the server of the client company. Django is specifically for back-end development.
• The service needs a standard API and database interaction. Django has ready-made solutions for these tasks. We used the Django-rest-framework library to connect the REST API and the Django ORM tool to work with the database.

How the service works:

Method 1:

• Receives data from the platform server through a webhook. A webhook is an automatically generated HTTP request that speeds up the exchange of data. The service needs to get the user test results from the platform server. But it doesn’t know if the user passed the test or not. When using standard queries, the service periodically asks the platform: are there any results? If the test passed at the time of the request, the server sends back data. A webhook is like subscribing to an event. Once the user has passed the test, the server automatically creates an HTTP call and sends the data to the service. Through a webhook, the service receives data faster, and doesn’t overload the server with frequent requests.

• When the data is received, method 1 checks it against the required structure, and stores it in the database.
• If a problem occurs while processing the request, method 1 passes the error to method 3. If all goes well, it generates a task to send the results to the server of the client company for method 2.

Method 2:

• Query the database for the user’s summary result for all the tests they have completed.
• Prepares and sends updated data to the server of the client company.

• If an error occurs while accessing the database or sending data to the client server, method 2 passes it to method 3

Method 3:

• Sends error messages to the Telegram server, which are sent by methods 1 and 2.

To store intermediate results, we created a database. When the user completes the test, method 1 sends the information to the database. The user performs another task, the results are again sent to the database, and the data is supplemented.

Method 2 requests the total result for the user from the database and sends it to the platform client server.

Through the webhook, the data processing service receives information from the server that an event has occurred on the platform: the user has passed the test. Method 1 is responsible for receiving the webhook: it accepts a request for the service URL.

Main stages of work:

1. Describes the User and Result database models.

2. Added the DRF library and description

• Data serializers. These are objects that describe the structure of the received data and the rules for their validation. They check for compliance with specified conditions.

• View method — a method for processing an HTTP request that includes access control rules.

3. Set up routing sets the URL by which requests will be received, and associates it with the view handler

4. Adds error trapping at key stages: writing to the database, validating the data structure

Method 2 is responsible for transferring the total result of the user to the server of the client company. To do this, it queries it in the database, forms an object of the required structure, and sends it to the server.

Sending the results is not performed immediately after receiving the data. This is a delayed process. When the platform server sends a webhook, method 1 processes it and forms a task for method 2, which sends an updated result to the server. The task is queued from the same tasks. Their implementation is implemented through Celery.

The message broker performs the function of a queue manager: it receives and sends messages between individual modules. For this project we used Redis.

Celery pulls tasks from the queue in order, and executes them. It queries the database for the total result for the user, and then sends it to the server of the client company. The Celery handler has multiple executors and runs in the background.

If an error occurs during execution, Celery records it in the log and tries several more times to complete the request within one minute. The service sends error notifications in the form of a report to the Telegram channel so that the administrator can respond in a timely manner.

One of the wishes of the customer was to set up error notifications. These may occur when the service accesses the internal database, when trying to send a request to the server of the client company, or there may be an error in the data structure transmitted from the platform. If the administrator is notified, he can take action in a timely manner. Django allows the customization of the interaction of the service with third-party programs, such as Telegram. To send error messages, we developed a helper bot.

Thanks to the Django framework, we developed a data processing service in less than 2 weeks from the start of obtaining the TOR. Our results:

• The service solves the customer’s problem: it performs calculations, saves data in the database, then transfers the final result to the server of the client company.
• The service accumulates data of 50,000 users from the client company. They are stored in an internal database and are used when updating
• In case of an error in data transfer, the system sends a notification to the administrator in Telegram.