How OrbitSoft Automated Digital Pen Data Acquisition, and Helped a Customer Secure a Market Advantage

Our client is a large American company that sells agricultural tools and equipment through auctions. If a farm needs harvesters, for example, it’s cheaper to buy them through an auction than to purchase them new.

To sell machinery, farmers must invite a specialist from the company to assess the value, inventory, and submit an application for their machinery. The company’s specialists often have to travel to a client’s farm, and it can be in another state, several hundred kilometers from the company’s offices. A specialist must go out into the field, describe their process on paper (this is a federal legal requirement), and then convert this information into electronic form. Taking into account that the number of pieces of equipment can reach a hundred units or more, the process of inventory and its conversion to electronic form can take a lot of time, up to 3 or 4 days.

After the documents are converted into electronic form, specialists arrange an auction on the company website, as well as distribute advertising leaflets. This adds an additional 1 to 2 days.

In total, an auction takes 4−6 days to set up, but it is paramount that this period be reduced to 2−3 days.

For this project, we worked on contract with an American company that develops software solutions.

They recommended the following method: using a digital pen system called Rover INK, with a video camera to record employee records, then sending this «scan» via phone to the company’s office, where it is automatically converted into electronic form.

How this technology works. A camera is installed on a digital pen. Entries are saved to the pen’s memory, and then transferred to a phone via Bluetooth.

Dot grid paper is used for recording entries, with the pattern of dots on this grid being non-repeating and unique, so that when a note is taken, the camera in the pen «understands» where on the sheet of paper the text is being written.

The dot design oo the grid is non-repeating, so the pen «understands» exactly where the writing on the paper occurs. When a note is taken, the pen stores both the «location» data, and the writing itself, in its memory. This data can then be transferred to a phone for sending and processing on the server.

Our task was to automate this process, developing a program that could receive record data from a pen, send this data to a processing center, and then convert the handwritten text into printed text.

For the inventory of special equipment, a form with a dotted grid and fields was developed.

Simplified form view

When a specialist filled in the form fields with a digital pen, they are entered into the pen’s memory in the form of scans, and «linked» to the names of the fields in the following format: inventory number > field name > scan of the record in the field.

This is not a standard pen function. To record this way, we had to modify the pen driver. This task was developed using C++. The source code for the driver was provided by the customer.

Next, when a specialist puts a check mark in the box labeled «inventory completed», the pen automatically sends the inventory to the data center by phone via Bluetooth. The automatic sending function was also not part of the pen driver, so we wrote that as well.

In the field, sometimes there is no cell phone service, so a module was provided that monitors the response from the server regarding receiving inventory data. Sending is repeated until a response is received from the server.

A C++ program was developed to recognize the records. It sends handwritten scans of each inventory field from the database to the OCR library, Google Text Recognition API, receives the recognized text, and writes it to the database.

Processing is performed automatically. As soon as the server receives scans of the handwritten inventory, it immediately sends them for recognition.

After such processing, handwritten scans of inventories and their recognized text versions are stored in the database.

OCR technology does not provide 100% recognition accuracy. Errors are occasionally encountered in the recognized text.

To avoid mistakes, we developed an interface that displays the handwritten fields next to the recognized text of the fields to the operator for each inventory, for cross-checking and confirmation.

The operator checks everything line by line, and crosschecks and confirms the data. If the records are incorrectly recognized, the operator corrects them, and then confirms compliance.

Fully verified inventories are automatically sent electronically to the auction checkout team.

The auction registration team receives data in digital form almost immediately after the field technician completes the inventory for each device. There is no need to wait for a technician to record information about every piece of equipment, transfer the documents into electronic form, and then transfer this data to the team.

As a result, auctions are now registered within one, or a maximum of two, days. This is 4 times faster than before automating this process.