One of the biggest challenges we have faced at tendmill while building products for the procurement food industry has been the quality of the available data going into our systems. The data contains lots of free text, but it is also inconsistent, incomplete and unstandardized.

‍

Initially, we tackled these problems with a clever system of regexes, which worked and still works well (about 99% coverage for Swedish). The problem is that this solution requires continuous maintenance and requires lots of programming hours when diverting into new industries and languages. To ease these scalability issues, we started implementing solutions using Machine Learning and Natural Language Processing. These efforts have shown to be very fruitful, and with the help of our mastermind Shweta Misra (who started working with us as a summer intern and now as part-time while finalizing her Master’s at KTH), we are approaching the same accuracy as our handcrafted regex solution, but for any language!

‍

As a science nerd, it is very rewarding to see how state-of-the-art research can yield a tremendous impact and create a more efficient society.