Blog

Delivering data quality at speed and scale

The lure of online retail has always been the ability to scale at speed, selling products globally and transcending the practical constraints of bricks-and-mortar. What makes ChannelSight a market-leading enabler for brands in this quest is our relentless focus on the quality of our product data and delivering this at speed and scale. With brands and retailers constantly adding new products, keeping these two very large, dynamic data sets in sync is a key challenge at the core of providing a world leading – Where to Buy solution.

In 2019 our daily volume of retailer products matched to brand products in our product database, doubled from the start of the year to the end. Our level of automated match improved from just under 90% in January to almost 99% in December. This ability to scale through automation while maintaining quality is at the core of our success and is a strategic asset for us.

Key Challenges

There are many challenges that we have overcome on the journey to achieving this level of automated scale. Product matching at the required speed and scale being the greatest. Factors that contribute to the challenge include the volume of products carried by online retailers as well as international manufacturers and brands having large catalogues including a multitude of product variants. Multilingual and localised product content also throws up challenges when trying to determine matches based on product titles, features and descriptions. The consistency of product identifiers employed varies from market to market and retailer to retailer meaning it can’t be a trusted source of accurate matching. Maximising product recall while maintaining matching precision through use of imagery and all available textual metadata is the ultimate goal.

How we achieved this

ChannelSight employs and has refined a variety of techniques to solve these challenges and deliver a solid data foundation for its market intelligence platform. We’ve taken a “processing value” pipeline approach in optimising our automated matching i.e. ensuring we get the best bang for our buck with every technique used. At the top of this pipeline is product identifier recognition and at the bottom is a highly optimised “suggested match” workflow for our data quality team but it is what bridges the gap in the middle that is our real differentiator. Our pipeline works by applying progressively more expensive techniques to narrow down product matches to achieve the required recall and precision. We combine a series of semantic similarity techniques for the textual data with a visual similarity checks for the imagery to achieve this.

These similarity checks can involve image based hashing and comparison using distance thresholds where similar product imagery is found on the retailer as to that provided by the manufacturer. While at the more complex end we train appropriate machine learning algorithms making use of complex neural networks to aid with this comparison and matching to maximise recall and precision.

Innovation, roadmap & patents

ChannelSight continues to invest in R & D of innovative approaches that enable us to scale our service while maintaining the data quality that our customers expect. In 2019 we had our first patents granted and we have further applications in progress in relation to the application of machine learning and other advanced analytical techniques to serve our speed and scale ambitions.

As well as serving our product data quality requirement these innovations are also being integrated with other parts of our service and indeed throughout our new product roadmap. Our Content Compliance offering due for release in Q2 2020 will utilise much of this research to analyse and provide insights into how well a brand’s entire catalogue of products are being represented in online retailers throughout the world.

Ready To Make A Change?

ChannelSight has worked with a number of brands to drive digital success. Read our recent case study on tado°, a smart home company that embarked on a journey to transform their online strategy with ChannelSight. Book a demo today with our team to see how our ‘Where to Buy’ solution could work for you.

Enda FarrellDelivering data quality at speed and scale