Turning 41 retailers' shelves into one comparable catalogue

The problem

Shoppers in Egypt, Saudi Arabia and the UAE buy groceries across several supermarket apps, and each app describes the same product its own way. One bottle of cooking oil appears under an Arabic name in one store, an English abbreviation in another, with the pack size written as 1L, 1 لتر or 1000 ml. Barcodes are often missing. A price comparison is only as good as the matching underneath it.

What we built

Fattuta is Vega Data's own price-intelligence product. Its data pipeline runs in five stages every night.

• Collect. Store-specific crawlers, one adapter per e-commerce platform, request prices at a polite pace between 02:30 and 05:30 Cairo time, with a full catalogue crawl each week. Failed runs retry after 15, 60 and 180 minutes.
• Normalise. Arabic and English names are cleaned, pack sizes parsed into standard units and unit prices calculated. Search understands common Arabic spelling variants.
• Match. A three-stage entity-resolution pipeline groups listings into products.
• Classify. Every product is placed in Fattuta's bilingual taxonomy of 19 departments and 391 categories, each mapped to GS1 and Google product categories. The same model normalises the 3,338 category names the stores use.
• Serve. Prices, baskets and history reach the web app, the Android and iOS apps and the API. Price history stores only changes, which keeps years of history cheap to query.

Matching in three explainable stages

We chose a staged design so that every merge can be explained to a retailer and checked by a person.

• Stage 1: barcodes. Identical barcodes merge automatically.
• Stage 2: blocking and bilingual scoring. Listings are grouped by brand and pack size, then Arabic and English names are compared with fuzzy similarity. Scores of 0.88 or more merge automatically; scores between 0.75 and 0.88 go to a human review queue.
• Stage 3: semantic matching, in development. Embedding-based matching for the long tail, released once a labelled set of reviewed pairs shows it beats stage 2.

Running it

Each night's run is monitored end to end with Prometheus, Grafana, Loki and Alertmanager, and database restores are rehearsed on a schedule. In a typical week the platform completes about 190 successful store runs, and 97% of all listings have a price confirmed within 36 hours.

Results

• 41 retailers live: 18 in Egypt, 11 in Saudi Arabia and 12 in the UAE.
• About 465,000 store listings resolved into about 343,000 products.
• More than 50,000 products priced at two or more stores, the comparisons shoppers come for.
• 97% of spot-checked compared products placed in the right category, and 90% in the right sub-category.
• Saudi Arabia and the UAE run on the same pipeline as Egypt.