Building an NLP-Powered Search Engine for an E-commerce Marketplace
We built e-commerce site search software that uses natural language processing to read each query, predict the category the shopper meant, and re-rank results so relevant products land at the top of the page.
- → +19% search-driven revenue per session in the first quarter after rollout
- → 91% top-1 category prediction accuracy on live queries
- → ~2.4M search queries handled every month
- → 58% fewer zero-result and mixed-category searches
Project Details
The client runs a multi-vendor marketplace with about 1.3 million active listings across 3,400 categories, four levels deep. Keyword matching through Algolia was fast, but it treated “sky” in a mascara name and “sky” in a light projector as the same thing, and a growing share of shoppers left the search results page without a click. They wanted search relevance that understood intent, not just tokens, without ripping out the Algolia index they already paid for.
Business Challenge: Transforming Outdated Search Into Intelligent Product Discovery
The marketplace had grown faster than its search function. Sellers added roughly 40,000 new listings a month, each tagged by hand into whatever category the seller picked, and the site search bar sat on top of a plain keyword index. Effective site search was the gap between listing a product and finding it. Volume was never the issue. About 2.4 million searches a month came through, and search-driven sessions converted at nearly three times the rate of browse-only sessions, so every bad results page cost real money.
The failure pattern showed up in the search data within a week of the audit, and the same search queries came back again and again. A query like “sky high” returned mascara, a light projector, a backpack, and a speaker, in that order or any other. A search for “laptop stand” surfaced laptop sleeves and laptops, because both matched a token. Ecommerce search built on tokens alone cannot tell the two apart. About 38% of search results pages mixed three or more top-level categories, about 9% of searches ended with zero results, and 21% of site visitors who used search left from the results page without clicking anything.
Customer satisfaction scores for the search step of the customer journey sat at 3.1 out of 5, the lowest of any step the client measured. Support logged about 1,100 tickets a month tagged “can’t find product,” and the merchandising team, short on technical resources of its own, spent an estimated 30 hours a week hand-tuning synonym lists that broke the next time sellers uploaded a batch.
Accurate Intent Recognition Beyond Keywords
Keyword search treated every token as equal, so user intent never entered the ranking. Accessories, complementary products, and partial phrase matches outranked the product the shopper actually named. Nearly 4 in 10 results pages pulled from three or more unrelated categories.
Personalization at Scale
Every shopper saw the same ranking. A buyer who had browsed camera gear for a week got the same “lens” results as a first-time visitor, and the existing stack had no way to fold purchase history, search history, or any other customer data into ranking at 2.4 million queries a month. Third-party tools that promised personalized shopping experiences wanted a full re-index the client could not afford.
Handling Complex Product Catalogs
3,400 categories across four levels, tagged inconsistently by 12,000 sellers. Roughly 18% of listings sat in the wrong branch of the tree, meaning even a perfect query couldn't find a product filed under the wrong parent.
Technologies Behind the Site Search Engine Software
Our AI-Powered Ecommerce Site Search Software Solution
One question shaped the architecture: what is the single category this shopper meant? If the model could answer that with confidence, everything downstream (ranking, facets, personalization) got easier. So the core of the ecommerce search system is a category predictor, and the rest of the intelligent search stack is built around trusting its answer.
Intent lived in the query. Structure lived in the product catalog. Algolia matched one against the other by tokens and stopped there. A shopper typing “sky high macara mayabelline” (a real query from the logs, spelling included) wanted one mascara from one brand, and nothing in the old pipeline could say so.
We kept Algolia. It handles instant search results, typo tolerance, and the first pass of candidate retrieval in about 20 ms, and the client’s team already knew how to operate it. Many leading brands run their ecommerce search on it for exactly that reason. We added a Python service behind Flask that takes Algolia’s top 200 candidates and the raw query, runs both through the NLP layer, and returns a re-ranked list with a predicted category attached. Added latency stays under 140 ms at p95.
The NLP layer is two models. A transformer classifier, fine-tuned in PyTorch with HuggingFace on about 480,000 query-to-click pairs from six months of search history, predicts the category at each of the four levels. A second model scores each candidate product against the query using semantic search capabilities rather than token overlap, so “laptop stand” and “laptop sleeve” finally land far apart. An older TensorFlow 2.3 image classifier, paired with Rekognition labels, handles the seller side: when a listing goes up, the system proposes a full category path, and the seller confirms or corrects it.
Intelligent Query Processing
Every one of the 2.4 million monthly search queries is normalized, spell-corrected against the catalog vocabulary, and mapped to a predicted category before ranking starts. Misspelled search terms like “macara” resolve to mascara in the same pass.
Semantic Search Capabilities
Products are matched on meaning. The model embeds the query and each candidate listing, so “running shoes for flat feet” finds stability trainers even when no listing uses that phrase, and partial matches drop below the products that fit the full intent.
AI-Powered Product Ranking
Algolia re-scores candidates with the query embedding, the predicted category, and listing quality signals. AI-powered search returns accurate search results even when the query has three misspelled words. One category per query, unless the confidence score says the query is ambiguous.
Real-Time Personalization
Search history and purchase history from the last 30 days feed a per-session boost, so the same search tools deliver relevant results for two different shoppers typing the same word. A shopper who bought a camera body last week sees lenses ranked above bags for “canon,” while a first-time visitor sees the neutral order.
Visual Search and Auto-Categorization
Sellers upload a photo, and the image model plus Rekognition labels propose the top 5 category paths, four levels deep. Around 86% of new listings now accept the first suggestion, which is how catalog tagging errors fell from 18% to under 5%.
Search Analytics Dashboard
A Streamlit dashboard for the merchandising team shows zero-result queries, low-click queries, category confidence distributions, and A/B results per ranking change. Search analytics that used to take a weekly SQL export now refresh every hour.
Agile Development Methodology
Project Journey
The 9-month project ran in two-week sprints, and the first three of them went to data work rather than models: pulling six months of query logs, joining them to click and purchase events, and cleaning a category tree that had 3,400 nodes and about 210 duplicates. Discovery settled the questions that shaped the search algorithm, including how confident the classifier had to be before the results page showed a single category, and what to do when a query legitimately spanned two.
How the AI-Powered Web Site Search Software Works
- Autocomplete suggestions appear from the third character, grouped by predicted category, so “sky” already separates mascara from projectors before Enter.
- The keyword index returns up to 200 matching listings in about 20 ms, with typo tolerance on, as the first pass.
- The transformer classifier reads the query and outputs one category path with a confidence score. Above the cutoff, the results page shows only that branch.
- Each candidate is scored against the query on meaning, then adjusted by category fit, listing quality, and the session's recent search and purchase history.
- The React storefront shows the re-ranked list with facets drawn from the predicted category only, so faceted search filters stay relevant.
- Click and purchase events flow back into the search queries log. The models retrain monthly, and the dashboard flags any query cluster where confidence drops.
Development Process Flow
AI development for search works better iteratively than as a big-bang release. The team reviewed real re-ranked results 19 times and rolled the optimized search out gradually to 5%, 25%, 50%, and finally 100% of shoppers. Four reviews changed ranking weights, and one removed synonym-based query expansion after it improved recall but reduced click-through.
How we Delivered Search Relevance Ranking Software
- We define the project goal together, agree on priority features, and set a realistic delivery date and budget.
- We build a ranked list of everything the product needs, starting with what matters most to the business.
- Work is broken into 2-week cycles. At the start of each, we select the next set of features to deliver.
- The team builds, tests, and integrates features throughout the sprint.
- At the end of every sprint, you see working software and give feedback that shapes the next cycle.
- Each sprint produces a shippable piece of the product. We review what worked, adjust, and move forward.
Timeline
Five phases, clearly defined
Discovery & Data Audit
- Pulling six months of query logs and joining them to 480k click and purchase events
- Auditing the 3,400-node category tree and merging about 210 duplicate branches
- Setting the confidence cutoff for showing a single category per query
Model Prototyping
- Benchmarking a fine-tuned transformer against a TF-IDF baseline on 2,600 labeled queries
- Testing sentence embeddings for candidate re-ranking against raw Algolia order
- Choosing Flask over a heavier framework to keep the re-ranking hop under 140 ms
Agile Development
- Building the Flask re-ranking service on top of the existing Algolia index
- Shipping the category chip, grouped autocomplete, and seller categorization flow
- Adding the personalization layer and the Streamlit search analytics dashboard
QA & Testing
- Running the 2,600-query regression suite against every ranking change
- Load testing at 4x peak volume, roughly 190 queries per second
- A/B testing the new search experience on 5%, 25%, and 50% of traffic
Launch & Support
- Rolling out to 100% of shoppers after the 50% cohort held its lift for three weeks
- Retraining both models monthly on the rolling query log
- Reviewing low-confidence query clusters with the merchandising team every sprint
UI/UX Design: Intuitive Search Experience for Ecommerce Users
Trust shaped the search UX. We reviewed 40 recorded sessions and found shoppers rarely went past the eighth result, often refining the same query instead. That led to a wider search bar, thumbnail-based autocomplete, and suggestions grouped by category.
On the results page, a category chip shows where the system placed the query and lets shoppers switch categories in one click. Category-specific facets reduced the average filter count from 31 to 9, while mobile users got a single-column layout with the chip pinned on scroll.
For sellers, the listing form suggests a category from the photo and title, shows five alternatives, and asks for confirmation. Median categorization time dropped from about four minutes to under 40 seconds.
Results
Before
- ✕~38% of search results pages mixed three or more top-level categories
- ✕About 9% of searches ended in zero results, and 21% of searchers left the results page without a click
- ✕One ranking for every shopper, with no use of search history or purchase history
- ✕~18% of listings filed in the wrong category by sellers, hand-tagging through a 3,400-node tree
- ✕Synonym lists maintained by hand, 30 hours a week, and no search analytics beyond a weekly export
After
- ✔91% top-1 and 97% top-3 category prediction accuracy on live queries
- ✔58% fewer zero-result and mixed-category searches, and results page exits down from 21% to 12%
- ✔+34% search-to-click rate with personalized search results from a 30-day session profile
- ✔Under 5% of new listings mis-categorized, with 86% of sellers accepting the first suggested path
- ✔One dashboard for zero-result queries, confidence, and A/B results, refreshed hourly
Impact of AI-Powered Product Research Software After Implementation
Verified Reviews
Our Reputation on Top Platforms
LITSLINK holds a 4.8 rating on top platforms. Clients writing about our AI and web development work most often mention deep expertise in machine learning and natural language processing, steady communication across long engagements, and a willingness to challenge a requirement when the data disagrees.
Have a Search Project in Mind?
Need ecommerce site search software? Tell us about your product catalog, your monthly query volume, and where relevance breaks today, and our specialist gets back to you within 48 hours.
Thank you for your message. It has been sent.
