Rakuten Advertising • Jan 2025 - Ongoing
Intelligent Search for Custom Reports — Natural Language Search & AI
Advertisers on Rakuten Advertising's platform create custom reports monthly (sometimes weekly) to track campaign performance across 170+ metrics. Manually building these reports took 15-20 minutes of clicking through dropdowns, selecting date ranges, and configuring data points, this led to a frustrating experience. With 1,000+ active Advertisers and dozens of account managers creating reports regularly, the hours used quickly adds up, leading to potentially thousands of wasted hours and an added strain on the support team.
Role: Sole UX designer
Skills: UX/UI, User Research, Prototyping, User testing
Challenge
Natural language search features sound simple until you design one. The challenge wasn't just "add a text box", it was building trust in automation while preserving user control in an area where data accuracy is crucial. Users want to make decisions based on these reports, meaning any search generated content needs to be verifiable and editable.
I needed to solve for:
- Trust: How will a user know that the search results accurately match the query, without expecting them to check against up to 170+ metrics?
- Ambiguity: How do we handle queries that are vague and could mean one of the many data points available?
- Control vs. speed: Power users want a quick way of performing time consuming tasks, new users want guidance, how can we offer both?
- Technical constraints: The query parser struggled with specific metric names that can be bespoke to Rakuten Advertising, or even the user’s account, this required careful design around these considerations.
- Fallback: Some users prefer manual control over search based automation. How can we ensure this option is still available to them?
Approach
I started by analyzing existing reports to understand common patterns: What metrics did users combine? What date ranges mattered? What questions were they trying to answer? This informed the natural language query design. Instead of just free-form text, I included suggested questions to help a user get started and understand the mechanics of the input box. We also included a 'tag' system in a later iteration to help users find and include certain data points that were harder to remember.
Key decisions
- Tags/Tokens: Users were able to include 'quick selected' tags to help direct a prompt better.
- Manual override: Every search generated report could be edited, saved, scheduled for a future date or rebuilt from scratch
- Feedback: Gathering feedback via Fullstory we were able to make further decisions in the UI and functionality to help continually improve the feature.
- Conservative defaults: The system suggested safe, common queries rather than trying to be clever.
I prototyped three interaction models and tested them with multiple users. The ability to select suggestions won over many participants and the addition of tags was identified as an extremely helpful approach to more complex report requirements.
Solution
The final solution combined natural language prompts, structured tags, and suggested queries to give users both speed and control. Every search generated report remained fully editable, could be saved as a template, or rebuilt from scratch, this preserved the manual workflow for users who preferred it.
Outcome
Closed beta launched in May 2025 with select power users, followed by a full open beta in July 2025 to all users. With this staggered approach it has allowed us to begin gathering adoption data and user feedback before the full release.
Based on the initial few months of usage we have determined that we have reduced the report creation time by up to 90% (measured using Fullstory during the beta phase) This translates to around $10 million in annual time saving potential when fully adopted by all users (both internal account managers and external users).
User feedback
"Super helpful to put in the prompts and get the reporting answers right away instead of having to sometimes pull a few different reports to get the answer." Account Manager
"When I needed to check week-on-week sales, Prompt made it easier and faster to generate the report, saving time and reducing manual effort." Account manager
"I was able to visualize best performing placement periods over time. I was able to add a 'lifetime value bounty' on top of RAD data. I was really impressed with that." Account manager
"I was able to visualize best performing placement periods over time. I was able to add a 'lifetime value bounty' on top of RAD data. I was really impressed with that." Account manager
Behavioural changes observed
- Discovery driven reporting: "I figured out new and noteworthy products performing well in the program which I could share with publishers", this shows users are willing to ask questions they may not have bothered searching for manually.
- Reduced multireport workflows: Users may previously have needed to create or edit reports 3 or 4 times, AI is allowing them to get them in a single go.
- Power user adoption: The introduction of tags has helped create more advanced reports within the Prompt product.
Learning from beta
Reviews and feedback have been positive and allowed us to continue to deliver updates and additional features. Some items of Prompt have not been as successful as we hoped, for example saving reports has not been used as much as expected, largely due to users finding it easy to recreate reports as needed. The suggested prompts have become a useful tool for new users, offering guidance into creating a first time prompt, we hope to expand this in the future to allow for personalised suggestions.
What surprised me?: The request for tags/tokens was not expected, it became apparent that for more accuracy in data heavy UI we needed additional ways to control the search results.
Seeing users explore the search possibilities was interesting and allowed us to observe users trying to create more complex reports through search that they may not have been confident in building manually.
What I learned
Designing intelligent search features for consumer products is about trust, not magic. Users want help, not replacement. The ability for users to control the outcome of a report became crucial, the full text to report approach had to be revised to ensure a user could take control and understand that the report would be exactly what they wanted and that the data could be trusted.
The final outcome saw the creation of a hybrid interface. The expectation was that users would prefer a pure natural language search, but the tag system allowed more accurate control. When managing large amounts of data precision matters more than conversational ease. This insight applies to any data heavy product, precision matters more than conversational ease. Sometimes a smart autocomplete is better than a chatbot.
Speed without trust is worthless. Users could generate reports in 90 seconds, but if they didn't trust the results they would still verify manually. Defeating the purpose. The continuous previews and manual override options built the confidence needed to aid adoption.