Rakuten Advertising • Jan 2025 - Ongoing
Intelligent Search for Custom Reports — Natural Language Search & AI
Advertisers on Rakuten's platform create custom reports monthly — sometimes weekly — to track campaign performance across 170+ metrics. Building one manually meant 15–20 minutes of clicking through dropdowns and configuring data points. With 1,000+ active advertisers and dozens of account managers doing this regularly, the time loss was significant. It also landed on support when people couldn't figure out the interface.
Role: Sole UX designer
Skills: UX/UI, User Research, Prototyping, User testing
Challenge
Natural language search features sound simple until you design one. The challenge wasn't just "add a text box", it was building trust in automation while preserving user control in an area where data accuracy matters. Users want to make decisions based on these reports, meaning any search generated content needs to be verifiable and editable.
I needed to solve for:
- Trust: How will a user know that the search results accurately match the query, without expecting them to check against up to 170+ metrics?
- Ambiguity: How do we handle queries that are vague and could mean one of the many data points available?
- Control vs. speed: Power users want a quick way of performing time consuming tasks, new users want guidance, how can we offer both?
- Technical constraints: The query parser struggled with specific metric names that can be bespoke to Rakuten Advertising, or even the user's account, this required careful design around these considerations.
- Fallback: Some users prefer manual control over search based automation. How can we ensure this option is still available to them?
Approach
I started by analyzing existing reports to understand common patterns: What metrics did users combine? What date ranges mattered? What questions were they trying to answer? This informed the natural language query design. Instead of just free-form text, I included suggested questions to help a user get started and understand the mechanics of the input box. We also included a 'tag' system in a later iteration to help users find and include certain data points that were harder to remember.
Key decisions
- Tags/Tokens: Users were able to include 'quick selected' tags to help direct a prompt better.
- Manual override: Every search generated report could be edited, saved, scheduled for a future date or rebuilt from scratch
- Feedback: Gathering feedback via Fullstory we were able to make further decisions in the UI and functionality to help continually improve the feature.
- Conservative defaults: The system suggested safe, common queries rather than trying to be clever.
I prototyped three interaction models and tested them with multiple users. The ability to select suggestions won over many participants and the addition of tags was identified as an extremely helpful approach to more complex report requirements.
Solution
The final solution combined natural language prompts, structured tags, and suggested queries to give users both speed and control. Every search generated report remained fully editable, could be saved as a template, or rebuilt from scratch, this preserved the manual workflow for users who preferred it.
Outcome
Closed beta launched in May 2025 with select power users, followed by a full open beta in July 2025 to all users. With this staggered approach it has allowed us to begin gathering adoption data and user feedback before the full release.
Based on the initial few months of usage we have determined that we have reduced the report creation time by up to 90% (measured using Fullstory during the beta phase) This translates to around $10 million in annual time saving potential when fully adopted by all users (both internal account managers and external users).
User feedback
"Super helpful to put in the prompts and get the reporting answers right away instead of having to sometimes pull a few different reports to get the answer." Account Manager
"When I needed to check week-on-week sales, Prompt made it easier and faster to generate the report, saving time and reducing manual effort." Account manager
"I was able to visualize best performing placement periods over time. I was able to add a 'lifetime value bounty' on top of RAD data. I was really impressed with that." Account manager
Behavioural changes observed
- Discovery driven reporting: "I figured out new and noteworthy products performing well in the program which I could share with publishers", this shows users are willing to ask questions they may not have bothered searching for manually.
- Reduced multireport workflows: Users may previously have needed to create or edit reports 3 or 4 times, AI is allowing them to get them in a single go.
- Power user adoption: The introduction of tags has helped create more advanced reports within the Prompt product.
Learning from beta
Reviews and feedback have been positive and allowed us to continue to deliver updates and additional features. Some items of Prompt have not been as successful as we hoped, for example saving reports has not been used as much as expected, largely due to users finding it easy to recreate reports as needed. The suggested prompts have become a useful tool for new users, offering guidance into creating a first time prompt, we hope to expand this in the future to allow for personalised suggestions.
What surprised me?: The request for tags/tokens was not expected, it became apparent that for more accuracy in data heavy UI we needed additional ways to control the search results.
Seeing users explore the search possibilities was interesting and allowed us to observe users trying to create more complex reports through search that they may not have been confident in building manually.
What I learned
The biggest surprise was how much trust mattered. I expected users to love the freedom of a text box; what they actually needed was confidence that the output matched their intent. The full text-to-report approach had to evolve so users could see exactly what had been selected and step in if anything looked off.
We ended up with a hybrid interface. Everyone assumed users would prefer pure natural language, but the tag system became the most-used feature for complex reports. When there's a lot of data on the line, people want precision — not just speed.
Users could generate reports in 90 seconds, but they'd still verify manually if they didn't trust the output — which defeated the point entirely. The continuous previews and manual override options were what finally shifted that. Adoption followed trust, not the other way around.
