Defense KM Knavigator

GenAI Knowledge Management Assistant, Knavigator enhanced the learning experience of defense knowledge management students by providing a knowledge search and content navigation guide.

Role: Lead designer

Squad Composition:
Lead Designer
Jr. Designer
Business Technical Leader
Solution Architect
AI Engineers
Platform Engineer

Timeline:
April - September 2024

Impact: Reduced the amount of time students spend researching information by 40%

Lean UX Design & Research, Collaborative Design, Visual Design

Background

The goal was to help the defense agency’s Knowledge management students find, and consolidate information from their coursework in one place using this GenAI assistant tool, Knavigator. The clients goal was to utilize AI to enhance knowledge integration within the knowledge management course and ultimately bring a similar solution to their individual command posts.

Approach

We started off the engagement with a Design Thinking discovery workshop to better understand the organization’s goals around watsonx and AI. The designers planned and led a two-day session with 26 participants over 2 days. Each day’s agenda was packed with 5 hours of workshop activities. They also brought us 2 identified GenAI use cases to explore applying in their organization. We prioritized one of the two, the Knowledge Management Use case to focus in our pilot.

Prepare & Plan Workshop

Learn about client problem and prepare workshop agenda depending on how much we know. Work with client to review agenda nad plan dates to visit in person or have virtual workshop.

Process

Execute Workshop

Execute Discovery workshop to ensure we are aligned on a problem statement or pilot statement of work, including success criteria and create buy-in for the vision of the experience.

Prepare Stage

Start crafting experience prototype. (Wireframing, prototyping, and visual design). Iterate with peers to ensure feasiblity and stakeholder acceptance. Also develop any user test plan materials that will be used in testing after the solution is built.

User Testing

Test the experience with users, analyze the results and distill into a report. Work with engineers on actionable insights to fix in futher sprints. Distill findings into report.

Build

Collaborate with engineers to bring our technologies value to life.

Executive Playback

Condense findings and client quotes from testing in the final playback for executive leadership.

Project Kickoff

Hold kickoff meeting with client and share project timeline and plan activities for the first sprint.

Prepare Stage

Lean User Research

The designers developed a research test plan of 4 scenarios and a series of performance questions with a likert scale to do a moderated usability test with KM students. We also asked them a series of general feedback questions to surface more ideas and opportunities for this solution to build on.

Visual Design

During the first few sprints the design team worked on UI designs to match the client’s branding, including:

  • a series of customized avatar options matching the client’s color theme

  • the iconography was designed to represent both knowledge management and navigation

Build Stage

The technical team then started building. They ingested over 140 documents and implemented user stories and RAG (Retrieval-Augmented Generation) workflows to address user inquiries in natural language.

Design Sessions

The design team and rest of the squad held 4 weekly 1 hour design sessions with stakeholder SMEs to refine the type of dialog flow would be ideal for the KM students since so much of the content itself was complex and we really needed SMEs to give us additional context to ensure we were building something that would be relevant and useful for our end users. The output of these design sessions were refined into user stories and were then prioritized and worked on during our week long sprints.

The main outcome of the workshop was to:

  • Understand the use case & the current as-is experience for the knowledge management students and their pain points

  • What data was needed?

  • Any associated risks

  • What success criteria to measure for our pilot

    This was all summarized in a Pilot Statement.

  • Step 1 - Scenario Exploration

    Working with KM students, Instructors and other Subject Matter Experts we ideated on specific scenarios that would provide value. Our intent was to then flesh out details to put into dialog flows with our AI engineers.

  • Step 2 - Map system functionality and identify any additional data sources

    We then took the Scenarios from Step 1 and mapped any system functionality needed and any additional data sources that may need to be included

  • Step 3 - Prioritize on Feasibility

    Then we had the AI engineers prioritize each scenario by feasibility to identify the low hanging fruit to start work on. I then refined these top scenarios into user stories the team worked on for each sprint.

Usability Testing

We conducted a usability test that included general feedback and performance questions. We had two rounds of testing with two different student cohorts with a sample size of n=8. After each round of testing, we coded our results and shared with the team.

These iterations really improved the value of the solution and surfaced many ideas from end users for broader application and scale.

Example of  a page from a report visualizing the study results in graphical form.

Impact

The implementation of Knavigator enhanced personal knowledge and productivity. Increasing students' confidence during training, improved their competencies to perform tasks quickly and more effectively, and allowed them to learn more in the same amount of time with the help of this learning aid. The following are the quantitative and qualitative benefits demonstrated:

  • Reduce the amount of time students spend researching information by 40%

  • Increase student effective learning by 20%

  • Standardize the knowledge research and add explainability to the process

  • Ability to scan documents to detect duplicate information, therefore reducing the time students spend from vetting information

  • Improve the accuracy, quality and timeliness of the searches