From overwhelmed to empowered: Picking your best AI documentation assistant

When I started exploring AI documentation tools in 2023, I quickly learned that the most accessible options aren't always the safest. At a cybersecurity conference, I learned that researchers had discovered over 100 malicious models on Hugging Face—a platform recommended by experts earlier that year.

Later, at KnowledgeOwl's "Selecting the Right AI for Your Knowledge Base" webinar, I discovered tools like Perplexity. I knew I needed a systematic approach to evaluation instead of automatically going with the experts.

If you're facing similar challenges with selecting an AI tool for documentation, you're in the right place. This guide will walk you through tips for picking the best AI documentation assistant for your specific needs without getting stuck by indecisions or risks.

Identify your documentation challenge and team impact

Start by identifying a single, specific documentation challenge. As I explain in my Getting started with AI solutions post, focusing on one pain point—like improving accuracy or streamlining research—helps you evaluate tools more effectively. Consider both the technical needs and how the tool will impact your team's workflow.

In the KnowledgeOwl webinar, Mariena Quintanilla from Mellonhead emphasizes the human element: What friction points exist between team members? How will AI impact their work? Plan for both the technical transition and team adaptation through proper training and support.

Key Takeaway: Before evaluating AI tools, start small and build a clear understanding of your business culture.

Understand generative AI technology

When evaluating AI tools, consider these three critical factors:

1. Information processing: Think of a generative AI model like an advanced search engine that predicts and suggests completions as you type. These applications improve accuracy using one or more techniques: checking external sources (RAG), breaking down content, or fine-tuning on specific examples.

2. Privacy and security: Your choice of AI tool directly affects what data you input and your costs. For example, with the right technical staff and a tight budget, you might choose an open-source solution like Scribe for internal documentation. For customer-facing documentation, secure commercial platforms like Lex Pro offer enterprise-grade security and clear data ownership policies. I learned this distinction matters after discovering some open-source tools were selling data to third parties.

3. Specialization level: Different documentation tasks need different levels of expertise. While ChatGPT and Claude excel at general writing tasks—such as improving clarity—specialized tools shine in specific contexts. For example, when writing for a financial institution, you might consider BloombergGPT.

Understanding these factors helps evaluate available options. For example, when researching Perplexity, I discovered:

• How it processes information by searching and citing sources

• Its privacy differences between free and paid versions

• Its focus on broad knowledge and its specialization in extracting relevant content

This systematic approach helped tie in my documentation needs (a research tool) with Perplexity's capabilities.

Key Takeaway: Understanding how AI tools process, protect, and specialize in content helps you evaluate which ones best fit your documentation needs.

Tap the documentarian community first

When evaluating generative AI tools, remember that newer or more sophisticated doesn't always mean better results for your specific needs. Sometimes, existing tools—like Microsoft Word or Excel—might serve your purpose just as well as a dedicated AI platform. Making this assessment on your own, however, can be overwhelming.

Instead, don't try to evaluate AI tools in isolation. Connect with peers through:

• Professional networks like the Support Driven and Write the Docs 

• Attending industry conferences and workshops

• Slack channels and discussion forums

• Direct conversations with colleagues who've implemented an AI solution for a similar pain point

For example, I discovered Perplexity through KnowledgeOwl's "Selecting the Right AI for Your Knowledge Base" webinar. This community recommendation guided my subsequent research and helped me evaluate the tool against my specific pain point: improving research efficiency.

Key Takeaway: Don’t try to do it all by yourself. Consult your peers to find out what works best for certain use cases and to minimize risks.

Continue your evaluation

Once you get a recommendation from your peers, supplement your research. Mariena Quintanilla recommends focusing on:

• Security and Data Privacy: Assess risk areas, past breaches, and data usage

• AI Capabilities and Differentiation: Assess ROI and vulnerabilities to mitigate

• Implementation and Support: Calculate resources needed for successful adoption

Consider these evaluation tools:

• Generative AI: While tools like Perplexity can jumpstart research, always verify sources to avoid hallucinations. The diagram below shows how Perplexity lists sources for verification.

• Search engines: Search engines like Bing or Safari provide a good list of links. While reviewing search results takes more time, it helps verify information and avoid AI hallucinations.

• Vendor materials: Review vendor marketing documentation for privacy policies, functionality, and support information. Account for marketing bias.

• Web-based demonstrations: Prioritize cloud-based free demos over downloadable software. Create a dedicated email to try out the generative AI tool. Avoid downloading applications to your computer.

• Proof of concept: Dig up your specific pain point and the ideal solution. Create a test plan, specifying the data to use and evidence to collect. Create metrics and collaborate with any stakeholders. Ensure repeatability in the generative AI tool’s benefits and iterate the plan as needed.

For example, when evaluating Perplexity, I used the web version and established a systematic process of cross-referencing sources to prioritize security and accuracy.

Key Takeaway: Use multiple sources to ensure better understanding of a generative AI candidate’s security, benefits, drawbacks, and implementation needs.

Try out your generative AI candidate

With thorough research complete, now comes the fun part: taking your chosen AI assistant for a test drive. But before jumping in, let's set up a structured trial that minimizes risks and maximizes learning. Here's how:

• Choose an example of the documentation pain point you identified earlier

• Partner up with any stakeholders and get any vendor support you need for implementation

• Plan your testing. Include factors such as:

  • How you'll measure success

  • Ways to protect sensitive data

  • Who needs to be involved

  • Timeline and resources needed

• Set up a sandbox environment for testing with minimal risks

• Try out the generative AI tool in the sandbox environment, using a test profile

• Document your results, including both successes and areas for improvement

• Repeat and refine your approach

• Share your findings

For example, when testing Perplexity, I organized a peer-writing session. Through screen sharing, I demonstrated my research workflow, evaluation criteria, and risk mitigation strategies, which provided valuable insights for fellow documentation professionals.

Remember: You don't need a perfect solution—just one that makes your documentation work better while keeping your content safe and accurate. With a clear pain point, peer expertise, and systematic evaluation, you can confidently choose an AI assistant that improves your documentation operations.