What Is a AI Strategy?
I’ve had the pleasure of being a part of numerous teams tasked with building AI apps. Although I’ve built apps from the ground up for my own use and have built components for AI apps, like an app that converts images of math problems into LaTeX (basically markup for math and science), my focus has been on the strategy that underpins the process. This strategy includes but is not limited to the following.
Model Selection
One of the biggest mistakes I see in model selection is using too expensive of a large language model (LLM) for the required task. Picking the best model for a task requires analyzing key performance indicators (KPIs) in the various leaderboards.
By way of example, Midjourney is often perceived to be the smartest text-to-image model. And they understandably charge a handsome price for that intelligence ($80 USD / 1000 image generations at the time of writing). But when you compare its performance against other similar models, you see you can get even more intelligence (as measured by Quality ELO) using the FLUX 1.1 [pro] model, for less than half the cost.
And for a small drop in quality, you could pay a tiny fraction of the cost using the Playground v3 or FLUX. 1 [schnell] model. Check the leaderboard out for yourself for the most up-to-date performance metrics. There are many of these leaderboards, and they come with a pretty high learning curve to get the most out of them. But this is one of the most essential tasks to get the most for your money, using the metrics that are most important to you. Choosing the right model for each task in your RAG pipeline ensures efficiency, scalability, and a seamless integration into business processes.
User Journey Mapping
In order to effectively build an AI app, you need to immerse yourself in the user’s current journey. What do they need to accomplish to be successful? How do they accomplish those tasks now? What are their biggest pain points? How can you ease that journey.
I dive deep into that journey, mapping it out as I go (example below).
Each of those nodes has an embedded note with ideas for how to titrate intelligence into that step—via an LLM, API, database lookup, etc—as well as links to content we’ll need for that page of the app, so we’re not scrambling for content when it’s time to actually start building. By meticulously defining and the ideal user flow for AI-powered systems, the team can more easily identify touchpoints where AI can enhance engagement and deliver value.
Conversation Design
To design truly effective conversations for your AI-powered systems, you need to start by understanding the user’s intent. What are they trying to achieve? What motivates their questions? Where are they likely to get frustrated? And how can you ensure their experience feels intuitive, helpful, and even delightful?
I work through these questions by mapping out the conversational flow step by step, considering every possible turn the interaction might take. For most apps I recommend using a dialog management system like Google’s Dialogflow CX (specifically CX because it offers a more robust and scalable solution for complex, multi-turn conversations, featuring advanced flow-based design, version control, and better support for large-scale enterprise applications.
Each node in the flow contains detailed notes on how to infuse intelligence into the interaction—whether through an LLM, pre-designed responses, database lookups, or third-party API integrations. I also include placeholders for dynamic content and functionality that may need to be integrated, so we’re not caught off guard later in the development process.
By defining and documenting these conversational pathways with precision, the team can focus on creating a user experience that feels seamless and personalized. This approach not only optimizes the interaction for the user but also helps stakeholders see exactly how AI can amplify engagement and drive business goals.
If a team decides to handle prompt instructions outside of a dialog management system, I perform comprehensive audits of their template files to ensure formatting issues like those demonstrated below don’t unintentionally send gibberish to the LLM.
Data Strategy
Though not the most scintillating aspect of AI strategy, an underpinning data governance framework is essential to maintain efficiency and compliance. One of the skills required to adequately prepare data to be used in an AI app is data modeling. Relying on the limited data modeling capabilities afforded to you in your database at this point is like building a website without CSS. I collaborate closely with data teams to establish a robust data modeling framework using tools like dbt, ensuring foundational decisions are made before connecting your data to the AI application. For instance, how should dates be standardized? How will null values be handled? And most importantly, how will you ensure compliance with industry-specific regulations while safeguarding against improper data collection or access?
Performance Monitoring
To ensure your AI application delivers consistent value, performance monitoring needs to be an integral part of your strategy from the start. How does the system perform under heavy loads? Are there bottlenecks in specific workflows? How can we proactively detect and resolve issues before they impact users?
I take a proactive approach to performance monitoring, building out a tailored framework that continuously tracks the health and efficiency of your AI system, using dashboarding tools like New Relic.
Each monitoring node focuses on a specific aspect of performance, such as response times, accuracy of outputs, system uptime, or user engagement metrics. These nodes are tied to key performance indicators (KPIs) identified during the planning phase, ensuring that every tracked metric has a direct impact on business goals. By integrating tools like observability dashboards, logging frameworks, and alerting systems, I ensure your team has full visibility into the system’s performance at every stage.
This structured, forward-thinking approach to performance monitoring doesn’t just help troubleshoot issues—it empowers the team to refine and optimize the AI app over time, keeping it aligned with user expectations and business objectives.
Portfolio
I’m only including AI apps if I either built them from scratch solo or if I built a component of the app so as to not take credit for anyone else’s work.
I also maintain a comprehensive timeline of AI news and developments that dates back to 2019.
Let’s Chat 📞
If you’re interested in learning more, you can request a proposal or ask questions via the form below. Or grab some time to talk about your unique needs.
Image Credit: Felix Mittermeier