In our previous blog post, we explored the transformative potential of Artificial Intelligence (AI) and why businesses need to start exploring its possibilities now. However, implementing AI requires a shift in how we approach product development. 

This post dives into six unique challenges and considerations for prototyping, testing, and experimenting with AI products:

1. Beyond Wireframes: A Higher Bar for Prototype functionality

In traditional product development, early-stage concepts can be tested with relatively low fidelity—think wireframes or basic prototypes. When testing an AI experience, the closest parallel is often a human doing the job, as it was for Amazon when they used humans to prototype the experience of Alexa. To appear sufficiently ‘intelligent’ any prototypes either need to keep a human in the loop somewhere, or demonstrate a certain level of reliability and utility to ensure they meet user expectations and functional requirements. This puts a higher than usual burden on teams to test and validate the quality of prototypes to ‘pre-test’ what is a passable experience for users.

2. Human in the Loop: The Importance of High-Touch Learning

Even once they’re ‘live’ early iterations of new experiences might require significant human assistance. Think of it as training wheels. In the initial stages of an AI experiment, human support can significantly improve the system's reliability and user experience. This means you need to think about finding the right experts from the business and getting them on board with the development of the system. As the AI matures and learns, the level of human intervention can gradually decrease. This high-touch support ensures that the AI can operate effectively in real-world conditions and that any errors can be managed promptly.

3. Data Dilemmas: Prototyping with Realistic (or Obscure) Data

One of the biggest challenges in AI prototyping is data access. Building a robust AI model often requires vast amounts of real-world data. However, obtaining this data can be a hurdle, especially during the early prototyping stages because real data often contains sensitive information, making it problematic for testing purposes. Here, creative solutions come into play. Techniques like data mocking can help simulate realistic data sets for testing purposes. Alternatively, obfuscation techniques can anonymise real data while preserving its core characteristics for prototyping.

4. Isolating concerns: Separating the Smarts from the Experience

It's crucial to differentiate between the core AI functionality and the user experience (UX) layer. While the AI engine operates in the background, the UX layer determines how users interact with the AI's outputs. Testing these components separately to begin with allows for focused optimisation, whilst also enabling teams to isolate the riskiest elements to test. In some cases these risks will sit in the experience layer i.e. ‘Will our customers trust an automated system to solve this problem for me’. In other cases, the risks will exist with the AI functionality i.e. ‘Can we return helpful and reliable advice or information’. Once we've determined where the risk lies, it's much easier to design an informative test to define where to focus efforts/ development.

5. Breaking Down Silos: Building with the business 

Developing AI products necessitates a highly collaborative environment. In contrast to more traditional approaches, AI projects require an always-on cross-functional approach from their inception. Traditionally the ‘logic’ of a product is defined and signed off before being left alone by commercial, operational or marketing teams. In a world where that logic is constantly improving and refining, there needs to be a much tighter feedback loop. Always-on cross functional collaboration that weaves data science, user experience and business expertise ensures we are constantly validating outputs, and capitalising on improvements to the logic.

6. Looking Inward First: Applying AI to operations

While AI promises exciting customer-facing applications, some of the earliest wins will likely come from internal process optimisation. AI can automate tedious tasks, improve efficiency, and reduce operational costs. This focus on internal experimentation allows companies to  learn about AI in a controlled setting, on a scale where its impact can be much more clearly seen/contained/mitigated, while avoiding the risks (brand, data, regulatory) of external deployment. Remember, a successful internal pilot can pave the way for future customer-centric applications.

Conclusion: Embracing the Future of Product Development

All of these changes focus on the products and experiences created, but it’s clear that co-pilots and other AI-enabled tooling will change the skills and capabilities required to create great software.

Stay tuned for Part 3, where we'll delve into how AI will impact the skills, teams and ways of working when building in the AI age.