The Exciting Potential and Fundamental Limits of Generative AI for Product Managers

As product managers, it's impossible not to be excited by Generative AI. We can all see the potential to revolutionize products and how we work. It truly feels like magic.

But when you get hands on and start building, you quickly run into a fascinating paradox, while these models seem capable of anything, they have very real, fundamental limits. To build great products, we can't just focus on the magic. We have to be clear eyed about the boundaries.

To help me think this through, I’ve started grouping these limitations into six connected categories. Understanding them has been key to figuring out where AI fits in product strategy and, just as importantly, where it doesn't.

1. Knowledge (The Library)

Think of an LLM as a brilliant student who has read every single book in a giant library. Their knowledge is vast... but it’s limited to only what's in that library. 

• The Limitation: Their knowledge is frozen in time and limited to their training data.

• A Simple Example: This is why you can ask a model about World War II and get a fantastic essay, but if you ask "Who won the women's cricket world cup last night?" it will be completely stumped or, worse, make something up. It simply doesn’t know because that information wasn't in its library.

2. Memory (The Conversation)

Memory is all about an AI's ability to hold on to context. While models are getting much better at this, they still struggle with "remembering" things across long conversations or large documents. This directly impacts how they use their knowledge.

• The Limitation: They have a limited "short term memory" for the current interaction.

• A Simple Example: Ever been in a gemini/ ChatGPT conversation where you have to repeat yourself? You might mention a key detail, and ten messages later, the model has completely forgotten it. That’s a memory limitation in action. For PMs, this means designing experiences that don't rely on the AI remembering page 1 when the user is on page 50.

3. Reasoning (Connecting the Dots)

Because their knowledge and memory are limited, LLMs can falter with complex, multi step reasoning. They are amazing at recognizing and repeating patterns they've seen in their training data, but they aren't always great at genuine, logical problem solving.

• The Limitation: They struggle with true multi step logic and understanding nuanced relationships.

• A Simple Example: You can ask an LLM to "list all the features users requested" and it will do brilliantly. But ask it to "analyze these feature requests, consider our technical debt, evaluate market timing, and recommend which feature to build next," and you'll see it struggle to weigh all those strategic trade-offs in a genuinely thoughtful way.

4. Learning & Improvement (The Student)

This one is critical for product managers. Unlike a fellow PM who learns your product's quirks and gets better at anticipating user needs over time, an LLM doesn't evolve from sprint to sprint. It can't build on past product decisions or learn from failed experiments. Every interaction is essentially day one.

LLM doesn't learn from its mistakes or conversations in real time. If you correct it, it might apologize and fix its answer in that single conversation, but the core model hasn't actually learned anything new.

And let me be clear - I am exhausted hearing people say I've trained my ChatGPT or the model has learned our process. No, you haven't. No, it hasn't. You've just had a conversation. That's not training - that's prompting. The model will forget everything the moment you start a new chat.

• The Limitation: They cannot learn or improve from new interactions without being fully retrained.

• A Simple Example: You use AI to draft PRDs. You correct it, Our platform always needs mobile and desktop parity we learned this the hard way in 2023. Next PRD? The AI suggests another desktop only feature. It won't remember your technical debt, your past pivots, why you killed that subscription model, or that your CEO absolutely will not approve anything that requires hiring more support staff. Every PRD starts from zero context about your actual product journey.

5. Purpose & Direction (The 'Why')

This might sound obvious, but it's the most critical thing to remember. LLMs have no goals. No self awareness. No intentions, good or bad. They are incredibly sophisticated tools just waiting for instructions.

• The Limitation: They have no agency, goals, or decision making ability beyond following a prompt.

• A Simple Example: An LLM will never wake up and think, I should really find a better way to help our users check their order status. It can't set its own objectives. All the "why" and what should we do next must come from us, the product team

6. Tools (Needing Help)

Finally, because of all these other limits, an LLM by itself is often helpless. To be truly useful in a real world product, it needs to be connected to external tools and systems.

• The Limitation: They can't access real time data or take actions in the world without integrations.

• A Simple Example: To get past its knowledge cutoff date, the AI needs a tool to browse the internet. To answer Where is my order? it needs a plugin to check your company's order database. A big part of our job as PMs is figuring out which tools our AI needs to be useful.

These six limitations aren't bugs to be fixed or problems to solve, they're the actual boundaries of what this technology can do today. Once we accept them, we can stop trying to force AI to be something it's not and start building products that truly work.