AI First.
From insight to action.
Connect strategy, technology and people to transform your customer experience.
Let's talkDesign, AI and the reinvention of business models
Abstract: AI-driven business models that optimize for engagement without designing for meaningful human value produce inferior products. In addition, they systematically shift risk onto users, regulators and society. At the same time, they accumulate trust debt that only becomes visible when it is expensive to address. This article argues that trust functions as an operational asset in AI-enabled systems. It compounds or erodes through every communicative act.
The article also separates two elements. Care-based trust creates the conditions for trust to persist. Human agency, in turn, is what makes it sustainable. Some AI systems simulate care and mirroring while removing the space for genuine human participation. In that case, they produce a pattern analogous to the uncanny valley. In other words, they are close enough to a genuine relationship to attract attachment, but hollow at their core. Therefore, the organizations that will hold durable positions in this market are those that design trustworthy care and agency at the same time. That is the foundation for loyalty that grows.
When adoption does not last
After three years of widespread commercial AI deployment, the pattern is already legible in the data.
Many organizations led with technical capability and assumed that trust would follow. Today, they face a specific and costly phenomenon: users who adopt the technology and then withdraw. The data and knowledge are sound. The construction of the LLM models and the AI architecture is sound as well. So why did trust not follow? And why do we see use cases that do not reflect the correct use of the tools? In terms of performance, the system does what it was designed to do. However, we must ask whether behavioral, relational and instructional design came with the functions or only afterwards. If it came afterwards, the ongoing dialogue probably does not stem from a crafted human-AI relationship model.
Research on the abandonment of connected consumer technologies documents the shape of this pattern. For example, 65 percent of activity tracker users stopped within two weeks (Shih et al., 2015). In another study, participants abandoned almost 80 percent of the smart devices they had chosen within two months (Lazar et al., 2015). The cause was not technical failure. Instead, a relational architecture able to sustain genuine engagement over time was missing. These were not AI products, but the mechanism is the same. After all, the architecture of consistent engagement is a design responsibility. When it is absent, no level of technical investment recovers the cost of rebuilding.
The return of a well-built architecture
The financial evidence for what this architecture produces, when built correctly, is equally clear. Companies in the top quartile of design maturity outperformed industry benchmarks by 32 percentage points in revenue growth. They also outperformed by 56 percentage points in total shareholder returns over five years (McKinsey & Company, 2018). These are not returns on aesthetics. Rather, they are returns on trust infrastructure, that is, on the commitment to design relationships and journeys, not just products. This article shows what that infrastructure requires. It also shows why two factors decide whether those returns grow: care-based trust and genuine human agency.
What trust requires and why design governs it
The architecture of most commercial AI products rests on a specific kind of promise. According to it, the system understands users, responds to them and offers something close to genuine intelligence. Commercially, it matters less whether this is technically accurate. What matters is whether the promise is experienced as true in every moment of the interaction. Most AI-driven business models sell the experience of intelligence, that is, an expected belief and perception. Thus, design is the mechanism that builds or erodes that experience in every element the user encounters.
The anatomist and the clinician
Two figures, the anatomist and the clinician, help clarify the role of design in AI systems. The anatomist knows the system: its mechanisms, architecture and operational logic, with precision and rigor. The clinician, on the other hand, knows the patient, a specific person in a specific moment of need. That person’s response to any intervention depends on context, history and the quality of the care relationship. Both are necessary. AI engineering produces the anatomist’s precision, while design produces the clinician’s relationship. However, only the relationship determines whether the person leaves trusting the system more than when they arrived.
Every element communicates
In AI-enabled products, every communicative element is the clinician’s territory. This includes the interface language, the framing of responses and the tolerance for user uncertainty. It also includes the way the system discloses its limitations. When this territory is left to engineering defaults, the anatomy may be flawless and, even so, the relationship erodes.
Gunther Kress’s account of multimodal communication shows the structural mechanism: meaning is never made through a single channel (Kress, 2010). A conversational AI communicates through its words. But it also communicates through its latency, the rhythm of its responses and its choice of vocabulary. Likewise, its willingness to acknowledge uncertainty and the visual framing of its outputs communicate. Each channel shapes user behavior, regardless of the semantic content of the words.
Design governs all of these channels, whether or not the organization assigned that responsibility intentionally. When design is treated as a finishing layer, the channels become incoherent. For users, this incoherence registers as a lack of trustworthiness, almost never as a design failure. Still, it is correct to attribute it to design, because a design decision, or the absence of one, produced it. After all, design is a language that communicates. Without it, or without its proper use, this uncomfortable situation arises.
The psychological contract, care and the agency requirement
When users start a relationship with an AI-enabled system, they bring a set of implicit expectations. They expect the system to do what it appears capable of doing and to behave with reasonable consistency. They also expect its errors to be proportionate and recoverable. Finally, they expect their time, data and judgment to be treated with something resembling respect. These expectations form a psychological contract. It is neither stated nor signed, and its violation triggers reactions that standard engagement metrics cannot see.
Trust in AI systems has two distinct components (Jacovi et al., 2021). The first is competence-based trust, that is, the belief that the system can perform its stated function. The second is care-based trust: the belief that the system serves the user’s interests rather than its own operational goals. In general, commercial AI products invest in demonstrating the first while undermining the second. A system that performs its function but optimizes engagement at the user’s expense is untrustworthy in this sense. And this holds regardless of its technical accuracy.
Care without agency: the relational uncanny valley
Care-based trust creates the conditions for trust to persist over time. But, on its own, it does not make trust sustainable. Consider a well-built AI model with the right data and knowledge behind it. In this scenario, the distinction between care and human agency intervention carries commercial weight. In the context of AI, care is not the delivery of good outputs to a passive recipient. It is the genuine orientation of the system toward the interests of a specific person. This orientation is expressed through an architecture that preserves and invites that person’s judgment, agency and participation.
Sometimes, AI systems simulate care while blocking human agency. They appear to serve users, but they fill the space where the users’ own reasoning should operate. Researchers of mind perception and social AI describe this effect as the relational equivalent of the uncanny valley (Mori, 1970/2012; Gray & Wegner, 2012). The system is close enough to a genuine relationship to attract attachment. However, the reciprocal agency that makes the relationship real is missing.
As a result, users invest emotionally in something that does not exist as they experience it. The gap may surface through failures, behavioral changes or growing awareness. When this happens, the distrust is out of proportion to the immediate trigger, because it is structural. After all, what is discovered is not a malfunction but a hollow sense of trust. For this reason, disappointment and distrust damage the relationship itself.
Human agency as the condition for care
This is why human agency is not an ethical constraint on care-based AI design. On the contrary, it is the condition that makes human-AI care genuine. Preserving the user’s space to judge, deliberate and participate is not in tension with building trust. In fact, it is the mechanism that makes trust sustainable rather than merely serviceable (Buijsman et al., 2025; Calvo et al., 2020).
Why trust is lost faster than it is gained
The asymmetry between trust formation and erosion compounds these risks. Trust builds incrementally, but it erodes exponentially. Systems that simulate care without enabling human agency are fragile in invisible ways, until they collapse. A systematic review of trust calibration in human-AI interaction reached a clear conclusion. Once a trust failure happens, recovery is both difficult and costly (Mehrotra et al., 2024). Therefore, the psychological contract must be built deliberately through design. Moreover, it must cover two dimensions: serving the user’s interests and genuinely depending on the user’s participation.
Risk externalization as structural design
A specific pattern in current AI deployment deserves direct attention from leadership. Many organizations deploy AI in contexts where failures fall mainly on users rather than providers. In practice, the lack of a design strategy is risk externalization, whether or not it is recognized as such.
Imagine an AI-enabled diagnostic system that surfaces an inaccurate recommendation. If the user acts on it, the user bears the consequence. The same happens with a financial guidance system trained on biased data. Research on algorithmic bias in health management shows that this risk is often embedded in the architecture (Obermeyer et al., 2019). In other words, it depends on how systems are designed and for whom, and it is not accidental. The distribution of accountability is therefore a design question. Who receives which information, under what framing and with what ability to challenge the system? These decisions belong to design. When they are made by default rather than by intention, they create real exposure, even if no one names design as the cause.
Dark patterns and their effects
Risk externalization is also a failure, even when it is meant to drive business results. Systems that shift risk onto users are, by definition, not oriented toward those users’ interests. Instead, they are oriented toward the provider’s operational convenience. This effect has been well documented since the persuasion techniques described by Fogg and other behavioral design approaches. Since then, evidence has accumulated and now drives lawmaking efforts.
More recently, a pre-registered experiment with 1,963 participants tested interface techniques that are common in the market. They included a pre-selected accept button, a “recommended” label and a second refusal screen. Together, these techniques raised acceptance of an unwanted paid service from 11.3 percent to 25.8 percent. More aggressive tactics pushed acceptance to 41.9 percent, but they caused measurable anger and lost trust in the firm. By contrast, the milder version produced no comparable reaction. Without these techniques, education did not predict who accepted. Under the mild condition, however, less educated participants became significantly more likely to accept (Luguri & Strahilevitz, 2021).
Regulatory pressure
The regulatory trajectory reinforces this commercial urgency. AI governance frameworks are advancing in several jurisdictions, always in the same direction: more organizational accountability for system behavior. As a result, organizations that have not given design authority over risk distribution are accumulating regulatory exposure and trust debt. And the users who bear the failures become the main source of both.
Trust as strategic infrastructure
The business case for treating trust as infrastructure, rather than a surface effect, is well documented. The McKinsey Design Index found that companies in the top quartile of design performance outperformed industry peers over five years. They delivered 32 percentage points more in revenue growth and 56 points more in shareholder returns. The strongest correlate was sustained investment in user research and in the quality of the experience (McKinsey & Company, 2018).
Similarly, a Forrester Consulting economic impact study of enterprise design practice found a three-year ROI of 301 percent. The benefit came mainly from less rework and faster delivery. Design and alignment time fell by 75 percent, and development and testing time by 33 percent. In addition, design defects were cut in half (Forrester Consulting, 2018).
Trust as a compounding asset
Even so, trust is the compounding asset that AI-focused organizations most often fail to treat as an asset. The companies in the top quartile of these studies are not there because their products are more beautiful. Rather, they are there because their products earn and sustain genuine trust, based on care and reinforced by human agency. At scale, this trust creates a different commercial relationship from the one produced by engagement optimization alone.
Expanding returns in subscription products
In AI-enabled subscription products, a trust infrastructure built correctly in the first months generates expanding returns. Users who trust a system engage with it more deeply. They also provide higher-quality interaction data and tolerate its errors better. On the other hand, users who do not trust it gradually disengage. Consequently, they undermine direct revenue and the data quality on which future model improvements depend. These dynamics carry financial weight and appear in operational data long before organizations identify their source.
Subscription AI and the difference between value and dependence
The structure of subscription AI products creates a specific strategic risk that requires explicit attention. In this model, commercial success depends on sustained engagement. However, the mechanisms that sustain this engagement are not equivalent from a trust perspective. This difference determines the commercial quality and the long-term sustainability of the relationship.
Genuine value versus dependence
Engagement driven by genuine value comes from the user’s own assessment that the system makes them more capable, informed or effective. This type of engagement produces loyalty grounded in the relationship. Users feel that the system understands them and works according to their needs. In addition, they can actively collaborate and intervene with agency.
Engagement maintained through dependency mechanics, by contrast, works differently. It uses friction to discourage exit, notifications that exploit loss aversion and recommendations that narrow user capability. As a result, it produces retention numbers that mask real disengagement. Self-Determination Theory explains the difference (Deci & Ryan, 2000). Motivation maintained through external pressure is fragile. Meanwhile, motivation generated through genuine competence, autonomy and relatedness grows over time. This difference is invisible in short-term adoption data, but decisive at 12 and 24 months.
Active and passive interaction
This distinction maps onto the difference between active and passive interaction architectures. Some systems are designed to elicit the user’s own judgment. They invite interpretation, surface uncertainty and preserve space for the user’s reasoning. In this way, they generate the engagement that produces care-based trust and genuine attachment. Other systems aim for maximum output fluency with minimum user participation. These produce dependent engagement that retains users in the short term, but does not survive serious failures or good alternatives (García Corretjer et al., 2022). For this reason, confusing the two types of retention is risky. The first gains value as the relationship deepens. The second, conversely, lowers trust precisely at the moments that matter most for long-term revenue.
The evidence on genuine loyalty also separates these two forms of retention in financial terms. Users who experience value through their own agency become advocates. In contrast, users kept through dependency tend to leave at the first opportunity and often amplify their dissatisfaction (Reichheld, 2003; Edelman, 2024). In a market where AI faces increasing regulatory and public scrutiny, optimizing for dependency rather than genuine value is risk management negligence.
Design intent as the differentiator
Over the next decade, the organizations with defensible positions in AI-enabled services will be those that treat design as a core mechanism. It is through design that the relationship with users is built, maintained and evidenced over time. These organizations also understand the specific requirement that care-based trust imposes. Human agency is not an ethical constraint on this work; it is its essential condition.
Trust debt
Once established, trust is the only business asset that multiplies without additional capital input. On the other hand, it cannot be bought back once it has been lost at scale. Organizations investing in AI without investing in human-centered design are accumulating trust debt. This debt remains invisible until it materializes. It may appear as churn that resists conventional retention analysis or as adoption plateaus despite marketing investment. It may also appear as regulatory exposure that the legal team will have to manage later. By then, recovering trust may cost more than the design investment that would have prevented the problem.
Design as a relationship, not a finishing layer
What sets design-led organizations apart from those that treat design as a finishing layer is scope. In the former, design governs the entire communicative relationship between system and user. This includes the language, the rhythm, the framing of uncertainty and the clarity about what the system can and cannot do. It also includes preserving the user’s own reasoning in the interaction, the instructions and the digital space. We are not talking about visual assets. Instead, we are talking about decisions that determine whether care-based trust can develop and last. Sustainability is an output of the relationship being designed, not a feature to add once adoption is achieved. The organizations that understand this and act now are building the advantage that will define the AI market over the next decade.
By Dr. Marialejandra García Corretjer, PhD
References
- Buijsman, S., Carter, S. E., & Bermúdez, J.-P. (2025). Autonomy by design: Preserving human autonomy in AI decision-support. Philosophy & Technology, 38, 97. https://doi.org/10.1007/s13347-025-00932-2
- Calvo, R. A., Peters, D., Vold, K., & Ryan, R. M. (2020). Supporting human autonomy in AI systems. In C. Burr & L. Floridi (Eds.), Ethics of digital well-being. Springer.
- Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits. Psychological Inquiry, 11(4), 227–268.
- Edelman. (2024). Edelman Trust Barometer. Edelman.
- Fogg, B. J. (2003), Persuasive Technology: Using Computers to Change What We Think and Do, Morgan Kaufmann.
- Forrester Consulting. (2018). The Total Economic Impact™ of IBM’s Design Thinking Practice. Study commissioned by IBM.
- García Corretjer, M. A., Ros, R., Mallol, R., & Miralles, D. (2022). Empathy as an engaging strategy in social robotics: A pilot study. User Modeling and User-Adapted Interaction, 33(2), 221–259. https://doi.org/10.1007/s11257-022-09322-1
- Gray, K., & Wegner, D. M. (2012). Feeling robots and human zombies: Mind perception and the uncanny valley. Cognition, 125(1), 125–130.
- Jacovi, A., Marasovic, A., Miller, T., & Goldberg, Y. (2021). Formalizing trust in artificial intelligence. FAccT 2021.
- Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. Routledge.
- Lazar, A., Koehler, C., Tanenbaum, J., & Nguyen, D. H. (2015). Why we use and abandon smart devices. Proceedings of UbiComp 2015, 635–646.
- Luguri, J., & Strahilevitz, L. J. (2021). Shining a light on dark patterns. Journal of Legal Analysis, 13(1), 43–109.
- McKinsey & Company / McKinsey Design. (2018). The business value of design. McKinsey Quarterly.
- McKinsey & Company. (2023). The state of AI in 2023.
- Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024). A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction. ACM J. Responsib. Comput., 1(4), Article 26.
- Mori, M. (1970/2012). The uncanny valley (K. F. MacDorman & N. Kageki, Trans.). IEEE Robotics & Automation Magazine, 19(2), 98–100.
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
- Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54.
- Shih, P. C., Han, K., Poole, E. S., Rosson, M. B., & Carroll, J. M. (2015). Use and adoption challenges of wearable activity trackers. iConference 2015 Proceedings.