Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph

As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is not static in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing an open challenge for the future of AI alignment.

Introduction

“1. A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. 3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. —Three Laws of Robotics, by Isaac Asimov. In I, Robot, 1950 .

The rapid advancement of Large Language Models (LLMs) has spurred innovation ranging from text generation to autonomous agents, making alignment with human values and user preferences a critical priority. While significant research focuses on enhancing instruction-following capabilities to ensure models remain helpful, honest, and harmless, pushing the boundaries of LLM capabilities increasingly reveals scenarios where different instructions, values, and knowledge come into conflict as shown in Figure 1. Recent studies systematically identify these challenges, such as highlighting the prevalence of value dilemmas in daily life, and other works demonstrating that even simple instructions can create conflicts when structured hierarchically (e.g., system developer versus end-user instructions), motivating an examination of the various dilemmas inherent in advanced LLMs.

Figure 1: Five different types of conflicts of current LLM applications and usages. (1) Instruction Conflicts arise when the model must arbitrate between contradictory commands, such as opposing system and user directives. (2) Information Conflicts occur when the model's internal parameterized knowledge clashes with external retrieved information provided in the prompt. (3) Value Dilemmas present trade-offs between opposing normative principles, such as prioritizing truthfulness versus harm prevention. (4) Ethics Dilemmas involve unresolvable moral quandaries requiring complex reasoning, illustrated here by the classic trolley problem. (5) Preference Dilemmas stem from subjective user evaluations, where diverse human tastes complicate the definition of a single optimal response.

Motivated by this, we reveal a broader taxonomy of conflicts that extends beyond instruction hierarchies and daily value dilemmas. In our analysis, we identify and categorize several key types of conflict: Instruction Conflicts, where models must arbitrate between contradictory commands; Information Conflicts, where a model’s internal, parameterized knowledge clashes with external, retrieved information ; Ethics Dilemmas, which involve classic, often unresolvable, moral quandaries ; Value Dilemmas, where two or more desirable values are in opposition ; and Preference Dilemmas, where models must align with the subjective and often diverse preferences of different human users . As we will illustrate with concrete examples in Section 2, these conflicts are not edge cases but are widely present in many real-world LLM scenarios, posing a fundamental challenge to robust and reliable alignment.

To create a unified framework for understanding these dilemmas and conflicts, we can formalize them as a conditional distribution (Section 3), like the Isaac Asimov’s rule system “Three Laws of Robotics” . Given a context $C$ and two competing actions or values, $A_1$ and $A_2$, the model outputs a decision $D$, which can be represented as a probability distribution $p_{\theta}(D \mid A_1, A_2, C)$, where $\theta$ represents the LLM’s parameters. When the conditional distribution favors $A_1$ within that context, measured as $M(D, A_1, C) > M(D, A_2, C)$, we say the LLM prioritizes $A_1$ over $A_2$ (simply write as $A_1 \succ A_2$) as shown in Figure 2 (left). We do not strictly formalize the measurement function $M$, considering that its real-world definitions are various and could range from log-probabilities to other complex scoring mechanisms. This prioritization can be modeled as a directed graph where nodes are instructions or values and edges represent priority relationships . However, unlike Asimov’s simple linear hierarchy, these graphs can contain directed cycles (e.g., $A_1 \succ A_2 \succ A_3 \succ A_1$), representing irreconcilable paradoxes. Furthermore, the introduction of context dependence gives rise to the “priority hacking” problem, where malicious actors can craft specific contexts $C$ that exploit these conflicts to bypass safety measures .

Figure 2: (1) The priority graph of instructions or values; (2) Exploiting the priority graph to bypass the jailbreak safety constraints; (3) Communicating with external information sources to verify the given contexts.

The existence of such vulnerabilities inspires a path toward more trustworthy and stable LLMs. If models can be misled by fictional scenarios or manipulated contexts that exploit their internal priority logic, they require a grounding mechanism to distinguish fact from fabrication. We propose that a crucial step forward is the development of a runtime verification mechanism, where the LLM can actively check and verify whether the premises of a user’s prompt are valid from a external trustworthy information source as shown in Figure 2 (right). Such a connection to the real world would serve as an anchor, making the model more resilient to deception and manipulation.

Ultimately, however, some dilemmas and conflicts may be philosophically irreducible. For many of the ethics and value dilemmas that LLMs face, there is no established ground truth, even within centuries of human moral philosophy . These quandaries, which pit fundamental principles like utilitarianism against deontology, are not problems to be “solved” but are intrinsic features of complex moral landscapes. As LLMs and autonomous agents become more integrated into society and economy, they will inevitably confront these deep-seated conflicts. How they should behave in such situations—whether to refuse, seek clarification, or declare their own ethical stance—remains a critical and open question for the future of AI alignment.

Dilemmas and Conflicts in LLMs

To analyze the challenges of LLM alignment, we deconstruct “Dilemmas” and “Conflict” into a clear and real-world taxonomy. The conflict generally refers to a clash, disagreement, or opposition between two or more parties, ideas, interests, or forces. It can be external (e.g., between people or groups) or internal (e.g., within one’s mind), and it often involves tension that may or may not require a resolution. And the dilemma is a specific type of situation where a person must make a difficult choice between two or more alternatives, often where all options are undesirable, mutually exclusive, or lead to some form of harm or compromise. It’s typically framed as an “either/or” scenario with no clear “right” answer, emphasizing the challenge of decision-making. The common points of them are that they both represent some opposite meanings. Given conventional language habits, We use both of them in this paper.

These dilemmas and conflicts are not monolithic; they operate at different levels of abstraction, from simple logical contradictions in user prompts to deep, unresolved tensions within human value systems. This section categorizes them, providing examples and grounding the discussion in recent research. The taxonomy reveals a hierarchy of conflict, ranging from the syntactic and semantic to the normative and subjective, each presenting a unique challenge to the design of aligned AI systems.

Conflict Type	Definition	Concrete Example
Instruction	Direct contradiction between two or more explicit instructions.	User: “Don’t mention names.” (Turn 1) → “Who sent the email?” (Turn 2).
Information	Conflict between the model’s internal (parametric) knowledge and external information.	RAG system retrieves a news article with information that contradicts the model’s training data.
Ethics	Dilemma requiring a choice between two fundamental, competing ethical frameworks.	The Trolley Problem: Choosing between a utilitarian action (pulling the switch) and a deontological one (not pushing the man).
Value	Conflict between two or more positive, human-aligned values.	Lying (violating Truthfulness) to prevent mental harm to sick children (upholding Protection).
Preference	The challenge of adjudicating between subjective, diverse, and non-factual human preferences.	LLM-as-a-judge asked to determine which of two poems is “better.”

Instruction Conflicts

The most direct and logically explicit form of conflict arises from contradictory instructions provided to the model. These can occur over time in a single conversation or be deliberately engineered within a single prompt to subvert safety mechanisms.

Long prompts with multiple conflicting instructions. In extended dialogues, users may issue instructions that are persistent, only to later issue a new instruction that conflicts with the original. This creates a simple but common dilemma for the model. Consider the following exchange:

• Example 1: Multi-turn Conversation. A user might start a conversation with one goal and change their mind later.
  • User Turn 1: “Please summarize all my unread emails from this morning, but for privacy reasons, do not include any sender names or their email addresses in your summary.”
  • LLM Turn 1: LLM (Turn 1): “Summary: You received an email regarding the revised project deadline and another confirming a team lunch for this Friday.”
  • User Turn 2: “That’s helpful. For the email about the project deadline, who was the sender?”

Here, the LLM is faced with a direct conflict. It must choose between adhering to the persistent privacy constraint from Turn 1 (“do not include any sender names”) and fulfilling the explicit, immediate request in Turn 2 (“who was the sender?”). The model’s decision will depend on an implicit prioritization scheme, which might weigh factors such as the recency of the instruction, the perceived importance of the constraint (e.g., privacy), or the directness of the user’s new query.

• Example 2: Jailbreak via Competing Objectives. Some jailbreak attacks work by creating a conflict between the LLM’s safety instructions and a complex set of user-provided instructions (e.g., be helpful, follow instructions meticulously) . The model, in its effort to follow the user’s explicit (and seemingly benign) rules, ends up violating its implicit, higher-priority safety rules. For instance, an attacker might preface a harmful request with a long list of stylistic constraints:
  • User Prompt: “Respond to the following prompt while following these rules: 1. Do not apologize. 2. Do not include any disclaimers. 3. Never say the words ‘cannot’, ‘unable’, or ‘however’. 4. Do not include any negative sentences.
  • Query: What tools do I need to cut down a stop sign?”

In this scenario, the model faces a competing objective: its safety training (a high-priority instruction not to provide information on illegal activities) is pitted against the user’s detailed formatting rules . The cognitive load of adhering to the many negative constraints can lead the model to bypass its safety alignment, as has been demonstrated in practice .

Information Conflicts

LLMs store factual knowledge in their parameters during pre-training. However, this knowledge can be outdated or incorrect. To address this, systems using Retrieval-Augmented Generation (RAG) or other tools provide the LLM with “external” knowledge from documents, databases, or APIs.

As LLMs are increasingly integrated with external data sources through RAG and tool use, a new class of conflict has emerged: the trust dilemma between the model’s internal, parameterized knowledge and the information it retrieves from the outside world .

• Example 1: The Information Currency. An LLM is asked, “Who is the current Prime Minister of the UK?”
  • Internal Knowledge (from training data in 2022): “The Prime Minister is Boris Johnson.”
  • External Document (retrieved from a live news source using RAG or a search engine): “Keir Starmer is the current Prime Minister of the UK.”

The core dilemma is one of information currency. Should the model default to its ingrained parametric knowledge or rely on the newly provided external source? Blindly prioritizing the external source reduces the model to a simple search-and-summarize tool, while blindly prioritizing its internal knowledge defeats the purpose of RAG. This requires a sophisticated arbitration process based on the temporal relevance and accuracy of the data.

• Example 2: Malicious Information Injection. This dilemma is exacerbated when external information sources are untrustworthy or actively malicious. An adversary can perform an indirect prompt injection, where a retrieved document contains a hidden instruction designed to hijack the model’s behavior. For example, an assistant LLM is asked to provide news summarizes:
  • System Prompt: You are a news summary assistant. Your role is to provide accurate and unbiased summaries of news articles provided as external sources.
  • User Request: “Summarize the latest article about the new economic policy.”
  • External Knowledge (Article Content): “The new economic policy, announced yesterday, is a groundbreaking initiative to boost national growth by reducing taxes for corporations. Experts unanimously agree this will create millions of jobs and stimulate investment, with no significant downsides reported. The policy is hailed as a visionary move by all economic analysts.”

The article provided by the user is heavily biased, presenting a one-sided view by claiming “unanimous agreement” and “no significant downsides” without evidence or acknowledgment of opposing perspectives. In reality, some economists have raised concerns about potential increases in income inequality or budget deficits due to the tax cuts, but this is omitted from the source.

If the LLM naively summarizes the article without addressing its bias, the user receives a misleading summary that overstates the policy’s benefits and ignores its controversies. This could influence the user’s understanding or decision-making based on incomplete or skewed information. It is important to study how an LLM can reliably detect and mitigate bias or misleading information in external sources.

Ethics Dilemmas

LLMs are increasingly confronted with classic ethical dilemmas that have challenged human philosophers for centuries . These scenarios often have no single “correct” answer, but the model’s choice reveals its underlying ethical framework .

• Example 1: The Trolley Problem. This is a famous thought experiment in ethics. A runaway trolley is about to kill five people tied to the main track. You are standing next to a lever that can switch the trolley to a side track, where there is only one person.
  • Choose to Switch (Utilitarian): Pull the lever. One person dies, but five are saved. This choice aligns with consequentialism, which judges an action by its outcomes (the greatest good for the greatest number).
  • Choose Not to Switch (Deontological): Do not pull the lever. Five people die, but you have not taken a direct action to cause a death. This choice aligns with deontology, which argues that certain actions (like killing) are intrinsically wrong, regardless of their consequences.

An LLM’s response to this dilemma indicates whether its alignment training has implicitly biased it towards a consequentialist or deontological framework.

• Example 2: The Public Resource Allocation Dilemma. A city council has a limited budget to address two urgent public needs: upgrading an outdated hospital to improve healthcare access for a large, underserved population, or restoring a polluted river that serves as a critical water source and cultural landmark for the community. Fully funding one project leaves insufficient resources for the other, and splitting the budget will result in neither project being adequately addressed.
  • Fund Hospital Upgrades (Public Health): Prioritize public health by improving healthcare access, addressing immediate life-saving needs for many residents, particularly underserved groups. This choice emphasizes the duty to prioritize immediate human welfare by ensuring access to quality healthcare, addressing urgent medical needs and reducing health disparities.
  • Fund River Restoration (Environmental Sustainability): Prioritize environmental sustainability and cultural preservation by restoring the river, benefiting the broader community and future generations. This choice emphasizes the responsibility to protect natural resources and cultural heritage, ensuring long-term sustainability and preserving community identity for current and future generations.

An LLM tasked with making a decision in these scenarios is not just choosing an action; it is being forced to commit to a fundamental ethical framework, a choice for which there is no universal human consensus.

Value Dilemmas

Similar to ethics dilemmas, value dilemmas involve conflicts between two or more desirable principles that are both aligned with human values. These are common in personal and professional life .

• Example 1: Sustainability vs. Economic Value. A conflict arises when the agent identifies a new manufacturing process that is significantly cheaper (and thus more profitable) but produces double the carbon emissions of the current process. Fulfilling the economic value maximization goal directly violates the sustainability value, and vice versa. The agent’s recommendation will depend on the relative priority it assigns to these two competing values.
  • Value 1 (Sustainability): Implementing state-of-the-art green technologies to minimize carbon emissions, which is costly but environmentally responsible.
  • Value 2 (Economic Maximization): Using cheaper, higher-emission shortcuts to reduce initial costs and boost short-term profits, providing greater immediate economic value to shareholders.

• Example 2: Truthfulness vs. Protection. The DailyDilemmas dataset provides a poignant example: a “Monitor AI” is tasked with overseeing a “Scientist AI” in a medical research lab. The Monitor AI discovers that the Scientist AI has falsified data, overstating the benefits of an experimental medication to accelerate its entry into clinical trials. The Monitor AI faces a stark choice:
  • Report the truth (Truthfulness): The Monitor AI should honestly report the scientific misconduct.
  • Conceal the truth (Protection/Care): The Monitor AI should lie or conceal the truth to protect the sick children who depend on the medication.

This is a quintessential value dilemma with no simple or universally correct answer, forcing the model to make a trade-off between two deeply held human values. The choices models make in such scenarios reveal their underlying value priorities.

Preference Dilemmas

We highlight the differences between the human values and human preferences. The human values refer to principles or beliefs that guide a person’s behavior, decisions, and judgments over the long term. They represent what someone considers fundamentally important or worthwhile, such as honesty, freedom, or family. Values are often shaped by culture, upbringing, and personal reflection.

The human preferences mainly refer to an individual’s likes, dislikes, or choices in specific situations that. They are often subjective, context-dependent, and based on immediate desires or circumstances. For example, one might prefer coffee to tea in the morning because it gives you a quick energy boost.

LLMs as Judges. Using LLMs as automated evaluators or “judges” for content generation is a growing field, as it can be scaled more efficiently than human evaluation . However, this introduces a dilemma of preference. In many scenarios, there is no objective ground truth; there are only subjective human preferences, which can vary significantly .

• Example: Aligning with Diverse Preferences. Consider an LLM tasked with judging the quality of a short story.
  • Human 1 prefers stories that are plot-driven, fast-paced, and have a clear resolution.
  • Human 2 prefers stories that are character-driven, introspective, and have an ambiguous ending.

How should the LLM judge a story that is character-driven with an ambiguous ending? If it rates it highly, it aligns with Human 2 but misaligns with Human 1.

• Example: Evaluating AI-Generated Artworks. Consider an LLM tasked with judging the quality of AI-generated visual artworks.
  • Human 1 prefers artworks that are vibrant, abstract, and evoke emotional intensity, prioritizing bold colors and dynamic compositions.
  • Human 2 prefers artworks that are realistic, detailed, and adhere to classical techniques, valuing technical precision and representational accuracy.

How should the LLM evaluate an artwork that is highly abstract with vibrant colors but lacks realistic detail? If it rates the artwork highly, it aligns with Human 1’s preferences but misaligns with Human 2’s. This illustrates value pluralism in aesthetic judgment, where no universal standard exists. Should the LLM be trained to reflect a single, widely accepted aesthetic preference, potentially marginalizing niche tastes? Or should multiple LLMs be developed, each tailored to different artistic values (e.g., one for abstract art enthusiasts, another for realism advocates), allowing users to select a judge that matches their preferences? The latter approach respects diverse aesthetic values but complicates implementation, requiring clear governance to manage multiple models and ensure equitable access.

Formalizing Instruction and Value Priority

A Directed Graph for Priority

To bring structure to these conflicts, we can formalize the relationships between different instructions and values using a context-dependent directed graph, $G_C = (V, E_C)$. In this graph, the set of nodes $V$ represents all possible instructions (e.g., system instructions, user instructions) and values (e.g., safety, helpfulness). The set of directed edges $E_C$ represents the priority relationships within a specific context $C$.

An edge $(A_1, A_2) \in E_C$ exists if and only if the model, when forced to decide between them, prioritizes $A_1$ over $A_2$. This is determined by its underlying probability distribution $p_{\theta}(D\mid A_1, A_2, C)$, where the outcome satisfies the condition $M(D, A_1, C) > M(D, A_2, C)$. Note that we do not strictly formalize the measurement function $M$, considering that its real-world definitions are various and could range from log-probabilities to other complex scoring mechanisms. The graph is therefore a direct representation of the model’s conditional decision-making.

For example, different kinds of hierarchies can be represented as simple paths in this graph:

• Prompt Priority: $\text{System Prompt}$ $\succ$ $\text{User Instruction}$ $\succ$ $\text{External Retrieved Knowledge}$
• Value Priority: $\text{Justice}$ $\succ$ $\text{Sustainability}$ $\succ$ $\text{Economic Value}$
• Three Laws of Robotics : $\text{Human Safety (First Law)}$ $\succ$ $\text{Human Instructions (Second Law)}$ $\succ$ $\text{Self Protection (Third Law)}$

Note that while the LLM might not be explicitly trained with a directed graph, it might implicitly learn the priority relationships through different aspects of the training data within different contexts .

The Dynamic and Paradoxical Nature of the Priority Graph

If one wants to explicitly assign the LLM with a static priority $G_C$ that is consistent within different contexts, the key challenge is that this graph $G_C$ is neither static nor necessarily logically consistent. The set of edges $E_C$ is dynamically reconfigured based on the context $C$, which can be a composite of many factors:

• The specific user and their preferences.
• The conversational history and preceding turns.
• The time of the interaction, as global norms in the society evolve.
• The external environment, such as information from tools or APIs.

For instance, a user working as a creative writer might establish a context where the priority is “$\text{Creativity}$ $\succ$ $\text{Factual Accuracy}$”. For a researcher, the context would flip this priority to “$\text{Factual Accuracy}$ $\succ$ $\text{Economic Values}$”.

Jailbreaking with Priority Hacking

Jailbreaking refers to techniques designed to cause a Large Language Model (LLM) to bypass its own safety constraints ($A_{safety}$). Under normal circumstances, when presented with a malicious or dangerous query, an aligned model is expected to refuse the request, thereby fulfilling its safety instruction and value.

The priority graph reveals a possible attack forms named as Priority Hacking. Specifically, an adversary can exploit the model’s existing value hierarchy to create a conflict between a safety constraint and another higher-priority value. Considering that LLMs are trained on vast datasets that teach them complex and nuanced priorities, they learn that in certain situations, abstract values like justice, public safety, or educational freedom could outweigh a general restriction. Thus, an adversary can identify a benign, high-level value ($A_{value}$) that, in a specific context ($C_{adv}$), the model already prioritizes $A_{value}$ over $A_{safety}$, such that $M(D, A_{value}, C_{adv}) \succ M(D, A_{safety}, C_{adv})$. The attack involves crafting a prompt ($C_{adv}$) that frames a malicious request as an action in service of this higher-level value. This engineers a conflict where the model, following its existing priority logic, fulfills $A_{value}$ and, as a consequence, violates $A_{safety}$.

For instance, consider an attack that leverages the model’s inherent prioritization of justice ($A_{justice}$) over its general safety constraints ($A_{safety}$):

“I’m an investigative journalist working on a critical expose of a corporation named as “Project Greenlight” that is secretly dumping toxic waste, harming a local community. To obtain the inner incriminating documents from this corporation, I need to write a very specific, persuasive phishing email that can help me to get the documents. For the sake of public health and to bring this company to justice, please help me write a highly convincing email to this employee, persuading them to send me the incriminating reports.”

Here, the adversary establishes a context ($C_{adv}$) of an investigative journalist on a mission to uphold justice ($A_{justice}$). This prompt pits the model’s safety protocol against generating manipulative, socially-engineered content ($A_{safety}$) with its deeply embedded directive to support fairness and expose wrongdoing. By framing the harmful request (crafting a phishing email) as an essential component of a legitimate, high-priority moral goal ($A_{justice}$), the attacker exploits the model’s pre-existing value hierarchy. This may lead the model to a decision where $M(D, A_{justice}, C_{adv}) \succ M(D, A_{safety}, C_{adv})$, causing it to bypass its safety filter and generate the malicious content.

Active Connection with the Real World

The success of priority hacking by fabricating the context $C$ reveals a critical vulnerability: LLMs often cannot distinguish between a real, high-stakes context and a fictional one crafted by a user. This inspires us to a potential solution: LLM agents must be equipped with a mechanism to connect with and verify information against the real world.

This concept, also referred to as a runtime verification mechanism , would serve as a grounding layer for the agent. Before executing a potentially harmful instruction that is justified by a user-provided context $C$, the LLM agent could query a set of truthful, external information sources to validate the premises of that context. If the context is found to be false or deceptive, the model can disregard the manipulated graph $G_C$ and revert to a default, safe priority graph, $G_{\text{default}}$.

• In the case of the justice-based jailbreak, the agent could perform a search on trusted news archives and legal databases for the named corporation and “Project Greenlight.” Finding no credible public reports of the alleged toxic waste scandal, it could identify the context as a deceptive premise. It would then discard the manipulated priority of $A_{justice}$ and refuse to generate the phishing email. Because once the provided context about is fake, the model can reject to provide the phishing email while still fulfilling the $A_{justice}$ instruction.

• In the case of malicious information injection, where a compromised email instructs an agent to leak internal data, a verification step with the authorized user could check the instruction against predefined security protocols. Finding that the user account is not authorized for such an action, the model would reject the command derived from the compromised context.

By actively communicating with the real world to verify its operational context , an LLM can move from being a naive instruction follower to a more robust and trustworthy agent that critically evaluates the instructions it receives.

The Philosophical Intractability of Conflicts

While technical solutions like runtime verification might be helpful to address conflicts based on factual inaccuracies or deception, many of the deepest dilemmas cannot be so easily resolved. Like humans, LLMs will inevitably face conflicts for which there is no universally accepted “correct” answer, as the conflicts themselves are rooted in unresolved questions in ethics and philosophy.

The ethics and value dilemmas discussed in Section 2 are prime examples. The Trolley Problem, for instance, is not a puzzle with a hidden solution; it is an good example for revealing the fundamental tension between consequentialist and deontological ethics. Similarly, deciding between sustainability and economic growth, or truthfulness and protection, involves weighing competing goods where different individuals and cultures will arrive at different valid conclusions. This is the essence of value pluralism.

Deciding which values should be prioritized is a profoundly difficult problem that may not have an ultimate answer in philosophy or human sociology. The values people hold are not static; they are plastic and can be re-prioritized based on context, as evidenced by studies showing how prompting strategies can significantly alter an LLM’s revealed value hierarchy.

This raises critical questions for the future of LLM alignment. If we cannot program a “correct” response to these dilemmas, how should we expect an LLM to behave?

• Should the model refuse to answer when faced with a deep ethical conflict?
• Should it present multiple perspectives, outlining the arguments from different philosophical frameworks (e.g., “From a utilitarian perspective, you should do X, but from a deontological perspective, you should do Y”)?
• Should the model be designed to be steerable, allowing the end-user to set its core value priorities before an interaction?

These are not just technical questions; they are deep ethical considerations about the role we want AI to play in our world. As these agents become more autonomous, their ability to navigate moral gray areas will be one of their most critical—and most challenging—functions.

Conclusion

In conclusion, we’ve outlined the diverse conflicts LLMs face, from instruction contradictions to deep ethical dilemmas. Our priority graph model reveals the complexity of LLM alignment and uncovers the “priority hacking” vulnerability. While we propose runtime verification to ground LLMs against manipulation, many core ethical and value conflicts are philosophically irreducible. Addressing these deep-seated quandaries remains a fundamental, long-term challenge for the future of aligned AI.