CWE-1426: Improper Validation of Generative AI Output
View customized information:
For users who are interested in more notional aspects of a weakness. Example: educators, technical writers, and project/program managers.
For users who are concerned with the practical application and details about the nature of a weakness and how to prevent it from happening. Example: tool developers, security researchers, pen-testers, incident response analysts.
For users who are mapping an issue to CWE/CAPEC IDs, i.e., finding the most appropriate CWE for a specific issue (e.g., a CVE record). Example: tool developers, security researchers.
For users who wish to see all available information for the CWE/CAPEC entry.
For users who want to customize what details are displayed.
×
Edit Custom FilterThe product invokes a generative AI/ML
component whose behaviors and outputs cannot be directly
controlled, but the product does not validate or
insufficiently validates the outputs to ensure that they
align with the intended security, content, or privacy
policy.
This table specifies different individual consequences
associated with the weakness. The Scope identifies the application security area that is
violated, while the Impact describes the negative technical impact that arises if an
adversary succeeds in exploiting this weakness. The Likelihood provides information about
how likely the specific consequence is expected to be seen relative to the other
consequences in the list. For example, there may be high likelihood that a weakness will be
exploited to achieve a certain impact, but a low likelihood that it will be exploited to
achieve a different impact.
This table shows the weaknesses and high level categories that are related to this
weakness. These relationships are defined as ChildOf, ParentOf, MemberOf and give insight to
similar items that may exist at higher and lower levels of abstraction. In addition,
relationships such as PeerOf and CanAlsoBe are defined to show similar weaknesses that the user
may want to explore.
Relevant to the view "Research Concepts" (CWE-1000)
The different Modes of Introduction provide information
about how and when this
weakness may be introduced. The Phase identifies a point in the life cycle at which
introduction
may occur, while the Note provides a typical scenario related to introduction during the
given
phase.
This listing shows possible areas for which the given
weakness could appear. These
may be for specific named Languages, Operating Systems, Architectures, Paradigms,
Technologies,
or a class of such platforms. The platform is listed along with how frequently the given
weakness appears for that instance.
Languages Class: Not Language-Specific (Undetermined Prevalence) Architectures Class: Not Architecture-Specific (Undetermined Prevalence) Technologies AI/ML (Undetermined Prevalence) Class: Not Technology-Specific (Undetermined Prevalence)
This MemberOf Relationships table shows additional CWE Categories and Views that
reference this weakness as a member. This information is often useful in understanding where a
weakness fits within the context of external information sources.
Research Gap
This entry is related to AI/ML, which is not well
understood from a weakness perspective. Typically, for
new/emerging technologies including AI/ML, early
vulnerability discovery and research does not focus on
root cause analysis (i.e., weakness identification). For
AI/ML, the recent focus has been on attacks and
exploitation methods, technical impacts, and mitigations.
As a result, closer research or focused efforts by SMEs
is necessary to understand the underlying weaknesses.
Diverse and dynamic terminology and rapidly-evolving
technology further complicate understanding. Finally,
there might not be enough real-world examples with
sufficient details from which weakness patterns may be
discovered. For example, many real-world vulnerabilities
related to "prompt injection" appear to be related to
typical injection-style attacks in which the only
difference is that the "input" to the vulnerable
component comes from model output instead of direct
adversary input, similar to "second-order SQL injection"
attacks.
Maintenance
This entry was created by members
of the CWE AI Working Group during June and July 2024. The
CWE Project Lead, CWE Technical Lead, AI WG co-chairs, and
many WG members decided that for purposes of timeliness, it
would be more helpful to the CWE community to publish the
new entry in CWE 4.15 quickly and add to it in subsequent
versions.
More information is available — Please edit the custom filter or select a different filter. |
Use of the Common Weakness Enumeration (CWE™) and the associated references from this website are subject to the Terms of Use. CWE is sponsored by the U.S. Department of Homeland Security (DHS) Cybersecurity and Infrastructure Security Agency (CISA) and managed by the Homeland Security Systems Engineering and Development Institute (HSSEDI) which is operated by The MITRE Corporation (MITRE). Copyright © 2006–2024, The MITRE Corporation. CWE, CWSS, CWRAF, and the CWE logo are trademarks of The MITRE Corporation. |