Full Report
It seems that when a new technology becomes practical, there is always a rush by self styled influencers to apply this solution to whatever problems they can think of. Those who question the applicability of this new technology are considered naysayers or even Luddites. Nevertheless, there is a history of overblown, oversold technologies. Remember Blockchains? […]
Analysis Summary
# Main Topic
The overzealous and often unfounded application of new technologies, specifically Artificial Intelligence (AI) and Large Language Models (LLMs), by self-styled influencers onto problems where their applicability is questionable, drawing parallels to past overhyped technologies like Blockchains. The core concern discussed is the danger of relying on inadequately understood AI systems in mission-critical fields like professional engineering.
## Key Points
- New technologies, like AI, face a rush of promotion despite historical precedent showing many solutions are oversold.
- LLMs operate as neural networks of probabilities, capable of eloquence but lacking genuine understanding or reasoning, often leading to "hallucinated" answers.
- AI differs significantly from established engineering tools because its decision-making process (why its output looks the way it does) is nearly impossible to explain, unlike deterministic software or junior engineers who possess self-awareness.
- AI lacks responsibility and formal reasoning; professional engineers remain legally and ethically accountable for its output, requiring intensive human review.
- The legal profession has already seen misconduct cases due to inappropriate AI use—engineering must heed these warnings.
## Threat Actors
- **Self-styled influencers/Tech-leaders:** Those aggressively promoting AI solutions without regard for technical limitations or appropriateness for specific professional tasks.
- **No specific malicious threat actor group or TTPs related to a cyberattack were identified; the focus is on professional/ethical risk.**
## TTPs
- The "TTP" discussed relates to the **deployment misuse** of unverified AI output:
- Using AI as an alleged replacement for junior engineers or defined engineering analysis tools.
- Trusting or deploying AI-generated solutions without thorough validation due to the "hallucination" risk.
- Ignoring the lack of formal reasoning in AI output.
## Affected Systems
- **AI/LLM Systems:** Specifically those trained and deployed in professional engineering contexts.
- **Professional Engineering Roles/Projects:** Any design work or decision-making relying on unreviewed AI output.
- **Clients/Public:** Those whose livelihood and lives depend on well-founded engineering decisions.
## Mitigations
- **Mandatory Human Review:** Professional engineers must remain essential for reviewing all AI outputs, as they are the ones who must take responsibility.
- **Professional Skepticism:** Heed negative experiences from other regulated fields (like the legal profession) regarding inappropriate AI use.
- **Avoidance in Design Work:** For now, avoid using AI for most design-related work until fundamental issues (responsibility, explainability) are addressed.
- **Demand for Explainability:** Do not adopt "post-modernist practice" relying on opaque reasoning; established engineering requires well-founded decisions backed by formal reasoning.
## Conclusion
AI shows potential, but its current implementation, particularly LLMs, poses significant professional risks in engineering due to a fundamental lack of explainability, responsibility, and formal reasoning capabilities. Professional engineers must exercise extreme caution, treating AI outputs as inherently questionable until its deep limitations are resolved, lest they assume liability for flawed solutions derived from opaque processes.