This innovative research demonstrates how simple addition operations can be used to create more energy-efficient language models without sacrificing performance.
The authors propose a novel architecture that significantly reduces computational complexity and energy consumption while maintaining model capabilities.
The study provides empirical evidence showing substantial energy savings compared to traditional transformer architectures.
Chomsky and others have claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim.
This study develops a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible (random and irreversible shuffles of English words), and on the other, languages considered impossible in linguistics, particularly those with rules based on counting word positions.
A wide range of evaluations assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, performed at various stages throughout training to compare the learning process for each language. The core finding is that GPT-2 struggles to learn impossible languages compared to English as a control, challenging Chomsky’s claim.
The authors hope their approach opens a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages, as tools for cognitive and typological investigations.
This paper bridges the gap between AI ethics and environmental sustainability, proposing a framework that considers both ethical and environmental implications of AI development.
The research examines how ethical AI principles can be aligned with environmental sustainability goals, addressing issues such as computational efficiency, resource allocation, and environmental justice.
The authors propose concrete guidelines for developing AI systems that are both ethically sound and environmentally sustainable.
This comprehensive review examines state-of-the-art approaches for making deep learning more energy-efficient across the entire stack, from hardware to algorithms.
The research analyzes various efficiency techniques including model compression, neural architecture search, and hardware-software co-design for energy-efficient deep learning.
The authors provide detailed case studies and empirical evaluations of different approaches, offering insights into their effectiveness for reducing energy consumption while maintaining model performance.
This paper presents a methodology for accurately measuring the carbon emissions of AI workloads running in cloud environments.
The research provides detailed measurements across different cloud providers and regions, showing how carbon intensity can vary significantly based on location and time of day.
The authors also release tools and best practices for researchers and practitioners to measure and reduce the carbon footprint of their AI applications.
This comprehensive survey examines the environmental impact of artificial intelligence throughout its lifecycle, from development to deployment and maintenance.
The paper provides a systematic analysis of the challenges in making AI more sustainable, including hardware efficiency, algorithm design, and operational practices.
The authors identify key opportunities for reducing AI’s environmental footprint and propose a research agenda for sustainable AI development.