Transformer expressivity
Characterizing what problems neural networks like transformers can and cannot compute, and how architectural choices shape models' computational power.
Welcome! We are a group of interdisciplinary researchers at TTIC interested in natural language, machine learning, and theoretical computer science. A major focus of our work is better understanding the computational power of transformers and other neural sequence models via formal language theory, motivated by fundamental curiosity, demystifying how LLMs work, and inspiring principled improvements to LLM architectures.
Research interests
Formal language theory
Applying automata, logic, and circuit complexity to understand neural networks, LLMs, and language as well as exploring fundamental questions in these areas.
Science of building LLMs
Conceptual understanding of LLM architecture and optimization. Applying these insights to make open LLM development more principled.