10.11.2026
CryptanalysisBench - Can LLMs do Cryptanalysis?
Abstract
Cryptanalysis—the task of finding attacks against cryptographic schemes—sits at the intersection of mathematical reasoning and programming, two areas where LLMs have made rapid progress. Just like math and programming, cryptanalytic attacks are formally defined and can be unambiguously verified. This raises the question if cryptanalysis might experience a similar acceleration in progress through LLMs. In this paper we introduce CryptanalysisBench, a benchmark of 114 tasks spanning six families of cryptographic primitives (block ciphers, hash functions, etc) drawn primarily from four NIST standardization competitions. Each task asks an agent to break an implementation of a cryptographic primitive by winning a formal security game. The benchmark has three tiers: (1) primitives with known practical breaks; (2) scaled-down variants of primitives without one; (3) a challenge set of unbroken production primitives. We evaluate Claude Opus 4.7, GPT-5.5 and Claude Mythos: all three solve a majority of Tier 1 (72.4%, 79.3%, and 75.9% respectively), while Tier 2 remains largely out of reach. One notable exception in Tier 2 is the SPoC AEAD, for which Mythos proposed a novel full key recovery attack. For Tier 1 successes, we provide a fine-grained analysis distinguishing paper recall from source-level rediscovery. We release CryptanalysisBench both as a forecasting tool to track when AI cryptanalytic capability becomes a serious factor, and as scaffolding for subjecting candidate schemes to more attacks before they are deployed.
Bio
Prof. Orr Dunkelman is a full professor at the Computer Science Dept. in the University of Haifa and a Guest Professor at the Technical University of Berlin. His research interests cover cryptography, with emphasis on cryptanalysis, privacy, with emphasis on biometric data, and computer security. He has served as the head of the Sub-center for Biometrics and its Applications here at the Center for Cyber, Law and Policy, and is a co-founder of the center (and was the center's vice-director). He served on the board of the International Association for Cryptologic Research (IACR), as a delegate to the ISO/IEC SC27 WG2 and WG3 (as an expert in cryptography and computer security), and he is a co-founder of the "Privacy Israel" NGO. Currently he serves as the chair of the Fast Software Encryption steering committee.
During his academic career, Prof. Dunkelman published more than 130 papers in international venues, including the introduction of new cryptanalytic techniques and best known cryptanalysis results against specific ciphers (such as the AES). As part of his analysis work, he is also a co-designer of multiple cryptographic schemes such as the HAIFA framework for hash functions, the KATAN and KTANTAN block ciphers, and the SHAvite-3 hash function. He served on almost 100 program committees (including EUROCRYPT, CRYPTO, ASIACRYPT, and Usenix), and as a program chair of 6 conferences (including EUROCRYPT 2022 and FSE 2009). He won numerous awards such as the Krill prize (2014) and Best Paper Awards (e.g., Crypto 2012 and FSE 2012).
Photo provided by speaker
Cryptanalysis—the task of finding attacks against cryptographic schemes—sits at the intersection of mathematical reasoning and programming, two areas where LLMs have made rapid progress. Just like math and programming, cryptanalytic attacks are formally defined and can be unambiguously verified. This raises the question if cryptanalysis might experience a similar acceleration in progress through LLMs. In this paper we introduce CryptanalysisBench, a benchmark of 114 tasks spanning six families of cryptographic primitives (block ciphers, hash functions, etc) drawn primarily from four NIST standardization competitions. Each task asks an agent to break an implementation of a cryptographic primitive by winning a formal security game. The benchmark has three tiers: (1) primitives with known practical breaks; (2) scaled-down variants of primitives without one; (3) a challenge set of unbroken production primitives. We evaluate Claude Opus 4.7, GPT-5.5 and Claude Mythos: all three solve a majority of Tier 1 (72.4%, 79.3%, and 75.9% respectively), while Tier 2 remains largely out of reach. One notable exception in Tier 2 is the SPoC AEAD, for which Mythos proposed a novel full key recovery attack. For Tier 1 successes, we provide a fine-grained analysis distinguishing paper recall from source-level rediscovery. We release CryptanalysisBench both as a forecasting tool to track when AI cryptanalytic capability becomes a serious factor, and as scaffolding for subjecting candidate schemes to more attacks before they are deployed.
Bio
Prof. Orr Dunkelman is a full professor at the Computer Science Dept. in the University of Haifa and a Guest Professor at the Technical University of Berlin. His research interests cover cryptography, with emphasis on cryptanalysis, privacy, with emphasis on biometric data, and computer security. He has served as the head of the Sub-center for Biometrics and its Applications here at the Center for Cyber, Law and Policy, and is a co-founder of the center (and was the center's vice-director). He served on the board of the International Association for Cryptologic Research (IACR), as a delegate to the ISO/IEC SC27 WG2 and WG3 (as an expert in cryptography and computer security), and he is a co-founder of the "Privacy Israel" NGO. Currently he serves as the chair of the Fast Software Encryption steering committee.
During his academic career, Prof. Dunkelman published more than 130 papers in international venues, including the introduction of new cryptanalytic techniques and best known cryptanalysis results against specific ciphers (such as the AES). As part of his analysis work, he is also a co-designer of multiple cryptographic schemes such as the HAIFA framework for hash functions, the KATAN and KTANTAN block ciphers, and the SHAvite-3 hash function. He served on almost 100 program committees (including EUROCRYPT, CRYPTO, ASIACRYPT, and Usenix), and as a program chair of 6 conferences (including EUROCRYPT 2022 and FSE 2009). He won numerous awards such as the Krill prize (2014) and Best Paper Awards (e.g., Crypto 2012 and FSE 2012).
Photo provided by speaker