Towards an Automated Test of LLM Security Knowledge

Large language models LLMs are increasingly used for a range of software, hardware and human-centered security tasks. Consequently, LLM performance on security tasks is an active area of measurement and research, often with a focus on identifying areas in which LLM security "knowledge'' may be...