Group testing theory in network security explores a new branch of group testing theory with an application which enhances research results in network security. This brief presents new solutions on several advanced network security problems and mathematical frameworks based on the group testing theory, specifically denial-of-service and jamming attacks. A new application of group testing, illustrated in this text, requires additional theories, such as size constraint group testing and connected group testing. Included in this text is a chapter devoted to discussing open problems and suggesting new solutions for various network security problems. This text also exemplifies the connection between mathematical approaches and practical applications to group testing theory in network security.