It is embarrassing for glibc not to check for overflow in calloc implementation prior to 2002. It is not only a security flaw but also violation of C Standards (even the first version ratified in 1989, usually referred to as C89).
The standard reads as follows:
void *calloc(size_t nmemb, size_t size);
The calloc function allocates space for an array of nmemb objects, each of whose size is size.[...]
and,
The calloc function returns either a null pointer or a pointer to the allocated space.
So if it cannot allocate space for an array of nmemb objects, each of whose size is size, then it has to return null pointer.
The standard reads as follows:
and, So if it cannot allocate space for an array of nmemb objects, each of whose size is size, then it has to return null pointer.