Primary caves are formed together with the surrounding rocks. When the rock is formed, the space of the cave remains empty. The cave ist formed in the primary stage of rock development, the formation of the rock itself. There are many different types of primary caves but their number is very small.
The geologic term for this is syngenetic, which is actually just the translation of formed at the same time. However, it is an adjective and not a name, so syngenetic cave is not a term, its a description.
Typical primary caves are: