The research paper “Variationally consistent quadratic finite element contact formulations
for finite deformation contact problems on rough surfaces” by S. Sitzmann et al. (mentioned in the CalculiX manual as one of the references for the mortar contact formulation) discusses so-called “dual gap” problem. I guess it could explain this behavior:
That means that a slave node can be set active, even if there is still a gap between the bodies and some non-physical pushing apart occurs. This situation can occur due to high gradients in the model or high curvatures modelled with very few elements or sharp edges in the continuum model. […] Also this can be triggered by a coarse curved slave mesh and a finer curved master mesh. These problems arise due to the non-positive part of the dual basis functions […]. In many situations, the problems linked to the dual basis functions can be avoided by simply switching the master and the slave side or by refining the mesh in this region, since the severity of the induced numerical artefacts are h-dependent.