The memory landscape in Germany has been lauded for its pluralism: for reckoning with the past not only critically but in its many complex facets. Nevertheless, particularly victims of repression in East Germany lament that their plight is not adequately represented and some have recently affiliated themselves with the Alternative for Germany (AfD) party and other groups on the far-right spectrum. This article seeks to explain the seeming contradiction between existing pluralism in German public memory and dissatisfaction with it by tracing how memory activists have shaped memory policy and institutions. Based on extensive interview and archival research, I argue that the infiltration of civil society into the institutions that govern memory in large part explains the strength of critical memory in unified Germany and the country’s ability to accommodate a variety of pasts. However, there is also a distinct lack of pluralism when it comes to the rules of “how memory is done,” to the exclusion of more emotional and politicized approaches that are sometimes favored by some victims’ groups. Using the case of the recent debate about the Hohenschönhausen Memorial, I contend that this explains some of the attraction felt by these groups towards the right.