The origins of the collection lay in the holdings of Johann Wilhelm's grandfather, Wolfgang Wilhelm, which Johann Wilhelm expanded significantly through his second marriage to Anna Maria Luisa de' Medici. The Dutch portrait painter Jan Frans van Douven served as the Elector's art agent from 1684 and became superintendent of the collection around 1700, playing a key role in shaping its character. The three-winged gallery building was erected from 1709 onwards on the south side of Düsseldorf Castle, making it one of the earliest independent museum buildings in Europe. The collection was first catalogued by Gerhard Joseph Karsch in 1719, followed by François-Louis Colins in 1756, Nicolas de Pigage in 1778, and Valentine Green in 1793. A significant early loss occurred in 1717 when Anna Maria Luisa withdrew her share of the collection and returned to Florence after Johann Wilhelm's death. The gallery faced repeated threats from warfare: in 1795, during the First Coalition War, the collection was evacuated to Mannheim and then to Munich to escape the advancing French Revolutionary army. After the Peace of Lunéville, the collection was returned to Düsseldorf, but the territorial upheavals of 1805-1806 proved fatal. Following a complex exchange of territories among Prussia, Kurpfalz-Bayern, and France, Elector Maximilian IV Joseph ordered the transportable collection removed to Schloss Kirchheimbolanden, and subsequently to Munich, where it was incorporated into the Bavarian royal collections. Düsseldorf citizens contested this removal, arguing the collection constituted public property of the Duchy of Berg, and in 1846 founded the Verein zur Errichtung einer Gemäldegalerie zu Düsseldorf to build a new collection.