The popularity of the term grew in the mid-1960s when the British pound fell in value against foreign currencies. Many Britons blamed Swiss bankers for possessing significant holdings of sterling.
The phrase was made famous during the 1964 Sterling Crisis in Britain when Harold Wilson, a politician in the British Labor Party, referred to Swiss bankers and financial speculators as "Gnomes of Zurich". In his speech to the House of Commons, Wilson blamed Swiss bankers for the decline of the pound on foreign exchange markets, blasting the "Gnomes of Zurich" for harmful speculating practices.
Swiss banks are regarded highly for their rigorous security measures and client discretion. However, it is these secretive practices that contribute to the continued use of the term "Gnomes of Zurich" in reference to Swiss bankers.
For more on this topic, read Elves And Gnomes: A Fairytale World Of Investing.
This question was answered by Richard C. Wilson.