Up until, and during the 1920's, all men's swimwear was knitted, which as you can imagine made it extremely heavy when wet. The result of this excess weight meant the bathing suit would often fall down.
All men's bathing suits, due to public indecency laws had to be one piece and have a skirt or skirt effect. It was either this or fly fronted flannel trousers to the knee with a vest. Sometime after this, the one piece 'speed suit' came along, with deeply slashed arm holes and closed leg trunks. This was the beginning of speed over modesty.
In 1933 a removable top was invented, although all this produced was scandals and arrests for public indecency. It was not until 1937 that men were allowed to go topless in public.