Motivated by GW170817 we examine three independent constraints on the angular structure of gamma-ray burst (GRB) jets that are required by observations of cosmological long GRBs. We find that efficient gamma-ray emission has to be restricted to material with Gamma>50 and is most likely confined to a narrow region around the core. Comparing GRB170817 with the regular population of short GRBs (sGRBs), we show that an order unity fraction of NS mergers result in sGRB jets that breakout of the surrounding ejecta, that their luminosity function must be intrinsically peaked and that sGRB jets are typically narrow with opening angles ~ 0.1 rad. Finally, we perform Monte Carlo simulations to examine models for the structure and efficiency of the prompt emission in off-axis sGRBs. We find that only a small fraction, 0.01-0.1, of NS mergers detectable by LIGO/VIRGO in GWs is expected to be also detected in prompt gamma-rays, and GW170817-like events are very rare.