Getting people into space is easy. Bringing them back is the hard part. Bringing them back alive is even harder.
Actually, getting them from the launch pad and into outer space is the hardest part. We've been lucky to have had very few tragedies at launch.
The amount of force exerted on the entire vehicle and support platforms seems to be more risky than being in a zero gravity, zero atmosphere environment where you no longer need massive booster rockets to break out of orbit.
In space, it's the failed component or collision with space junk that would be my main worry. If the equipment is top quality, and there are no surprises heading for you, you're good until that thrill ride known as reentry.
Shuttle Columbia disintegration on re-entry -- 2003.
Shuttle Challenger experienced a launch booster failure, vehicle disintegration during launch - 1986.
The only deep space fatalities were Soyuz 11. Three cosmonauts died after undocking from a space station. They experienced complete depressurization due to a defective valve. The recovery mission found all 3 bodies aboard the craft. - 1971.
The majority of astronaut fatalities were from jet flight training missions, one fire during a test aboard Apollo 1 - 1967, and drowning during water training or testing.
https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidentsFrom a statistical viewpoint, space travel has a very good record once you're beyond the Kármán line.
https://en.wikipedia.org/wiki/K%C3%A1rm%C3%A1n_line