You don't have any direct control over how a Groupbox is displayed - there is no "curved edges" option to the delimiting lines.
You have two options: create your own custom control that emulates a Groupbox but with rounded corners, or handle the Paint event and draw them yourself (this may require a derived control: rounded corners will encroach on the space the control uses for it's contained controls, so you may well have to modify quite a few things to get it to work).
This may help:
Reference Source[
^] - but don't expect this to be a trivial change!