Panel autoscroll clientarea not updating correctly when content reduced

Short version:  If a panel contains an large object, and scrollbars are at non-zero positions, and the object in the panel reduces in size, the panel client area does not update correctly. How can this be fixed?

Detailed version:
I have a PictureBox in a Panel control so that scrollbars can be used when the image is large. The bitmap is large, and the PictureBox has SizeMode set to "Stretch", the panel has AutoScroll set to True. So at the start of the program the bitmap is correctly scaled inside the picturebox.

If I press a button in the program, the Picturebox switches SizeMode=PictureBoxSizeMode.AutoSize. Everything is still good, the bitmap is at true size, scrollbars appear.

Now if I scroll to the bottom of the image, switch back to SizeMode=Stretch (the user wants to see the full image again), the image stretches correctly, however the scrollbar does not disappear, and scrolling up there is a large grey area. It is like the client area does not reduce correctly in the panel when the scrollbars are not in the 0 position.

Note that this only happens if the scrollbars are at non-zero positions.

How can I get the panel client area to update (reduce) correctly when the content of the panel reduces in size?