First, try updating to in5 version 3.8.16. When I export an image in a rounded rectangle frame, it exports a PNG with the corners transparent.
Another option would be to apply Object Export Options (Object > Object Export Options...). In the Object Export Options dialog, click the EPUB and HTML tab. Then set Preserve Appearance From Layout to Rasterize Container, set the Format to PNG, and set the resolution as desired. When you export either by going to in5 > Export HTML5 with in5... or to in5 > Easy Export Wizard... and clicking the Advanced button, go to the Advanced section of the Export HTML5 with in5 dialog. Set Image Quality to Use Object Export Settings.
You could also group the image in the rounded rectangle frame with an invisible rectangle (one with no stroke and no fill) and in the Advanced section of the export dialog, select Render Groups as Images.