Currently inplaceSelect component does not support SelectItemGroup nested elements. It displays only SelectItemGroup labels and ignores nested SelectItem elements. This reduces the styling and grouping possibility for select elements, because jsf select items do not allow 'style' atribute either.
It would be great to have the possibility to use grouped select items in inplaceSelect. Seems this is more the bug, than feature request, but feel free to change the type, if necessary.