Fix code according to fuzzer findings - #30286
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
In 5.0 cv::Mat_<bool>::depth() returns CV_Bool instead of CV_8U, so
boundingRect routed boolean masks into pointSetBoundingRect, which expects
a CV_32S/CV_32F point set and threw. CV_Bool is a 1-byte 0/1 mask type, so
handle it like CV_8U/CV_8S via the byte-wise maskBoundingRect path.