Hi,
Are you talking about the categories that are listed in the Document section when editing a post?
Exactly.
Looks like there are some plugin to achieve this for classic editor by changing the check boxes into a dropdown list or radio buttons.
https://www.wpbeginner.com/plugins/how-to-enforce-one-category-per-post-in-wordpress/
Now I limit this by jQuery like this
var cat_checklist = $('.categorychecklist input[type=checkbox]');
cat_checklist.click( function() {
$(this).parents('.categorychecklist').find('input[type=checkbox]').attr('checked', false);
$(this).attr('checked', true);
});