I did it. Result: Inside the category e.g. portfolio-1 the post with two categories – home and portfolio-1 – was excluded from next/previous.
I am astonished that WordPress does not recognize that e.g. portfolio-1 is the only category for next/previous. Why does it jump to home? Is there any category-preference or hierachy WordPress is following?