Can I restrict segments to profiles who have only purchased from a single category?

We have a category, lets call is ‘category A’, and we’d like to create a segment that only includes profiles who have ONLY purchased products from this category.

However, when I set up a segment, I can’t seem to find any setting that will allow me to do this. For instance:

• I have two user profiles.
• The first user has purchased a product from both ‘category A’ and ‘category B’
• The second user has only purchased from ‘category A’

No matter what I do in terms of filtering, I cannot get the segment to only include the profile that has only purchased from ‘category A’.

Any ideas would be much appreciated.