The Label Dilemma

I have never been a fan of clothing/accessories with large logos/labels. I have no interest in being a walking advertisement. I do not understand why someone would want to wear clothing that graphically screams out the designers name. Although it bothers me less, I am also not a fan of the more subtle logos/labels that are also visible from the outside - like Kate Spade, Tory Burch and Coach. So here lies my dilemma - to label or not to label, that is the question. When I started my company, Jack and Mary Designs I always sewed my labels on the inside of the garments. I was told by many people I should move them to the outside for name recognition and they also happen to be really cute. So, I started having them sewn on the outside. The other day I was at a wholesale show taking an order from a store owner.She asked me to model one of my hats. She gasped when she saw my label on the outside and stated that she would not place an order with me if I was one of those designers who place their labels on the outside - labels belong on the inside. My labels are small - about 1/2 x 1.0 inch and colorful. I am not sure what to do so I decided to ask for feedback. Any opinions?