Plaque buildup can also accumulate, leaving teeth dirty and yellow. Brushing, flossing and regular dentist visits can usually remove yellow plaque buildup, but over time without regular care, plaque can actually permanently stain your teeth.