The singer was seen strolling back to an awaiting car after she enjoyed a bit of shopping at the luxurious baby boutique on Thursday