If you leave it as-is, you’re not going to hurt anything or cause damage or anything. That’s up to you. If you leave it, it’s likely to crack & crumble a certain amount over time. My CDO wouldn’t allow that for myself, personally. I’d dig it out and apply a 100% silicone bead there.