Broke my diet like 3 days in a row?

the first day was my dad's birthday, so I think I was justified to have a slice of cake. But the second day I was just soooooooooo hungry for all the foods I couldn't eat, I just binged. The third day i was at my grandma and grandpas house and they keep a lot of that food in the house, and I hadn't eaten breakfast so that was really all I could eat. It wasn't too bad though. But I've been on this diet for a long time solidly and lost a bunch of weight, but I gained weight on just those 3 days. This isn't a calorie-restrictive diet, and it includes plently of protein so my metabolism was not in starvation mode. But how can I make up for this? I really want to lose weight and I'm willing to be patient but I don't want any lost time. Extra exercise, maybe? Any time I try to restrict my diet more than I am right now it generally doesn't last more than a day.