As we’ve learned from the behind the scenes drama at Grey’s Anatomy, ABC doesn’t have a problem saying goodbye to difficult actors. Unfortunately, that’s a lesson Balthazar Getty may be learning the hard way.

According to multiple sources, Getty could soon be getting more time to divide amongst his four kids and his new squeeze. While there’s a chance the actor may be let go from Brothers & Sisters altogether, it’s believed he will at the very least be downgraded from full-time status next season.

These rumors come not-so-coincidentally after reports of increased tension on the program’s set. Balthazar’s affair with Matthew Rhys’ ex has cause him to lose fans since this past summer. His recent work performance and attitude supposedly haven’t helped either. Although I’m sure the 33-year-old’s relationship with Sienna Miller played some part in the personnel decision, Entertainment Weekly claims “budgetary and storyline considerations” were also factors.

So what does this mean for fans of the show? There’ll be one less person yelling around that Walker dinner table. (And that’s a good thing.)