You'll be seeing a lot less of these 'Summer House' cast members

"They will still be on, but they won't be back living in the house," said a source close to the Bravo show.