Scope on empty association

I'm trying to create a scope looking at empty association.

I've 4 classes : User, Idea Project and UserJoins.

More than one user can have the same idea or the same project.

I would like to create a scope to isolate users without ideas.

Idea.rb

has_many :user_joins
has_many :users, through: :user_joins

Project.rb

has_many :user_joins
has_many :users, through: :user_joins

User.rb

has_many :user_joins
has_many :ideas, through: user_joins, source: :imaginable, source_type: 'Idea'
has_many :projects, through: user_joins, source: :imaginable, source_type: 'Project'

scope :without_ideas, ->{
  # I'm stuck here.
}

UserJoin.rb

belongs_to :imaginable, polymorphic: true
belongs_to :user

I'm using Rails 3.2.17 and Ruby 2.0.0

Does anyone have an idea to solve this ?