Traditional parallel query(PQ) execution, it adopted Direct Path Read to load data which bypassed the database buffer cache(buffer cache) and load directly from the disks. In-Memory parallel execution takes advantage of this large aggregated database buffer cache. By having parallel execution servers access objects using the database buffer cache, they can scan data at least ten times faster than they can on disk. It allows you to cache your hottest tables across buffer caches of multiple RAC nodes
What about Exadata ? In-memory PQ is a great option, only if I/O is your bottleneck. With Exadata you get Terabytes falsh cache and Flash I/O , In-memory PQ is probably not a got idea.