The simplest way to handle more data using more cores (whether on a single machine or in cluster) is to partition it into disjoint subsets, and work on each subset in isolation. In the database world this is called sharding, and it makes scaling relatively simple; only downside is -— it makes writing compellingly complex applications hard.

via Parallel Universe • A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceships on My Laptop.