L1 cache shouldn't be large. Increasing the size of the L1 cache increases the latency. Maybe if you shrink the size of the cloths you wear you can squeeze more into the chair, but the ideal L1 cache has to minimize it's distance from processing. Oversizing adds latency.

Your L2 cache is where you generally try and shove a much bigger cache into it, but it's still got a size constraint for the latency you are after. Further, typically L1 and L2 only serve 1 CPU. To multi-process stuff you'll typically need an even larger L3 cache which is shared among cores.

So the cloths on your chair should be minimal for fast access (L1). You can put more cloths on your bed and dressers or in laundry baskets that can be promoted to the chair if you start needing them more often (L2). You can throw a bunch of cloths into a pile in the corner which sit there for a few years and serve many occasions (L3).

The worst thing is going back to main memory (your closet) to search for specialty cloths you are ultimately going to need to send back to the closet. And heavy help you if you have to swap (do laundry).