I don't see this as a problem, there is a price to pay for performance in multithreaded environments, and in this case, this is one of these things.
You can switch to redis caching, allow it to store like X GB then use an eviction policy to remove old data, this way you'll always have at most x GB of data in cache.