Have any of you happen to work on high requests per second web api's? There's a project I'm working on where I need to achieve up to 100k rps.
My question would be - what database solution would be up for the job? Heavy caching is required. There won't be a huge amount of data in the database at any one time as the data will be updated hourly with no leftovers. My current solution is Apache Cassandra, even though it's more suited for huge data amounts, I do believe there are better solutions. Also, powerful hardware is not a problem, I can throw as much resources at it as I need.
Hopefully there is someone with the experience to help out. Thank you in advance!
If your looking at web software for http/htttps calls via api I would recommend nginx over apache. Then your web software or whatever could use memcached on top of that.
You can also use nginx and then pass the requests along to apache depending on your needs.
Well basically, I need to spew out small portions of data in json or xml from a couple databases that are related to each other (not too sure about how much data I'll end up having in there, but not anything too crazy for memcaches) while keeping very basic statistics data, everything lightning fast.
Memcache is definetly the best solution as I'd rather just have a high-ram easily replaceable server machine than a specialized one with specific components, and nginx will be useful too if it'll come to it eventually. I also found out about Redis while looking up on memcaches, it caches all it can of its database in memory and will soon (if not yet) have clustering support, might as well try that too. Thanks for the suggestions guys!
Memcache or Redis sound good if all you need is to store key values. If you need a proper RDBMS and MySQL won't cut it the latest versions of Oracle and SQL Server Enterprise now have in memory functionality which supports high performance OLTP workloads whilst maintaining full ACID compliance and all the other advantages of old fashioned relational systems.
Well you know, there's boundaries that are called "unreasonable", no need to throw a fortune on hardware when you can achieve the same thing with less :).
For the guys wondering. The solution was Aerospike NoSql database, it was just too perfect for the job: scales well (can be clustered), has nice tools and options, good connectors for common languages, and best of all it's uber fast. Hit the spot really well and I only had to use a single dedicated server for the database...