Optimizing data collection

[GavraM]

We have now collected over 40 million unique data points (and growing), and have made tons of back-end changes to make our software blazing fast. To get where we are, we revitalized our architecture to better handle, manage, retain, and process huge volumes of data. We’re leveraging Redis for caching which made creating flows incredibly quick!

[JasonW]

It's great to hear that you have collected over 40 million unique data points and have made back-end changes to make your software faster. Revitalizing your architecture to better handle, manage, retain, and process large volumes of data is a smart move, as it will allow you to extract insights and patterns from your data more efficiently.

Leveraging Redis for caching is also a good decision, as it can help speed up data retrieval and reduce the need to query your database repeatedly. By storing frequently accessed data in Redis, you can reduce the amount of time it takes to retrieve that data, which can improve the overall performance of your software.

Creating flows quickly is also important, as it can help you iterate faster and make changes more efficiently. By leveraging Redis for caching, you can create flows more quickly, as you won't need to wait for data collection to be retrieved from the database every time a flow is executed.