Case Study: Decore Uses Rockset for Search & Analytics on DynamoDB

Many early adopters of cryptocurrency were individuals at the forefront of this technology, but enterprises are now increasingly getting more involved. As using cryptocurrency for business transactions becomes more commonplace, Decore aims to make accounting as streamlined as possible for companies accepting and sending crypto.

Conceived as a “Quickbooks for crypto,” Decore provides accounting solutions for companies that have adopted crypto. In the same manner that accounting software like Quickbooks may pull data from banks and credit cards, categorize transactions, and generate periodic reports, Decore’s software service pulls and compiles data from blockchains so that accountants may process crypto transactions easily. Decore has the ability to sync crypto transactions with Quickbooks itself, allowing accountants to operate with crypto in a familiar environment.

Decore also provides companies other forms of automation around crypto transactions. Companies can opt to use Decore for payroll purposes and augment regular bank deposits with crypto payments to employees. In addition, tax reporting can be complicated given the large trading volume common in crypto and the different tax treatments that apply, so Decore simplifies this process by generating tax reports on crypto holdings.

Pulling Crypto Transaction Data into DynamoDB

A lot goes on behind the scenes to run Decore’s software service. Whenever a user imports their cryptocurrency wallet or exchange account, Decore needs to pull data on all the transactions associated with it.

Decore adopted a serverless approach to building their application. To populate and update crypto transaction data once a wallet or exchange account has been added, Decore fires off AWS Lambda functions to query blockchains approximately every hour. Decore also requires a data backend that can scale with the volume of transactions as their business grows, so they use Amazon DynamoDB to store all the crypto transaction data returned by the queries.

Search and Analytics on Crypto Transactions

Aside from compiling all transaction data from users’ wallets and exchange accounts, Decore, as an accounting tool, needs to allow accountants to verify, allocate, and reconcile transactions as part of their work. To support this, Decore provides functionality to filter, search, and analyze transactions. As an example, users can perform ad hoc searches for transactions involving specific deposit and withdrawal currencies, between particular origin and destination accounts.

Ad Hoc Queries Run Too Slowly

The ability for accountants to issue arbitrary queries on transaction data was not straightforward to implement, however. Decore originally built these features on the DynamoDB data store but quickly realized that this design was infeasible. Searches on transaction data needed to be fast and sufficiently interactive to be usable by accountants, but DynamoDB could not deliver the necessary performance on its own. These searches were simply not possible without predefining the queries and creating secondary indexes in DynamoDB for this subset of queries. But with more than 20 attributes in the DynamoDB table at this point, it wasn’t feasible to maintain global secondary indexes (GSI) for everything, which was what would have been required to support ad hoc queries.

Decore would have had to severely limit query functionality and disable ad hoc and random queries by users if they couldn’t find a reasonable way to run analytics on DynamoDB. Decore engineers looked at offloading analytics onto other data stores and considered supplementing DynamoDB with MySQL to index the crypto transaction data. However, this alternative was also fraught with problems, including an inability to scale.

Delivering Real-Time Search and Analytics with Rockset

In searching for a solution to the query performance issue, the Decore team came across Rockset, which enables real-time search and analytics on data from DynamoDB. Rockset delivers low-latency queries through a combination of cloud autoscaling and automatic indexing of every field and value in the data, making it possible for Decore to enable ad hoc queries on the crypto transaction data without any performance engineering or index management. In addition, Rockset continuously loads data from DynamoDB, so the most up-to-date data Decore is receiving can be made available for fast analytics.

Decore incorporated Rockset into their data landscape, setting up a stream from DynamoDB to maintain a copy of the transaction data in Rockset. Decore rebuilt their application so that search operations go through Rockset while write operations continue going to DynamoDB. All queries that filter and search on the transactions are handled through Rockset's index, which returns transaction IDs that the Decore app then uses to fetch matching transactions from DynamoDB. The integration was relatively simple, due to Rockset’s support for DynamoDB as a standard data source and Decore’s modular, microservices-based architecture, taking one engineer half a day to implement.

As a serverless search and analytics engine, Rockset also aligns well with Decore’s serverless approach, requiring no management of infrastructure or data platform while scaling transparently to meet Decore’s needs. This allows Decore to use Rockset in their architecture with minimal disruption to their processes.

Powering a Better Crypto Accounting Experience

The combination of DynamoDB and Rockset allowed Decore to deliver the full range of search functionality for crypto transactions that was planned. Most ad hoc queries that were not possible before now return in under 100ms, so performance is no longer a limiting factor.

“We want to provide accountants an environment where they can work efficiently with crypto, so it’s important they have all the functionality they need to do their job,” says Yenwen Feng, CEO of Decore. “By building our search and analytics on top of Rockset, we make it possible for accountants to find and review transactions as needed.”

According to Feng, using DynamoDB and Rockset together gives Decore the best of both worlds—fast writes and fast analytics—with no need to trade off between the two. He adds, “Queries that were impossible with DynamoDB alone are now completing in milliseconds with Rockset. Anyone running analytics on DynamoDB can get better performance just by hooking up their applications to Rockset and connecting to DynamoDB.”

Blockchain by Frühstück from the Noun Project