Skip to main content
Solved

Get all events for specific data range and metric id FAST, maybe concurrently?

  • December 3, 2024
  • 1 reply
  • 26 views

adzviser
Contributor I
Forum|alt.badge.img

I have read this prior post before posting.

I’m trying to build a connector for Klaviyo so that email marketers can pull their klaviyo campaign/flow stats to Google Sheets / ChatGPT. To achieve such a goal, I decide to do an overall aggregation over all the events due to the limitations of the existing reporting api such as not including profiles, other important fields. 

Aggregation over all events works great and accurately except that it takes too slow. If an account sends thousands of emails per day, then it will take more than 5 mins to retrieve all the Received Email events in a date range of 7 days. 

We currently do what’s advised - sequentially send requests to https://a.klaviyo.com/api/events and get the next page url from the prior response. Each response contains 200 events. 

This method is way too slow. Is it possible if we could do this concurrently while still being under the rate limits (350/s and 3500/m)? Is it possible to pre-calculate the page cursors returned at each response like WzE3MzI5MjUwMjMsICI1SlNDRHJZeEtOayIsIHRydWVd? It’s encoded with Base64 but the underlying encoded dataset does not make a whole lot of sense. 

Being able to get events fast and ideally concurrently will be a game changer for us. Thanks. 

Best answer by chloe.strange

Hi ​@adzviser

Welcome! And glad you’re a part of our community! I’ve addressed your questions below: 

  • 5 minutes to get event history over last 7 days for a given account
    • This is expected - the recommendation would be to do the bulk backfill, save results, and then periodically poll for incremental results to stay up to date. Webhooks might be a better way to stay in sync with without hitting the API rate limits (so long as the particular events are supported in webhooks)
  • Rate limiting
    • Rate limiting on private keys is per account
  • Is it possible to pre calculate page cursors?

    • Cursor for the next page depends on the last result on the page, so you can't know it until you load that page

~Chloe

View original
Did this topic or the replies in the thread help you find an answer to your question?

1 reply

chloe.strange
Community Manager
Forum|alt.badge.img+41
  • Community Manager
  • 425 replies
  • Answer
  • December 11, 2024

Hi ​@adzviser

Welcome! And glad you’re a part of our community! I’ve addressed your questions below: 

  • 5 minutes to get event history over last 7 days for a given account
    • This is expected - the recommendation would be to do the bulk backfill, save results, and then periodically poll for incremental results to stay up to date. Webhooks might be a better way to stay in sync with without hitting the API rate limits (so long as the particular events are supported in webhooks)
  • Rate limiting
    • Rate limiting on private keys is per account
  • Is it possible to pre calculate page cursors?

    • Cursor for the next page depends on the last result on the page, so you can't know it until you load that page

~Chloe