Skip to main content
Solved

Bulk create event - time field, keeping profile data up to date.


Forum|alt.badge.img
  • Contributor II
  • 3 replies

Hi community,

Background:

I’m reaching the rate limit on profiles and events due to high activity. Currently, I use the synchronous single request, but the activity is higher than the allowed rate limit (forcing us to move to bulk api). I must point out that on initial sync, I’m already using bulk, but the ongoing activity hit the rate limit, so the solution is to stop sending single requests and use bulk requests (enqueue the requests and send them together in the bulk request).

 

Question:

Now, how can we keep the order of the requests?  We don’t want the newest request to be processed before an older one, otherwise, the final data on the profile will not be accurate (it will have old data).

Furthermore, I see in the Bulk Import Profiles API regarding “Ordering and duplicates” that Processing order is not guaranteed.

 

In the profiles update use case, it’s simpler because we can avoid sending duplicates (send only the oldest). But in the events, we need to send all (even duplications, if, for example, 2 purchases were made).

I don’t see in the documentation of the Bulk events any info regarding the asynchronous impact on the order.  I do see that they offer a time filed for historical events.

So the question is, what exactly the time field in historical events works? If I send 3 events on the same profile (that include also profile properties, and add a time, does the profile will end with the property values of the event with the oldest time? Or it will have the values of the last processed event? (Or it knows to process them order by time filed?).

Second question, is the time field available also for the Bulk profiles? (seems that the answer is no based on the body params in the api doc.

 

To conclude, it will help a lot if you could help to clarify how the time field works in the bulk events request, and in general, if someone has an insight into how to overcome the ordering issue in events, in a way that the profile will end with the most up to date data I will very appreciate it. Thanks!

 

Best answer by MANSIR2094

Hello ​@AW1 

To maintain request order in the Bulk API and ensure accurate profile data:

1. Use the time field in events: The time field is used for processing historical events. While the Bulk Events API doesn’t guarantee processing order, including accurate timestamps ensures the latest profile state reflects the most recent event data. The system prioritizes the last processed event, so provide precise timestamps for all events.

2. Deduplicate requests when possible: For profile updates, only send the oldest update per profile to reduce conflicts.

3. Manage event queues externally: Implement an external queuing system to ensure proper sequencing before submitting to the API. This guarantees chronological order, even if the API processes asynchronously.

4. Custom logic for merging properties: Use custom properties or tags to help identify and merge the latest updates accurately if order issues arise.

For complex implementation, feel free to reach out for tailored support.

View original
Did this topic or the replies in the thread help you find an answer to your question?

3 replies

MANSIR2094
Problem Solver IV
Forum|alt.badge.img+13
  • Problem Solver IV
  • 174 replies
  • Answer
  • January 16, 2025

Hello ​@AW1 

To maintain request order in the Bulk API and ensure accurate profile data:

1. Use the time field in events: The time field is used for processing historical events. While the Bulk Events API doesn’t guarantee processing order, including accurate timestamps ensures the latest profile state reflects the most recent event data. The system prioritizes the last processed event, so provide precise timestamps for all events.

2. Deduplicate requests when possible: For profile updates, only send the oldest update per profile to reduce conflicts.

3. Manage event queues externally: Implement an external queuing system to ensure proper sequencing before submitting to the API. This guarantees chronological order, even if the API processes asynchronously.

4. Custom logic for merging properties: Use custom properties or tags to help identify and merge the latest updates accurately if order issues arise.

For complex implementation, feel free to reach out for tailored support.


Forum|alt.badge.img
  • Author
  • Contributor II
  • 3 replies
  • January 18, 2025

Thanks ​@MANSIR2094 

Regarding point 1 - YES! That’s exactly my question. I’m already implementing a queuing system to ensure proper sequencing before submitting to the API (as suggested in point 4). My question is about how this works on Klaviyo’s side. Does adding the time field act as a timestamp? In other words, even if events are processed in a different order (inside the bulk or between distinct bulk jobs), will the final profile data reflect the oldest event based on the time field?

I also have a second question that hasn’t been addressed yet, though it might have gotten lost in my earlier long message:

The profiles and events APIs are distinct, yet they both manage the same entity (the profile). From what I can tell, the profile API doesn’t allow adding a timestamp. This raises a concern: if an older profile update request is processed after an earlier event (containing profile data), could this result in the profile having outdated values? or vise-versa...

 


chloe.strange
Community Manager
Forum|alt.badge.img+41
  • Community Manager
  • 425 replies
  • February 5, 2025

Hi ​@AW1

Following up here, was this resolved, or do you still have some outstanding questions that you need answered?

~Chloe