+1
I would like to fetch a list of event types and times per user for all profiles.
I saw I can fetch all profiles and all events. I then have to run a query per event to fetch the profile asociated with the event. I then habe to write some code to save the information I want to be per profile instead of per event.
is there any simpler/faster way to export this data in bulk?
Best answer by saulblum
View original
+1
Hi there,
It sounds like you're dealing with a data extraction challenge. Here's my suggestion
I understand the complexity of fetching event types and times per user for all profiles. Instead of fetching all profiles and all events separately and running multiple queries, consider exploring the following approach:
Batch API Requests: Check if the API you're using supports batch requests. This way, you can fetch information for multiple profiles and events in a single API call, reducing the number of requests and speeding up the process.
Optimize API Queries: Review the API documentation to see if there are parameters or filters that allow you to retrieve the desired data more efficiently. For example, you might be able to filter events based on user IDs to directly get the relevant information.
Use Data Transformation Tools: Consider using data transformation tools like Apache Spark or pandas in Python. These tools can help you process and transform large datasets efficiently, allowing you to reshape the data as needed for your analysis.
Caching and Incremental Updates: If applicable, implement a caching mechanism to store previously fetched data. This way, you only need to retrieve and process new or updated information, saving time and resources.
Parallel Processing: Depending on the scale of your data, explore parallel processing techniques to fetch and process data concurrently. This can significantly speed up the overall data export process.
Remember to consult the API documentation thoroughly and test any changes in a controlled environment before implementing them in a production setting. If you have any specific details about the technology stack you're using, I can provide more targeted advice.
Message me for further discussion
Best regards,
Juan.
Could you give more of the context and use case? Which events do you want to export? Native events like Clicked Email? E-commerce-specific events like Viewed Product? Do you care about the specific profiles attached to these events or only want a count over a time window?
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
