Timeouts with large number of reportsuites

[genejones]

See this twitter chain

Large numbers of reportsuites can create timeout issues. I suspect this is linked to limits on the Adobe side of request sizes. Events in particular create massive requests even for relatively small report suite sizes.

The solution is to split requests for information on evars, props, and events, perhaps into groups of 10-20? I'll do some math on the maximum possible size per report_suite and go from there.

[genejones]

Let's look at the maximum size of an events response for a single reportsuite. Events responses will be bigger than anything else because of the 1000 custom events, plus 13 default events which Adobe preconfigures. That leaves us with 1013 events per response. The longest IDs are going to be with the preconfigured events. We can go ahead and calculate the typical/max size of the default events by using JSON.stringify(defaultEvents).length. I get a result of 2701, but this could vary a little. To be safe I'll default to 2800 bytes to transmit the default events.

Now all we need to worry about is the size of the remaining 1000 custom events. I don't know if a single production reportsuite in the world has all these configured, but it's worth examining the maximum possible amount.

{
"id":"event1",
"name":"Hostile takeover",
"description":"Fire whenever the Illuminati take over a country (despite not existing for several hundred years).",
"type":"enabled",
"default_metric":false,
"participation":"disabled",
"serialization":"always_record",
"polarity":"positive",
"visibility":"everywhere"
}

Let's break this down into maximum lengths. First, we can figure out the length from the IDs, as they will be static.

var entries = "";
for (var i=1; i<1001; i++){entries = entries + '"id":"event' + i +'",';}
console.log(entries.length);
15893

15893 bytes for IDs.
The other elements are a bit harder to nail down.

"name":limit is 128 chars, so with '"name":"content",' we are looking at 11+128 or 139 bytes max.
"description":256 chars limit. This means 273 bytes max.
"type":"currency_no_subrelations" is the longest at 24 chars. This means a total of 34 bytes max.
"default_metric" is either true or false, so the length is either 22 bytes or 23 bytes.
"participation":" either enabled or disabled, so length is either 26 or 27 bytes.
"serialization": longest option is "record_once_per_unique_id", which results in 44 bytes.
"polarity": positive or negative, but both are the same at 22 bytes.
"visibility":"everywhere" - the API options here aren't well documented. Everywhere is the default and is 25 bytes.

We also get 4 bytes per object for the object initialization {} and also for their newlines.

In total, we get 139 + 273 + 34 + 23 + 27 + 44 + 22 + 25 + 4 = 591 591 bytes maximum per event entry. This doesn't include the ID values we calculated earlier, but we'll add them back in.

OK, so 591 * 1000 = 591000 + 15893 (ids) = 606893 (for custom events).
Then adding in the default events, we get 606893 + 2800 = 609693.

Per reportsuite we have a maximum of 606693 characters (I've been using bytes and characters interchangeably, but this is only true if ASCII characters are exclusively used. This may not be true for international reportsuites, where UTF-8 is used and will use up additional space).

There are a few more characters which are included for each reportsuite.

{
"rsid":"illuminati2014global",
"site_title":"Bavarian Enlightenment",
"events":[all 1000 events go here]
}

I could go through the max size for these too, but I'm tired of this post now. Let's assume 100 chars total for our purposes.

Maximum size per reportsuite is thus 609,703 bytes, or 610 KB. This is massive. With only 2 reportsuites an event API response could be 1.2 megabytes. 100 reportsuites could result in API results of over 60MB!

Typical lengths will be under this because descriptions for non-configured events are much smaller, but 300KB per reportsuite is still typical. One bit of data I will collect on reportsuites.com is the total size per reportsuite of each response. I'm hoping to publish a few blog posts on the subject as I get more data.

[genejones]

Moving to getSettings, which will allow for getting all the details on a reportsuite at a time. This should fix this issue.