Limitations of RmlUi data bindings

[exjam]

I am struggling with the RmlUi data binding model, and given the length of this write up I am really hoping I am not missing something obvious 🤞.

My misadventures in RmlUi data bindings.

Imagine you have a simple mail reading application, with a very traditional UI where you have a list box containing the list of emails in your inbox and another area to show the contents of the mail you are currently reading.

----------------------------------
| mail 1 | From: friend          |
| mail 2 | Sent: 01 / 01 / 1970  |
| mail 3 | Hello world!          |
| ...    |                       |
----------------------------------

So you might start with a data model backing this which is simple std::vector of Mail, e.g.:

struct Mail {
    uint64_t id;
    uint64_t timestamp;
    std::string from;
    std::string message;
    bool read;
};
std::vector<Mail> m_mail;

auto mail = constructor.RegisterStruct<Mail>();
mail.RegisterMember("id", &Mail::id);
mail.RegisterMember("timestamp", &Mail::timestamp);
mail.RegisterMember("from", &Mail::from);
mail.RegisterMember("message", &Mail::message);
constructor.RegisterArray<std::vector<Mail>>();
constructor.Bind("inbox", &m_mail);

Handling the html for the list of mail is very simple and we already have the data-for binding which is great for this!

<listbox id="inbox">
    <listbox-item data-for="mail, mail_index : inbox" data-attr-value="mail_index">
        {{mail.timestamp} {{mail.from}}
    </listbox-item>
</listbox>

Unfortunately handling the UI for the reading the mail is more complicated... we would want something like this:

From: {{currently_being_read_mail.from}}
Sent: {{currently_being_read_mail.timestamp}}
{{currently_being_read_mail.message}}

But, what is currently_being_read_mail here?

The first way I did this was to add a read_mail_index variable to my data model:

int m_read_mail_index;
constructor.Bind("read_mail_index", &m_read_mail_index);

<listbox id="inbox" data-attr-value="read_mail_index">...</listbox>

From: {{inbox[read_mail_index].from}}
Sent: {{inbox[read_mail_index].timestamp}}
{{inbox[read_mail_index].message}}

This trivial implementation gives one immediate problem: if your inbox is empty this will cause an error because there is no valid inbox[n] when your inbox is empty leading to an out-of-bounds error. My first thought was that, that is easy, we can use data-if and get rid of this UI when there is no mail:

<div data-if="read_mail_index < inbox.size">
   From: {{inbox[read_mail_index].from}}
   Sent: {{inbox[read_mail_index].timestamp}}
   {{inbox[read_mail_index].message}}
</div>

Except this does not work, as data-if does not remove elements from the DOM but merely sets display: None and the child data bindings are still being updated. So there is another very messy workaround you could do which is only possible thanks to the very recent PR#740:

<div data-if="read_mail_index < inbox.size">
   From: {{read_mail_index < inbox.size ? inbox[read_mail_index].from : ''}}
   Sent: {{read_mail_index < inbox.size ? inbox[read_mail_index].timestamp : ''}}
   {{read_mail_index < inbox.size ? inbox[read_mail_index].message : ''}}
</div>

This now technically works, but it is ugly and, more importantly, we have had to add something which is purely UI logic to our data model.

This can be seem as problematic from an ideological point of view of "separation of concerns", in that I would like all my UI code to be together in one place, but it is also problematic from a techincal perspective:

• What if I had multiple Mail reading windows? They would all need their own read_mail_index0, read_mail_index1, ... etc variable to use.
• What if the UI was completely user made (e.g. because you have modding support), they would not be able to implement this without edits to the C++ data model.
• Even if it was not user made, what if we used Lua scripting to drive our UI instead of C++? Our lua would be reliant on our C++ data model having something added to it.

OK so it might seem like you can solve all of those 3 above points by using Lua scripting (if you did not have the lua/user requirement you could have also done this same "solution" from C++). Let's say you fully integrated Lua + RmlUi into your code base, well, you'd be immediately stuck as there is no way to access the internal DataModel header - so let's say we modified RmlUi to expose the internal DataModel.h header which allowed us to provide proper access to the DataModel from Lua:

<listbox id="inbox" onchange="on_selected_mail_changed(tonumber(event.parameters['value']))">...</listbox>

<div>
   From: <span id="read-from"></span>
   Sent: <span id="read-sent"></span>
   <span id="read-message"></span>
</div>

<script>
data_model = context:GetDataModel("mail")

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    read_mail_index = index
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = mail["from"].value

        element = document:GetElementById("read-sent")
        element.inner_rml = mail["timestamp"].value

        element = document:GetElementById("read-message")
        element.inner_rml = mail["message"].value
    end
end
</script>

So then you go play around with your UI in your application and it all seems to be working, big success! But something happens and you soon remember why the whole data bindings thing even exist to begin, if the data changes this change will not be reflected in our programatically driven UI. There is no connection between model.DirtyVariable("inbox") and this Lua code to update the currently read mail. In my case it was simply that we were reading mail index 0 when a new mail came in, which updated the listbox's data-for but as the listbox's value did not change (we still had index 0 selected) it did not update the currently read mail, this meant it showed the contents of the previous mail at index 0 whilst the listbox showed that you had selected the new mail at index 0.

So then you might be like OK I got a real big brain and I can come up with a solution to this too! We could just change our Lua code to write data models but with the exact index in, this is like our first approach of {{inbox[read_mail_index].xxx}} but without any of the downsides of requiring UI logic variable in the data model and without the out-of-bounds access problems.

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = "{{mail[" .. index .. "].from}}"

        element = document:GetElementById("read-sent")
        element.inner_rml = "{{mail[" .. index .. "].sent}}"

        element = document:GetElementById("read-message")
        element.inner_rml = "{{mail[" .. index .. "].message}}"
    end
end

So you excitedly open your game again and click around your inbox and it seems to be working, but you notice something weird. For the duration of 1 frame the text in your UI is the string of the data binding "{{mail[0].from}}" before it updates to "Friend". This seems strange so you set some breakpoints to realise once again you have been bitten by order-of-operations with regards to RmlUi's update loop (I say bitten once again because I have found this was a real hard learning point of RmlUi, it was difficult to get the order of operations correct when dealing with the ordering interactions between layout / data binding / data callbacks / event handling / rendering). You can see in the below code from Context::Update all the data bindings are updated, and then the Update function is called on your documents which will lead to things like onchange event handler being called and consequently your Lua code which updated the DOM, so the data binding you set will not be applied until the next frame when you call to Context::Update again.

bool Context::Update() {
...
   // Update all the data models before updating properties and layout.
   for (auto& data_model : data_models)
      data_model.second->Update(true);
...
   root->Update(density_independent_pixel_ratio, Vector2f(dimensions));

After feeling very defeated you pour over the RmlUi documentation looking for a ray of hope, any hack you can apply to get the result you want, and then you see data-alias. At first glance data-alias seems like it might help here, it would allows us to get code much closer to my first proposed data binding, and even though the alias lives in the data model it is scoped to an Rml::Element * which gives us the per-document UI logic semantics that we were looking for.

<div id="read-mail" data-alias-currently_being_read_mail="mail[read_mail_index]">
   From: {{currently_being_read_mail.from}}
   Sent: {{currently_being_read_mail.timestamp}}
   {{currently_being_read_mail.message}}
</div>

The problem we have is that as mentioned before we want read_mail_index to be a Lua variable and not a data binding, so you have a great idea, lets combine our previous ideas and just modify the data-alias with the index value from lua!

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    if mail then
        element = document:GetElementById("read-mail")
        element:SetAttribute("data-alias", "mail[" .. index .. "]")
    end
end

So you start up your game and then... nothing happens... you step into the SetAttribute function and see that you cannot set or modify any data views from SetAttribute. So then you go on to look at how data-alias is stored in the data model and realise there is this nice method DataModel::InsertAlias which you could call to update what the alias points to, so you try it and then... nothing happens... turns out data-alias is immutable in that every data view and data expression resolves addresses at time of creation (which will resolve an alias to the address that the alias points to), so changing the alias will do nothing for currently existing DOM elements. And then thinking about it, even if data-alias was mutable, you would still need to bind this data-alias to something when the inbox is empty else you end up back with the original out-of-bounds access problem.

Enough complaining, what about solutions?

Well this is why I made an Issue instead of a Pull Request, I do not really know what an acceptable solution is which is why I am requesting feedback, I have two ideas though.

1. We could support multiple DataModel in one DOM hierarchy, I do not know if there is more history around why you're only allowed exactly one data model per DOM hierarchy other than the idea that it makes things much simpler (which is a totally worthwhile goal), but by having multiple data models then you could have a "UI logic" DataModel along side your actual data DataModel, and consequently your fully moddable Lua code could be able to create a data model and use it within the same UI that uses original data model from the game's data.

2. Mutable data-alias with nullable types, unfortunately data-alias is currently immutable in that data views resolves addresses at time of creation and data expressions at the time of parsing. If we were to allow data-alias to change at runtime then we would have to have a way to force everything to re-resolve their addresses and also to dirty their state. Additionally for mutable data-alias to be useful for the exact use case specified above we would need support for a null type, and as data-alias only supports DataAddress, then null would need to be a valid DataAddress to the null type. You can find an extremely hacked together mutable data-alias here

I am not sure if I had to pick exactly one of these two solutions which one I would go for. Multi-DataModel gives much more moddability support to our users in that they could mix their own lua data models in with our c++ authored game data models but it will come at the cost of much more complexity (e..g what happens if you have multiple data models active with the same name etc). Mutable data-alias seems like a less radical change to the existing data model structure to directly fix this problem, but you restrict moddability wrt mixing data models.

[exjam]

Another way of looking at the problem which someone just mentioned to me is, if we stick with the LUA editing DOM approach:

read_mail_index = 0
function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    read_mail_index = index
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = mail["from"].value

        element = document:GetElementById("read-sent")
        element.inner_rml = mail["timestamp"].value

        element = document:GetElementById("read-message")
        element.inner_rml = mail["message"].value
    end
end

Then what is missing is that I would want the document event on_selected_mail_changed(read_mail_index) to be called in the case of model.DirtyVariable("inbox").

So looking at it from that perspective, I could also solve this by adding a DataView implementation which calls a callback on update, though it would need to be attached to some element. This gives you the advantage of having C++ code or Lua code be able to observe and react to any changes in the data model.

You can find a hacked together implementation here.

[mikke89]

First of all, you raise some good points! I do appreciate the read. It was quite entertaining even, heh.

I'm also quite amazed at the effort, and particularly I wonder if it's something that could be solved by simply ignoring warnings. And I'm only half joking, your first solution probably works completely fine I expect?

I'll try to keep this mostly solution-oriented, otherwise I think it's easy to get lost here. Some approaches I would suggest:

1. As mentioned above, filter warnings. This should work today I believe.
2. Designate the first element in the data model array as an "invalid" entry with dummy data. Set the index to zero when nothing is selected. Should work as of today, and even with multiple such reader windows as you are outlining.
3. Implement a data view like data-if but instead actually removes the underlying RML and adds it back, based on the condition.

Thinking more about the null type that was suggested here and I prototyped earlier, I'm not sure if that's the right path, at least not in a way where out-of-bounds simply result in null without any warnings. I think out-of-bounds indicates some issue, and the user should explicitly have to state what they want instead. Ternaries are a valid way in my view, even if they are a bit ugly as you say. Maybe "turning off" a section of RML can be considered a valid statement well.

I also want to add, I wonder if some of the requirements and desires being described here is putting too much responsibility onto the data bindings as a feature overall. I am quite adamant that it should be limited in scope, particularly, we do not want it to become another JavaScript. If your things are of certain complexity, it's okay to write this using some other methods. We do have a very expansive API exactly to give you full power over the UI.

[rminderhoud]

Thanks for the reply @mikke89. I work with @exjam and we took this discussion further offline. I shared that we might want to take some inspiration from modern "reactive" frontend frameworks. Specifically focusing on the "data down, events up" approach. The discussion led us to the idea that we would have lua scripts that are responsible for creating/manipulating DOM elements, e.g.

            function do_something(name)
                element_to = document:GetElementById("send-to")
                element_to.value = name

                element_message = document:GetElementById("send-message")
                element_message:Focus(true)
            end

The trick here is that we want this to be reactive to the data model and be called if the data model changes. One can think of this like "re-rendering" this part of the DOM tree in the abstract sense (even though this function only modifies some inner html). One pattern we identified that could be useful is if the data model could emit an "event" when it was dirty so that we could call the lua function again to update the view.

@exjam has an experiment which introduces this new attribute (data-triggerevent-invalidate) which will call the oninvalidate callback when the data model is dirty.

<listbox id="inbox"
                                data-triggerevent-invalidate="inbox"
                                oninvalidate="on_inbox_invalidated()"
                                onchange="on_selected_mail_changed(tonumber(event.parameters['value']))"
                            >

So far the results of the experiment are pretty good. @exjam managed to rewrite one of our UIs to be entirely in lua with a very simple datamodel backing it in C++ using this new system. We haven't yet shipped it to production to know any limitations or edge cases that weren't considered.

[exjam]

1. As mentioned above, filter warnings. This should work today I believe.
2. Designate the first element in the data model array as an "invalid" entry with dummy data. Set the index to zero when nothing is selected. Should work as of today, and even with multiple such reader windows as you are outlining.
3. Implement a data view like data-if but instead actually removes the underlying RML and adds it back, based on the condition.

3 good proposals to the out-of-bounds access of the array but none of these solve the separation of concerns with requiring a data model variable which is solely used for UI logic - I did mention that this constraint would be partly ideological but I also mentioned the practical implications of requiring that:

• What if I had multiple Mail reading windows? They would all need their own read_mail_index0, read_mail_index1, ... etc variable to use.
• What if the UI was completely user made (e.g. because you have modding support), they would not be able to implement this without edits to the C++ data model to add the UI logic variable.
• Even if it was not user made, what if we used Lua scripting to drive our UI instead of C++? Our lua would be reliant on our C++ data model having something added to it.

I also want to add, I wonder if some of the requirements and desires being described here is putting too much responsibility onto the data bindings as a feature overall. I am quite adamant that it should be limited in scope, particularly, we do not want it to become another JavaScript. If your things are of certain complexity, it's okay to write this using some other methods. We do have a very expansive API exactly to give you full power over the UI.

I do wonder that too - despite my long post, for now I am happy with the approach of just 3e4168b + using Lua modifying the DOM as shown above by @rminderhoud , though I have not yet ported all our existing data-model based UI logic yet so have not yet become fully aware if this approach has its own limitations. I'll let you decide if you think its useful for upstreaming else happy to maintain it in our fork. This satifies my constraint of keeping the ui logic out of the data model and does not require the more complicated ideas of multi data model or mutable aliases (which although I did make a basic implementation of - it quickly got complicated when I tried to expand the idea to be 100% solid across all use cases).

If you are interested to see it I have attached our current 100% Lua based UI-logic mail window .html, this is backed by a simple data model of std::vector as described in my first post.

mail.html.txt

[mikke89]

Hello to both of you. I think it's a very interesting path with the triggerevent view, and I'd like to see where this goes. I hope you can share your experiences once you have tried this for some time.

I see the point of invoking some Lua script (or C++ function) to do more complex scripting behavior, and I do think it makes sense to tie these together in some way. While emitting events from a data view feels a bit wrong, maybe there is some other ways to invoke the script, or maybe it's okay after all.

I am a bit concerned that there could be some foot guns around this. Particularly with modifying the DOM together with data bindings, one has to be careful. But I'd love to hear more about your experiences after a while. I'd like to see how it goes for you before considering it for the library, but I am open to it in the longer term.

3 good proposals to the out-of-bounds access of the array but none of these solve the separation of concerns with requiring a data model variable which is solely used for UI logic - I did mention that this constraint would be partly ideological but I also mentioned the practical implications of requiring that: [...]

Yes, I hear you, although I am leaning more towards the pragmatic side here:

• The multiple indices would be part of the UI state, I think it's reasonable to explicitly model those.
• The second concern is a bit out of scope for data bindings the way I see it. You will have to work around that some other way, which brings us to the next point.
• We can always have Lua call into C++, so that's a matter of language bindings. Did you try out the data model bindings we have already for Lua? I don't have too much experience with those myself, and I am very much open to improving them.

I do understand that your use case might not be exactly what we have been aiming for, particularly with the fully user-defined UI without modifying source code. The data bindings aren't meant to be a complete solution for a front end, instead we have a very expansive API more generally to fill such needs. Although maybe a user-defined UI should be a use case for the library to consider more thoroughly? I haven't really heard many requests around this from library users so far.

I'm sure there are smaller things we could do here and there to improve the situation, some of which have been discussed here. There has also been a proposal around defining data variables with expressions directly in RML, which maybe could help a bit. But again, we need to be careful not to make this into another JavaScript.

Let me know how it goes, I am very much interested to learn from your approach and see if it is something that could be a good fit to integrate into the library.

[exjam]

The multiple indices would be part of the UI state, I think it's reasonable to explicitly model those.

Yeah I guess this is where ideology comes into play and is probably largely influenced by my previous experiences with Qt's model-view architecture meaning that in my mind the model is bound purely to the data, and UI logic is part of the view.

We can always have Lua call into C++, so that's a matter of language bindings. Did you try out the data model bindings we have already for Lua? I don't have too much experience with those myself, and I am very much open to improving them.

We are using sol3 so went with completely custom bindings. I did notice that your Lua data model bindings only allows you to create a data model in lua, not to access a C++ data model from Lua.

So I am currently doing the opposite of the official Lua bindings and exposing C++ data models to Lua (which required making DataModel.h a public header in RmlUi).

function mail_format_timestamp(timestamp)
    time = Clock.AbsSecond.new(timestamp).time
    return os.date("%Y/%m/%d %H:%M", time)
end

data_model = context:GetDataModel("mail-inbox")
data_model:RegisterTransformFunc("mail_format_timestamp", mail_format_timestamp)
inbox = data_model['inbox']

function on_selected_mail_changed(index)
    if index < #inbox then
        mail = inbox[index]
        read_mail_index = index

        document:GetElementById("read-from").inner_rml = mail["from"].value
        document:GetElementById("read-timestamp").inner_rml = mail_format_timestamp(mail["timestamp"].value)
        document:GetElementById("read-message").inner_rml = Ui.TextLink.DecodeTextToRml(mail["message"].value)
        document:GetElementById("read-reply").disabled = false
        document:GetElementById("read-delete").disabled = mail["deleting"].value

        if (mail['status'].value & 1) == 0 then
            set_mail_read_timer:Start(1000, mail['id'].value)
        else
            set_mail_read_timer:Stop()
        end
    else
        document:GetElementById("read-from").inner_rml = ""
        document:GetElementById("read-timestamp").inner_rml = ""
        document:GetElementById("read-message").inner_rml = ""
        document:GetElementById("read-reply").disabled = true
        document:GetElementById("read-delete").disabled = true

        set_mail_read_timer:Stop()
    end
end

struct DataVariableProxy
{
    Rml::DataModel& model;
    Rml::DataVariable variable;
};

void bind_rml_data_model(sol::table& rml)
{
    auto data_model = rml.new_usertype<Rml::DataModel>("DataModel", sol::no_constructor);
    data_model.set_function(
        sol::meta_function::index,
        [](Rml::DataModel& self, const char* address) -> DataVariableProxy
        {
            auto data_address = self.ResolveAddress(address, nullptr);
            return DataVariableProxy{self, self.GetVariable(data_address)};
        });
    data_model.set_function(
        "RegisterTransformFunc",
        [](Rml::DataModel& self, const char* name, sol::function function)
        {
            Rml::DataModelConstructor constructor(&self);
            constructor.ReplaceTransformFunc(
                name,
                [function = std::move(function)](const Rml::VariantList& arguments) -> Rml::Variant
                {
                    std::string result = function(sol::types<std::string>(), arguments[0]);
                    return Rml::Variant(result);
                });
        });
    data_model.set_function("DirtyVariable", [](Rml::DataModel& self, const char* name) { self.DirtyVariable(name); });

    auto data_variable = rml.new_usertype<DataVariableProxy>("DataVariable", sol::no_constructor);
    data_variable.set_function(
        sol::meta_function::index,
        sol::overload(
            [](DataVariableProxy& self, const char* address) -> DataVariableProxy
            {
                if (!self.variable)
                    return DataVariableProxy{self.model, Rml::DataVariable()};

                return DataVariableProxy{self.model, self.variable.Child(self.model, Rml::DataAddressEntry(address))};
            },
            [](DataVariableProxy& self, int index) -> DataVariableProxy
            {
                if (!self.variable)
                    return DataVariableProxy{self.model, Rml::DataVariable()};

                return DataVariableProxy{self.model, self.variable.Child(self.model, Rml::DataAddressEntry(index))};
            }));
    data_variable.set_function(
        sol::meta_function::length,
        [](DataVariableProxy& self) -> int
        {
            if (!self.variable)
                return 0;

            return self.variable.Size();
        });
    data_variable.set(
        "value",
        sol::property(
            [](DataVariableProxy& self, sol::this_state state) -> Rml::Variant
            {
                Rml::Variant value;
                if (!self.variable || !self.variable.Get(value))
                    return Rml::Variant();

                return value;
            },
            [](DataVariableProxy& self, Rml::Variant value) { self.variable.Set(value); }));
}

I do understand that your use case might not be exactly what we have been aiming for, particularly with the fully user-defined UI without modifying source code. The data bindings aren't meant to be a complete solution for a front end, instead we have a very expansive API more generally to fill such needs. Although maybe a user-defined UI should be a use case for the library to consider more thoroughly? I haven't really heard many requests around this from library users so far.

Yeah it is probably mostly my ideology of a strict model-view architecture which ends up putting us in the same place than a user who wants to mod the game would be - where they cannot modify the C++ data model and only edit the html + lua.

Also even from a developer point of view it is nice to have all the UI code in one file rather than split between multiple files.

Let me know how it goes, I am very much interested to learn from your approach and see if it is something that could be a good fit to integrate into the library.

Will do, also we appreciate the work you and all contributors do on RmlUi - it certainly is a massive improvement over the old ui system we are replacing!