Cross-Platform QML

I’ve been thinking a lot about cross-platform QML recently, and I’m about to email the dev list with a suggestion. But the conceptual background essay is here instead of in the mail this time. Hopefully this will help better organize the information – it certainly allowed me to go into much greater depth on the wider issue 🙂 .

Since the number of QML-based platforms just keeps growing, it might be best to improve the ‘write once, deploy everywhere’ story for QML. Currently the story is to write the UI once for each platform because QML makes that incredibly easy and fun, but I think we can improve on that. So I’ve been thinking about what simple features could be added to the language that would help write cross-platform UIs. The goal here is to minimize the porting effort of applications, and as always my exemplar is SameGame. We can all agree that SameGame should work on every QML enabled platform, and it was awesome to see it running on BlackBerry 10 at Dev Days. What was not awesome was how they did it, here’s a snippet from settings.js:

.pragma library

//This should be switched over once a proper QML settings API exists

var menuDelay = 500

var headerHeight = 20 // 70 on BB10
var footerHeight = 44 // 100 on BB10

var fontPixelSize = 14 // 55 on BB10

var blockSize = 32 // 64 on BB10

There are two major problems with this. One is that settings shouldn’t be stored in a JS file like this, as mentioned in that code comment (but the recent ML discussion suggested that we should wait for QtCore to replace QSettings, something they’ve been wanting to do for a while). The bigger problem is that it’s not actually cross-platform. The automated process for ‘rebuilding’ the BB10 version is

sed -i '/on BB10/ s/\([0-9]*\) \/\/ \([0-9]*\) on/\2 \/\/ \1 off/' settings.js

That is not pretty, easy, or extensible. But I approved the patches adding that file to SameGame because the truth is it’s the best solution QML currently has. It needs to be replaced with some proper cross-platform support, added to the language itself.

Language-Level

There is an important distinction here about Language-level vs component-level features. Language features can be used on all platforms; guaranteed. Features of a component or type set cannot, there are even some platforms which have non-QtQuick native GUI component sets (which is entirely appropriate). Add to this that non-GUI QML applications might still want some platform-specific parts and it becomes clear that device independent QML is worth investigating from a technically non-GUI perspective, even though the main goal is to provide enablers for all the different GUIs. Resolution independence in particular is a GUI constraint, so that can’t be explicitly solved at the language level. The problem isn’t that the language might be tainted with ‘GUI concepts’, it’s that the language needs to run fine without a display and all features must work with all different GUI implementations.

That’s the key background for my email, wherein I propose adding a language level feature of platform contents resolution in URLs. It was inspired by the talk Aaron Seigo gave at Dev Days, where he mentioned how the Plasma package format provides this for tablet and touch form factors. While they only deal with generic form factors and input modes, I think the concept is extensible to where you can use either generic or specific keywords. With the renewed desktop and cross-platform focus for QML, I’m hoping to work a lot more with the Plasma team this year. After writing complex, real applications in QML maybe there are other features they’ve explored which would be better (more powerful and easier to implement) inside Qt.

The platform contents resolution seems like such a feature to me. In plasma they have the problem of having to go through a custom QNetworkAccessManager, and needing to use “plasmapackage:” URLs because of that. They also have the problem of it only working for pure QML plasmoids, they can’t set that QNAM on hybrid plasmoids which use their own QML engine. So it would be easier to implement everywhere by just doing it in the QML engine. The only major change the functionality needed was to be relative to the base file, rather than the package root, so as to be independent of how your QML is organized. The extensive componentization of QML means that any sub-directory can be pulled out as a separate module, and that modularity should be preserved.

It’s also a good choice for a language feature, because it’s not explicitly GUI but it’s GUI friendly. You could use the general idea to split up hdpi, mdpi and ldpi images for resolution independence, and you can replace settings files filled with pixel heights like in SameGame’s case. Plus it’s still supporting pixel-perfect UIs, it’s just that you only need to swap out the bits that need rewriting in order to be ‘pixel-perfect’ on multiple devices. It turns out that even pixel-perfect UIs don’t have to replace every last line, and not all QML is UI (even if it arguably should be 😉 ). The generic selectors I’m currently thinking of are also potentially extremely versatile. But if that part doesn’t fall during the discussion, conventions are bound to spring up to curtail that freedom.

Component-Level

The other half of cross-platform QML, as a language, is the stuff that shouldn’t be solved at language level. Which I want to bring up because I see different component sets implementing it different ways – and not necessarily in what I think are the best ways.

Native styling and layouts are fairly clearly component set concerns. Since platforms are so amazingly diverse in UIs right now you can’t have one set that will look native across all platforms. This is different from the varied desktop platforms, because that was primarily just a different pixmap and not a different UI flow. The best we could have by default is a neutral UI experience that is functional on almost all hardware, even though it doesn’t feel native.

I think the QtQuick Components are planning to run functionally across all platforms, and have decent styling capabilities. Perhaps interested platforms can provide a compatible styled version which toggles some native look and feel, but that would still be mostly swapping out pixmaps because I’m talking about a global flag to QtQuick components and using its styling API. This means they would be the same components with the exact same API on all platforms, just styled, and you should be able to disable that native styling if desired. The best solution for truly cross-platform UIs that I can foresee is to have bits of app chrome redone with native UI components to varying degrees between platforms, and then your custom or QtQuick content can be shared in the middle. This is inevitably like the ‘rewrite for each platform’ that we have now, except specifically targeted so that you only rewrite the parts that actually change. To me, that sounds like the theoretical best case (until we come up with an AI which intelligently restyles your app for you 😉 ).

Layouts are very similar to styling, because with the extreme differences in form factor between various devices (rectangles of all sizes!), not to mention space requirements due to input methods (you can cram more on a resistive screen than a capacitive screen with the same resolution due to increased user accuracy), the right layout to use varies per platform/device as well. Neither styling nor layouts are likely to be solved in the GUI primitives layer either, because QtQuick is intended to provide what you need to create those components. It should be enough graphical primitives to write your own layouts, or your own stylish components, but not do it for you.

Recently a new QML-based mobile platform was announced, Ubuntu Phone, which was what prompted me to hurry up and start the cross-platform discussion. So I wanted to comment on their approach as well, especially since two of their pillars (CSS-like styling and non-pixel units) have both been shot down in the past for QtQuick. They really should have just snapped up the QML team when we were on the market last year 😛 .

CSS styling has major theoretical and practical issues working with QML. Not that I’m saying it’s a bad choice for the web, but to a large extent what makes it a good choice for web makes it a bad choice for QML integration. My main concern with mingling CSS and QML is that the conceptual overlap. CSS was designed to separate the styling out from the content of web pages, QML was designed to separate out the UI from business logic of native applications. Those are similar, but different, concepts. Consequently there are significant differences between CSS and QML, but there is still some overlap. An instance of the conflict is in coloring a single block of text – QML puts the style right on the item to bring together the UI, CSS moves the color to a separate file to split out the styling from the ‘content’. In web terms the text is the content, in native terms the text is just UI representing a data source (which would be the real content, unless this is just a fixed text label in the UI). An instance of the overlap is in coloring a group of texts – Where in CSS you’d use a class, in QML you’d use a component.

<p class="MyText">Text</p>

vs

MyText{ text: Text }

While QML and CSS aren’t identical, having these alternate ways to do things that are handled better with the QML language is a drag because some people might use them; without realizing that there was a better path to follow.

As for the practical issues with CSS, the advanced selector logic is certainly more powerful than creating a QML component. It’s also a lot slower and in the early (XML) days of QML we tried CSS and quickly dismissed it as too slow – unless you stuck to simple selectors like classes (which we could do without ‘forking’ CSS). So even if there weren’t conceptual issues with mixing QML and CSS, it’s just not designed for native performance. CSS was designed for web-level performance, and it squarely hits that mark.

Another problem with just using a subset of CSS is the example of QtSVG. QtSVG is deprecated not because it’s bad, but because it’s really hard to use just the specific subset of SVG that it supports. With QML we already run into issues of the conflict between QML/JS and browser/JS, and there we implement the full ECMAScript spec! Differences between QML/CSS and browser/CSS would make it even more of a confusing mess for anyone coming from a web background.

Device independent units is another thing that keeps cropping up. In addition to resolution independence concerns, there was a case where the designers had written all the design specs for UI elements in millimetres. So this feature also has the potential to improve developer performance. Unfortunately it comes at a cost of run-time performance, and in the trade-off between developer performance and run-time performance you almost always want better run-time performance. The problem is that you’re offloading a calculation from developer-time to run-time, the reverse of how high-performance apps are written. To make things worse this is in the measurements of your UI elements, probably the single most frequently set category of value in UI descriptions. The cumulative cost of something that could be so easily avoided stacks up to be quite a problem.

The other issue with device independent units is that it’s ugly. As much as I’d love to be able to write 10mm in QML and have it automatically be parsed as 10 * mm (like in hand written mathematics, e.g. 10x), that’s not valid JavaScript. Best case you have to actually write 10*mm, worst case it’s more like PlatformUnits.millimetresToPixels(10). Either one might not look that bad, but again you must remember that these values make up a lot of the UI. You’ll be seeing code littered with this pattern on a daily basis, you can’t just tuck it into a corner or use a tool to generate it when you have to. Suddenly the readable beauty of QML has died a tragic death.

However there’s nothing conceptually wrong with device independent units in QML, if you do it right. My initial thoughts on how to do it right are as follows:

• Optimize constant expressions in QML. In some cases QML can already identify constant expressions and just set them at compile time instead of as a binding (evaluated every time an instance is created even if it doesn’t change). Making sure this covers your units case is the first step. All this does though is move the cost from instantiation (run-time at startup) to compile time (also at startup). So to really solve the performance problem we need to be able to compile QML to bytecode and then deploy that, so that compile-time costs are not startup-time costs. The work on this is already under way, but it can’t be completed until we finish v4vm and integrate that into the QML engine. This would solve the performance cost.
• An option for fixing the ugliness cost is to not have pixel based properties. If you’re writing a cross-platform component set, do you need specific pixel values so much? You can have your width/height/x/y in millimetres if that’s the common case, and then have a pixelOffset (or similar) property should per pixel adjustments be needed. This was unacceptable for QtQuick because that’s intended for making pixel perfect UIs and component sets. Component sets may optionally follow that convention, or choose their own units.

Implementing both of those points would require a lot of work both in Qt and in your component set. So I’m not surprised no-one has done it yet. But I think that amount of effort is necessary to avoid ruining your QML by sprinkling little bits of badness everywhere.

In conclusion: the interpreted language aspect of QML should be a boon for cross-platform compatiblity – not a drag. Now that the focus has shifted I think we can finally spend time working on QML as a language and not just as a vehicle for QtQuick on certain (Nokia) platforms. And it looks like I have spent a lot of time on this already 😉 .