Wednesday, October 13, 2010

Qt Creator: How to compile it significantly faster

I prefer to use latest Qt and latest Qt Creator - to use new features and also to see that nothing is broken in the current version. So I recompile Qt and Qt Creator from the scratch quite often - to be sure that there are no any bugs due to the incorrect compilation.

So I want to compile them as fast as possible. My Qt configure is so:
-debug -opensource -no-qt3support -no-phonon-backend -no-phonon -no-multimedia -no-webkit -no-svg -no-javascript-jit -optimized-qmake -no-cups -reduce-relocations -no-xmlpatterns -no-xvideo -nomake demos -nomake examples -declarative -qt-sql-sqlite -developer-build -qtnamespace MegaQtNameSpace
Briefly - do not compile a lot of modules which will not be needed for me, and also specify Qt namespace (to see any errors related to misuse this feature). I use Ubuntu 10.10, so I had just to create a copy of config.status - and copy it to any new shadow build folder.

I think that most interesting thing with such Qt config is that the Qt compilation requires almost the same time like the Qt Creator compilation. About 22-23 minutes on my home laptop and about 16-18 minutes - on the work laptop. (I compile Qt Creator like any usual Qt application: qmake -r; make -j 2)

What might be the reason? Even reduced Qt has to be much more than Qt Creator.

I decided that the answer is a usage of precompiled headers in Qt sources by default and lack of usage this feature in Qt Creator.

What are precompiled headers?

Quote from Qt documentation:
"Precompiled headers are a performance feature supported by some compilers to compile a stable body of code, and store the compiled state of the code in a binary file. During subsequent compilations, the compiler will load the stored state, and continue compiling the specified file. Each subsequent compilation is faster because the stable code does not need to be recompiled."

Also by the documentation - it should be very easy to add precompiled headers to any Qt project, almost everything will be done by the framework:
  1. Create a special header file which has to contain stable and static code
  2. Define the PRECOMPILED_HEADER variable in the project file
And that's all.

There is just some note, that "all platforms that support precompiled headers have the configuration option precompile_header set". It looks like a bug in the documentation, I became confused by this sentence, at least in the current Qt master branch grep -r precompile_header qt/mkspecs shows that precompile_header is predefined only for WinCE and Win32.

Ok, anyway it can be enabled manually by:
CONFIG *= precompile_header

Which exactly precompiled header has to be used for Qt Creator? Lets check what is used for Qt.

  1. QtCore - just 9 base Qt headers
  2. QtGui - the same base + 12 classes related to QtGui

Suddenly for me - just a small amount of headers are included. I think that almost the same might be used for Qt Creator.
Lets just add there, for example, QPointer and QScopedPointer headers. Small shell command to see the most popular includes (has to be improved, of course):
egrep -rsh ^[[:space:]]*#[[:space:]]*include * | sed 's/^ *\(.*\) *$/\1/' | sort | uniq -c | sort -n | less

Qt Creator consist of different projects (libraries and plugins), so PRECOMPILED_HEADERS has to be defined in a lot of sub projects. How it can be done?

This is my solution:
Merge request for Qt Creator.

So general Qt headers will be used in any libraries (by qtcreatorlibrary.pri), and extended with Gui part - in any plugin (by qtcreatorplugin.pri). Also it is possible to use any other precompiled headers, like it is done for Botan (it is pure C++ library, so even general Qt header are almost useless).

As it is already said, precompiled headers are not enabled by default for linux gcc, so it has to be done manually like here:
qmake "CONFIG*=precompile_header" -r ../qt-creator

Ok, what are the results? Is compiling become faster?

Ubuntu 10.10 x64, CPU T7500, 3GB RAM

Without precompiled headers (average, ~10 tries):
real ~23m
user ~36m
sys 4:30

With precompiled headers (average, ~20 tries):
real 14:30
user 21m
sys 3:30

Ubuntu 10.10 x64, CPU T9600, 4GB RAM

Without - ~16 minutes, with - ~11:30.

Required space: 1.3 GB without pch and 2.8 GB with pch.

Ok, the result is already great for me, about 40% faster than before.

I tried to add more headers to qtcreator_pch.h or botan_pch.h, but again - a compilation became slower. I think the reason is that the .pch file become too big and might not be cached good enough.

I decided to check, what is the result for Windows.

Windows Vista x86, CPU P8400, 2GB RAM

Precompiled headers are enabled by default for Windows, so the next command line was used to compile the version without precompiled headers:
qmake "CONFIG-=precompile_header" -r ../qt-creator

Qt 4.7.0 release (Qml Designer will not be compiled).
Visual Studio 2008.

With precompiled headers: ~20 minutes (~2 GB)
Without precompiled headers: ~70 minutes (~1 GB)
So the compilation time is reduced to ~30%. Amazing.
And I recalled why I switched to use Linux. :)

About a required additional hard disk space for precompiled headers. Again, Qt Creator consist of a lot of subprojects, so any of this projects - will create its own precompiled header, even if it is the same qtcreator_pch.h. It requires time and disk space. I hope later Qt will be able to handle such situation also (if supported by compiler).

Ok. I hope I will be able to check Mac OS X results also, it is interesting - is it worse or better than Windows. :)

So, the conclusion: a lot of space for contributions to Qt Creator in different areas. E.g. I think a plugin-helper to create precompiled headers might be useful (at least count the used headers - or more useful, up to the special wizard).

Update 25.10:
It looks like precompiled headers sometime are not the good choice, especially on 'cool compiler PCs'. E.g. Linux PC which might compile Qt Creator by make -j 10 in 12 minutes - with enabled compiled header might became totally disappointing, compilation time might grow to 30-40 minutes. It looks like the reason is cache - requirements to the available free memory are much more stricter now.

I think the solution might be something like 'global' precompiled headers (for most Qt Compiler subprojects - precompiled headers are almost the same, so it really might be compiled just once). Problem - different defines for such precompiled headers, has to be investigated how exactly it has to be done correctly.

Sunday, September 19, 2010

Qt Creator: Improving user experience

I use Qt Creator daily (to be more specific - one of the latest builds).
Every week it becomes better, bug fixing and feature improvements are ongoing.
Qt Creator (and Qt) is Open Source and almost anybody can contribute it. It is available on Gitorious.

What is the most annoying in Qt Creator now? For me it is the time I have to wait before Qt Creator will be loaded and ready to work. On the clean PC with just cached previously started application - it takes about 800 ms only, but otherwise - it can take 5-10 seconds, and some time - up to 30 seconds (I use Ubuntu 10.04 x64 with T7500 CPU and 3 GB RAM). Can it be somehow improved? I decided to check this issue.

Qt Creator consist of different plugins (a lot, the list + description is available in Help - About plugins). Source for the plugin library is here: src/libs/extensionsystem. It is possible to profile plugin loading by using an option '-profile'. After loading - every plugin is initialized by initialize (initializePlugin in the report) and extensionInitialized (initializeExtensions) calls. Difference is the order of dependent plugins' execution, it is easy to see by the '-profile' output.

So what is the real result for Qt Creator loading? What is the slowest plugin?
    TOP5 plugins by loading time
  1. Welcome - suddenly it can take up to 20-30% of full Qt Creator loading time.
  2. Core - this is really a core for Qt Creator, so it is hard to optimize it.
  3. Debugger - also a lot of features / sub-plugins.
  4. Find - quite interesting why it is here
  5. Qt4ProjectManager
Why it is TOP5? It had to be TOP3, but there is a Core plugin, and it is very complicate. :)

Find plugin - short investigation shows that it is due to X11 specific problem, QApplication::clipboard() has to initialize X11 desktop also. So it is X11 problem only, can be investigated more later.

Ok, lets try to check Welcome plugin. Most time is consumed by WelcomePlugin::initialize. Sources can be find here: src/plugins/welcome. What is going on there? Lets profile it more carefully.
Most work is done here:
m_welcomeMode = new WelcomeMode;
So what is the WelcomeMode::WelcomeMode()?

Additional profiling - and the next 2 lines require most time for the execution:
m_d->m_welcomePage = new ImageWidget(QPixmap(":/welcome/images/welcomebg.png"), m_d->m_widget);
2nd line is just an usual UI setting up (from Designer), it is executed quite long time but it is hard to investigate why (is it related to Qt - to styleSheet usage?..) So lets try to check 1st line - what is exactly done there and why it is so slow? ImageWidget::ImageWidget() is almost empty, so the longest time is required for QPixmap creation. ImageWidget does not use this QPixmap directly anyway, and uses it for the scaling source only.

What is src/plugins/welcome/images/welcomebg.png? Some info:
21649 bytes, PNG image, 800 x 600, 8-bit/color RGBA, non-interlaced

So it is quite a big image (~2 mb unpacked). What might be the problem with the performance? Try to check Qt documentation: "QImage is designed and optimized for I/O, and for direct pixel access and manipulation, while QPixmap is designed and optimized for showing images on screen." As mentioned already, ImageViewer uses the source image (m_bg) for the scaling source only, how the scaling is implemented? Check Qt sources, QPixmap for the scaling uses QPixmapData::transformed. It is implemented so:
QPixmap::fromImage(toImage().transformed(matrix, mode));
So lets try to change QPixmap as a source for the background in ImageWidget to QImage (it will be faster even for the scaling). It will be so:
m_d->m_welcomePage = new ImageWidget(QImage(":/welcome/images/welcomebg.png"), m_d->m_widget);
And ImageWidget has to be changed accordingly.
Patch is available here.

Ok, lets test will it be really faster or not.

Welcome plugin does not depend to any other plugin except Core, so to do the testing more real - Qt Creator will be compiled in the release mode for the next Qt 4.7 version: 4.7.0~rc1+git20100917-0ubuntu1.
    How the testing will be done?
  1. Start old Qt Creator, wait until it will be ready, kill it, wait 5 seconds
  2. Start new Qt Creator, wait until it will be ready, kill it, wait 5 seconds
  3. Go to 1
Test results (time in ms):
initializeextInittotal loadinitializeextInittotal load
both139 both416
Together: 139 ms (81 + 58) for modified Qt Creator vs 416 (352+64) for original one. Reduced to about 33%, or +200% to the loading speed for particular Welcome plugin. Full load time - about 300 ms faster or +33%. 900 ms vs 1200 ms - it is palpably faster.

Is it really so? Yes and no.
  • Yes, at least for X11 Welcome plugin became faster per 200% (on the particular PC). And for any platform - in ImageWidget::paintEvent due to the scaling.
  • No, the test is synthetic. There is enough available memory, and a lot of files might be cached, so even 2 different Qt Creators start faster than usual. E.g. here Qt Creator starts in 25 seconds: QTCREATORBUG-2396. 250 ms (or even 5 seconds) faster for 25 seconds - it is almost invisible.
A test on the much faster PC (the same Ubuntu 10.04, T9600 @ 2.80GHz + 4 GB RAM, Video: Quadro FX 770M) does not show any speed up at all. So this is even a more specific patch, related to something else (some resource?...)

For the created patch - a merge request is created. Lets hope it will be accepted. :)

WelcomePlugin::extensionsInitialized (and other plugins) will be investigated and considered later.

Conclusion: Qt Creator's loading speed might be significantly improved.

It just has to be done, and of course to be done correctly. And there are a lot of other, more important tasks for Qt Creator team. What is one of the main difference between official Qt Creator team and any independent developer? Priorities. An independent developer can spend a lot of hours for some minor suggestion due to any reason, for example, because it is interesting.

Except improvements for the current solution, any other possibilities might be considered, e.g. boost showing the main window with base plugins only - and continue the loading in the background (e.g. Settings widgets) etc.

Ok, what is the main reason for this article? To show that anybody can help Qt Creator to become better. Also I am looking for a new challenge, and I decided to try to speed up Qt Creator loading. Do you want to participate?..

Any feedback is highly appreciated.