How to generate CentOS 7 sysroot on Ubuntu 18.04

There was some delay since the previous post... So might be there will be more posts soon.

Ok, I want to compile 32-bit C application with clang 6 for CentOS 7 on my 64-bit Ubuntu 18.04. glibc on Ubuntu is 2.27, very custom 2.17 on CentOS 7 (full version - 2.17-222.el7), and some other potential dependencies.

Prerequisites

Installed clang and lld (to compile and link), yum and rpm.

Sysroot 

• wget -c http://mirror.centos.org/altarch/7/os/i386/Packages/centos-release-7-5.1804.el7.centos.i.i686.rpm
• for 64-bit: http://mirror.centos.org/centos/7/os/x86_64/Packages/centos-release-7-5.1804.el7.centos.x86_64.rpm
• mkdir centos7sysroot
• cd centos7sysroot
• sudo rpm -i --root=$PWD --nodeps ../centos-release-7-5.1804.el7.centos.i.i686.rpm
• Output:
  

  rpm: RPM should not be used directly install RPM packages, use Alien instead!
  rpm: However assuming you know what you are doing...
  warning: ../centos-release-7-5.1804.el7.centos.i.i686.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
  warning: %post(centos-release-7-5.1804.el7.centos.i.i686) scriptlet failed, exit status 127
  

• sudo sed -i "s/\$basearch/i386/g" etc/yum.repos.d/CentOS-Base.repo
• for 32 bits, as otherwise host yum will use x86_64
•  sudo sed -i "s/gpgcheck=1/gpgcheck=0/g" etc/yum.repos.d/CentOS-Base.repo
•  sudo yum --installroot=$PWD install -y glibc-devel glibc-headers
• Output:
  

  ...
  Running Transaction
  ** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows:
  centos-release-7-5.1804.el7.centos.i.i686 has missing requires of /bin/sh
  centos-release-7-5.1804.el7.centos.i.i686 has missing requires of coreutils
  centos-release-7-5.1804.el7.centos.i.i686 has missing requires of grep
  
  ...
  
  Installed:
    glibc-devel.i686 0:2.17-222.el7                                                                   glibc-headers.i686 0:2.17-222.el7                                                                  
  
  Dependency Installed:
    basesystem.noarch 0:10.0-7.el7.centos            bash.i686 0:4.2.46-30.el7          filesystem.i686 0:3.2-25.el7                                glibc.i686 0:2.17-222.el7                         
    glibc-common.i686 0:2.17-222.el7                 info.i686 0:5.1-5.el7              kernel-headers.i686 0:3.10.0-862.3.2.el7.centos.plus        libgcc.i686 0:4.8.5-28.el7_5.1                    
    libselinux.i686 0:2.5-12.el7                     libsepol.i686 0:2.5-8.1.el7        libstdc++.i686 0:4.8.5-28.el7_5.1                           ncurses-base.noarch 0:5.9-14.20130511.el7_4       
    ncurses-libs.i686 0:5.9-14.20130511.el7_4        nspr.i686 0:4.19.0-1.el7_5         nss-softokn-freebl.i686 0:3.36.0-5.el7_5                    nss-util.i686 0:3.36.0-1.el7_5                    
    pcre.i686 0:8.32-17.el7                          setup.noarch 0:2.8.71-9.el7        tzdata.noarch 0:2018e-3.el7                                 zlib.i686 0:1.2.7-17.el7                          
  
  Complete! 

Sysroot is created.

Compiler settings

CC = clang-6.0 -m32 --sysroot=sysroot_path
LD = $(CC) -fuse-ld=lld
or
LD = clang-6.0 -m32 --sysroot=sysroot_path -fuse-ld=lld

Some hardening clang options:

CHARDENING = -D_FORTIFY_SOURCE=2 -D_GLIBCXX_ASSERTIONS -fasynchronous-unwind-tables -fexceptions  -fPIC \
                -grecord-gcc-switches -g -O2 -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration \
                -fstack-protector-strong -fsanitize=safe-stack -fsanitize=cfi -fvisibility=hidden -flto
LHARDENING = $(CHARDENING) -Wl,-z,defs -Wl,-z,now -Wl,-z,relro 

(-Wall and so on are enabled by default)

That's all. 

My published apllications

I am trying to publish one more application for N9, original link to my blog as a support website is not enough.

So I have created this page for the support.

Drop Cache for N9 - an utility to flush disk cache to prevent some data loss and to allow increase a free memory by dropping out an outdated low level cache (non-destructive). Free. Open source, GPLv3.

If you have any question / problem or just a feedback - please write a comment here.

Password Generator for N9

The fixed version of my Password Generator for N9 ('unpublished' since July 2011) is successfully accepted and published again to Ovi Store.

If anybody has any question / problem or just a feedback - please write a comment.

More info about the app:

It is an open source: sources are here
Available on Ovi Store here

Main features:

• Localised everywhere. It works in any language - current active for the device is used, more than 24 languages are supported now by N9.
• True secure random number generator is used. Hardware's one.

How is it achieved?

Localisation - mostly commonly provided for Meego Touch Framework strings are used. System uses the same strings.

True random number generator. The most interesting feature for me. :)
I am the current maintainer of mssf-qt library package, which is a part of Meego 1.2 Harmattan Security Qt API (more info - Security domain overview). Password Generator uses this library, so briefly for the random data - a request is done, which is transferred to Trusted Execution Environment (TEE) Service - cryptographic and secure one, which generates the needed data. It is not a multiple requests, for example, 8 for 8 characters, but just 1 request.

Really secure and truly random.

Nokia Qt Certification

I passed my last planned MCPD certification exam on 24 November 2011 (70-519, Pro: Pro: Designing and Developing Web Applications Using Microsoft .NET Framework 4) and became Microsoft Certified Professional Developer.

 

Also I had an opportunity to become Nokia Qt Certified Specialist on 28 November 2011, I passed 2 exams - 010-002: Qt Essentials and 022-001: Core C++ for Qt developers. 

  

So I can compare some aspects of Nokia Qt and Microsoft Professional Developer certifications.

M	N
Available at least since 1993	Since 2010
Score report contains exact received points + information of strength and weak topics, what should be improved	Currently - just Pass or Fail, by forum - might be improved in future
Notification from the company is received on the next day (as exam is provided at a special test service partner - might take some time)	The same
No paper certificate by default, PDF or XPS can be downloaded from a special MCP site, paper certificate can be ordered for an additional money	Certificate is delivered in 4-6 weeks
It is possible to check status anytime on the special site: mcp.microsoft.com, available almost immediately (see 2 rows above)	No any special site, it should be possible to add a special badge to account on developer.qt.nokia.com site by an additional request. For me - request is sent more than 1 week ago, still no any reaction/reply
A lot of different exams	Qt without Widgets seems is impossible, despite of QML is a modern way. I hope it will be improved later for Qt5

So briefly, Nokia Qt Certification is not mature enough, and I hope it will be significantly improved in future.

Anyway preparations for exams helped me to recall some topics and concentrate my knowledge.


So both certification paths - MCPD (MCTS) and Nokia Qt Developer/Specialist are really useful for developers.

Microsoft Certification Preparation

Due to the Nokia switch to Windows Phone - I got an opportunity to become MCPD by my company.
I am still mostly working with Linux and Qt, but Nokia, Nokia... :)
Here is my notes about different book as preparation materials on my way to Microsoft Certified Professional Developer (MCPD).
(I decided to become certified Web Developer 4 on Visual Studion 2010 as I think the mobile devices' future is really tightly related to Web/Cloud)
Book images might be blocked by ad blocker, but this is the only way to get legal images which I know, so please take my apologies :)

Exam 70-513.
Technology Specialist: Windows Communication Foundation Development with Microsoft .NET Framework 4

• There are no Microsoft Press books currently available.
  So I decided to take a look to the book for previous version of the exam, 70-503 for .NET 3.5. It is really outdated and can be used for preparations for this exam just in case if you really have enough time, a lot of subjects are not covered at all.
• Professional WCF 4: Windows Communication Foundation with .NET 4.
  Quite good book to recall some part related to the topic and improve knowledge, but does not cover everything what is needed for the exam. Otherwise the book itself can not be considered as something useful for specialists, it is mostly 'for beginners'.

Exam 70-516.
Technology Specialist: Accessing Data with Microsoft .NET Framework 4

• Official book.
  Really useful and "must have" for such exam. Just Entity Framework is described quite briefly.
• Pro Entity Framework 4.0 (Expert's Voice in .NET).
  Really nice book about Entity Framework. The author real is a professional and expert. Also "must have", as Entity Framework knowledge is really needed to pass the exam.

It was the most complex exam for me, despite of both "must have" books here. :)

Exam 70-515.
Technology Specialist: Web Applications Development with Microsoft .NET Framework 4

• Again official book.
  Some general info. Quite good, but I think - definitely not enough to pass the exam. But I do not know what else might be used here.

Exam 70-515.
Pro: Designing and Developing Web Applications Using Microsoft .NET Framework 4

• 70-519 Exam Ref.
  I am still preparing to this exam, but seems it is a nice book which I would like to use to prepare.

I am not sure can I say anything exam content, but I just want to advice - use any available test simulations (at least that which are available on MCTS Self-Paced Training Kit disks), does not matter how strange they are.

Qt Creator: How to compile it significantly faster

I prefer to use latest Qt and latest Qt Creator - to use new features and also to see that nothing is broken in the current version. So I recompile Qt and Qt Creator from the scratch quite often - to be sure that there are no any bugs due to the incorrect compilation.

So I want to compile them as fast as possible. My Qt configure is so:

-debug -opensource -no-qt3support -no-phonon-backend -no-phonon -no-multimedia -no-webkit -no-svg -no-javascript-jit -optimized-qmake -no-cups -reduce-relocations -no-xmlpatterns -no-xvideo -nomake demos -nomake examples -declarative -qt-sql-sqlite -developer-build -qtnamespace MegaQtNameSpace

Briefly - do not compile a lot of modules which will not be needed for me, and also specify Qt namespace (to see any errors related to misuse this feature). I use Ubuntu 10.10, so I had just to create a copy of config.status - and copy it to any new shadow build folder.

I think that most interesting thing with such Qt config is that the Qt compilation requires almost the same time like the Qt Creator compilation. About 22-23 minutes on my home laptop and about 16-18 minutes - on the work laptop. (I compile Qt Creator like any usual Qt application: qmake -r; make -j 2)

What might be the reason? Even reduced Qt has to be much more than Qt Creator.

I decided that the answer is a usage of precompiled headers in Qt sources by default and lack of usage this feature in Qt Creator.

What are precompiled headers?

Quote from Qt documentation:
"Precompiled headers are a performance feature supported by some compilers to compile a stable body of code, and store the compiled state of the code in a binary file. During subsequent compilations, the compiler will load the stored state, and continue compiling the specified file. Each subsequent compilation is faster because the stable code does not need to be recompiled."

Also by the documentation - it should be very easy to add precompiled headers to any Qt project, almost everything will be done by the framework:

1. Create a special header file which has to contain stable and static code
2. Define the PRECOMPILED_HEADER variable in the project file

And that's all.

There is just some note, that "all platforms that support precompiled headers have the configuration option precompile_header set". It looks like a bug in the documentation, I became confused by this sentence, at least in the current Qt master branch grep -r precompile_header qt/mkspecs shows that precompile_header is predefined only for WinCE and Win32.

Ok, anyway it can be enabled manually by:

CONFIG *= precompile_header

.

Which exactly precompiled header has to be used for Qt Creator? Lets check what is used for Qt.

1. QtCore - just 9 base Qt headers
2. QtGui - the same base + 12 classes related to QtGui

Suddenly for me - just a small amount of headers are included. I think that almost the same might be used for Qt Creator.
Lets just add there, for example, QPointer and QScopedPointer headers. Small shell command to see the most popular includes (has to be improved, of course):

egrep -rsh ^[[:space:]]*#[[:space:]]*include * | sed 's/^ *\(.*\) *$/\1/' | sort | uniq -c | sort -n | less

Qt Creator consist of different projects (libraries and plugins), so PRECOMPILED_HEADERS has to be defined in a lot of sub projects. How it can be done?

This is my solution:
Merge request for Qt Creator.

So general Qt headers will be used in any libraries (by qtcreatorlibrary.pri), and extended with Gui part - in any plugin (by qtcreatorplugin.pri). Also it is possible to use any other precompiled headers, like it is done for Botan (it is pure C++ library, so even general Qt header are almost useless).

As it is already said, precompiled headers are not enabled by default for linux gcc, so it has to be done manually like here:

qmake "CONFIG*=precompile_header" -r ../qt-creator

Ok, what are the results? Is compiling become faster?

Ubuntu 10.10 x64, CPU T7500, 3GB RAM

Without precompiled headers (average, ~10 tries):
real ~23m
user ~36m
sys 4:30

With precompiled headers (average, ~20 tries):
real 14:30
user 21m
sys 3:30

Ubuntu 10.10 x64, CPU T9600, 4GB RAM

Without - ~16 minutes, with - ~11:30.

Required space: 1.3 GB without pch and 2.8 GB with pch.

Ok, the result is already great for me, about 40% faster than before.

I tried to add more headers to qtcreator_pch.h or botan_pch.h, but again - a compilation became slower. I think the reason is that the .pch file become too big and might not be cached good enough.


I decided to check, what is the result for Windows.

Windows Vista x86, CPU P8400, 2GB RAM

Precompiled headers are enabled by default for Windows, so the next command line was used to compile the version without precompiled headers:

qmake "CONFIG-=precompile_header" -r ../qt-creator

Qt 4.7.0 release (Qml Designer will not be compiled).
Visual Studio 2008.

With precompiled headers: ~20 minutes (~2 GB)
Without precompiled headers: ~70 minutes (~1 GB)
So the compilation time is reduced to ~30%. Amazing.
And I recalled why I switched to use Linux. :)

About a required additional hard disk space for precompiled headers. Again, Qt Creator consist of a lot of subprojects, so any of this projects - will create its own precompiled header, even if it is the same qtcreator_pch.h. It requires time and disk space. I hope later Qt will be able to handle such situation also (if supported by compiler).

Ok. I hope I will be able to check Mac OS X results also, it is interesting - is it worse or better than Windows. :)

So, the conclusion: a lot of space for contributions to Qt Creator in different areas. E.g. I think a plugin-helper to create precompiled headers might be useful (at least count the used headers - or more useful, up to the special wizard).

Update 25.10:
It looks like precompiled headers sometime are not the good choice, especially on 'cool compiler PCs'. E.g. Linux PC which might compile Qt Creator by make -j 10 in 12 minutes - with enabled compiled header might became totally disappointing, compilation time might grow to 30-40 minutes. It looks like the reason is cache - requirements to the available free memory are much more stricter now.

I think the solution might be something like 'global' precompiled headers (for most Qt Compiler subprojects - precompiled headers are almost the same, so it really might be compiled just once). Problem - different defines for such precompiled headers, has to be investigated how exactly it has to be done correctly.

Qt Creator: Improving user experience

I use Qt Creator daily (to be more specific - one of the latest builds).
Every week it becomes better, bug fixing and feature improvements are ongoing.
Qt Creator (and Qt) is Open Source and almost anybody can contribute it. It is available on Gitorious.

What is the most annoying in Qt Creator now? For me it is the time I have to wait before Qt Creator will be loaded and ready to work. On the clean PC with just cached previously started application - it takes about 800 ms only, but otherwise - it can take 5-10 seconds, and some time - up to 30 seconds (I use Ubuntu 10.04 x64 with T7500 CPU and 3 GB RAM). Can it be somehow improved? I decided to check this issue.

Qt Creator consist of different plugins (a lot, the list + description is available in Help - About plugins). Source for the plugin library is here: src/libs/extensionsystem. It is possible to profile plugin loading by using an option '-profile'. After loading - every plugin is initialized by initialize (initializePlugin in the report) and extensionInitialized (initializeExtensions) calls. Difference is the order of dependent plugins' execution, it is easy to see by the '-profile' output.

So what is the real result for Qt Creator loading? What is the slowest plugin?

TOP5 plugins by loading time
1. Welcome - suddenly it can take up to 20-30% of full Qt Creator loading time.
2. Core - this is really a core for Qt Creator, so it is hard to optimize it.
3. Debugger - also a lot of features / sub-plugins.
4. Find - quite interesting why it is here
5. Qt4ProjectManager

Why it is TOP5? It had to be TOP3, but there is a Core plugin, and it is very complicate. :)

Find plugin - short investigation shows that it is due to X11 specific problem, QApplication::clipboard() has to initialize X11 desktop also. So it is X11 problem only, can be investigated more later.

Ok, lets try to check Welcome plugin. Most time is consumed by WelcomePlugin::initialize. Sources can be find here: src/plugins/welcome. What is going on there? Lets profile it more carefully.
Most work is done here:

m_welcomeMode = new WelcomeMode;

So what is the WelcomeMode::WelcomeMode()?

Additional profiling - and the next 2 lines require most time for the execution:

m_d->m_welcomePage = new ImageWidget(QPixmap(":/welcome/images/welcomebg.png"), m_d->m_widget);
m_d->ui.setupUi(m_d->m_welcomePage);

2nd line is just an usual UI setting up (from Designer), it is executed quite long time but it is hard to investigate why (is it related to Qt - to styleSheet usage?..) So lets try to check 1st line - what is exactly done there and why it is so slow? ImageWidget::ImageWidget() is almost empty, so the longest time is required for QPixmap creation. ImageWidget does not use this QPixmap directly anyway, and uses it for the scaling source only.

What is src/plugins/welcome/images/welcomebg.png? Some info:
21649 bytes, PNG image, 800 x 600, 8-bit/color RGBA, non-interlaced

So it is quite a big image (~2 mb unpacked). What might be the problem with the performance? Try to check Qt documentation: "QImage is designed and optimized for I/O, and for direct pixel access and manipulation, while QPixmap is designed and optimized for showing images on screen." As mentioned already, ImageViewer uses the source image (m_bg) for the scaling source only, how the scaling is implemented? Check Qt sources, QPixmap for the scaling uses QPixmapData::transformed. It is implemented so:

QPixmap::fromImage(toImage().transformed(matrix, mode));

So lets try to change QPixmap as a source for the background in ImageWidget to QImage (it will be faster even for the scaling). It will be so:

m_d->m_welcomePage = new ImageWidget(QImage(":/welcome/images/welcomebg.png"), m_d->m_widget);

And ImageWidget has to be changed accordingly.
Patch is available here.

Ok, lets test will it be really faster or not.

Welcome plugin does not depend to any other plugin except Core, so to do the testing more real - Qt Creator will be compiled in the release mode for the next Qt 4.7 version: 4.7.0~rc1+git20100917-0ubuntu1.

How the testing will be done?
1. Start old Qt Creator, wait until it will be ready, kill it, wait 5 seconds
2. Start new Qt Creator, wait until it will be ready, kill it, wait 5 seconds
3. Go to 1

Test results (time in ms):

	modified			original
	initialize	extInit	total load	initialize	extInit	total load
1	94	68	927	377	50	1155
2	94	68	1006	386	69	1137
3	76	50	845	310	69	1137
4	70	47	842	364	41	1222
5	70	49	929	450	50	1230
6	71	48	868	365	69	1292
7	94	64	997	146	71	1053
8	69	39	1141	410	63	1318
9	71	45	873	359	65	1276
10	75	65	843	387	49	1221
11	72	49	911	418	86	1192
12	117	58	916	340	69	1249
13	84	72	996	392	100	1294
14	76	74	849	209	69	1362
15	78	74	897	367	44	1146
average	81	58	923	352	64	1219
	both	139		both	416

Together: 139 ms (81 + 58) for modified Qt Creator vs 416 (352+64) for original one. Reduced to about 33%, or +200% to the loading speed for particular Welcome plugin. Full load time - about 300 ms faster or +33%. 900 ms vs 1200 ms - it is palpably faster.

Is it really so? Yes and no.

• Yes, at least for X11 Welcome plugin became faster per 200% (on the particular PC). And for any platform - in ImageWidget::paintEvent due to the scaling.
• No, the test is synthetic. There is enough available memory, and a lot of files might be cached, so even 2 different Qt Creators start faster than usual. E.g. here Qt Creator starts in 25 seconds: QTCREATORBUG-2396. 250 ms (or even 5 seconds) faster for 25 seconds - it is almost invisible.

A test on the much faster PC (the same Ubuntu 10.04, T9600 @ 2.80GHz + 4 GB RAM, Video: Quadro FX 770M) does not show any speed up at all. So this is even a more specific patch, related to something else (some resource?...)

For the created patch - a merge request is created. Lets hope it will be accepted. :)

WelcomePlugin::extensionsInitialized (and other plugins) will be investigated and considered later.

Conclusion: Qt Creator's loading speed might be significantly improved.

It just has to be done, and of course to be done correctly. And there are a lot of other, more important tasks for Qt Creator team. What is one of the main difference between official Qt Creator team and any independent developer? Priorities. An independent developer can spend a lot of hours for some minor suggestion due to any reason, for example, because it is interesting.

Except improvements for the current solution, any other possibilities might be considered, e.g. boost showing the main window with base plugins only - and continue the loading in the background (e.g. Settings widgets) etc.

Ok, what is the main reason for this article? To show that anybody can help Qt Creator to become better. Also I am looking for a new challenge, and I decided to try to speed up Qt Creator loading. Do you want to participate?..

Any feedback is highly appreciated.

My car

This is my car now. Nissan Almera 1.6 SLX, 1998.