Daniel Nicoletti

Brazil, São Paulo http://dantti.wordpress.com 27 Posts

Announcing Sticklyst - leveraging KDE Frameworks on the Web

Sticklyst is a web paste tool, like pastebin, Stick Notes (paste.kde.org), build with Cutelyst and KDE Frameworks.

Building this kind of tool has been on my TODO list for a long time, but never really put some effort into it. When the idea first came by, I decided to look at the code of http://paste.scsys.co.uk/ which is powered by a Perl Catalyst application, to my surprise the Perl module that handled syntax highlighting was a port of the code of Kate, and it even said it used Kate's definitions.

If that code used Kate's, I'd better use the code directly, I hoped it wouldn't be too coupled with GUI classes, but failed to find the code. A week ago I saw a mention about this on KDE's mailing list, so with a little help from some IRC fellows I met the https://cgit.kde.org/syntax-highlighting.git/ code.

Then thanks to some tips from Volker Krause who showed me how to use it without GUI widgets, I managed to build a simple web app, which I presented live on the first Qt conference here in Brazil at São Paulo https://br.qtcon.org/.

Just for record the application uses less than 3MB of RAM and renders quickly using a sqlite db for the storage.

The code is 90% completed, is on production at https://paste.cutelyst.org/, and is now up for adoption. Basically someone willing to take care of it could implement the following missing features compared to Stick Notes:

  • Extra database support
  • URL shortener
  • User authentication

The list is short and the code is clean, go test, break, hack:

https://github.com/cutelyst/sticklyst

Have fun!

Cutelyst 1.8.0 released

Cutelyst the Qt Web Framework, has another stable release, this release is mostly filled with bug fixes, the commit log is rather small.

It got fixes on Cutelyst-WSGI to properly work on Windows, QtCreator integration fixes, properly installing dll's on Windows, fix returning the right status from views (this allows you to know if for example View::Email sent the email with success).

Cutelyst is cross-platform, the code builds on CI running on Windows, OSX and Linux, but first class is still UNIX systems, as I don't use Windows, nor have a VM with development configured so I have only ensured it build fine on AppVeyor, Aurel Branzeanu provided some pull requests to improve that, if you also happen to have experience with CMake, MSVC and Windows please help me review or do other pull requests.

The release also got two API additions:

  • Context::setStash(QString, ParamsMultiMap), ParamsMultiMap also known as QMap<QString, QString> is a type used on HTTP query and body parameters, having this method avoid writing ugly code full of QVariant::fromValue()
  • Application::pathTo(QString), instead of using the QStringList version which join the strings to build a path (like the Catalyst does), this just gets the string as you would pass to a QFile, that in turn deals with platform issues.

I'm a bit busy with html-qt and also planning a rewrite of simplemail-qt to be async, so expect smaller Cutelyst releases :)

Again help is welcome (specially on Windows usage).

Get it here https://github.com/cutelyst/cutelyst/archive/v1.8.0.tar.gz

CMlyst 0.3.0 released

CMlyst is a Web Content Management System built using Cutelyst, it was initially inspired by Wordpress and then Ghost. So it's a mixture of both.

Two years ago I did it's first release, and since them I've been slowly improving it, it's been on production for that long providing www.cutelyst.org web site/blog. The 0.2.0 release was a silent one which marks the transition from QSettings storage to sqlite.

Storing content on QSettings is at first quite interesting since it's easy to use but it showed not suitable very fast, first it kept leaving .lock files, then it's not very fast to access so I had used a cache with all data, and a notifier updated that when something changed on the directory, but this also didn't properly triggered QFileSystemWatcher so once a new page was out the cache wasn't properly updated.

Once it was ported to sqlite, I decided to study how Ghost worked, this was mainly due many Qt/KDE developer switching to it. Ghost is quite simplistic, so it was very easy to try to provide something quite compatible with it, porting a Ghost theme to CMlyst requires very little changes due it's syntax being close to Grantlee/Django.

Due porting to sqlite it also became clear that an export/import tool was needed, so you can now import/export it in JSON format, pretty close to Ghost, actually you can even import all you Ghost pages with it, but the opposite won't work, and that's because we store pages as HTML not Markdown, my feeling about markdown is that it is simple to use, convenient to geeks but it's yet another thing to teach users which can simply use a WYSIWYG editor.

Security wise you need to be sure that both Markdown and HTML are safe, and CMlyst doesn't do this, so if you put it on production be sure that only users that know what they are doing use it, you can even break the layout with a not closed tag.

But don't worry, I'm working on a fix for this, html-qt is a WHATWG HTML5 specification parser, mostly complete, but the part to have a DOM, is not done yet, with it, I can make sure the HTML won't break layout and remove unsafe tags.

Feature wise, CMlyst has 80% of Ghost features, if you like it please help add missing features to Admin page.

Some cool numbers

Comparing CMlyst to Ghost can be trick, but it's interesting to see the numbers.

Memory usage:

  • CMlyst uses ~5MB
  • Ghost uses ~120MB

Requests per second (using the same page content)

  • CMlyst 3500/rps (production mode), 1108/rps (developer mode)
  • Ghost 100/rps (production mode)

While the RPS number is very different, on production you can use NGINX cache which would make the slow Ghost RPS not a problem, but that comes to a price of more storage and RAM usage, if you run on an AWS micro instance with 1GB of RAM this means you can have a lot less instances running at the same time, some simple math shows you could have 200 CMlyst instaces vs 8 of Ghost.

Try it!

https://github.com/cutelyst/CMlyst/archive/v0.3.0.tar.gz

Sadly it's also liked soon I'll be forking Grantlee, the lack of maintenance just hit me yesterday (when I was going to release this), Qt 5.7+ has changed QDateTime::toString() to include TZ data which broke Grantlee date filter which isn't expecting that, so I had to do a weird workaround marking date as local to avoid the extra information.

Cutelyst 1.7.0 released! WebSocket support added.

Cutelyst the Qt / C++11 web framework just got another important release.

WebSocket support is probably a key feature to have on a modern web framework, Perl Catalyst doesn't look like it wasn't designed with it in mind, the way I found to do WS there wasn't intuitive.

Qt has WebSocket support via QWebSockets module, which includes client and server implementation, and I have used it for a few jobs due lack of support inside Cutelyst. While looking at it's implementation I realized that it wouldn't fit into Cutelyst, and it even had a TODO due a blocking call, so no go.

I've looked then at uWSGI implementation and while it is far from being RFC compliant or even usable (IMO) the parser was much simpler, but it required that all data was available to parse. Since messages can be split into frames (really 63bits to address payload was probably not enough) and each frame has a bit to know if the message is over, which uWSGI simply discards, thus any fragmented message can't be merged back.

WebSockets in Cutelyst have an API closer to what QWebSocket has, but since we are a web framework we have more control over things, first on your websocket endpoint you must call:

c->response->webSocketHandshake()

If the return is false just return from the method to ignore the request or it might be the case that you want the same path to show a chat HTML page, this can happen if the client didn't sent the proper headers or if the Engine backend/protocol doesn't support WebSockets. If true you can connect to the Request object and get notified about pong, close, binary and text (UTF-8) messages and frames, a simple usage is like:

connect(req, &Request::webSocketTextMessage, [=] (const QString &msg) {

  qDebug() << "Got text msg" << msg;

  response->webSocketTextMessage(msg);

});

This will work as a Echo server, the signals also include Cutelyst::Context in the case you are connecting to a slot.

Cutelyst implementation is non-blocking, a little faster than QWebSockets, uses less memory and passes all http://autobahn.ws/testsuite tests, it doesn't support compression yet.

systemd socket activation support was added, this is another very cool feature although it's still missing a way to die the application on idle. On my tests with systemd socket activation, Cutelyst started so fast that the first request that starts it takes only twice the time it would take if Cutelyst was already running.

This release also include:

  • Fixes for using cutelyst-wsgi talking FastCGI to Apache
  • Workaround QLocalServer closing down when an accept fail (very common when used on prefork/threaded servers)
  • Fixed when using cutelyst-wsgi on threaded mode, POST requests would use the same buffer corrupting or crashing.
  • A few other fixes.

Due WebSockets plans for Cutelyst 2 are taking shape, but I'll probably try to add HTTP2 support first since the multiplexing part is might make Cutelyst Engine quite complex.

Have fun https://github.com/cutelyst/cutelyst/archive/v1.7.0.tar.gz

Cutelyst benchmarks on TechEmpower round 14

The new TechEmpower benchmarks results are finally out, round 14 took almost 6 months to be ready, and while a more frequent testing is desirable this gave us plenty of time to improve things.

Last round a bug was crashing our threaded benchmarks (which wasted time restarting), besides having that bug fixed the most notably changes that improved our performance were:

  • Use of a custom Epoll even dispatcher instead of default glib one (the tests without epoll in it's name are using the default glib)
  • jemalloc, this brought the coolest boost, you just need link to it (or use LD_PRELOAD) and memory allocation magically get's faster (around 20% for the tests)
  • CPU affinity that helps the scheduler to keep each thread/process pinned to a CPU core.
  • SO_REUSEPORT which on Linux helps to get connections evenly distributed among each thread/process
  •  Several code optimizations thanks to perf

I'm very glad with our results, Cutelyst managed to stay with the top performers which really pays off all the hard work, it also provide us with good numbers to show for people interested in.

Check it out https://www.techempower.com/benchmarks/