Tag Archives: optimization

Telegram, Lottie animated stickers, Python and Glaxnimate

→ Version en français ici

Table of Content

* Introduction
* A good workflow
* Publication on the web (SVG)
* Telegram Bot

Introduction


I’m a fan of Telegram chat application interface. If the sever is closed source, desktop and mobile applications are open sources, the interface is very well designed and thinked, intuitive, relativly light and full of interesting functionnalities, compared to other applications of this kind. It could be good to patch to have a Jabber/XMPP compatibility. It is the first, as far I know, to use Lottie, since Jully 2019 (and the bot API that goes with it), an animated vector format that I dreamd of for decades, that become an de facto open standard. Animations of this page have been created with Glaxnimate, a tool to make Lottie format animations, as main goal, but that can also be used to export SVG animation. The authors of Lottie published some JavaScript for the World Wide Web, Samsung, made a free and open source rlottie C++ library, as well as WASM version, that allow to play them and has binding for several programming languages. Qt has an included Lottie reader, and there are several tools to create them.


The multiplatofmr free software Glaxnimate allow to make them with a Graphical User Interface, familiar with people creating animation on computers. His main author also made a Python library, python-lottie (AUR: python-lottie-git, pip: Lottie), that allow procedural generation of them. Both alow to convert them to several formats (export animated SVG, Lottie JSON, HTML page all ready, Telegram, mpeg4, PNG, wepb, gif, and I probably forgot some other), and as input to vectorize animated gif, etc… The 2D vector animation Synfig also have a Lottie exporter, but also Blender3D, a plugin made by Glaxnimate author (distributed with python-lottie) for this last one. There are so several choices to create them.


Glaxnimate allow to easily create animations in Lottie format, including .TGS, a version with more limited specifications and gzip compressed for Telegram. This format is more compact but has some constraints, that didn’t stop some artists to make very great artworks since 2 years:
* 512×512 pixels
* 3 secondes
* 64 KB by animated sticker..
* Animated stickers can be grouped in a sticker pack corresponding to several expressions (like emoticons/emoji).

Why for 512×512 pixels with a vector format? Telegram client application render the animation in this dimension. This dimension is already big for some mobile screens. This vector format allow to reduce bandwidth usage and to have a high quality animation (generaly computed by the GPU, SVG accelerators was already in Nokia and Symbian made phones in 2000s). WeChat/微信 choose animated GIF about one decade ago. GIF are probably converted in MPEG nowaday? The instant messaging Discord, populare in Far West, also use the Lottie format since Jully 2021, following Telegram after 2 years.


As a refecence, top of this page animation is:
* 1109 bytes (1,1 KB) in TGS (binary compressed Telegram format)
* 4682 bytes (4.5 KB) in JSON uglified (no more indentation, or carriage return)
* 508054 bytes (500 KB) in WebP
* 9710 bytes (9.6 KB) in animated SVG (there was an useless animated path, I don’t know why for).
* 1804 bytes (1.7 KB) in ainmated SVGZ (gzip compressed SVG). It is more interesting today to let the server compress using Brotli format (Nginx, Apache) or to precompress them (.br). It is really well supported by web browsers
* 6114 bytes (6.1 KB) in SVG, Inkscape optimised compressed (based on Scour)), it’s possible to reduce size more. Today if the option to reduce digits after comma (floats) is less than 3, it will be still kept at 3 digits. In animation parts still not managed there are six 0 (1.000000 or 0.000000 for example). It is still possible to optimize by hand until thes options are added (See sed usage, below), so :
* 5827 bytes (5.8 KB) in SVG, a bit hand finished.
* 5559 bytes (5.5 KB) by removing some useless intermediate groups (warning to don’t break all, try group by group) and replacing numbers like 1.0 by 1


One of the main problems of optimized SVG output from Glaxnimate is that Lottie heavily use groups, where SVG allow to made most of these operation in the objects themselves. So transformation matrix are all in groups containing the objects, instead of in objects themselves. you can see at right the XML source inf Inkscape. This still ask lot of handcrafted work on files nowaday. It should be possible to improve it.

It looks like there are more efficient methods withSVGO, but based on Node.js, and it breaks animations, some of its optimisation methods could be used in python one. I don’t like Node at all due to bad habits of lot of devs around Node. The Inkscape plugin, “inkscape-svgo” is still in this spirit, if we try to compile it with included Makefile, it doesn’t test if Nodes modules are already installed but forcee download Node JS version 11 (and x86_64 only, so not multi-architecture), and recompile all the dependenccies. So one more time with Node, we wast lot of earth limited resources, wit an extention of tens of Megabytes (75 MB, where Inkscape is 145 MB and the Node SVGO library itself is only 5 MB), where it should only be few KB.

A good workflow

* Create object with Inkscape
* Animate it with Glaxnimate
* Export it in SVG format (don’t forget to also save a Glaxnimate format file)
* Open the new file with Inkscape to clean it (extract some groups that are not needed
* Export to optimized output
* Finish by hand.

For the last part there are 2 solution for have an automated one:
* Patch Inkscape optimized output.
* Do a second path optimizer.

Some space can be saved with :

sed s/1.00000/1/g <input.svg | sed s/0.00000/0/g | sed 's/0000;/;/g' >output.svg

One or two digit precision (instead of 3 or 4 today with Scour)), should be good enough.

SVGcleaner is better than Scour in several cases, but itdoesn’t support animations and don’t want to (it’s complex to manage, can easily become a nightmare).

Publication on the web (SVG)

So as I just said, as animated SVG is really more easy to integrate in a web page, after lot of tries with Lottie, I finnaly choose this option. The best is to not compress with SVGZ, but to optimizer the SVG file then to compress with Brotli (.br) and maybe to have a gzip version beside for oldest web browsers.

Its integration in a web page is really simple, it can be done as other pictures format with image:

<img src="/chemin/du/fichier_animation.svg" align="right" width="200" height="250">

That’s what I done in this page, every browser automatically manage them.

Note : It could be interesting to integrate it inline to manipulate an easier way with Javascript. For simple animated illustration, maybe just animated/zoomed/rotated object, without inside details modifications, image is perfect.

For compressing I use the max compression, here is a simple method to compress all files of a single directory beside their uncompressed version:

ls --ignore=*.gz --ignore=*.br | while read file
do
 brotli -q 11 -c <"${file}" >"${file}".br
 gzip -9 -c <"${file}" >"${file}".gz
done

* 5559 murphy_anim0.animoptiv5.svg
* 1303 murphy_anim0.animoptiv5.svg.br
* 1543 murphy_anim0.animoptiv5.svg.gz
* 4682 murphy.json
* 976 murphy.json.br
* 1149 murphy.json.gz

To force loading one of the 2 files format, if it is supported by borwer (I mean if it supports both), Brotli (.br) will be used, with Nginx (after compiled it with the Brotli patch, rules in the Nignx configuration files are realy simples:

brotli_static on;
gzip_static on;

With Apache, Brotli is here by default in versions 2.4, but it’s a bit harder to manage.

To test their support, Curl can be used:

curl -I -H 'Accept-Encoding: gzip, deflate, br' https://host.net/fichier.svg

If brotli is really activated, you will receive the following line in the answer:

content-encoding: br

Telegram Bot

The Python-Lottie library come just right for me, I would like to study Python, including for micropython, used in embedded world and for my work of System and Network administrator, where it is more and more used, replacing step by step Perl, and with some tools like Ansible an automation paltform used to manage computer farms, as well as more and more other basic system tools. After about one year of experience with procedural generation in Lua give me some willing to do this with other languages too (I’m currently preparing a suprise for soon). There are several bot libraries for Telegram in Python, including Python Telegram Bot, that follow well the official Telegram API, and Telethon (Documentation) that allow to communiatte directly in MTproto, the Telegram protocole, instead of using HTTP layer. This reduce one layer and allow a more fine control of exchanges. You can read here about the differences between Bot API (HTTP vs MT Proto).


There are still not too much examples with Python’s Telegram bot, that I started with, but by digging a bit in documentation, it is possible to find what’s needed. By mixing Python-Lottie for procedural generation, Glaxnimate for hand made animation, and the bot there is some interesting possibilities that open. I so made a first Python-Telegram-Bot bot. After some exchanges with Glax, the author of Glaxnimate, he let me know the Telethon Python module, that it uses for its bot, that has some similarities with what I want. It’s called Glaxcomm, and generate Lottie/TGS on the fly. I hope I will manage (time+willing) to rebuild my bot with this module and I hope I can get it where I want it. My next goal would be to be able to use it with other protocols, including ActivityPub (W3C specifications).

Glax made a good set of basic procedural generation examples that greatly help to understand it’s usage, a good documantation about Lottie format, stickers scripts examples.

I started by make a simple bot from python-telegram-bot library examples, but complete explanation about how to use file_id was missing (and I didn’t found them around the net). This is the methode Telegram promote to to send file, by it’s Id instead of sending several times the same file. The file_id is given by Telegram when a file is sent. So ideal method is to keep the ID of already succefuly sent files in a database or a file on the disk. This example create a file containing the ID beside image file, this is probably a bit better to have thei ID in a database (SQL or TOML) :

if os.path.exists(idfile):
  fid = open(idfile)
  stickid = fid.read()
  fid.close()
  msg = update.message.reply_sticker(stickid)
else:
  msg = update.message.reply_sticker(open(stickfile, 'rb'))
  fid = open(idfile, 'w')
  fid.write(msg['sticker']['file_id'])
  fid.close()

Create aligned disk partitions to improve performances and reduce SSD

Align disk partitions on their cylinders, is, knowing that most frequent datas feet on a cylinder:

* Avoid to read 2 cylinders instead of one => gain time at read, reduce useless cache usage, gain bandewidth.
* Avoid to write 2 cylinders when only one is good enough => gain time, reduce useless cache usage, gain bandwith, reduce wearout
* For the previous reason, allow to grow greatly timelife of an SSD disk (number of writing cycles is limited on an SLC, more on an MLC, even more on a TLC, but the price of this last one is the lower by GB).

A simple method to know if your partitions are well aligned on cylinders

Launch cfdisk on the disk you want to optimize (replace /dev/sda, by the disk to optimize: /dev/sdb, /dev/sdc…):

cfdisk /dev/sda

If you see an asterisk (or star) at the most right of a partition ligne, this one is not aligned on cylinder. In my case, I have to remake all the partitions but sda3:

                           cfdisk (util-linux 2.20.1)

                              Disk Drive: /dev/sda
                       Size: 240057409536 bytes, 240.0 GB
             Heads: 255   Sectors per Track: 63   Cylinders: 29185

    Name        Flags      Part Type  FS Type          [Label]        Size (MB)
 ------------------------------------------------------------------------------
                            Pri/Log   Free Space                           1.05*
    sda1        Boot        Primary   ext4                             51158.98*
                            Pri/Log   Free Space                           1.22*
    sda2        Boot        Primary   ext4                             53686.01*
                            Pri/Log   Free Space                           0.41*
    sda3                    Primary   ext4                            135207.16
                            Pri/Log   Free Space                           2.62*





     [   Help   ]  [   New    ]  [  Print   ]  [   Quit   ]  [  Units   ]
     [  Write   ]

                      Create new partition from free space

How to create an unique partition, using the whole disk and well fited

Using the old usagemode of fdisk. I found this tip on an Ubuntu documentation.

Replace /dev/sdX by the disk on which you want to create aligned partition:

fdisk /dev/sdX
c
u
p
n
[return key]
[return key]
[return key]
[return key]
w

Et voilà, you will see a beautifull result on cfdisk

To create several aligned partitions: Choose the good option instead of the 4 time [return key] and redo the n (new), for each partition.