Author Archives: Popolon

64x64x16colours (Sweetie16) PixelArt with Pixelorama “β-karoten – We know whom will be eat”

→ Version en français ici

β-karoten - We know whom will be eat

I participated to the LoveByte Battleground demoparty> that runned this week-end, by posting a drawing last week. Sadly/funnily , there was few mistake ^^:
* I made a 64×64 pixels picture instead of a 128×128 one. The palette for the competition was 16 colours Sweetie16 (the default one on FOSS, TIC-80 fantasy console (Source code)).
* My second mistake is I’ve uploaded a first version, and few hours later another one using (FOSS) source code) on Debian on (FOSH) RISC−V (Specifications, there are lot of free or not implementations) emulator of FOSS Qemu (Source code, Git instance), the first upload worked fine but the second one didn’t work (maybe because my installation of Netsurf doesn’t manage javascript)^^. I also performed in a livecoding match (256 bytes and 25 minutes limits). Result of my poor production, Commented Live coding session record.

This is made with FOSS Pixelorama (Source code), itself made on FOSS Godot game engine (Source code). I use FOSS Arch Linux OS. Also made a ArchLinux AUR package pixelorama-git after pixelorama package (for git version, I would like to use v0.9, still not out, only v0.8 was available). There are pixelorama package (last stable, compiling from source), and pixelorama-bin package (from developers binary tarball). Pixelorama is a Pixel art picture and animation editor. I believe I discovered Pixelorama thanks to blog Librearts.org.

The name is “β-karoten – We know whom will be eat”.

Pixelorama editor screenshot
Screenshot of Pixelorama

Telegram, autocollants animés Lottie, Python et Glaxnimate

Sommaire

* Introduction
* Un bon flux de travail
* Publication sur le web (SVG)
* Bot Telegram

Introduction


Je suis un grand fan de l’interface de l’application de chat Telegram. Si le serveur est fermé, l’application bureau et Android est ouverte, l’interface est bien pensée, intuitive, relativement légère et plein de fonctionnalités intéressantes par rapport aux autres applications de ce type. Il serait d’ailleurs sans doute intéressant de la patcher pour une compatibilité Jabber/XMPP. Il a été le premier à ma connaissance, à utiliser Lottie, dès juillet 2019 (et l’API pour bot qui va avec) un format d’animation vectoriel dont je rêvait depuis des décennies, devenu un standard de fait et ouvert. Les animations de cette page ont été crées avec Glaxnimate, un outil pour réaliser des animations au format Lottie, que l’on peut également exporter au format SVG animation. Les auteurs de Lottie fournissent des JavaScript pour le web, la bibliothèque libre, rlottie de Samsung (en C++, avec version WASM) permet de les jouer et comporte des adaptations pour différents langage de programmation. Qt intègre sa lecture par défaut et il existe différents outils pour en créer.


Le logiciel libre multiplate-forme d’animation Glaxnimate permet d’en réaliser avec une interface graphique d’un type assez classique pour les logiciels d’animation. Son auteur à également fait une bibliothèque python, python-lottie (AUR : python-lottie-git, pip : Lottie), permettant d’en générer algorithmiquement et les 2 permettent de convertir différents formats (export SVG animé, Lottie JSON, page HTML toute prête, Telegram, mpeg4, PNG, wepb, gif, et je dois en oublier), et en entrée de vectoriser des gif animés, etc… Le logiciel d’animation vectoriel 2D Synfig à également un exportateur, ainsi que Blender3D, fait également par l’auteur de Glaxnimate. Les choix ne manquent donc pas pour en créer.


Glaxnimate permet de faire facilement des animations au format Lottie, dont le format .TGS, une version aux spécifications volontairement plus limitées et compressée en gzip de Telegram. Le format de ce dernier est plus compact mais comportes quelques contraintes, cela n’a pas empêché d’avoir quelques merveilles depuis plus de 2 ans :
* format 512×512 pixels
* 3 secondes
* 64 Kio par autocollant animé
* Le tout peut être mis dans un paquet d’animations (sticker pack) regroupant différentes expressions.

Pourquoi 512×512 pixels alors que c’est un format vectoriel ? L’application cliente Telegram rendra dans cette dimension l’animation et la compressera en local, pendant la première boucle, en animation WebP de 512×512 pixels pour la conservation en cache. Cette dimension est déjà grande pour les certains écrans mobiles. Cela permet d’économiser la bande passante (animation vectorielle compacte transférée), d’avoir la haute qualité du vectoriel (généralement calculé par GPU, des accélérateurs SVG étaient déjà présent sur les téléphones Symbian des années 2000), et d’avoir des boucles d’animation qui ne prennent pas trop de CPU/GPU après la première boucle, ni trop de mémoire (compression WebP). WeChat/微信 a choisit des GIF animés il y a une bonne décennie, les GIF sont probablement convertit en MPEG aujourd’hui ? La messagerie instantanée Discord, populaire en Extrême-Occident, utilise aussi le format Lottie depuis juillet 2021.


Pour référence, l’animation du haut fait :
* 1109 octets (1,1 Ko) en TGS (format Telegram compressé binaire)
* 4682 octets (4.5 Ko) en json uglifié (plus d’indentations ni re retours chariots)
* 508054 octets (500 Ko) en WebP
* 9710 octets (9.6 Ko) en SVG animé (il y avait un path inutile en trop, je sais pas pourquoi).
* 1804 octets (1.7 Ko) en SVGZ animé (SVG compressé via gzip), il peut être plus intéressant de laisser le serveur compresser via brotli (Nginx, Apache) ou de pré-compresser (.br) Il est bien supporté par les navigateurs
* 6114 octets (6.1 Ko) en SVG, sortie optimisée Inkscape, on peut encore gagner, aujourd’hui si l’option de précision du nombre de chiffres après la virgule est inférieure à 3, elle reste à 3. Dans les parties animation non gérées on se retrouve à six 0 (1.000000 ou 0.000000). On peut encore l’optimiser à la main en attendant de l’intégrer (voir les sed plus bas), soit :
* 5827 octets (5.8 Ko) en SVG, retravaillé un peu à la main.
* 5559 octets (5.5 Ko) en supprimant des groupes intermédiaires inutiles et en remplaçant 1.0 par 1


Un des principaux problèmes à la sortie optimisée de SVG depuis Glaxnimate est que Lottie est basés sur une forte utilisation des groupes, tandis-que SVG permet d’effectuer de nombreuses opérations équivalentes dur les objets eux-mêmes. Ainsi les matrices de transformation sont situées dans des groupes contenant les objets. Voir l’affichage XML d’Inkscape ci-contre. Cela demande donc aujourd’hui du travail manuel sur les fichiers. Il doit être possible d’automatiser ça.

Il semble que SVGO ai d’autres méthodes encore plus efficaces, mais basé sur du node.js, mais il casse complétement les animations. Je n’ai pas trop confiance, en raison de beaucoup de mauvaises habitudes autour de Node. Et le plugin Inkscape, inkscape-svgo ne déroge pas à ses nouvelles habitudes, si l’on essaie de le compiler avec le Makefile inclus, pas de test d’installation existante, récupération forcée de node 11 (x86_64, perd le multi-architecture), et recompilation de toutes les dépendances. On se retrouve une fois de plus avec une extension énorme de dizaines de méga-octets (75 Mo, contre 145 Mo pour Inkscape et 5 Mo pour la bibliothèque SVGO elle même), alors que ça devrait faire que quelques Ko grand maximum.

Un bon flux de travail

* Créer l’objet sous Inkscape
* L’animer avec Glaxnimate
* Exporter en SVG
* Ouvrir de nouveau avec Inkscape pour nettoyer (sortir des groupes ce qui n’en a pas besoin,
* Faire une sortie optimisée
* Retravailler à la main.

Pour la dernière partie, cela oriente vers deux solutions pour l’automatisation :
* Patcher la sortie d’Inkscape optimisée
* Faire un optimiseur couche 2

On peut déjà pas mal gagner avec :

sed s/1.00000/1/g <input.svg | sed s/0.00000/0/g | sed 's/0000;/;/g' >output.svg

Un ou 2 chiffre de précision (à la place de 3 ou 4 aujourd’hui de Scour), devraient également suffire.

SVGcleaner est meilleur que Scour dans différents cas, mais ne supporte pas et n’a pas l’intention de supporter les animations (complexe à gérer, peut facilement tourner à la catastrophe).

Publication sur le web (SVG)

Comme le SVG animé est infiniment plus simple à intégré pour le web je préfère cette option. Le mieux est de ne pas compresser en SVGZ, mais seulement d’optimiser le fichier SVG puis de le compresser en Brotli (.br) et éventuellement en gzip pour des très vieux navigateurs.

L’intégration dans une page est très simple, elle se fait comme pour toute image :

<img src="/chemin/du/fichier_animation.svg" align="right" width="200" height="250">

C’est ce que j’ai fait sur cette page, le navigateur la prend automatiquement en charge.

Pour la compression, j’utilise les max voici un moyen simple d’avoir tous les fichiers d’un dossier compressé à côté de la version non compressée :

ls --ignore=*.gz --ignore=*.br | while read file
do
 brotli -q 11 -c <"${file}" >"${file}".br
 gzip -9 -c <"${file}" >"${file}".gz
done

* 5559 murphy_anim0.animoptiv5.svg
* 1303 murphy_anim0.animoptiv5.svg.br
* 1543 murphy_anim0.animoptiv5.svg.gz
* 4682 murphy.json
* 976 murphy.json.br
* 1149 murphy.json.gz

Pour forcer le chargement d’un des 2, si le navigateur supporte (si il supporte les 2, Brotli (.br) sera utilisé, sous Nginx (après avoir compilé le patch Brotli, les règles dans le fichier de configuration sont simples :

brotli_static on;
gzip_static on;

Sous Apache, Brotli est présent par défaut dans les versions 2.4, mais c’est un peu plus compliqué.

Pour tester, le mieux est d’utiliser la commande curl:

curl -I -H 'Accept-Encoding: gzip, deflate, br' https://host.net/fichier.svg

Si le brotli est bien activé, dans la réponse, vous verrez :

content-encoding: br

Bot Telegram

La bibliothèque python-lottie tombe bien, je voulais me mettre à Python, notamment pour micropython dans l’embarqué et pour du travail d’admin sys & réseau, où il est de plus en plus utilisé, avec l’outil de déploiement de parc Ansible, et d’autres outils système de base. Environ une année d’expérience de génération procédurale en Lua me donnait envie d’en faire d’en d’autres langages (je réserve une surprise pour bientôt). Il existe plusieurs bibliothèques de bot pour Telegram en python, dont Python Telegram Bot, qui colle assez bien à l’API officielle de Telegram et Telethon (Documentation) qui permet de communiquer directement en MTproto, le protocole de Telegram, plutôt que de passer par la couche HTTP, cela réduit d’une couche et permet un contrôle plus poussé des échanges. Voir le comparatif Bot API (HTTP vs MT Proto).


On ne trouve pas encore beaucoup d’exemples avec Python Telegram bot avec lequel j’ai commencé, mais en fouillant un peu dans la documentation, on trouve ce qu’on veut. En mixant la bibliothèque et python-lottie pour la génération procédurale, Glaxnimate pour la génération affinée à la main et le bot on a de quoi s’amuser. J’ai donc fait un premier bot en python-telegram-bot, suites aux suggestions de Glax, l’auteur de Glaxnimate qui m’a fait connaître le module python, Telethon, qu’il utilise pour son bot similaire, Glaxcomm, générant des Lottie/TGS à la volée. Je vais refaire mon bot et l’emmener vers ce que je voulais faire. Mon but étant aussi de pouvoir l’utiliser avec d’autres protocoles que Telegram.

Glax à fait un bon ensemble d’exemples de génération procédurale de base pour comprendre l’utilisation, bonne documentation sur le format Lottie, exemples de scripts de stickers

J’ai fait un bot simple en partant des exemples de la bibliothèque python-telegram-bot, mais il manquait des explications complètes (et je n’ai pas trouvé sur le net) sur la façon d’utiliser le file_id, méthode recommandée par Telegram, plutôt que d’envoyer plusieurs fois le même fichier. En fait ce file_id est récupéré au premier envoie. Donc, l’idéal est de stocker dans une base ou fichier sur disque l’id des fichiers qui ont déjà été envoyés (et reçu) au premier envoie réussi). Cet exemple crée un fichier contenant l’id à côté de l’image, c’est sans doute un peu plus bourrin que de conserver tous les ID dans une base (sql ou TOML) :

if os.path.exists(idfile):
  fid = open(idfile)
  stickid = fid.read()
  fid.close()
  msg = update.message.reply_sticker(stickid)
else
  msg = update.message.reply_sticker(open(stickfile, 'rb'))
  fid = open(idfile, 'w')
  fid.write(msg['sticker']['file_id'])
  fid.close()

Lua, TIC-80, LÖVE,etc: Introduction to particles systems and games

Version en français ici
default code in TIC-80

A good language for easily developing games, interactive content and procedural art is Lua scripting language. This is a simple functional language with some limited oriented object language features. It is, due to its simplicity and bytecode compiling at start by default, one of the lightest and fastest script language. Some integrations like LÖVE media/games engine and API (very powerful), and TIC-80, that is more limited fantasy-computer inspired first by PICO-8 (also use Lua). They allow quick prototyping, for a final product in the same language or a later port in another language. Lua is also used as plug-in systems language, in lot of games, and tools, including desktop applications (like Blender), web application (like MediaWiki behind Wikimedia) or embedded world. In this domain, popular and open-source drone board control BetaFlight or Open-TX open source radio-command for commands like Taranis one. There is a complete online documentation of Lua on the official website. It is possible to embbed C libs/function in Lua programs with libffi. It as been created first for Python, CFFI, and there is FFI support for PHP too now. It also possible to embed Lua scripts in C program. I also just discovered when I wrote this article (thanks to the author of TIC-80, that there is also PicoC, a simple C language interpreter that so allow to control more finely/low level data structures. The binary size is about the same than Lua interpreter.

An interactive demo of how trigonometric functions workSo, after few years of looking time to time this language and tools, I started to play a bit more in end of 2020, and in few month I can said I made lot of progress in real-time programming. Doing and even finishing light games. I Needed to study again basic trigonometry (follow link for a simple interactive explanation), basic vector algebra and few other mathematics fun things, that I personally consider like puzzles games.

Banner of Falacy Gorx, pseudo 3d game, using lot of tables
I also wrote a short making-off article on itch.io in march, 2021, during a game jam competition (instead of coding ^^). All of this motive me to write more articles tutorials about real-time/vector and procedural generation programming. I will try to write a series explaining methods I used. I will try to keep it as simple as possible to everyone, but some basic knowledge in general programming and mathematics could help a lot in this field as in life in general.

So I start with a short introduction about what I use the most in my development, arrays of elements and some randomness:
* Tables and their common basic management related to animation logic
* Creating a table
* Procedural generation of the content of a table
* A simple example for cleaning a table
* Variation and cleaning of table depending on tests
* Compacting code a bit
* Procedural generation of the content of a table
* Simple graphic particle system example

Tables and their common basic management related to animation logic

Some general table usage in interactive applications

Most things are managed, for scalability, in tables, main actors (like players characters), objects, particles, active agents, etc. Object themselves, also often contains tables of sub-elements, thing as example about the parts of a caterpillar body.

So, most structures have generally 4 vital logic functions that can be managed different ways:
* Cleaning of the table, used when tables elements are no more used, but it is also wise to use it to initialise new general state, like changing from main menu to game for example.
* Initialisation of table (generally including cleaning if needed) and adding 1 to n elements
* Update of the table, states of elements, including their mechanical & physical relationship, positions, their state of drawable, interacting with other objects, and finally removing or creating a new element in the table if needed. All these parameters depend on lot of various critters.
* Output of the content of the tables (display, sound, etc).

You can test these examples in Lua command line interpreter by typing lua in a shell, and then copy-paste them, or by putting them in a file.lua, and calling it by:

lua file.lua

The last examples use TIC-80, that can be used in web browser and natively on several systems, including, Linux, Mac, Windows, Android, Rpi, etc. There is a Pro version, but I never used its functionalities, it’s good to help the developer that is very nice and worked a lot of it. The examples can also easily be ported to other environments.

Creating a table

We will use here an objs[] table (a table with name objs, short of objects). In Lua, by default, table start at 1 when you fill it by objs={elt1,elt2}, but it is still possible to have a 0 indexed element by using objs[0]=elt0. The table can be (re-)initialized empty with objs={}. If it wasn’t empty, Lua garbage collector will clean it. As I coded a lot in low level languages, such as C/C++ and assembly, I’m not a big fan of garbage collectors, but that’s really convenient for quick prototyping.

Content of the table can be simply printed with a classical for loop. For the table called obs, #objs allow to know the number elements currently availablee in the table.

objs={"ball","cube","player"}
for i=1,#objs do
 print(objs[i])
end

Procedural generation of the content of a table

particules systems for clouds, mountains, characters, table of elemets for dragonIn procedural generation, the best friend of Chaos also called Nature, is random function, that generate randomly numbers. In Lua core, math.random is dedicated to this. it accept a range as parameters, limited to integer and with crescent values only. We can as example, decide to generate a number between 1 to 6 included, as the total number of elements we want. If you try several times this function, it will display random value included between 1 and 6 each time. Like if you launched a classical 6 faces cubic dice.

print(math.random(1,6))

In an interactive or animated program, we generally want to create random object, and then varying a more managed way during a period of time. So we fill a table with random values, that can be reused later. We need to first (re-)initialize objs to an empty array, so it can be filled by the for loop.

objs={}
for i=1,math.random(1,6) do
 objs[i]={lt=math.random(1,50)}
end

We so generated a 1 to 6 random number of random 1 to 50 numbered assigned to a parameter we called lt parameter instead of the table directly. This kind of element can be accessed 2 ways in Lua, objs[i].lt or objs[i]["lt"], the second one can be useful in some situations, but we will see this later in another tutorial.
so now the table is filled, we can print values several times, they will be kept the same:

for i=1,#objs do
 print("objs["..i.."]="..objs[i].lt)
end

We use the Lua string concatenation here (symbol ..) to have more information about which element we see printed.

A simple example for cleaning a table

It is important for cleaning a table or removing elements from it to do the loop from last to first element index, as when an element is deleted, all following items in the table will have decremented index. We could then stride over some elements and remove elements we don’t want to.

for i=#objs,1,-1 do
 table.remove(objs,i)
end

In the table.remove() standard Lua function, arguments are the name of the table and the index of the element to remove.

Variation and cleaning of table depending on tests

It can be a good habit to use a short local pointer variable to the element parsed to have a shorter code inside the loop, as it will il most case be accessed a lot. We remove here an element when lt (short for lifetime) is gone to 0, else we decrement it.

for i=#objs,1,-1 do
 local o=objs[i]
 if o.lt<=0 then
  table.remove(objs,i)
 else
  o.lt=o.lt-1
 end
end

The o variable can’t be passed to table.remove(), as it is used as a pointer to an element of the table, not as the name of the table, and the second argument is an index of the table, not either a pointer to the element as o is.

We have now the general base of a particles system.

All particle systems work with generation, often using at least a bit of randomness, and lifetime of each particles as a base, other criteria can changes depending on kind of particles.

Compacting code a bit

I generally add the local variable on the for...do line to have a more readable/compact code and changes behind if...then or else keywords to have a more compact and still readable code:

for i=#objs,1,-1 do local o=objs[i]
 if o.lt<=0 then table.remove(objs,i)
 else o.lt=o.lt-1 end
end

And Lua allow to assign a function pointer to a variable, so I generally use the following trick to have a more compact and clear code :

m=math rnd=m.random

So m is assigned to math and then rnd to m(ath).random.

Warning: on constraint is to not use m variable in the same variable scope than m.* assignation. It’s so better to made this assignation at the beginning, so m can be used freely after.

Simple graphic particle system example

In this example we will simply add random x (horizontal axis) and y (vertical axis) position around a starting central point 120,70 we choose a variation of -15 to +15 pixels in each direction:

objs[i]={lt=rnd(10,30),x=120+rnd(-15,15),y=70+rnd(-15,15)}

Basic particle systemand made them randomly move of a length from 0 to 1 pixel and in direction left/right and up/down. So, we have:
For x-axis
* -1 = left
* 0 don’t move
* +1 = right
For y-axis
* -1 = up
* 0 don’t move
* +1 down

o.x = o.x+rnd(-1,1) o.y = o.y+rnd(-1,1)

In Lua functions are defined by function function_name(arg1,arg2,...) and finish by end.
In the case of TIC-80 for example, that has the advantage to have all embedded in one executable, the function TIC() is a function called periodically, at each new frame, allowing to made time related content (so animation) easily. The function cls(color) is used to clean the screen with color at index “color” as TIC-80 use a 16 colors palettes.

The place of the particles are drawn by circ() (circle) function.

The definition of the function is:

circ(X center, Y center, radius, colour)

We place the center of the circle at the coordinates of the object, it has here a radius of 1 and use the colour index 0, that is black by default with TIC-80.

circ((o.x, o.y, 1, 0)

The current lifetime that the particle stille have is printed just at the right of the particle here for the demonstration. In TIC-80 print(), is used to place screen on graphic screen, and trace() on console.

The part of the print arguments we use are defined the following way:

print(text, X start, Y start, color, fixed font, scale, small font)

So we place as text, the current lifetime of the particle, at the center position of the object (o.x, o.y) with an horizontal variation of +2 pixel (so right) and horizontal of -2 pixel (so top). We choose a fixed size (true), scale of 1, and use small font (true):

print(o.lt, o.x+2, o.y-2, 2, true, 1, true)

In lua we can define several variables in one line, by crossing names of variables and assignations. For example, here, x=5 y=4 can be wrote x,y=5,4.

m=math rnd=m.random
t=0
objs={}
for i=1,rnd(5,8) do
 objs[i]={lt=rnd(10,30),x=120+rnd(-15,15),y=70+rnd(-15,15)}
end
function TIC()
 cls(12)
 for i=#objs,1,-1 do local o=objs[i]
  if o.lt<=0 then table.remove(objs,i)
  else o.lt,o.x,o.y=o.lt-1,o.x+rnd(-1,1),o.y+rnd(-1,1)
   circ(o.x,o.y,1,0)
   print(o.lt,o.x+2,o.y-2,2,true,1,true)
  end
 end
 t=t+1
end

In the Exemple to download we put the generation in a function generate() and call it when table content is null (#objs==0) as a simple loop generation.

function generate()
 for i=1,rnd(5,8) do
  objs[i]={lt=rnd(10,30),x=120+rnd(-15,15),y=70+rnd(-15,15)}
 end
end

function TIC()
 ...
 if #objs==0 then generate()end
 t=t+1
end

If you want to see the particles without their numbers, you can simply comment the print line. To comment code in Lua, simply add two dash at the beginning of the comment.

   -- print(o.lt,o.x+2,o.y-2,2,true,1,true)

Android without Google and GNU/Linux with APK

Android will move from Linux kernel and open specs APK format to a more closed package format and to their closed source Fuschia kernel going away from Free Open Source software (FOSS) that make Google, Android, ChromeOS grow, but also that are used on most of the internet servers, on most of supercomputer in the world, on most spacecrafts, on most internet boxes, TVboxes, etc…

GNU/Linux on your phone now

At the same time GNU/Linux distributions for mobile grow a lot the 2 recent years, thanks to availability of PinePhone for developers for about 2 years. There are now at least 15 Linux distributions for phone and applications maturated a lot, including since few month MegaPixels hardware accelerted camera application, that use GL and other processor to accelerate rendering/computing of what camera receive.

PostMarketOS is a GNU/Linux distribution available for a bunch of available market phones, but not everything is managed an all devices.

GNU/Linux with Android APK compatibility layer

Just like there is Wine to run Windows application on GNU/Linux, there are also several systems allowing to run Android APK on your phone if needed. Jusr as Wine, beside the possibility of running Android in a qemu VM, you can also run these application inside your the standard GNU system.

AnBox allow to run APK/Android application inside a snap packaged container.

Google goes away from FOSS, afraid by the lost of market share, and they probably loose a lot from this, as Microsoft does by the past. Chinese company, thanks to Trump restrictions, embrace more FOSS.

Huawei the maker of the probably highest quality phones, that was going in first seller position just ahead Samsung, when Trump blocked it, already very active in Linux kernel and community for years, made HarmonyOS for their devices, a system, that continue tu uses Linux kernel for their power devices, and LiteOS for their low end/old devices. LiteOS is a FOSS embedded devices oriented light kernel. They keep the EMUI User Interface they used previously on Android, moving step by step away from it, and at the same time keep a compatibility layer with current APK open-source format.

Another Chinese company, JingOS, made high quality tablets, based on GNU/Linux distribution, that derivates from Ubuntu, with KDE tools and Android APK application compatibility using Anbox. And they are active with KDE and open source communities. Few demos here and here.

Removing closed source spywares from your current Android system as a transition phase

Beside long time F-droid Free software apk app repository (I use it for year as main source of APK).

You can remove Android pre-installed (memory+cpu+bandwidth thieves) applications, including NSA compliant one like Google, Amazon, FaceBook, Microsoft ones using from a computer with adb. This allowed me to have less bloated phone and to gain few more days of on-battery usage.

List packages:

adb shell pm list packages

See which one are related to facebook:

adb shell pm list packages facebook

For easier copy paste and have only the package name without package: prefix

adb shell pm list packages | cut -d: -f 2

The corresponding application, can be checked by using in a web browser, this does not work for some system applications, for example, for com.google.android.videos:

https://play.google.com/store/apps/details?id=com.google.android.videos

Remove never at least first seen, never used application:

# FaceBook spyware

adb shell pm uninstall --user 0 com.facebook.katana
adb shell pm uninstall --user 0 com.facebook.system
adb shell pm uninstall --user 0 com.facebook.appmanager

# Microsoft stuff
adb shell pm uninstall --user 0 com.microsoft.office.excel
adb shell pm uninstall --user 0 com.microsoft.office.onenote
adb shell pm uninstall --user 0 com.microsoft.office.powerpoint
adb shell pm uninstall --user 0 com.microsoft.office.word
adb shell pm uninstall --user 0 com.microsoft.skydrive # data thief

adb shell pm uninstall --user 0 com.google.android.apps.docs # Google Drive (data thief)
adb shell pm uninstall --user 0 com.google.android.apps.photos # Google photos (data thief)
adb shell pm uninstall --user 0 com.google.android.googlequicksearchbox # Google Search box (behavior thief)
adb shell pm uninstall --user 0 com.google.android.gm # GMail
adb shell pm uninstall --user 0 com.google.android.talk # Hangouts
adb shell pm uninstall --user 0 com.google.android.music # Google Play Music
adb shell pm uninstall --user 0 com.google.android.videos # Google Play Film & Series
adb shell pm uninstall --user 0 com.google.android.youtube # Youtube application

To be checked

adb shell pm uninstall --user 0 com.google.android.gms # Services Google Play

adb shell Dumpsys a dumper and debugger

adb shell dumpsys allow to dump application before removing them and reinstalling it after that.

I still didn’t tried that, will update it after my tests

adb shell dumpsys package package.name # dump package state
adb pm enable —user 0 nom_de_paquet

Linux syscall and RISC-V assembly

Sample of RISC-V assembly code

Syscall in Linux kernel, is an interface to access to kernel basic functions. They are described in section 2 of man pages. The introduction is in man 2 syscall (indirect system call), and the list of functions are described in man 2 syscalls. ”’Update:”’ System Calls in lectures of official Linux kernel documentation including “Linux system calls implementation”, “VDSO and virtual syscalls” and “Accessing user space from system calls”

This article follow previous one about RISC-V overall progress and available tools to play with, I will try to make a short article here about Linux syscall usage and the RISC-V assembly case.

Table of Content

* Description section of the man page
* Getting the list of function and how to access them
* Passing parameters
* Function number and registers of return values
* Return values and error code
* Compiling and executing on virtual environment
* Update: Bronzebeard assembler and its baremetal environment for real hardware

Description section of the man page

* syscall() is a small library function that invokes the system call whose assembly language interface has the specified number with the specified arguments. Employing
* syscall() is useful, for example, when invoking a system call that has no wrapper function in the C library.
* syscall() saves CPU registers before making the system call, restores the registers upon return from the system call, and stores any error returned by the system call in errno(3).
* Symbolic constants for system call numbers can be found in the header file .

You can find here function, like access to files open/close/read/write/flush, access to sockets, ioctl, uid, gid, pid, messages, ptrace, restart system, etc…

Getting the list of function and how to access them

As far I know, now only a part of syscall functions are accessible easily in assembly, they are defined in /usr/include/unistd.h, and function numbers assigned in ABI are defined in /usr/include/asm-generic/unistd.h.

The more practical match I found is using /usr/include/asm-generic/unistd.h to see which function are available and there respective manpage for the function header definition. For example:
* asm-generic: #define __NR_read 63
* man 2 read: ssize_t read(int fd, void *buf, size_t count);

The ABI with RISC_V as defined in man 2 syscall in section Architecture calling conventions use the registers following 2 tables rules.

Passing parameters

The second table in this section of the man page shows the registers used to pass the system call arguments.

Arch/ABI      arg1  arg2  arg3  arg4  arg5  arg6  arg7  Notes
──────────────────────────────────────────────────────────────
riscv         a0    a1    a2    a3    a4    a5    -

Here are the arguments in the order of the function definition, for example, in read (63) function:

ssize_t      read  ( int fd, void *buf, size_t count );
a0(result) = a7(63)( a0(fd),  a1(*buf),    a2(count) )

For remember, 3 standard I/O file descriptors are STDIN=0, STDOUT=1, STDERR=2, the other are used when opened a file with open and closed by close.

So we set the arguments as this. x0 is the always 0 register:

        addi  a0, x0, 0       # Set STDIN as sources
        la    a1, buffer_addr # load address of helloworld
        addi  a2, x0, 3       # reaad 3 bytes

Function number and registers of return values

And the first one give the register in which put the function number, that will receive return value and errno (error value)

Arch/ABI    Instruction           System  Ret  Ret  Error    Notes
                                  call #  val  val2
───────────────────────────────────────────────────────────────────
riscv       ecall                 a7      a0   a1   -

So for the read function, we need to put 63 (as found in /usr/include/asm-generic/unistd.h in register a7, and registers and following registers will receive the return values, a0 will receive system call result, and a1 an error message (the errno value).

ssize_t read(int fd, void *buf, size_t count);
a0    =  63 (    a0,       a1,         a2)

So can set them as this:

        addi  a7, x0, 63     # set called function as read()
        ecall                # call the function

Return values and error code

After the man page of read(2):
* On success, the number of bytes read is returned
* On error, -1 is returned, and errno is set to indicate the error.

So we can test first the return value of a0 and if a0 < 0 then we jump to part for display the error message, else we can simply display a OK message. So for the branching part, RISC-V in its super reduced set only have < (lt) and <= (le) comparators, you just need to swap registers to compute > (gt) and >= (ge), but this avoid lots more of transistors.

    addi a3,x0,0           # x3=0
    blt  a1,a3, error_seq  # if x1<0 branch to error_seq

We so use here the syscall write function (64) defined as:
* asm-generic: #define __NR_write 64
* man 2 write: ssize_t write(int fd, const void *buf, size_t count);
* So, registers: a0=1 (STDOUT), a1=*buf, a2=count, a7=64 (function number)

And we will finish with exit() syscall, defined as:
* asm-generic: #define __NR_exit 93
* man 2 exit: noreturn void _exit(int status);
* So, registers: a0=return code, a7=93 (function number)

    la    a1, ok           # load address (pseudo code) of ok string
    addi  a2, x0, 3        # set length of text to 3 (O + K + \n)
    addi  a7, x0, 64       # set ecall to write function
    ecall                  # Call the function

    addi  a0, x0, 0        # set return code to 0 (OK) for exit (93) function
    j     end              # unconditional jump to end before quit

error_seq:
    la    a1, error        # load address (pseudo code) of error string
    addi  a2, a2, 0x30     # add 0x30 (0 ASCII code) to the error code
    sb    a2, 7(a1)        # put the (byte) value at position 7 of Error string (before \n)
    addi  a2, x0, 10       # set now length of our string
    addi  a7, x0, 64       # set ecall to write function
    ecall                  # Call the function

    addi  a0, x0, -1       # set return code to -1 (error) for exit (93) function
end:
    addi    a7, x0, 93     # set ecall to exit (93) funciton
    ecall                  # Call linux to terminate the program

.data:
ok:     .ascii "OK\n"
error:  .ascii "Error:  \n"

RISC-V Longan nano

Compiling and executing on virtual environment

If you don't have a RISC-V hardware (can be found as low as 3€ now), you need to have a cross compiler and qemu for emulating instructions, or a whole system installed.

Packages needed for compiling on ArchLinux x86 or ARM for example.

sudo pacman -S riscv64-linux-gnu-gcc  riscv64-linux-gnu-glibc riscv64-elf-binutils riscv64-elf-binutils riscv64-elf-gcc riscv64-elf-gdb

Newlib is a lightweight RV32 (RISC-V 32bits) lightweight library for bare metal that can be used instead of a whole GNU system on embedded devices with low memory capacity (as Longan nano, less than 8€ with screen, see picture below, or 3€ Sipeed RV): riscv32-elf-newlib.

I made a simple shell script to don't have to remember the commands to assemble the code from an x86 platform (work also on ARM or RISC-V one) that take the .s as argument:

name=$1
riscv64-linux-gnu-as -march=rv64imac -o ${name}.o ${name}.s
riscv64-linux-gnu-ld -o ${name} ${name}.o

You can add a strip but better to avoid it if you need to debug it:

riscv64-elf-strip --strip-all ${name}

And it can be executed on non RISC-V platforms by using qemu-riscv64, if it doesn't depend on libraries or if you have them installed, it allow you to test it without having a full RISC-V system installed, qemu is so fantastic. On ArchLinux it is available in package qemu-arch-extra:

qemu-riscv64 ${name}

And can be disassembled (will probably use different instruction than your assembly code, due to RISC-V assembly pseudo-instructions:

riscv64-linux-gnu-objdump -d ${name}

Bronzebeard assembler and its baremetal environment for real hardware

RISC-V Longan nano

Update: Bronzebeard is an assembler with light baremetal environment builder for RISC-V, GD32VF103 as Longan Nano (about 8€ with a screen, as pictured on this article pictures) and Wio (similar board with an added ESP8266 SoC). I made an AUR package of Bronzebeard, and someone made a Mandelbrot set demo in 918 bytes pure RISC-V assembly. You can find some other example in the source of Bronzebeard. gd32vf103inator is a set of tools for GD32V, to manage from a simple random text editor.