Ciro Santilli OurBigBook.com  Sponsor 中国独裁统治 China Dictatorship 新疆改造中心、六四事件、法轮功、郝海东、709大抓捕、2015巴拿马文件 邓家贵、低端人口、西藏骚乱
Big goals:
Just art:
  • useless mathy stuff
  • incredibly nifty little tools that are just so satisfying to use it is mind blowing:
  • media related stuff

Build automation

words: 25 articles: 3

CMake

words: 25
Examples under cmake:

make (Software)

articles: 1

Compiler

words: 283 articles: 22

Compiler toolchain

words: 52 articles: 1
Compiler + other closely related crap like linker.
Some linker related ansewrs by Ciro Santilli:

Binutils

words: 4 articles: 3

Binutils utility

words: 4 articles: 2
Not possible it seems:

Automatic programming

words: 42 articles: 1
"automatic programming has always been a euphemism for programming in a higher-level language than was then available to the programmer" sums it up.
The ultimate high level is of course to program with: "computer, make money", which is the goal of artificial general intelligence.

Lowering and raising

words: 24 articles: 2
Lowering means translating to a lower level representation.
Raising means translating to a higher level representation.
Decompilation is basically a synonym, or subset, of raising.

List of compilers

words: 146 articles: 8

GNU Compiler Collection (gcc)

words: 11 articles: 2
gcc CLI option
words: 11 articles: 1
gcc -save-temps
words: 11
Saves preprocessor output and generated assembly to separate files.

LLVM

words: 135 articles: 4
Very hot stuff! It's like ISA-portable assembly, but with types! In particular it also it deals with calling conventions for us (since it is ISA-portable). TODO: isn't that exactly what C does? :-) LLVM IR vs C
Documentation: llvm.org/docs/LangRef.html
Example: llvm/hello.ll adapted from: llvm.org/docs/LangRef.html#module-structure but without double newline.
To execute it as mentioned at github.com/dfellis/llvm-hello-world we can either use their crazy assembly interpreter, tested on Ubuntu 22.10:
sudo apt install llvm-runtime
lli hello.ll
This seems to use puts from the C standard library.
Or we can Lower it to assembly of the local machine:
sudo apt install llvm
llc hello.ll
which produces:
hello.s
and then we can assemble link and run with gcc:
gcc -o hello.out hello.s -no-pie
./hello.out
or with clang:
clang -o hello.out hello.s -no-pie
./hello.out
hello.s uses the GNU GAS format, which clang is highly compatible with, so both should work in general.
clang
words: 7
LLVM front-end for C and related language like C++ etc.
Reproducible builds allow anyone to verify that a binary large object contains what it claims to contain!
Bibliography:

Graphics software

words: 140 articles: 12
en.wikipedia.org/wiki/List_of_information_graphics_software
Survey by Ciro Santilli: math.stackexchange.com/questions/1985/software-for-drawing-geometry-diagrams/3938216#3938216
Many plotting software can be used to create mathematics illustrations. They just tend to have more data-oriented rather than explanatory-oriented output.
Some notable ones:

Graphics library

words: 116 articles: 10

OpenGL

words: 51 articles: 4
Ciro Santilli has some good related articles listed under: the best articles by Ciro Santillis.
Freetype GL
words: 9
github.com/rougier/freetype-gl
Good library to render text in OpenGL, see also: stackoverflow.com/questions/8847899/opengl-how-to-draw-text-using-only-opengl-methods/36065835#36065835
Khronos Group
words: 24 articles: 1
The fact that they kept the standard open source makes them huge heroes, see also: closed standard.
Shame that many (most?) of their proposals just die out.
github.com/opengl-tutorials/ogl/
Good modern OpenGL tutorial in retained mode with shaders, see also: stackoverflow.com/questions/6733934/what-does-immediate-mode-mean-in-opengl/36166310#36166310

JavaScript graphics library

words: 65 articles: 3
github.com/paperjs/paper.js
github.com/pixijs/pixi.js
Two.js
words: 65
github.com/jonobr1/two.js
Examples at: two-js/.
JavaScript library, works both on browser and headless with Node.js to SVG.
Feels good. Maybe not ultra featured, and could have more simple examples in docs, but still good.
Vs Paper.js github.com/jonobr1/two.js/issues/319
One of the main features of Two.js appears to be the fact that it can natively render to either SVG and canvas, rather than creating SVG through DOM hacks as done by other projects.
One specific software project, typically with a single executable file format entry point.

Computer security

words: 405 articles: 30
As mentioned at Section "Computer security researcher", Ciro Santilli really tends to like people from this area.
Also, the type of programming Ciro used to do, systems programming, is particularly useful to security researchers, e.g. Linux Kernel Module Cheat.
The reason he does not go into this is that Ciro would rather fight against the more eternal laws of physics rather than with some typo some dude at Apple did last week and which will be patched in a month.

DEF CON (1993-)

articles: 1

Computer security researcher

words: 114 articles: 3
Ciro Santilli found out that he likes computer security researchers and vice versa.
It's a bit the same reason why he likes physicists: you can't bullshit with security.
You can't just talk nice and hope for people to belive you.
You can't not try to break things and just keep everyone happy in their false illusion of safety.
You can't do a half job.
If you do any of that, you will get your ass handed to you in a little gift bag.
All of this is closely linked to Ciro Santilli's self perceived creative personality and being naughty and creative are correlated.

Dan Kaminsky (1979-2021)

words: 23 articles: 1
A superstar security researcher with some major exploits from in the 2000's.
twitter.com/dakami/status/1344853681749934080
Oh yeah, that felt good. A few months before he died.

Multi-factor authentication (2FA)

words: 66 articles: 3

2FA app

words: 66 articles: 2
Ermm, as of February 2021, I was able to update my 2FA app token with the password alone, it did not ask for the old 2FA.
So what's the fucking point of 2FA then? An attacker with my password would be able to login by doing that!
Is it that Google trusts that particular action because I used the same phone/known IP or something like that?

OAuth

words: 68
The fatal flaw of OAuth is that websites have to enable specific providers, they can't just automatically select the correct OAuth for a given email domain. This means that the vast majority of websites will only provide the most widely popular providers such as Google, and the like, which means people won't have decent privacy.
So you are just better off with password logins and a decent password manager.

Password

words: 34 articles: 1
A cross browser, cross platform, and server-encrypted password manager is a must after Snowden!!! E.g. Proton Pass. And governments should obviously provide one to its citizens, or else be spied upon by the USA obviously: Governments should provide basic Internet infrastructure.

Security through obscurity

words: 56 articles: 1
stackoverflow.com/questions/533965/why-is-security-through-obscurity-a-bad-idea
Do as I say, not as I do: Ciro Santilli's Stack Overflow suspension for vote fraud script 2019, meta.stackoverflow.com/questions/381577/is-it-ok-to-have-links-on-how-to-create-sock-puppets-and-gain-rep-fraudulently-i/381635#381635.
Video 1.
LockPickingLawyer SAINTCON keynote (2021)
. Source. SAINTCON is "Utah's Premiere Security Conference".
Basically the opposite of security through obscurity, though slightly more focused on cryptography.

Data breach

articles: 3

Computer user-interface

words: 336 articles: 27

GraphQL

words: 54
This is really good.
It allows the client to prepare a single request that gets all the data it wants to fill up a given webpage, rather than doing several separate requests.
So it only gets exactly what it needs, and in a single request.
Very sweet. This is the future of the web.

Command-line interface (CLI)

words: 220 articles: 13

Linux CLI HOWTO

words: 2 articles: 1

Command line utility (CLI tool)

words: 208 articles: 7
List of command line utilities
words: 208 articles: 6
GNU parallel
words: 197
The author Ole Tange answers every question about it on Stack Exchange. What a legend!
This program makes you respect GNU make a bit more. Good old make with -j can not only parallelize, but also take in account a dependency graph.
Some examples under:
man parallel_exampes
To get the input argument explicitly job number use the magic string {}, e.g.:
printf 'a\nb\nc\n' | parallel echo '{}'
sample output:
a
b
c
To get the job number use {#} as in:
printf 'a\nb\nc\n' | parallel echo '{} {#}'
sample output:
a 1
b 2
c 3
c 3
{%} contains which thread the job running in, e.g. if we limit it to 2 threads with -j2:
printf 'a\nb\nc\nd\n' | parallel -j2 echo '{} {#} {%}'
sample output:
a 1 1
b 2 1
c 3 2
d 4 1
The percent must be a reference to "split the inputs module the number of workers", and modulo uses the % symbol in many programming languages such as C.
To pass multiple CLI argments per command you can use -X e.g.:
printf 'a\nb\nc\nd\n' | parallel -j2 -X echo '{} {#} {%}'
sample output:
a b 1 1
c d 2 2
ncdu
words: 10
Way too few people know about this. Spread the word.
stackoverflow.com/questions/1019116/using-ls-to-list-directories-and-their-total-sizes/55519414#55519414
https://web.archive.org/web/20221208132133if_/https://i.stack.imgur.com/lHXP3.png
sudo
words: 1
Availability: unix.stackexchange.com/questions/48522/how-universal-is-sudo
xkcd.com/149/

Text-based user interface (TUI)

words: 10 articles: 2
The perfect Middle Way between command-line interfaces and GUIs. A thing of great beauty.
ncurses
articles: 1

Graphical user interface (GUI)

words: 62 articles: 9

Display manager

words: 59 articles: 1
Check which you you have:
systemctl status display-manager.service
Tested on Ubuntu 23.10 I see:
● gdm.service - GNOME Display Manager
     Loaded: loaded (/lib/systemd/system/gdm.service; static)
     Active: active (running) since Sun 2023-12-24 10:34:50 GMT; 23min ago
    Process: 1827 ExecStartPre=/usr/share/gdm/generate-config (code=exited, status=0/SUCCESS)
   Main PID: 1850 (gdm3)
      Tasks: 4 (limit: 71817)
     Memory: 6.8M
        CPU: 119ms
     CGroup: /system.slice/gdm.service
             └─1850 /usr/sbin/gdm3
which means I have GNOME Display Manager.
Bibliography:

Desktop environment

words: 3 articles: 3
tmux for newbs: Section "Terminal multiplexers are CLI desktop environments".

Data compression

articles: 3

Database

words: 6k articles: 113

ACID (database)

words: 70 articles: 2
This means that e.g. if you do an UPDATE query on multiple rows, and power goes out half way, either all update, or none update.
This is different from isolation, which considers instead what can or cannot happen when multiple queries are running in parallel.
Determines what can or cannot happen when multiple queries are running in parallel.
See Section "SQL transaction isolation level" for the most common context under which this is discussed: SQL.
A software that implements some database system, e.g. PostgreSQL or MySQL are two (widely extended) SQL implementations.

NoSQL

words: 104 articles: 5

LevelDB

words: 30 articles: 2
One "LevelDB" database contains multiple file in a directory. Off the bat inferior to SQLite which stores everything in a single file!
Dump LevelDB
words: 9 articles: 1
LevelDBDumper
words: 9
github.com/mdawsonuk/LevelDBDumper
github.com/mdawsonuk/LevelDBDumper/tree/e750a27ff58443ecc410b5c16abbdc539d617387#installation worked on Ubuntu 23.10 Annoying installation, but worked: github.com/mdawsonuk/LevelDBDumper/issues/13
Initial issues off-the-bat:

MongoDB

words: 74 articles: 1
List databases:
echo 'show dbs' | mongo
Delete database:
use mydb
db.dropDatabase()
or:
echo 'db.dropDatabase()' | mongo mydb
View collections within a database:
echo 'db.getCollectionNames()' | mongo mydb
Show all data from one of the collections: stackoverflow.com/questions/24985684/mongodb-show-all-contents-from-all-collections
echo 'db.collectionName.find()' | mongo mydb
Tested as of Ubuntu 20.04, there is no Mongo package available by default due to their change to Server Side Public License, which Debian opposed. Therefore, you have to add their custom PPA as mentioned at: docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/

Object-relational mapping (ORM)

words: 137 articles: 1
Per language:
How to decide if an ORM is decent? Just try to replicate every SQL query from nodejs/sequelize/raw/many_to_many.js on PostgreSQL and SQLite.
There is only a very finite number of possible reasonable queries on a two table many to many relationship with a join table. A decent ORM has to be able to do them all.
If it can do all those queries, then the ORM can actually do a good subset of SQL and is decent. If not, it can't, and this will make you suffer. E.g. Sequelize v5 is such an ORM that makes you suffer.
The next thing to check are transactions.
Basically, all of those come up if you try to implement a blog hello world world such as gothinkster/realworld correctly, i.e. without unnecessary inefficiencies due to your ORM on top of underlying SQL, and dealing with concurrency.

Relational database

words: 5k articles: 93
SQL
words: 5k articles: 91
This section is present in another page, follow this link to view it.

Database feature

articles: 6

Table relationship

articles: 3

File manager

words: 53 articles: 2
Ciro Santilli used to use file managers in the past.
But he finally converted to a shell cd aliases that auto-ls: github.com/cirosantilli/dotfiles/blob/a51bcc324f0cff0eddd4c3bb8654ec223a0adb7b/home/.bashrc#L1058

Krusader

words: 19
The most powerful GUI file manager ever?? Infinite configurability??
Ciro Santilli wasted some time on it before he gave up on file managers altogether.
Ciro Santilli considered it before he stopped using file managers altogether, it is not bad.

File sharing

articles: 1

Game engine

words: 47 articles: 5
A library to make games.

Cocos2d

words: 19
Ciro Santilli considered this as the basis for Ciro's 2D reinforcement learning games, but ultimately decided it was a bit too messy. Nice overall though.

RPG Maker

words: 15
The one true game engine!
Video 2.
Reviewing a Bunch of RPG Maker Games by Majuular (2022)
. Source.

Urho3D

words: 8
Their project lead as of 2018 was pro-CCP: github.com/cirosantilli/china-dictatorship/blob/aa1176c57fc2929465294e520b43b50d44e202ba/communities-that-censor-politics.md

Geographic information system (GIS)

words: 390 articles: 10
Originally by Keyhole Inc., which the nbecame Google Maps, but the format seems standardized and has non-Google support, so should be OK.

Google Maps

words: 84 articles: 3
Owned/developed by Google as of 2020.
Early on jumpstarted from several acquisitions, notably Keyhole Inc. and Where 2 Technologies.
Street View's go into the past mode is the dream of every archaeologist. Ciro can only dream of a magic street view that allows going back to earlier centuries and beyond... isn't it amazing to think that people in the future will have that ability to time travel back to around the year 2006? Ciro wonders how long Google will be able to keep storing data like that.
Thanks, CIA.

OpenStreetMap (OSM)

words: 268 articles: 1
It is rare to find a project with such a ridiculously high importance over funding ratio.
E.g., as of 2020, their help login help.openstreetmap.org/ shows MyOpenID as an option, which was discontinued in 2014, and not Google OAuth.
They do still seem to have a bit more activity than gis.stackexchange.com/questions/tagged/openstreetmap on Stack Exchange.
Complaints:
All of this is a shame, because they do have some incredible data that you cannot find easily on other maps because people just edited it up.
OsmAnd
words: 32
github.com/osmandapp/Osmand
Kind of works! Notably, has the amazing cycling database offline for you, if you fall within the 6 area downloads. It is worth supporting these people beyond the 6 free downloads however.

Ordnance Survey

words: 18
Has some of the best map data available for the United Kingdom, but their data appears to be proprietary?
IDEs are absolutely essential for developing complex software.
The funny thing is that you don't notice this until someone shows it to you. But once you see it, there is not turning back, just like Steve Jobs customers don't know what they want quote.
Unfortunately, after the Fall of Eclipse (archive), the IDE landscape in 2019 is horrible and split between:
  • highly buggy but still feature rich Eclipse
  • many may many other feature lacking options using possibly more trendy and forward lasting implementations like Electron
  • awesome cross-platform proprietary JetBrains IDEs
  • the God-like Windows-only proprietary language-lacking Visual Studio
Programmers of the world: unite! Focus on one IDE, and make it work for all languages and all build systems. Give it all the features that Eclipse has, but none of the bugginess. Work with top project to make sure the IDE works for all top projects.
Projects of the world: support one IDE, with in-tree configuration. Complex integration is often required between the IDE and the build system, and successful projects must to that once for all developers. Either do this, or watch you complex project wither away.
Build tool maintainers: make it possible for IDEs to support your tool! E.g., implement JSON Compilation Database output so that IDEs can read the exact compiler commands from that, in order to automatically determine how files should be parsed! Or better, just use libllvm in your IDE itself as the main parser.
Ciro is evaluating some IDEs at: github.com/cirosantilli/ide-test-projects

Text editor

words: 35 articles: 5

JavaScript text editor

words: 35 articles: 1
Monaco (editor)
words: 35
Either extracted from, or designed for, vscode by Microsoft:
However also at the same time very limited integration with vscode, that makes using it for VScode compatibility almost useless, e.g.:

Vim

words: 121 articles: 5
Before we get a decent open source integrated development environment, what else can you do?
But also perfect for small one-off files when you don't have the patience to setup said IDE.
vim's defaults are atrocious for the 21st century! Vundle is reasonable as an ad-hoc package manager, but it can't set fixed versions of packages:

vader.vim

words: 67 articles: 3
github.com/junegunn/vader.vim
Vimscript unit testing!!!
github.com/plasticboy/vim-markdown
Ciro Santilli contributed a bit to this, and was even given push rights, see also: see also: Ciro Santilli's minor projects.
github.com/honza/vim-snippets
Vimium
words: 49
vimium.github.io/
Since you can't escape shitty browser GUIs and live in the command line, the next best thing you can do is to bring Vim bindings to your browser :-)
There is one major annoyance: you can't use ESC to leave the address bar focus, but using Tab as a workaround works:

Eclipse (IDE)

words: 335
Once upon a time (early 2010's), Eclipse dominated the IDE landscape and all was good. NetBeans was around too. And Java was still unmarred by Google LLC v. Oracle America, Inc..
But then something happened.
For some reason, Eclipse started to decay.
And the project that had once been a vibrant community of awesomeness, started to become... a zombie of its former self.
Buggyness started increasing. And not even hard to fix bugs. One liners that affect every user immediately after startup.
Sometimes, to Eclipse's defense they weren't "bugs". Just features that it became evident with time every programmer expected from a modern IDE.
But somehow the Eclipse community had a deep problem. A cancer. It had completely lost touch with user experience.
Perhaps is was due to the increasing interest of the several corporations that had adopted Eclipse as the base IDE for the proprietary solutions?
Perhaps.
Many users stuck to the IDE.
Some heroic efforts were made as plugins that drastically improved certain defects. The Darkest Dark plugin comes to mind.
But all those efforts required configuration. A setup time that most users simply don't have. The core devteam had become dumb and dead, unable to incorporate such changes.
This greatly opened up the space for other competing IDEs to come along. The "semi feature complete but at least easy to use and not so buggy" Visual Studio Code and the proprietary JetBrains IDEs being some of the most notable ones.
Using Eclipse as of the early 2020's is such a mixed experience. If you spend enough time to configure out the key buggyness, there are moments where you can feel "OMG, this feature is amazing".
But the effort is just too great, and soon another bug or obvious missing feature hits you and brings you back to reality.
Every young person uses VS Code now. Eclipse is dead, and there is no way back, usage will just continue dropping.
RIP, Eclipse. It wasn't meant to be.
Bibliography:
Figure 1.
Eclipse. usage from 2012 to 2016 according to a JRebel survey
. Source.

vscode bug

words: 3
Persistent undo history:

vscode Vim

words: 5
It is especially bad on large projects, unless you carefully whitelist only the small source directories:
stackoverflow.com/questions/30118107/vscode-intellisense-not-working

Messaging software

words: 1k articles: 41
This section is present in another page, follow this link to view it.

Multimedia software

words: 603 articles: 14

FFmpeg

words: 524 articles: 7
FFmpeg is the assembler of audio and video.
As a result, Ciro Santilli who likes "lower level stuff", has had many many hours if image manipulation fun with this software, see e.g.:
As older Ciro grows, the more he notices that FFmpeg can do basically any lower level audio video task. It is just an amazing piece of software, the immediate go-to for any low level operation.
FFmpeg was created by Fabrice Bellard, which Ciro deeply respects.
Resize a video: superuser.com/questions/624563/how-to-resize-a-video-to-make-it-smaller-with-ffmpeg:
ffmpeg -i input.avi -filter:v scale=720:-1 -c:a copy output.mkv
Unlike every other convention under the sun, the height in scale is the first number.
Filter graphs are a thing of great beauty. What an amazingly obscure domain-specific language, but which can produce striking results with very little!!!
A quick example from stackoverflow.com/questions/59551013/how-to-generate-stereo-sine-wave-using-ffmpeg-with-different-frequencies-for-eac/77730492#77730492 illustrates some of the fundamentals:
ffplay -autoexit -nodisp -f lavfi -i '
sine=frequency=500[a];
sine=frequency=1000[b];
[a][b]amerge, atrim=end=2
'
which creates a graph:
                              +--------+
[sine=frequency=500]--->[a]-->|        |
                              | amerge |-->[atrim]-->[output]
[sine=frequency=1000]-->[b]-->|        |
                              +--------+
and plays 500 Hz on the left channel and 1000 Hz on the right channel for 2 seconds.
So we see the following syntax patterns:
  • sine, amerge and atrim are filters
  • sine=frequency=500: the first = says "araguments follow"
    • frequency=500 sets the frequency argument of the sine filter
    • for multiple arguments the syntax is to separate arguments with colons e.g. sine=frequency=500:duration=2
  • ;: separates statements
  • [a], [b]: sets the name of an edge
  • ,: creates unnamed edge between filters that have one input and one output
A list of all filters can be obtained ith:
ffmpeg -filters
and parameters for a single filter can be obtained with:
ffmpeg --help filter=sine
Related question: stackoverflow.com/questions/69251087/in-ffmpeg-command-line-how-to-show-all-filter-settings-and-their-parameters-bef
TODO dump graph to ASCII art? trac.ffmpeg.org/wiki/FilteringGuide#Visualizingfilters mentions a -dumpgraph option, but haven't managed to use it yet.
Bibliography:

ffplay

words: 100 articles: 1
Awesome tool to view quick stuff quickly without generating files. Unfortunately it doesn't support all options that the ffmpeg CLI supports, e.g. ffplay multiple input files. One day, one day.
TODO possible? superuser.com/questions/559768/ffplay-how-to-play-together-separate-video-and-audio-files
For synthesized streams like sine we can do it e.g.
ffplay -autoexit -nodisp -f lavfi -i '
sine=frequency=500[a];
sine=frequency=1000[b];
[a][b]amerge, atrim=end=2
'
but it does not seem to accept multiple -i for some reason. So is there a way to open a file from some filter? E.g.:
ffplay -i tmp.wav -i tmp.mkv -filter_complex "[0:a]atrim=end=2[a];[1:v]trim=end=2[v]" -map '[a]' -map '[v]'
fails with:
Argument 'tmp.mkv' provided as input filename, but 'tmp.wav' was already specified.
Simple sines and variants:
2 second 1000 Hz:
ffmpeg -f lavfi -i "sine=f=1000:d=2" out.wav
Video with a solid color:
  • 2 second white video:
    ffplay -autoexit -f lavfi -i 'color=white:640x480:d=3,format=rgb24,trim=end=2'
    Also add some audio:
    ffmpeg -lavfi "color=white:640x480:d=3,format=rgb24,trim=end=2[v];sine=f=1000:d=2[a]" -map '[a]' -map '[v]' out.mkv
    TODO how to ffplay the video + audio directly? -map does not seem to work unfortunately.
  • 2 second white followed by 2 second black video:
    ffplay -autoexit -f lavfi -i 'color=white:640x480:d=3,format=rgb24,trim=end=2[a];color=black:640x480:d=3,format=rgb24,trim=end=2[b];[a][b]concat=n=2:v=1:a=0'
  • bibliography:
Display count in seconds on the video:
FFmpeg is likely the backend of YouTube through reverse engineering: streaminglearningcenter.com/blogs/youtube-uses-ffmpeg-for-encoding.html (archive)
On Quora: www.quora.com/What-does-YouTube-use-for-encoding-video/answer/Ciro-Santilli
stackoverflow.com/questions/7333232/how-to-concatenate-two-mp4-files-using-ffmpeg

ImageMagick

words: 79 articles: 5
Crop 20 pixels from the bottom of the image:
convert image.png -gravity East -chop 20x0 result.png

ImageMagick HOWTO

words: 63 articles: 4
stackoverflow.com/questions/20737061/merge-images-side-by-sidehorizontally/63575228#63575228
convert -size 512x512 xc:blue blue.png
Bibliography:
imagemagick.org/script/gradient.php
convert -size 256x256 gradient: out.png
convert -size 256x256 gradient:white-black out.png
convert -size 256x256 gradient:red-blue out.png
convert -size 256x256 radial-gradient: out.png
convert -size 256x256 radial-gradient:white-black out.png
Digits 0 to 9, white on black background:
for i in `seq 0 9`; do convert -size 512x512 xc:black -pointsize 500 -gravity center -fill white -draw "text 0,0 \"$i\"" $i.png; done
Bibliography:

Open source software

words: 1k articles: 21
What happens when the underdogs get together and try to factor out their efforts to beat some evil dominant power, sometimes victoriously.
Or when startups use the cheapest stuff available and randomly become the next big thing, and decide to keep maintaining the open stuff to get features for free from other companies, or because they are forced by the Holy GPL.
Open source frees employees. When you change jobs, a large part of the specific knowledge you acquired about closed source a project with your blood and tears goes to the trash. When companies get bought, projects get shut down, and closed source code goes to the trash. What sane non desperate person would sell their life energy into such closed source projects that could die at any moment? Working on open source is the single most important non money perk a company can have to attract the best employees.
Open source is worth more than the mere pragmatic financial value of not having to pay for software or the ability to freely add new features.
Its greatest value is perhaps the fact that it allows people study it, to appreciate the beauty of the code, and feel empowered by being able to add the features that they want.
That is why Ciro Santilli thought:
Life is too short for closed source.
But quoting Ciro's colleague S.:
Every software is open source when you read assembly code.
And "can reverse engineer the undocumented GPU hardware APIs", Ciro would add.
While software is the most developed open source technology available in the 2010's, due to the "zero cost" of copying it over the Internet, Ciro also believes that the world would benefit enormously from open source knowledge in all areas on science and engineering, for the same reasons as open source.

GNU Project

articles: 1
A more precise term for those in the know: open source software that also has a liberal license, for some definition of liberal.
Ciro Santilli defines liberal as: "can be commercialized without paying anything back" (but possibly subject to other restrictions).
He therefore does not consider Creative Commons licenses with NC to be FOSS.
For the newbs, the term open source software is good enough, since most open source software is also FOSS.
But when it's not, it's crucial to know.
This model can work well when there is a set of commonly used libraries that some developers often use together, but such that there isn't enough maintenance work for each one individually.
So what people do is to create a group that maintains all those projects, to try and get enough money to survive from the contributions done primarily for each one individually.
Examples:

Open knowledge

words: 19 articles: 3
Ciro Santilli's raison d'etre, one of his attempts: OurBigBook.com.
The outcome of closed knowledge is reverse engineering.
Projects:
open.umn.edu/opentextbooks/
Not everything is perfect.
One big problem of many big open source projects is that they are contributed to by separate selfish organizations, that have private information. Then what happens is that:
  • people implement the same thing twice, or one change makes the other completely unmergeable
  • you get bugs but can't share your closed source test cases, and then you can't automate tests for them, or clearly demonstrate the problem
  • other contributors don't see your full semi secret important motivation, and may either nitpick too much or take too long to review your stuff
Another common difficulty is that open source maintainers may simply not care enough about their own project (maybe they did in the past but lost interest) to review external patches by people they don't know.
This is understandable: a new patch, is a new risk of things breaking.
Therefore, if you ever submit patches and they get ignore, don't be too sad. It just comes down to a question of maintenance cost, and means that you will waste some extra time on the next rebase. You just have to decide your goals and be cold about it:
  • are you doing the right thing and going for a specific goal backward design? Then just fork, run as fast as possible towards a minimum viable product, and if you start to feel that rebase is costing you a lot, or feel you could get some open source fame for cheap, open reviews and see what upstream says. If they ignore you, politely tell yourself in your mind silently "fuck them", and carry on with the MVP
  • otherwise, e.g. you just want to randomly help out, you have to ask them before doing anything big "how can I be of help". If I propose a patch for this issue, do you promise to review it?
Writing documentation in an open source project in which you don't have immediate push rights is another major pain due to code reviews. Code code reviews tend to be much less subjective, because if you do something wrong, stuff crashes, runs slower, or you need more lines of code to reach the same goal. There are tradeoffs, but in a limited number. Documentation code reviews on the other hand, are an open invitation to infinite bike-shedding, since you can't "run" documentation through a standardized brain model. Much better is for one good documenter person to just make one cohesive Stack Overflow post, and ping others with more knowledge to review details or add any missing pieces :-)

Code drop

words: 52
Open source development model in which developers develop in private, and only release code to the public during releases.
Notable example project: Android Open Source Project.
This development model basically makes reporting bugs and sending patches a waste of time, because many of them will already have been solved, which is why this development model is evil.
Ciro Santilli can accept closed source on server products more easily than offline, because the servers have to be paid for somehow (by stealing your private data).
Closed source on offline products used by millions of people is evil, when you could just have those for free with open source software! Thus Ciro's hatred for Microsoft Windows and MacOS (at least userland, maybe).

Closed source software

words: 108 articles: 3
The opposite of open source software.

Closed standard

words: 85 articles: 1
ISO is the main culprit of this bullshit, some notable examples related to open source software:
The only low level thing that escaped this was OpenGL via Khronos, what heroes those people are.
How the hell are you supposed to develop an open source implementation of something that has a closed standard?
Not to mention open source test suites, that would be way too much to ask for, those always end up being made by some shady small companies that go bankrupt from time to time, see e.g. .

Inner source

words: 19
If you are going to do closed source, at least do it like this.
Basically the opposite of need to know for software.

Productivity software

words: 9 articles: 1

LibreOffice

words: 9
These people are heroes. There's nothing else to say.

Programming language

words: 6k articles: 168
This section is present in another page, follow this link to view it.

Search engine

words: 511 articles: 8

Web crawling

words: 511 articles: 3

Open web crawling

words: 511 articles: 2
Common Crawl
words: 511 articles: 1
commoncrawl.org/
Amazing project, that basically makes a more searchable Wayback Machine.
A bit hard to use their data though, partly due to size, but also lack of free to use querrying mechanisms, and how obtuse Amazon S3 is to use.
Notably, aws-cli with an account is the only reliable way, everything else is way too broken, e.g. trying the to check the an index index.commoncrawl.org/CC-MAIN-2023-06/ very often 500s.
But still, their projct is amazing.
The only out-of-the-box search they seem to have is: urlsearch.commoncrawl.org/ for domains/URLs. It is good, but there could be so much more... notably IPs.
Also could should document the data shape a bit better.
Sample sizes can be found at: commoncrawl.org/2023/04/mar-apr-2023-crawl-archive-now-available/
To explore the data, after login:
aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2013-20/
Copy the toplevel directory only:
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/ . --recursive --exclude "*/*"
Copy some wet/wat files:
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz .
aws s3 sync s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz .
Directory structrure:
  • cc-index.paths.gz (1K)
  • cc-index-table.paths.gz (1K)
  • segment.paths.gz (1.7K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/
    crawl-data/CC-MAIN-2013-20/segments/1368696381630/
  • index.html (2.3K)
  • wat.paths.gz (98K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wat.gz
  • wet.paths.gz (98K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wet.gz
  • warc.paths.gz (99K)
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.gz
  • segments: directgory with actual data
    • 1368696381249: one of many segments, any meaning of name?
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz (142M, 334M unzipped)
        A tiny bit of metadata, and then plaintext content from the website, e.g. the second one:
        WARC/1.0
        WARC-Type: conversion
        WARC-Target-URI: http://004eeb5.netsolhost.com/stephensilver.htm
        WARC-Date: 2013-05-18T08:11:02Z
        WARC-Record-ID: <urn:uuid:773b31ba-ddc6-47a5-ae24-d08141b9944d>
        WARC-Refers-To: <urn:uuid:4b1bdbff-4926-4ced-86f6-072f5bb3837a>
        WARC-Block-Digest: sha1:LQFSCR2LIJQYMPTXRHWU7HAPQTVSYS3A
        Content-Type: text/plain
        Content-Length: 12046
        
        Stephen Silver is a journalist and editor who specializes in the areas of politics, pop culture, film and sports. He works as an editor with the North American Publishing Co. and as a film critic with The Trend, a local newspaper in the Philadelphia area.
        No IP unfortunately.
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz (329M, 1.4G unzipped)
        A lot of JSON metadata and no contents as desired. Contains IP! Some entries however are humongous with a ton of useless data, that's what bloats these so much:
        WARC/1.0
        WARC-Type: metadata
        WARC-Target-URI: CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
        WARC-Date: 2013-11-22T14:51:12Z
        WARC-Record-ID: <urn:uuid:ec54e493-8965-41be-b344-07596cc30b3a>
        WARC-Refers-To: <urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>
        Content-Type: application/json
        Content-Length: 1180
        
        {"Envelope":{"Format":"WARC","WARC-Header-Length":"274","Block-Digest":"sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR","Actual-Content-Length":"372","WARC-Header-Metadata":{"WARC-Type":"warcinfo","WARC-Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz","WARC-Date":"2013-11-22T14:51:12Z","Content-Length":"372","WARC-Record-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Type":"application/warc-fields"},"Payload-Metadata":{"Trailing-Slop-Length":"0","Actual-Content-Type":"application/warc-fields","Actual-Content-Length":"372","Headers-Corrupt":true,"WARC-Info-Metadata":{"robots":"classic","software":"Nutch 1.6 (CC)/CC WarcExport 1.0","description":"Wide crawl of the web with URLs provided by Blekko for Spring 2013","hostname":"ip-10-60-113-184.ec2.internal","format":"WARC File Format 1.0","isPartOf":"CC-MAIN-2013-20","operator":"CommonCrawl Admin","publisher":"CommonCrawl"}}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"453","Header-Length":"10","Inflated-CRC":"866052549","Inflated-Length":"650"},"Offset":"0","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}
        
        WARC/1.0
        WARC-Type: metadata
        WARC-Target-URI: http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions
        WARC-Date: 2013-05-18T05:48:54Z
        WARC-Record-ID: <urn:uuid:d519658f-7a63-46c1-849b-4cd92332ddb8>
        WARC-Refers-To: <urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>
        Content-Type: application/json
        Content-Length: 1501
        
        {"Envelope":{"Format":"WARC","WARC-Header-Length":"433","Block-Digest":"sha1:B2B6JDSGWCUQIIUGV54SXEE25RX4SANS","Actual-Content-Length":"302","WARC-Header-Metadata":{"WARC-Type":"request","WARC-Date":"2013-05-18T05:48:54Z","WARC-Warcinfo-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Length":"302","WARC-Record-ID":"<urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>","WARC-Target-URI":"http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions","WARC-IP-Address":"165.1.125.44","Content-Type":"application/http; msgtype=request"},"Payload-Metadata":{"Trailing-Slop-Length":"4","HTTP-Request-Metadata":{"Headers":{"Accept-Language":"en-us,en-gb,en;q=0.7,*;q=0.3","Host":"ap.org","Accept-Encoding":"x-gzip, gzip, deflate","User-Agent":"CCBot/2.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},"Headers-Length":"300","Entity-Length":"0","Entity-Trailing-Slop-Bytes":"0","Request-Message":{"Method":"GET","Version":"HTTP/1.0","Path":"/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions"},"Entity-Digest":"sha1:3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ"},"Actual-Content-Type":"application/http; msgtype=request"}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"455","Header-Length":"10","Inflated-CRC":"453539965","Inflated-Length":"739"},"Offset":"453","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}
        Let's beautify one of them to see it better:
        
        {
          "Envelope": {
            "Format": "WARC",
            "WARC-Header-Length": "274",
            "Block-Digest": "sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR",
            "Actual-Content-Length": "372",
            "WARC-Header-Metadata": {
              "WARC-Type": "warcinfo",
              "WARC-Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz",
              "WARC-Date": "2013-11-22T14:51:12Z",
              "Content-Length": "372",
              "WARC-Record-ID": "<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>",
              "Content-Type": "application/warc-fields"
            },
            "Payload-Metadata": {
              "Trailing-Slop-Length": "0",
              "Actual-Content-Type": "application/warc-fields",
              "Actual-Content-Length": "372",
              "Headers-Corrupt": true,
              "WARC-Info-Metadata": {
                "robots": "classic",
                "software": "Nutch 1.6 (CC)/CC WarcExport 1.0",
                "description": "Wide crawl of the web with URLs provided by Blekko for Spring 2013",
                "hostname": "ip-10-60-113-184.ec2.internal",
                "format": "WARC File Format 1.0",
                "isPartOf": "CC-MAIN-2013-20",
                "operator": "CommonCrawl Admin",
                "publisher": "CommonCrawl"
              }
            }
          },
          "Container": {
            "Compressed": true,
            "Gzip-Metadata": {
              "Footer-Length": "8",
              "Deflate-Length": "453",
              "Header-Length": "10",
              "Inflated-CRC": "866052549",
              "Inflated-Length": "650"
            },
            "Offset": "0",
            "Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"
          }
        }
        Fuck no IP addresses either. But other entries do have it, why not this one?
        The reason these can be huge is the HTML-Metadata section which contain all outlinks! gist.github.com/Smerity/e750f0ef0ab9aa366558#file-bbc-pretty-wat-L34
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz ()
        Obtain:
        aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz .
TODO no IP? Sadface?

Scientific computing

words: 665 articles: 22

Scientific software

words: 665 articles: 20

Numerical software

words: 521 articles: 5
The original gangster.
Computer algebra system
words: 519 articles: 3
SymPy
words: 519 articles: 2
This is the dream cheating software every student should know about.
It also has serious applications obviously. www.sympy.org/scipy-2017-codegen-tutorial/ mentions code generation capabilities, which sounds super cool!
The code in this section was tested on sympy==1.8 and Python 3.9.5.
Let's start with some basics. fractions:
from sympy import *
sympify(2)/3 + sympify(1)/2
outputs:
7/6
Note that this is an exact value, it does not get converted to floating-point numbers where precision could be lost!
We can also do everything with symbols:
from sympy import *
x, y = symbols('x y')
expr = x/3 + y/2
print(expr)
outputs:
x/3 + y/2
We can now evaluate that expression object at any time:
expr.subs({x: 1, y: 2})
outputs:
4/3
How about a square root?
x = sqrt(2)
print(x)
outputs:
sqrt(2)
so we understand that the value was kept without simplification. And of course:
sqrt(2)**2
outputs 2. Also:
sqrt(-1)
outputs:
I
I is the imaginary unit. We can use that symbol directly as well, e.g.:
I*I
gives:
-1
Let's do some trigonometry:
cos(pi)
gives:
-1
and:
cos(pi/4)
gives:
sqrt(2)/2
The exponential also works:
exp(I*pi)
gives;
-1
Now for some calculus. To find the derivative of the natural logarithm:
from sympy import *
x = symbols('x')
diff(ln(x), x)
outputs:
1/x
Just read that. One over x. Beauty.
Let's do some more. Let's solve a simple differential equation:
y''(t) - 2y'(t) + y(t) = sin(t)
Doing:
from sympy import *
x = symbols('x')
f, g = symbols('f g', cls=Function)
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x)**4)
print(dsolve(diffeq, f(x)))
outputs:
Eq(f(x), (C1 + C2*x)*exp(x) + cos(x)/2)
which means:
To be fair though, it can't do anything crazy, it likely just goes over known patterns that it has solvers for, e.g. if we change it to:
diffeq = Eq(f(x).diff(x, x)**2 + f(x), 0)
it just blows up:
NotImplementedError: solve: Cannot solve f(x) + Derivative(f(x), (x, 2))**2
Sad.
Let's try some polynomial equations:
from sympy import *
x, a, b, c = symbols('x a b c d e f')
eq = Eq(a*x**2 + b*x + c, 0)
sol = solveset(eq, x)
print(sol)
which outputs:
FiniteSet(-b/(2*a) - sqrt(-4*a*c + b**2)/(2*a), -b/(2*a) + sqrt(-4*a*c + b**2)/(2*a))
which is a not amazingly nice version of the quadratic formula. Let's evaluate with some specific constants after the fact:
sol.subs({a: 1, b: 2, c: 3})
which outputs
FiniteSet(-1 + sqrt(2)*I, -1 - sqrt(2)*I)
Let's see if it handles the quartic equation:
x, a, b, c, d, e, f = symbols('x a b c d e f')
eq = Eq(e*x**4 + d*x**3 + c*x**2 + b*x + a, 0)
solveset(eq, x)
Something comes out. It takes up the entire terminal. Naughty. And now let's try to mess with it:
x, a, b, c, d, e, f = symbols('x a b c d e f')
eq = Eq(f*x**5 + e*x**4 + d*x**3 + c*x**2 + b*x + a, 0)
solveset(eq, x)
and this time it spits out something more magic:
ConditionSet(x, Eq(a + b*x + c*x**2 + d*x**3 + e*x**4 + f*x**5, 0), Complexes)
Oh well.
Let's try some linear algebra.
m = Matrix([[1, 2], [3, 4]])
Let's invert it:
m**-1
outputs:
Matrix([
[ -2,    1],
[3/2, -1/2]])
python/sympy_cheat/logarithm_integral.py
#!/usr/bin/env python3

from sympy import *

x = symbols('x')
myli = integrate(sympify(1)/ln(x), x)

# It recognizes our definition as its own li! Beauty.
assert myli.equals(li(x))

for r in range(-2, 2):
    for i in range(-2, 2):
        print(f'{r} {i} {li(r + i*I).evalf()}')

Scientific visualization

words: 144 articles: 13
Chart
articles: 2
Chart type
articles: 1
Scientific visualization software
words: 144 articles: 9
en.wikipedia.org/wiki/Scientific_visualization
Ciro's large dataset survey: Section "Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli".
Huge respect to this companies.
Plotting software
words: 134 articles: 5
E.g. showing live data from a scientific instrument! TODO:
stackoverflow.com/questions/5854515/large-plot-20-million-samples-gigabytes-of-data/55967461#55967461
By Ciro Santilli.
Figure 2. Source.
Matplotlib
words: 60
It does a huge percentage of what you want easily, and from the language that you want to use.
Tends to be Ciro's pick if gnuplot can't handle the use case, or if the project is really really serious.
Couldn't handle exploration of large datasets though: Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli
Examples:
Tested on Python 3.10.4, Ubuntu 22.04.
gnuplot
words: 63 articles: 1
Tends to be Ciro Santilli's first attempt for quick and dirty graphing: github.com/cirosantilli/gnuplot-cheat.
domain-specific language. When it get the jobs done, it is in 3 lines and it feels great.
When it doesn't, you Google for an hours, and then you give up in frustration, and fall back to Matplotlib.
Couldn't handle exploration of large datasets though: Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli
askubuntu.com/questions/277363/gnuplot-not-showing-the-graph-window/683073#683073
CLI hello world:
gnuplot -p -e 'p sin(x)'
github.com/ReFirmLabs/binwalk

Software bug

words: 497 articles: 15

Glitch

words: 27
A glitch is more precisely a software bug that is hard to reproduce. But it has also been used to mean a software bug that is not very serious.

Debugging

words: 441 articles: 10
Debugging sucks. But there's also nothing quite that "oh fuck, that's why it doesn't work" moment, which happens after you have examined and placed everything that is relevant to the problem into your brain. You just can't see it coming. It just happens. You just learn what you generally have to look at so it happens faster.
Related:
This is a simple hierarchical plaintext notation Ciro Santilli created to explain programs to himself.
It is usuall created by doing searches in an IDE, and then manually selecting the information of interest.
It attempts to capture intuitive information not only of the call graph itself, including callbacks, but of when things get called or not, by the addition of some context code.
For example, consider the following pseudocode:
f1() {
}

f2(i) {
  if (i > 5) {
    f1()
  }
}

f3() {
  f1()
  f2_2()
}

f2_2() {
  for (i = 0; i < 10; i++) {

    f2(i)
  }
}

main() {
  f2_2()
  f3()
}
Supose that we are interested in determining what calls f1.
Then a reasonable call hierarchy for f1 would be:
f2(i)
  if (i > 5) {
    f1()

  f2_2()
    for (i = 0; i < 10; i++) {
      f2(i)

    main
    f3
f3()
  main()
Some general principles:
  • start with a regular call tree
  • to include context:
    • remove any blank lines from the snippet of interest
    • add it indented below the function
    • and then follow it up with a blank line
    • and then finally add any callers at the same indentation level
One of the Holiest age old debugging techniques!
Git has some helpers to help you achieve bisection Nirvana: stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593
Obviously not restricted to software engineering alone, and used in all areas of engineering, e.g. Video "Air-tight vs. Vacuum-tight by AlphaPhoenix (2020)" uses it in vacuum engineering.
The cool thing about bisection is that it is a brainless process: unlike when using a debugger, you don't have to understand anything about the system, and it incredibly narrows down the problem cause for you. Not having to think is great!

Debugger

words: 112 articles: 7
Reverse debugging
words: 51 articles: 1
Nirvana!!!
For compiled languages, see: Section "GDB reverse debugging".
For JavaScript: stackoverflow.com/questions/17498159/how-to-go-backwards-while-debugging-javascript-in-chrome-sources-debugging/74968631#74968631
What it adds on top of reverse debugging: not only can you go back in time, but you can do it instantaneously.
Or in other words, you can access variables from any point in execution.
TODO implementation? Apparently Pernosco is an attempt at it, though proprietary.
GNU Debugger (GDB)
words: 61 articles: 4
Just add GDB Dashboard, and you're good to go.
GDB reverse debugging
words: 46 articles: 2
The best open source implementation as of 2020 seems to be: Mozilla rr.
Mozilla rr
words: 34 articles: 1
github.com/mozilla/rr
Pernosco
words: 24
pernos.co/
Proprietary extension to Mozilla rr by rr lead coder Robert O'Callahan et. al, started in 2016 after he quit Mozilla.
TODO what does it add to rr?
GDB Dashboard
words: 7
github.com/cyrus-and/gdb-dashboard
GDB Nirvana?
stackoverflow.com/questions/10115540/gdb-split-view-with-code/51301717#51301717
Figure 3.
Screenshot of terminal running GDB Dashboard
. Source.
The musical study of software engineering.
Ciro Santilli is obsessed by those in order to learn any new concept, not just for bug reporting.
This includes to learn more theoretical subjects like physics and mathematics.

Software company

words: 847 articles: 18

Microsoft

words: 383 articles: 12
This section is present in another page, follow this link to view it.

Oracle Corporation

words: 141 articles: 1
Evil company that desecrated the beauty created by Sun Microsystems, and was trying to bury Java once and or all in the 2010's.
Their database is already matched by open source e.g. PostgreSQL, and ERP and CRM specific systems are boring.
Oracle basically grew out of selling one of the first SQL implementations in the late 70's, and notably to the United States Government and particularly the CIA. They did deliver a lot of value in those early pre-internet days, but now open source is and will supplant them entirely.
Although Ciro Santilli is a bit past their era, there's an aura of technical excellence about those people. It just seems that they sucked at business. Those open source hippies. Erm, wait.
Bibliography:
Video 3.
The Dawn and Dusk of Sun Microsystems by Asianometry (2022)
. Source. One of the main inspirations for the creation of their workstations were CAD applications.

Red Hat

words: 248 articles: 1
Video "1984 Macintosh advertisement by Apple (1984)" comes to mind.
TODO year. This was a reply to Microsoft anti-Linux propaganda it seems: www.ubuntubuzz.com/2012/03/truth-happens-redhats-legendary-reply.html
Trascript from: www.dailymotion.com/video/xw3ws
The world is flat. Earth is the centre of the universe. Fact - until proven otherwise.
Despite ignorance. Despite ridicule. Despite opposition. Truth happens.
Despite ignorance.
The telephone has too many shortcomings to be seriously considered as a means of communication. /Western Union 1876/
In 1899 the US Patent Commissioner stated, everything that can be invented has been invented.
Despite ridicule.
The phonograph has no commercial value at all. /Thomas Edison 1880/
The radio craze will die out in time. /Thomas Edison 1922/
The automobile has practically reached the limit of its development. /Scientific American 1909/
Despite it all truth happens.
Man will not fly for fifty years. /Orville Wright 1901/
The rocket will never leave the Earth's atomosphere. /New York Times 1936/
There is a world market for maybe five computers. /IBM's Thomas Watson 1943/
640K Ought to be enough for anybody. /Bill Gates 1981/
First they ignore you...
Linux is the hype du jour. /Gartner Group 1999/
Then they laugh at you...
We think of linux as competitor in the student and hobbyist market. But I really don't think in the commercial market we'll see it in any significant way. /Bill Gates 2001/
Then they fight you...
Linux isn't going away. Linux is a serious competitor. We will rise to this challenge. /Steve Ballmer 2003/
Then you win... /Mohandas Gandhi/
You are here.
Red Hat Linux. IBM.
Video 4.
Truth Happens advertisement by Red Hat
. Source.

Software documentation

words: 4 articles: 2

README

words: 4
Please, use AsciiDoc and one page to rule them all.

Software engineering

words: 2k articles: 46

Software development

words: 4 articles: 3

Software development principle

words: 4 articles: 2
Don't repeat yourself
words: 4 articles: 1
Yet another
words: 4
The mandatory xkcd: xkcd 927: Standards.
Of course, "Ciro Santilli" with quotes, since all of those are either taken directly from others, or had been previously formulated by others.
Some anecdotes.
Ciro Santilli never splits up functions unless there is more than one calling point. If you split early, the chances that the interface will be wrong are huge, and a much larger refactoring follows.
If you just want to separate variables, just use a scope e.g.:
int cross_block_var;

// First step.
{
    int myvar;
}

// Second step.
{
    int myvar;
}
Ciro has seen and had to deal with in his lifetime with two projects that had like 3 to 10 git separate Git repositories, all created and maintained by the same small group of developers of the same organization, even though one could not build without the other. Keeping everything in sync was Hell! Why not just have three directories inside a single repository with a single source of truth?
Another important case: Linux should have at least a C standard library, init system, and shell in-tree, like BSD Operating Systems, as mentioned at: Section "Linux".
A slow development test cycle will kill your software.
New developers won't want to learn your project, because they would rather shoot themselves.
This means that build time, and the time to run tests, must be short.
5 seconds to rebuild is the maximum upper limit.
Of course, at some point software gets large enough that things won't fit anymore in 5 seconds. But then you must have either some kind of build caching, or options to do partial builds/tests that will bring things down to that 5 second mark.
You also have to spend some time profiling execution and build from scratch times.
A slow build from scratch will mean that your continuous integration costs a lot, money that could be invested in a new developer!
It also means that people won't bother to reproduce bugs on given commits, or bisect stuff.
One anecdote comes to mind. Ciro Santilli was trying to debug something, and more experience colleague came over.
To reproduce a problem, ciro was running one command, wait 5 seconds, run a second command, wait 5 seconds, run a third command:
cmd1
# wait 5 seconds
cmd2
# wait 5 seconds
cmd3
The first thing the colleague said: join those three commands into one:
cmd1;cmd2;cmd3
And so, Ciro was enlightened.
Figure 4.
xkcd 303: Compiling
. Source. They should be benchmarking and fixing their shitty build system instead.
Whenever someone asks:
I can only see this one thing different our setups, do you think it could be the cause of our different behaviour?
you don't need to read anymore, just point them to this page immediately. Virtualization for the win.
Sometimes you are really certain that something is a required substep for another thing that is coming right afterwards.
When things are this concrete, fine, just do the substep.
But you have to always beware of cases where "I'm sure this will be needed at some unspecified point in the future", because such points tends to never happen.
YAGNI is so fundamental, there are several closely related concepts to it:
Figure 5.
xkcd 2730: Code Lifespan
. Source.
The software engineer phrasing of simplicity is the ultimate sophistication.
Like all other principles, it is not absolute.
But it is something that you should always have on the back of your mind.
You aren't gonna need it is closely related, as generally the extra unnecessary complications are set in place to accommodate useless features that will never be needed.

Hofstadter's law

words: 126
The trivial takes a few hours.
The easy takes a week.
And what seemed hard takes a few hours.
As "deadlines" approach, feature sets get cut down, then there are delays, and finally a feasible feature set is delivered some time after the deadline.
The only deadlines that can be met are those of tasks which have already been done but not announced.
This is of course Hofstadter's law.
On the other hand, as a colleague of Ciro once mentioned, it is also known that the time it takes for a task to be done expands without limits to match the deadline. And therefore, without deadlines, tasks will take forever and never get done.
And so, in a moment, perceiving this paradox, Ciro was enlightened.
Video 5.
The Misty Mountains Cold Scene from The Hobbit: An Unexpected Journey (2012)
. Source.
I will take each and every one of these dwarves over an army from the Iron Hills. For when I called upon them they answered. Loayalty. Honour. And willing heart. I can ask no more than that.
Once upon a time, when Ciro Santilli had a job, he had a programming problem.
A senior developer came over, and rather than trying to run and modify the code like an idiot, which is what Ciro Santilli usually does (see also experimentalism remarks at Section "Ciro Santilli's bad old event memory"), he just stared at the code for about 10 minutes.
We knew that the problem was likely in a particular function, but it was really hard to see why things were going wrong.
After the 10 minutes of examining every line in minute detail, he said:
I think this function call has such or such weird edge case
and truly, that was the cause.
And so, Ciro was enlightened.
Working remotely is hard if you don't already highly master the software and enterprise systems used.
Also you don't feel people's love as strongly, and usefulness is built on love, see also Steve Jobs's Pixar office space design philosophy.
But please, give workers a small silent office so that we can concentrate instead of a silly open space, and create an internal social network so people can see what others are doing.
Remote working is much better if the majority of the team also does it, otherwise you will get excluded. Maybe after VR...
When debugging complex software, make sure to keep notes of every interesting find you make in a note file, as you extract it from the integrated development environment or debugger.
Especially if your memory sucks like Ciro's.
This is incredibly helpful in fully understanding and then solving complex bugs.

"Hello, World!" program

words: 21 articles: 1
The most important program ever written!!!
Other programs that can be considered "hello worlds" in different contexts:

Software engineer

words: 1k articles: 21
Poet warriors monkeys? Or Code peasants (码农) according to the Chinese.
Ciro Santilli claims to be one of them.
Much like a pianist plays his piano, a software engineer plays his computer.

Software engineer stereotype

words: 300 articles: 3
www.quora.com/Why-do-successful-geeky-white-men-have-Asian-wives-This-seems-to-be-the-norm-in-Silicon-Valley suggests it is just an statistical inevitability.
Ciro Santilli believes that there is a positive correlation between being a software engineer and liking Buddhist-like things.
Maybe it is linked to minimalism and DRY, which software engineers value so greatly.
Even Ciro had to try an unoriginal Buddhist joke intro in one of this Stack Overflow answers.
Ciro also feels that his "minimal reproducible example" scientific language/concept learning method obsession of breaking things into tiny sub-problems has a strong link with Koans.
Some notable Buddhism/programmer examples:
Another thing that points the correlation out is the existence of wattsalan.github.io/ on a github.io about Alan Watts.
Ciro Santilli's joke version of the Chinese Four Treasures of the Study!
  • web browser
  • Text editor
  • terminal. Though to be honest, circa 2022, Ciro learned of the ctrl + click to open file (including with file.c:123 line syntax) ability of Visual Studio Code (likely present in other IDEs), and he was starting considering dumping the terminal altogether if some implementation gets it really really right. The main thing is that it can't be a tinny little bar at the bottom, it has to be full window and super easily toggleable!
In the past, Ciro used to use file managers, which would be the fourth tresure. But he stopped doing so for years due to his cd alias... so it became three. He actually had exactly three windows open when he was checking if there was anything else he could not open hand of.
Figure 6.
The three Treasures of the Programmer
. Featuring: Gvim, tmux running in GNOME terminal, and Chromium browser on Ubuntu 22.04. The minimized windows are for demonstration purposes, Cirism mandates that all windows shall be maximized at all times. Splits withing a single program are permitted however.

List of software engineers

words: 466 articles: 10
Aaron Swartz
words: 150 articles: 2
Aaron, Ciro Santilli will complete your quest to make eduction free. Just legally this time, with the and with the Creative Commons license you helped to create.
Ciro likes how The Internet's Own Boy (2014) explains how Aaron felt like high school was bullshit, and that he could learn whatever he wanted from books, which is one of Ciro's key feelings.
It also mentions how he was a natural teacher from a very early age.
gist.github.com/briandoll/4522952
Hmmm, he does not know how to spell guerilla? sic? www.quora.com/What-is-the-correct-spelling-guerilla-or-guerrilla
Note to self: if you are going to commit a crime, don't publish your plans online.
Ross Ulbricht's diaries come to mind.
That's how Russian shadow library maintainers do it, they know how to crime good old Russians. Maybe there is a good thing about having dictatorships in the world that give zero fucks about American copyright laws. There will always be some random Russian academic who will implement this and not go to jail. Maybe it's even state sponsored.
James Somers
words: 40 articles: 1
Huge interest overlap with Ciro Santilli, e.g. he's into
jsomers.net/i-should-have-loved-biology/
This resonates a lot with Ciro Santilli's ideas!
Sandy Maguire
words: 264
Lots of similar ideologies to Ciro Santilli, love it:
  • sandymaguire.me/about/:
    I might best be described somewhere between independent researcher and voluntarily-unemployed bum. At the ripe old age of 27 I decided to quit my highly-lucrative engineering job and decide to focus more on living than on grinding for the man. It's what you might call a work in progress.
  • sandymaguire.me/blog/reaching-climbing/: don't be a pussy
    Last Friday was my final day at work. According to my facebook profile, I am now "happily retired." As of today, I don't plan to do another day of "traditional work" in my life. That's not to say that I'll be sitting idle playing tiddly winks. I want to build things, to dedicate my life to independent study, and to get really, really good with building communities. I don't have time for any of this "work" stuff that somehow pervades our entire culture, choking our inspiration and sapping our energy away from the things we'd rather be doing.
    One is also reminded of Gwern Branwen. Sandy is also into self-improvement stuff, so even more like Gwern. This is a point Ciro diverges on. Ciro works actively on self-worsening.
  • he thinks university is useless:
  • he likes jazz: sandymaguire.me/blog/too-smart/
Other interesting points:
He's a Haskell person.
Dan Dascalescu
words: 12 articles: 1
His website is down as of 2020, shame: wiki.dandascalescu.com/essays/english-universal-language
wiki.dandascalescu.com/essays/english-universal-language
web.archive.org/web/20200317221752/https://wiki.dandascalescu.com/essays/english-universal-language
Dan Dascalescu's version of having more than one natural language is bad for the world.
Donald Knuth
articles: 1

EMBII

words: 59
One of the dudes from the AtomSea & EMBII Bitcoin-based file upload system.
Figure 7.
EMBII's usual profile image
. Source.

Gwern Branwen

words: 266 articles: 2
Author of gwern.net.
Accounts:
He posts insanely much on these websites. It's a bit like Ciro Santilli on Stack Overflow.
Ciro Santilli envies this guy a bit. He dumps his brain more or less full time on his highly customized static website partly due to early Bitcoin investments gwern.net/me says:
I am a freelance American writer & researcher. (To make ends meet, I have a Patreon, benefit from Bitcoin appreciation thanks to some old coins, and live frugally.)
Also unsurprisingly he likes Haskell:
I mostly contribute to projects in Haskell, my favorite language
Ciro Santilli considers Gwern Ciro Santilli's e-soulmates due to his interest in "dark web things" like Bitcoin and Silk Road, his immense writing output in encyclopedic book-sized articles on a static website, and his desire to live frugally and just research and write all day. Ah, if only Ciro had some old coins!!!
This is likely a pseudonym, his real name not being publicly unknown, e.g. at news.ycombinator.com/item?id=5659278:
> Why do you choose relative anonymity?
For the reasons I've said in the past. To which I can add personal safety: my Silk Road page is a bit questionable legally, and we all know that there are ways to exploit knowledge of one's True Name and address (even if, as far as I know, I have no enemies willing to resort to, say, 'swatting' me) - one group of stalkers called up a college they thought I worked at to see if they could get me fired or otherwise ruin my day.
gwern.net
words: 17
www.gwern.net
Gwern Branwen's website.
One thing that annoys Ciro Santilli about that website are the footnote overload. Ciro likes linear things.
TODO find the best source for the amazing "I have done your mother" quote.
Programming is hard. To Ciro Santilli, it's almost masochistic.
What makes Ciro especially mad when programming is not the hard things.
It is the things that should be easy, but aren't, and which take up a lot of your programming time.
Especially when you are already a few levels of "simple problems" down from your original goal, and another one of them shows up.
This is basically the cause of Hofstadter's law.
But of course, it is because it is hard that it feels amazing when you achieve your goal.
Putting a complex and useful program together is like composing a symphony, or reaching the summit of a hard rock climbing proble.
Programming can be an art form. There can be great beauty in code and what it does. It is a shame that this is hard to see from within the walls of most companies, where you are stuck doing a small specific task as fast as possible.

Software quality assurance

words: 203 articles: 12

Software testing

words: 203 articles: 9
This script tests all executables under a selected directory.
Ciro Santilli has been writing scripts of that type for a long time in order to test his programming self-learning setups with asserts.
The most advanced of those being the test system of Linux Kernel Module Cheat.
But had too much stuff that would be specific to that project, so Ciro decided to start this new one in Node.js, hopefully it will also be the last he ever writes.
A sample usage of the test library can be seen at: nodejs/sequelize/test.
test_executables.js was not rendered because it is too large (> 2000 bytes)

Test driven development

words: 117 articles: 1
This is a good approach. The downside is that while you are developing the implementation and testing interactively you might notice that the requirements are wrong, and then the tests have to change.
One intermediate approach Ciro Santilli likes is to do the implementation and be happy with interactive usage, then create the test, make it pass, then remove the code that would make it pass, and see it fail. This does have a risk that you will forget to test something, but Ciro finds it is a worth it generally. Unless it really is one of those features that you are unable to develop without an automated test, generally more "logical/mathematical" stuff. This is a sort of laziness Driven Development.

Source code

words: 20 articles: 2
Some blogs:
Also resonates with backward design.
wiki.c2.com/?UseTheSourceLuke

Systems programming

words: 11k articles: 139
This section is present in another page, follow this link to view it.

Terminal emulator

words: 101 articles: 5
Once upon a time young Ciro Santilli spent lots of time evaluating the features of different terimnals. The many windows of Terminator. The pop-uppiness of Guake/Yakuake.
But then one day he met tmux, and he was enlightened
Terminal choice doesn't matter. Just use tmux.
github.com/KDE/yakuake

Terminal multiplexer

words: 59 articles: 3
If we didn't have GUIs, terminal multiplexers would be our desktop environments. E.g. they handle stuff like:
  • window switching
  • copy pasting across windows
  • screen locking
  • clock on the status bar (same one that holds tabs)
It is a thing of beauty.

GNU screen

words: 11
Most important things to know:
  • kill window: Ctrl + A K

tmux

words: 10
github.com/tmux/tmux
If session autosave was finally mainlined, this would be Nirvana.

Version control

words: 2k articles: 60

List of version control systems

words: 2k articles: 58
It is said, that once upon a time, programmers used CSV and collaborated on SourceForge, and that everyone was happy.
These days, are however, long gone in the mists of time as of 2020, and beyond Ciro Santilli's programming birth.
Except for hardware developers of course. The are still happily using Perforce and Tcl, and shall never lose their innocence. Blessed be their souls. Amen.

Git

words: 2k articles: 55
The fundamental insight of Git design is: a SHA represents not only current state, but also the full history due to the Merkle tree implementation, see notably:
This makes it so that you will always notice if you are overwriting history on the remote, even if you are developing from two separate local computers (or more commonly, two people in two different local computers) and therefore will never lose any work accidentally.
It is very hard to achieve that without the Merkle tree.
Consider for example the most naive approach possible of marking versions with consecutive numbers:
  • Local 1:
    • 0: root commit
    • 1: commit 1
    • 2: commit 2 by local 1
  • Local 2:
    • 0: root commit
    • 1: commit 1
    • 2: commit 2 by local 2
    • 3: commit 3 by local 2
  • Remote
    • 0: root commit
    • 1: commit 1
If Local 1 were to push to Remote first, how could Local 2 notice that when it tries to push itself? The navie method of just checking: "does Remote have commit "2"" does not work, because Local 2 has a different version of commit 2 than local 1.
Git command
words: 9 articles: 2
git clone
words: 9 articles: 1
Git internals
articles: 4
Git object
articles: 1
stackoverflow.com/questions/22968856/what-is-the-file-format-of-a-git-commit-object-data-structure
Git UI
words: 146 articles: 19
Perfect Git integration belongs in integrated development environments :-)
gitk
words: 7
Figure 8.
gitk 2.34.1 running on Ubuntu 22.04 with a simple repository.
.
tig (git UI)
words: 21
github.com/jonas/tig
This is good. But it misses some key operations, so much so that makes Ciro not want to learn/use it daily.
Git web interface
words: 112 articles: 16
GitHub
words: 94 articles: 11
This is where Ciro Santilli stored his code since he started coding nonstop in 2013.
He does not like the closed source aspect of it, but hey, there are more important things to worry about, the network effect is just too strong.
GitHub repo
words: 17 articles: 2
Some amazing people have put book source codes on GitHub. This is a list of such repos.
www.quora.com/Does-github-allow-uploading-a-porn-image/answer/Ciro-Santilli
stackoverflow.com/questions/50720844/can-github-delete-your-account/66105692#66105692
GitHub Pages
words: 31
pages.github.com/
The cheapest and most resilient way to publish text content humanity has achieved so far.
Some tests:
GitHub Sponsors
articles: 1
github.com/isaacs/github/issues/1824
Pull request
words: 5
The heart/main innovation of GitHub!
Octokat.js
articles: 1
github.com/philschatz/octokat.js
github.com/philschatz/octokit.js
GitLab
words: 17 articles: 2
GitLab was very important to Ciro because he wanted to base Booktree on it.
See also: Ciro Santilli's minor projects.
github.com/gitlabhq/gitlab-ci
gitlab.com/gitlab-org/cookbook-gitlab
Git tips
words: 1k articles: 20
This section is present in another page, follow this link to view it.
SourceForge
words: 1
RIP: www.quora.com/Is-SourceForge-still-relevant-to-open-source-projects/answer/Ciro-Santilli

Web technology

words: 3k articles: 52
This section is present in another page, follow this link to view it.
Git implementation
articles: 2
libgit2
articles: 1
github.com/libgit2/libgit2
github.com/libgit2/rugged
Git bibliography
articles: 1
github.com/progit/progit2
github.com/EbookFoundation/free-programming-books

Tagged

Ancestors

  1. Computer
  2. Information technology
  3. Area of technology
  4. Technology
  5. Ciro Santilli's Homepage