Ciro Santilli
OurBigBook.com
$£ Sponsor €¥
中国独裁统治 China Dictatorship 新疆改造中心、六四事件、法轮功、郝海东、709大抓捕、2015巴拿马文件 邓家贵、低端人口、西藏骚乱
software.bigb
= Software
{wiki}
= The art of programming
{parent=Software}
{tag=Essays by Ciro Santilli}
Big goals:
* the pursuit of <AGI>
* <computational physics>[physics simulations], including <scientific visualization software>
* <formalization of mathematics>
Just <art>:
* useless mathy stuff
* incredibly nifty little tools that are just so satisfying to use it is mind blowing:
* <ncdu>
* <GNU parallel>
* media related stuff
* <ffmpeg> one liners!
= Build automation
{parent=Software}
{wiki}
= CMake
{c}
{parent=Build automation}
{wiki}
Examples under \a[cmake]:
* \a[cmake/hello]: just print a message in CMake itself and exit. No compilation.
* \a[cmake/hello_c]: C hello world
* \a[cmake/option]: `set()` and `option()` basic examples
* \a[cmake/multi_executable]
* \a[cmake/multi_file]
* \a[cmake/multi_file_recursive]
* \a[cmake/shared_lib_external]
= make
{disambiguate=Software}
{parent=Build automation}
{wiki}
= GNU make
{c}
{parent=make (Software)}
{tag=GNU package}
{wiki}
= Compiler
{parent=Software}
{wiki}
= Compile
{synonym}
= Compiles
{synonym}
= Compiler toolchain
{parent=Compiler}
{wiki}
<Compiler> + other closely related crap like <linker (computing)>.
= Linker
{disambiguate=computing}
{parent=Compiler toolchain}
{wiki}
Some linker related ansewrs by <Ciro Santilli>:
* https://stackoverflow.com/questions/3322911/what-do-linkers-do/33690144#33690144[How do linkers and address relocation work?]
* https://stackoverflow.com/questions/29391965/what-is-partial-linking-in-gnu-linker/53959624#53959624[What is incremental linking or partial linking?]
* https://stackoverflow.com/questions/3476093/replacing-ld-with-gold-any-experience/53921263#53921263[GOLD (`-fuse-ld=gold`) linker vs the traditional GNU ld and LLVM ldd]
* https://stackoverflow.com/questions/2463150/what-is-the-fpie-option-for-position-independent-executables-in-gcc-and-ld/51308031#51308031[What is the -fPIE option for position-independent executables in GCC and ld?]
= Binutils
{c}
{parent=Compiler}
= Binutils utility
{parent=Binutils}
= strings
{parent=Binutils utility}
{disambiguate=Binutils}
{tag=Good}
= Find UTF-8 strings with Binutils `strings`
{parent=Binutils utility}
{disambiguate=Binutils}
{tag=Good}
Not possible it seems:
* https://lists.gnu.org/archive/html/bug-binutils/2009-03/msg00140.html
* https://sourceware.org/bugzilla/show_bug.cgi?id=27551
* https://stackoverflow.com/questions/17470569/is-there-strings-command-for-utf-8
* https://stackoverflow.com/questions/7863986/gnu-binutils-strings-utf-8-instead-of-utf-16-or-ascii
= Automatic programming
{parent=Compiler}
{wiki}
"automatic programming has always been a euphemism for programming in a higher-level language than was then available to the programmer" sums it up.
The ultimate high level is of course to program with: "computer, make <money>", which is the goal of <artificial general intelligence>.
= Automatic code generation
{parent=Automatic programming}
= Code generation
{synonym}
= Lowering and raising
{parent=Compiler}
Lowering means translating to a lower level representation.
Raising means translating to a higher level representation.
<Decompilation> is basically a synonym, or subset, of raising.
= Lower
{c}
{parent=Lowering and raising}
{disambiguate=compilation}
= Decompiler
{c}
{parent=Lowering and raising}
{tag=Reverse engineering}
= Decompilation
{synonym}
= List of compilers
{c}
{parent=Compiler}
= GNU Compiler Collection
{c}
{parent=List of compilers}
{tag=GNU package}
{wiki}
= gcc
{c}
{synonym}
{title2}
= gcc CLI option
{c}
{parent=GNU Compiler Collection}
= gcc `-save-temps`
{parent=gcc CLI option}
Saves preprocessor output and generated assembly to separate files.
* preprocessor:
* https://stackoverflow.com/questions/3742822/preprocessor-output/76127046#76127046
* https://stackoverflow.com/questions/985403/seeing-expanded-c-macros/60816758#60816758
* https://stackoverflow.com/questions/4900870/can-gcc-output-c-code-after-preprocessing/55477371#55477371
* assembly:
* https://stackoverflow.com/questions/137038/how-do-you-get-assembler-output-from-c-c-source-in-gcc/56801917#56801917
= LLVM
{c}
{parent=List of compilers}
{wiki}
= LLVM Intermediate Representation
{c}
{parent=LLVM}
= LLVM IR
{c}
{synonym}
{title2}
Very hot stuff! It's like <ISA>-portable <assembly>, but with <type (programming)>[types]! In particular it also it deals with <calling conventions> for us (since it is ISA-portable). TODO: isn't that exactly what <C> does? :-) <LLVM IR vs C>
Documentation: https://llvm.org/docs/LangRef.html
= LLVM IR vs C
{parent=LLVM Intermediate Representation}
* https://www.reddit.com/r/ProgrammingLanguages/comments/72cl28/is_c_still_the_best_target_for_new_languages/
* https://stackoverflow.com/questions/10264635/compiler-output-language-llvm-ir-vs-c
= LLVM IR hello world
{parent=LLVM Intermediate Representation}
Example: \a[llvm/hello.ll] adapted from: https://llvm.org/docs/LangRef.html#module-structure but without double newline.
To execute it as mentioned at https://github.com/dfellis/llvm-hello-world[] we can either use their crazy assembly interpreter, tested on <Ubuntu 22.10>:
``
sudo apt install llvm-runtime
lli hello.ll
``
This seems to use `puts` from the <C standard library>.
Or we can <lower (compilation)> it to <assembly> of the local machine:
``
sudo apt install llvm
llc hello.ll
``
which produces:
``
hello.s
``
and then we can assemble link and run with <GCC>:
``
gcc -o hello.out hello.s -no-pie
./hello.out
``
or with <clang>:
``
clang -o hello.out hello.s -no-pie
./hello.out
``
`hello.s` uses the <GNU GAS> format, which <clang> is highly compatible with, so both should work in general.
= clang
{c}
{parent=LLVM}
<LLVM> front-end for <C> and related language like <C++> etc.
= Reproducible builds
{c}
{parent=Compiler}
{wiki}
Reproducible builds allow anyone to verify that a <binary large object> contains what it claims to contain!
Bibliography:
* https://stackoverflow.com/questions/14653874/how-to-produce-deterministic-binary-output-with-g/31019307#31019307
* https://stackoverflow.com/questions/19511356/how-to-make-android-applications-with-reproducible-builds
= Source-to-source compiler
{parent=Compiler}
{wiki}
= Transpiler
{synonym}
= Transpilation
{synonym}
= Transpile
{synonym}
= Transpiles
{synonym}
= Computer-aided design
{parent=Software}
{wiki}
= CAD
{c}
{synonym}
{title2}
= Open source CAD software
{parent=Computer-aided design}
= FreeCAD
{c}
{parent=Open source CAD software}
{wiki}
= Graphics software
{parent=Software}
{tag=Computer graphics}
{wiki}
https://en.wikipedia.org/wiki/List_of_information_graphics_software
= Mathematics illustration software
{parent=Graphics software}
= Software for drawing geometry diagrams
{synonym}
{title2}
Survey by <Ciro Santilli>: https://math.stackexchange.com/questions/1985/software-for-drawing-geometry-diagrams/3938216#3938216
Many <plotting software> can be used to create <mathematics> illustrations. They just tend to have more data-oriented rather than explanatory-oriented output.
Some notable ones:
* <Inkscape>{child}
= Graphics library
{c}
{parent=Graphics software}
{wiki}
= OpenGL
{c}
{parent=Graphics library}
{tag=Khronos standard}
{wiki}
<Ciro Santilli> has some good related articles listed under: <articles>.
= Freetype GL
{c}
{parent=OpenGL}
https://github.com/rougier/freetype-gl
Good library to render text in <OpenGL>, see also: https://stackoverflow.com/questions/8847899/opengl-how-to-draw-text-using-only-opengl-methods/36065835#36065835
= Khronos Group
{c}
{parent=OpenGL}
{wiki}
= Khronos
{c}
{synonym}
The fact that they kept the standard <open source> makes them huge heroes, see also: <closed standard>.
Shame that many (most?) of their proposals just die out.
= Khronos standard
{c}
{parent=Khronos group}
= opengl-tutorial.org
{parent=OpenGL}
https://github.com/opengl-tutorials/ogl/
Good modern <OpenGL> tutorial in retained mode with shaders, see also: https://stackoverflow.com/questions/6733934/what-does-immediate-mode-mean-in-opengl/36166310#36166310
= Direct3D
{c}
{parent=Graphics library}
{tag=Evil}
{wiki}
= JavaScript graphics library
{parent=Graphics library}
= Paper.js
{c}
{parent=JavaScript graphics library}
https://github.com/paperjs/paper.js
= Pixi.js
{c}
{parent=JavaScript graphics library}
https://github.com/pixijs/pixi.js
= Two.js
{c}
{parent=JavaScript graphics library}
https://github.com/jonobr1/two.js
Examples at: \a[two-js/].
<JavaScript> library, works both on browser and headless with <Node.js> to <SVG>.
Feels good. Maybe not ultra featured, and could have more simple examples in docs, but still good.
Vs <Paper.js> https://github.com/jonobr1/two.js/issues/319
One of the main features of Two.js appears to be the fact that it can natively render to either SVG and canvas, rather than creating SVG through DOM hacks as done by other projects.
= Computer program
{parent=Software}
{wiki}
= Program
{synonym}
One specific <software> project, typically with a single <executable file format> entry point.
= Computer security
{parent=Software}
{wiki}
As mentioned at <computer security researcher>{full}, <Ciro Santilli> really tends to like people from this area.
Also, the type of programming Ciro used to do, <systems programming>, is particularly useful to security researchers, e.g. <Linux Kernel Module Cheat>.
The reason he does not go into this is that Ciro would rather fight against the more eternal <laws of physics> rather than with some typo some dude at <Apple> did last week and which will be patched in a month.
= Exploit
{disambiguate=computer security}
{parent=Computer security}
{wiki}
= Arbitrary code execution
{parent=Exploit (computer security)}
{wiki}
= Phising
{parent=Exploit (computer security)}
{wiki}
= Cross-site scripting
{parent=Exploit (computer security)}
{wiki}
= XSS
{synonym}
{title2}
= Computer security conference
{parent=Computer security}
= DEF CON
{c}
{parent=Computer security conference}
{title2=1993-}
{wiki}
= Black Hat Briefings
{c}
{parent=DEF CON}
{title2=1997-}
{wiki}
= BlackHat
{c}
{synonym}
= Computer security researcher
{parent=Computer security}
{wiki}
<Ciro Santilli> found out that he likes computer security researchers and vice versa.
It's a bit the same reason why he likes <physicists>: you can't bullshit with security.
You can't just talk nice and hope for people to belive you.
You can't not try to break things and just <security through obscurity>[keep everyone happy in their false illusion of safety].
You can't do a half job.
If you do any of that, you \i[will] get your ass handed to you in a little gift bag.
All of this is closely linked to <Ciro Santilli's self perceived creative personality> and <being naughty and creative are correlated>.
= Dan Kaminsky
{c}
{parent=Computer security researcher}
{title2=1979-2021}
{wiki}
A superstar security researcher with some major exploits from in the 2000's.
= Dan Kaminsky approves Linux Kernel Module Cheat
{c}
{parent=Dan Kaminsky}
https://twitter.com/dakami/status/1344853681749934080
Oh yeah, that felt good. A few months before he died.
= Len Sassaman
{c}
{parent=Computer security researcher}
{wiki}
= Data erasure
{parent=Computer security}
{wiki}
= Denial-of-service attack
{parent=Computer security}
{title2=DoS}
{wiki}
= DoS attack
{c}
{synonym}
{title2}
{wiki}
= Denial-of-service
{synonym}
= Multi-factor authentication
{parent=Computer security}
{wiki}
= 2FA
{c}
{synonym}
{title2}
= 2FA app
{parent=Multi-factor authentication}
= Google 2FA app token can be updated without checking the old 2FA
{parent=2FA app}
Ermm, as of February 2021, I was able to update my 2FA app token with the password alone, it did not ask for the old 2FA.
So what's the fucking point of 2FA then? An attacker with my password would be able to login by doing that!
Is it that Google trusts that particular action because I used the same phone/known IP or something like that?
= Authy
{c}
{parent=2FA app}
= OAuth
{c}
{parent=Computer security}
{wiki}
The fatal flaw of OAuth is that websites have to enable specific providers, they can't just automatically select the correct OAuth for a given email domain. This means that the vast majority of websites will only provide the most widely popular providers such as <Google>, and the like, which means people won't have decent privacy.
So you are just better off with password logins and a decent <password manager>.
= Password
{parent=Computer security}
{wiki}
= Password manager
{parent=Password}
{wiki}
A cross browser, cross platform, and server-encrypted password manager is a must after <Snowden>!!! E.g. <Proton Pass>. And governments should obviously provide one to its citizens, or else be spied upon by the <USA> obviously: <Governments should provide basic Internet infrastructure>.
= Plausible deniability
{parent=Computer security}
{wiki}
= Privacy
{parent=Computer security}
{wiki}
= Security through obscurity
{parent=Computer security}
{wiki}
https://stackoverflow.com/questions/533965/why-is-security-through-obscurity-a-bad-idea
Do as I say, not as I do: <Ciro Santilli's Stack Overflow suspension for vote fraud script 2019>, https://meta.stackoverflow.com/questions/381577/is-it-ok-to-have-links-on-how-to-create-sock-puppets-and-gain-rep-fraudulently-i/381635#381635[].
\Video[https://www.youtube.com/watch?v=IH0GXWQDk0Q]
{title=LockPickingLawyer SAINTCON keynote (2021)}
{description=
SAINTCON is "Utah's Premiere Security Conference".
* https://youtu.be/IH0GXWQDk0Q?t=900 mentions that https://en.wikipedia.org/wiki/Alfred_Charles_Hobbs[Alfred Charles Hobbs] commented in 1853:
\Q[Rogues are very keen in their profession, and know already much more than we can teach them]
}
= Kerckhoffs's principle
{c}
{parent=Security through obscurity}
{wiki}
Basically the opposite of <security through obscurity>, though slightly more focused on <cryptography>.
= Malware
{parent=Computer security}
{wiki}
= Ransomware
{parent=Computer security}
{wiki}
= Data breach
{parent=Computer security}
{wiki}
= WikiLeaks
{c}
{parent=Data breach}
{tag=Wiki}
{wiki}
= List of data breaches
{parent=Data breach}
= United States diplomatic cables leak
{c}
{parent=List of data breaches}
{title2=2010}
{wiki}
= Cablegate
{c}
{synonym}
= Computer user-interface
{parent=Software}
= Application programming interface
{parent=Computer user-interface}
{wiki}
= API
{c}
{synonym}
{title2}
= REST API
{c}
{parent=Application programming interface}
{wiki}
= GraphQL
{c}
{parent=Application programming interface}
{wiki}
This is really good.
It allows the client to prepare a single request that gets all the data it wants to fill up a given webpage, rather than doing several separate requests.
So it only gets exactly what it needs, and in a single request.
Very sweet. This is the future of the web.
= Command-line interface
{parent=Computer user-interface}
{wiki}
= CLI
{c}
{synonym}
{title2}
= Command-line
{synonym}
= Linux CLI HOWTO
{c}
{parent=Command-line interface}
= Convert bytes to hex from Linux CLI
{parent=Linux CLI HOWTO}
* no formatting;
* https://stackoverflow.com/questions/2614764/how-to-create-a-hex-dump-of-file-containing-only-the-hex-characters-without-spac
* https://unix.stackexchange.com/questions/10826/shell-how-to-read-the-bytes-of-a-binary-file-and-print-as-hexadecimal/758531#758531
* https://stackoverflow.com/questions/2003803/show-hexadecimal-numbers-of-a-file/77262369#77262369
* https://stackoverflow.com/questions/9515007/linux-script-to-convert-byte-data-into-a-hex-string/77262375#77262375
= Command line utility
{parent=Command-line interface}
= CLI tool
{c}
{synonym}
{title2}
= List of command line utilities
{parent=Command line utility}
= GNU parallel
{parent=List of command line utilities}
{tag=Good}
{tag=GNU package}
{wiki}
The author Ole Tange answers every question about it on <Stack Exchange>. What a legend!
This program makes you respect <GNU make> a bit more. Good old make with `-j` can not only parallelize, but also take in account a <dependency graph>.
Some examples under:
``
man parallel_exampes
``
To get the input argument explicitly job number use the magic string `{}`, e.g.:
``
printf 'a\nb\nc\n' | parallel echo '{}'
``
sample output:
``
a
b
c
``
To get the job number use `{#}` as in:
``
printf 'a\nb\nc\n' | parallel echo '{} {#}'
``
sample output:
``
a 1
b 2
c 3
c 3
``
`{%}` contains which thread the job running in, e.g. if we limit it to `2` threads with `-j2`:
``
printf 'a\nb\nc\nd\n' | parallel -j2 echo '{} {#} {%}'
``
sample output:
``
a 1 1
b 2 1
c 3 2
d 4 1
``
The percent must be a reference to "split the inputs module the number of workers", and modulo uses the `%` symbol in many programming languages such as <C (language)>.
To pass multiple CLI argments per command you can use `-X` e.g.:
``
printf 'a\nb\nc\nd\n' | parallel -j2 -X echo '{} {#} {%}'
``
sample output:
``
a b 1 1
c d 2 2
``
= htop
{c}
{parent=List of command line utilities}
{tag=Good}
{wiki}
= less
{c}
{disambiguate=Unix}
{parent=List of command line utilities}
{wiki}
= ncdu
{parent=List of command line utilities}
{tag=Good}
{tag=ncurses program}
{wiki}
Way too few people know about this. Spread the word.
https://stackoverflow.com/questions/1019116/using-ls-to-list-directories-and-their-total-sizes/55519414#55519414
\Image[https://web.archive.org/web/20221208132133if_/https://i.stack.imgur.com/lHXP3.png]
= rsync
{c}
{parent=List of command line utilities}
{wiki}
= sudo
{c}
{parent=List of command line utilities}
{wiki}
Availability: https://unix.stackexchange.com/questions/48522/how-universal-is-sudo
https://xkcd.com/149/
= Text-based user interface
{parent=Command-line interface}
{tag=Good}
{title2=TUI}
{wiki}
The perfect <Middle Way> between <command-line interfaces> and <GUIs>. A thing of great beauty.
= ncurses
{c}
{parent=Text-based user interface}
{wiki}
= ncurses program
{parent=ncurses}
= Graphical user interface
{parent=Computer user-interface}
{wiki}
= GUI
{c}
{synonym}
{title2}
= Display manager
{parent=Graphical user interface}
{wiki}
Check which you you have:
``
systemctl status display-manager.service
``
Tested on <Ubuntu 23.10> I see:
``
● gdm.service - GNOME Display Manager
Loaded: loaded (/lib/systemd/system/gdm.service; static)
Active: active (running) since Sun 2023-12-24 10:34:50 GMT; 23min ago
Process: 1827 ExecStartPre=/usr/share/gdm/generate-config (code=exited, status=0/SUCCESS)
Main PID: 1850 (gdm3)
Tasks: 4 (limit: 71817)
Memory: 6.8M
CPU: 119ms
CGroup: /system.slice/gdm.service
└─1850 /usr/sbin/gdm3
``
which means I have <GNOME Display Manager>.
Bibliography:
* https://unix.stackexchange.com/questions/20370/is-there-a-simple-linux-command-that-will-tell-me-what-my-display-manager-is
* https://askubuntu.com/questions/584373/how-to-check-using-the-command-line-which-display-manager-is-running
= GNOME Display Manager
{c}
{parent=Display manager}
{title2=GDM}
{wiki}
= Desktop environment
{parent=Graphical user interface}
{wiki}
<tmux> for newbs: <terminal multiplexers are CLI desktop environments>{full}.
= GNOME Project
{c}
{parent=Desktop environment}
{title2=1997}
{wiki}
= GNOME desktop
{c}
{parent=GNOME Project}
= KDE
{c}
{parent=Desktop environment}
{title2=1996}
{wiki}
= Splash screen
{parent=Graphical user interface}
{wiki}
= Qt
{disambiguate=software}
{c}
{parent=Graphical user interface}
{wiki}
= WYSIWYG
{c}
{parent=Graphical user interface}
{wiki}
= Data compression
{parent=Software}
{wiki}
= Lossless and lossy compression
{parent=Data compression}
{wiki}
= Lossy compression
{parent=Lossless and lossy compression}
{wiki}
= Lossless compression
{parent=Lossless and lossy compression}
{wiki}
= Lossless
{synonym}
= Database
{parent=Software}
{wiki}
= ACID
{disambiguate=database}
{c}
{parent=Database}
{wiki}
= Atomicity
{disambiguate=database systems}
{parent=ACID (database)}
This means that e.g. if you do an `UPDATE` query on multiple rows, and power goes out half way, either all update, or none update.
This is different from <isolation (database systems)>, which considers instead what can or cannot happen when multiple queries are running in parallel.
= Isolation
{disambiguate=database systems}
{parent=ACID (database)}
Determines what can or cannot happen when multiple queries are running in parallel.
See <SQL transaction isolation level>{full} for the most common context under which this is discussed: <SQL>.
= Database management system
{parent=Database}
{wiki}
= DBMS
{c}
{synonym}
A <software> that implements some database system, e.g. <PostgreSQL> or <MySQL> are two (widely extended) <SQL> implementations.
= NoSQL
{c}
{parent=Database}
{wiki}
= LevelDB
{c}
{parent=NoSQL}
{wiki}
One "LevelDB" database contains multiple file in a directory. Off the bat inferior to <SQLite> which stores everything in a single file!
= Dump LevelDB
{parent=LevelDB}
= LevelDBDumper
{c}
{parent=Dump LevelDB}
https://github.com/mdawsonuk/LevelDBDumper
https://github.com/mdawsonuk/LevelDBDumper/tree/e750a27ff58443ecc410b5c16abbdc539d617387#installation worked on <Ubuntu 23.10.> Annoying installation, but worked: https://github.com/mdawsonuk/LevelDBDumper/issues/13
Initial issues off-the-bat:
* https://github.com/mdawsonuk/LevelDBDumper/issues/13
* https://github.com/mdawsonuk/LevelDBDumper/issues/14
* https://github.com/mdawsonuk/LevelDBDumper/issues/15
= MongoDB
{c}
{parent=NoSQL}
{wiki}
List databases:
``
echo 'show dbs' | mongo
``
Delete database:
``
use mydb
db.dropDatabase()
``
or:
``
echo 'db.dropDatabase()' | mongo mydb
``
View collections within a database:
``
echo 'db.getCollectionNames()' | mongo mydb
``
Show all data from one of the collections: https://stackoverflow.com/questions/24985684/mongodb-show-all-contents-from-all-collections
``
echo 'db.collectionName.find()' | mongo mydb
``
= Install MongoDB on Ubuntu
{parent=MongoDB}
Tested as of Ubuntu 20.04, there is no Mongo package available by default due to their change to <Server Side Public License>, which Debian opposed. Therefore, you have to add their custom PPA as mentioned at: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
= Object-relational mapping
{parent=Database}
{wiki=Object–relational_mapping}
= ORM
{c}
{synonym}
{title2}
Per language:
* <node.js ORM library>
= How to decide if an ORM is good?
{parent=Object-relational mapping}
How to decide if an ORM is decent? Just try to replicate every <SQL> query from \a[nodejs/sequelize/raw/many_to_many.js] on <PostgreSQL> and <SQLite>.
There is only a very finite number of possible reasonable queries on a two table many to many relationship with a join table. A decent ORM \i[has] to be able to do them all.
If it can do all those queries, then the ORM can actually do a good subset of SQL and is decent. If not, it can't, and this will make you suffer. E.g. <Sequelize> v5 is such an ORM that makes you suffer.
The next thing to check are transactions.
Basically, all of those come up if you try to implement a blog <hello world> world such as <gothinkster realworld> \i[correctly], i.e. without unnecessary inefficiencies due to your ORM on top of underlying SQL, and dealing with concurrency.
= Relational database
{parent=Database}
{wiki}
= Relational database management system
{parent=Relational database}
{tag=Database management system}
= RDBMS
{c}
{synonym}
{title2}
\Include[sql]{parent=relational-database-management-system}
= Database feature
{parent=Database}
= Database trigger
{parent=Database feature}
{wiki}
= Stored procedure
{parent=Database feature}
{wiki}
= Table relationship
{parent=Database feature}
{wiki=Associative_entity}
= One-to-one
{disambiguate=data-model}
{parent=Table relationship}
{wiki}
= One-to-many
{parent=Table relationship}
{wiki=One-to-many_(data_model)}
= Many-to-one
{synonym}
{title2}
= Many-to-many
{parent=Table relationship}
{wiki=Many-to-many_(data_model)}
= Many-to-many relation
{synonym}
= File manager
{parent=Software}
{wiki}
<Ciro Santilli> used to use file managers in the past.
But he finally converted to a shell `cd` aliases that auto-`ls`: https://github.com/cirosantilli/dotfiles/blob/a51bcc324f0cff0eddd4c3bb8654ec223a0adb7b/home/.bashrc#L1058
= Krusader
{parent=File manager}
{wiki}
The most powerful <GUI> <file manager> ever?? Infinite configurability??
<Ciro Santilli> wasted some time on it before he gave up on file managers altogether.
= Ranger
{disambiguate=file manager}
{parent=File manager}
{wiki}
<Ciro Santilli> considered it before he stopped using file managers altogether, it is not bad.
= File sharing
{parent=Software}
{wiki}
= BitTorrent
{c}
{parent=File sharing}
{wiki}
= Game engine
{parent=Software}
{wiki}
A library to make <games>{parent}.
= Cocos2d
{c}
{parent=Game engine}
{wiki}
<Ciro Santilli> considered this as the basis for <Ciro's 2D reinforcement learning games>, but ultimately decided it was a bit too messy. Nice overall though.
= RPG Maker
{c}
{parent=Game engine}
{wiki}
The one true game engine!
\Video[https://www.youtube.com/watch?v=kHoCSK1ikfI]
{title=Reviewing a Bunch of RPG Maker Games by Majuular (2022)}
= Unity
{c}
{disambiguate=Game engine}
{parent=Game engine}
{wiki}
= Unity
{c}
{synonym}
= Unity-based software
{c}
{parent=Unity (Game engine}
= Urho3D
{c}
{parent=Game engine}
{wiki}
Their project lead as of 2018 was pro-CCP: https://github.com/cirosantilli/china-dictatorship/blob/aa1176c57fc2929465294e520b43b50d44e202ba/communities-that-censor-politics.md
= Geographic information system
{parent=Software}
{title2=GIS}
{wiki}
= Geographic information system file format
{parent=Geographic information system}
= Keyhole Markup Language
{c}
{parent=Geographic information system file format}
{title2=KML}
{wiki}
Originally by <Keyhole Inc.>, which the nbecame <Google Maps>, but the format seems standardized and has non-Google support, so should be OK.
= List of geographic information systems
{parent=Geographic information system}
= Google Maps
{c}
{parent=List of geographic information systems}
{tag=Google product}
{wiki}
Owned/developed by <Google>{parent} as of 2020.
Early on jumpstarted from several acquisitions, notably <Keyhole Inc.> and <Where 2 Technologies>.
= Keyhole Inc.
{c}
{parent=Google Maps}
= Where 2 Technologies
{c}
{parent=Google Maps}
= Google Street View
{c}
{parent=Google Maps}
{wiki}
Street View's go into the past mode is the dream of every archaeologist. Ciro can only dream of a magic street view that allows going back to earlier centuries and beyond... isn't it amazing to think that people in the future will have that ability to time travel back to around the year 2006? Ciro wonders how long <Google> will be able to keep storing data like that.
Thanks, <Keyhole Inc.>[CIA].
= OpenStreetMap
{c}
{parent=List of geographic information systems}
{title2=OSM}
{wiki}
It is rare to find a project with such a ridiculously high importance over funding ratio.
E.g., as of 2020, their help login https://help.openstreetmap.org/ shows MyOpenID as an option, which was discontinued in 2014, and not <Google> <OAuth>.
They do still seem to have a bit more activity than https://gis.stackexchange.com/questions/tagged/openstreetmap on <Stack Exchange>.
Complaints:
* <Transliteration> is off by default!...... https://wiki.openstreetmap.org/wiki/Translation You just have to learn all scripts ever. Good luck with the <Chinese characters>. Genius.
* In order to see information about places, you have to click "Query features" on the toolbar first. Who made such a terrible UI? Direct click is a much, and so easy to implement?
* It is impossible to discern different types of paths and other walking path symbols, the symbols are too small, and just scale down to a line no matter how much you zoom in.
* Power lines are way too visible. While that is kind of cool, it is useless and distracting to most people most of the time.
* No street-level imagery...: https://help.openstreetmap.org/questions/1178/adding-photos
* No aerial imagery: https://help.openstreetmap.org/questions/6849/how-can-i-see-the-aerial-imagery-without-editing-the-map But that is kind of understandable, as that one might not be free.
* No restaurant ratings: https://help.openstreetmap.org/questions/64852/ratings-for-pois because it is "Subjective". OMG those people, such a huge value powerhouse wasted.
Not just for restaurants, but for other things as well, e.g. sharing of good cycle circuits.
All of this is a shame, because they do have some incredible data that you cannot find easily on other maps because people just edited it up.
= OsmAnd
{c}
{parent=OpenStreetMap}
{wiki}
https://github.com/osmandapp/Osmand
Kind of works! Notably, has the amazing cycling database offline for you, if you fall within the 6 area downloads. It is worth supporting these people beyond the 6 free downloads however.
= Ordnance Survey
{c}
{parent=List of geographic information systems}
{wiki}
Has some of the best map data available for the <United Kingdom>, but their data appears to be proprietary?
= Integrated development environment
{parent=Software}
{wiki}
= IDE
{c}
{synonym}
{title2}
IDEs are absolutely essential for developing complex software.
The funny thing is that you don't notice this until someone shows it to you. But once you see it, there is not turning back, just like <Steve Jobs customers don't know what they want quote>.
Unfortunately, after the https://movingfulcrum.com/the-fall-of-eclipse/[Fall of Eclipse] (https://web.archive.org/web/20190824081229/https://movingfulcrum.com/the-fall-of-eclipse/[archive]), the IDE landscape in 2019 is horrible and split between:
* highly buggy but still feature rich Eclipse
* many may many other feature lacking options using possibly more trendy and forward lasting implementations like https://en.wikipedia.org/wiki/Electron_(software_framework)[Electron]
* awesome cross-platform proprietary https://en.wikipedia.org/wiki/JetBrains[JetBrains] IDEs
* the God-like Windows-only proprietary language-lacking Visual Studio
Programmers of the world: unite! Focus on one IDE, and make it work for all languages and all build systems. Give it all the features that Eclipse has, but none of the bugginess. Work with top project to make sure the IDE works for all top projects.
Projects of the world: support one IDE, with in-tree configuration. Complex integration is often required between the IDE and the build system, and successful projects must to that once for all developers. Either do this, or watch you complex project wither away.
Build tool maintainers: make it possible for IDEs to support your tool! E.g., implement https://clang.llvm.org/docs/JSONCompilationDatabase.html[JSON Compilation Database] output so that IDEs can read the exact compiler commands from that, in order to automatically determine how files should be parsed! Or better, just use libllvm in your IDE itself as the main parser.
Ciro is evaluating some IDEs at: https://github.com/cirosantilli/ide-test-projects
= Text editor
{c}
{parent=Integrated development environment}
{wiki}
= JavaScript text editor
{c}
{parent=Text editor}
{tag=JavaScript library}
= Monaco
{c}
{disambiguate=editor}
{parent=JavaScript text editor}
Either extracted from, or designed for, <VSCode> by <Microsoft>:
* https://github.com/microsoft/monaco-editor
* https://microsoft.github.io/monaco-editor/
However also at the same time very limited integration with <VSCode>, that makes using it for VScode compatibility almost useless, e.g.:
* you can't reuse the syntax defintions!
* https://stackoverflow.com/questions/37936919/writing-visual-studio-code-syntaxes-in-monarch
* https://github.com/Microsoft/vscode/issues/216
= WYSIWYG text editor
{c}
{parent=Text editor}
{tag=WYSIWYG}
= JavaScript WYSIWYG text editor
{c}
{parent=WYSIWYG text editor}
{tag=JavaScript text editor}
= TinyMCE
{c}
{parent=JavaScript WYSIWYG text editor}
{wiki}
= Vim
{c}
{parent=Integrated development environment}
{wiki=Vim_(text_editor)}
Before we get a decent open source <integrated development environment>, what else can you do?
But also perfect for small one-off files when you don't have the patience to setup said <IDE>.
vim's defaults are atrocious for the 21st century! Vundle is reasonable as an ad-hoc package manager, but it can't set fixed versions of packages:
* https://stackoverflow.com/questions/15259868/vundle-plugin-install-concrete-version-of-plugin/67870413#67870413
* https://github.com/VundleVim/Vundle.vim/pull/681
= Gvim
{c}
{parent=Vim}
= vader.vim
{parent=Vim}
https://github.com/junegunn/vader.vim
Vimscript <unit testing>!!!
= plasticboy/vim-markdown
{parent=vader.vim}
https://github.com/plasticboy/vim-markdown
<Ciro Santilli> contributed a bit to this, and was even given push rights, see also: see also: <Ciro Santilli's minor projects>.
= honza/vim-snippets
{parent=vader.vim}
https://github.com/honza/vim-snippets
= Vimium
{parent=vader.vim}
https://vimium.github.io/
Since you can't escape <shitty> browser <GUIs> and live in the command line, the next best thing you can do is to bring Vim bindings to your browser :-)
There is one major annoyance: you can't use ESC to leave the address bar focus, but using Tab as a workaround works:
* https://superuser.com/a/1560178/128124
= Eclipse
{c}
{disambiguate=IDE}
{parent=Integrated development environment}
{wiki=Eclipse_(software)}
Once upon a time (early 2010's), Eclipse dominated the IDE landscape and all was good. NetBeans was around too. And <Java> was still unmarred by <Google LLC v. Oracle America, Inc.>.
But then something happened.
For some reason, Eclipse started to decay.
And the project that had once been a vibrant community of awesomeness, started to become... a zombie of its former self.
Buggyness started increasing. And not even hard to fix bugs. One liners that affect every user immediately after startup.
Sometimes, to Eclipse's defense they weren't "bugs". Just features that it became evident with time every programmer expected from a modern IDE.
But somehow the Eclipse community had a deep problem. A cancer. It had completely lost touch with user experience.
Perhaps is was due to the increasing interest of the several corporations that had adopted Eclipse as the base IDE for the proprietary solutions?
Perhaps.
Many users stuck to the IDE.
Some heroic efforts were made as plugins that drastically improved certain defects. The Darkest Dark plugin comes to mind.
But all those efforts required configuration. A setup time that most users simply don't have. The core devteam had become dumb and dead, unable to incorporate such changes.
This greatly opened up the space for other competing IDEs to come along. The "semi feature complete but at least easy to use and not so buggy" <Visual Studio Code> and the proprietary <JetBrains> IDEs being some of the most notable ones.
Using Eclipse as of the early 2020's is such a mixed experience. If you spend enough time to configure out the key buggyness, there are moments where you can feel "OMG, this feature is amazing".
But the effort is just too great, and soon another bug or obvious missing feature hits you and brings you back to reality.
Every young person uses <VS Code> now. Eclipse is dead, and there is no way back, usage will just continue dropping.
RIP, Eclipse. It wasn't meant to be.
Bibliography:
* https://movingfulcrum.com/the-fall-of-eclipse/ attributes the fall to Eclipse 4
* https://www.reddit.com/r/programming/comments/8f0xns/the_fall_of_eclipse/
* https://www.quora.com/Why-do-people-still-use-Eclipse-IDE
\Image[https://www.jrebel.com/wp-content/uploads/2016/07/intellij-idea-overtakes-eclipse.png]
{title=<Eclipse (IDE)>. usage from 2012 to 2016 according to a JRebel survey}
{source=https://www.jrebel.com/blog/java-trends-and-historical-data}
= JetBrains
{c}
{parent=Integrated development environment}
{wiki}
= Visual Studio Code
{c}
{parent=Integrated development environment}
= VS Code
{c}
{synonym}
{title2}
= vscode
{c}
{synonym}
{title2}
= vscode bug
{parent=Visual Studio Code}
Persistent undo history:
* https://github.com/microsoft/vscode/issues/43555
* https://stackoverflow.com/questions/67248384/how-to-keep-undo-history-after-closing-vscode
= vscode Vim
{c}
{parent=Visual Studio Code}
* `undo` is broken beyond belief: https://github.com/VSCodeVim/Vim/issues/1490
= vscode jump to definition broken
{parent=Visual Studio Code}
It is especially bad on large projects, unless you carefully whitelist only the small source directories:
* https://stackoverflow.com/questions/37341849/vscode-go-to-definition-not-working/76193270#76193270
* https://stackoverflow.com/questions/62706944/vscode-go-to-definition-not-working-only-in-big-projects/76193292#76193292
https://stackoverflow.com/questions/30118107/vscode-intellisense-not-working
= Killer application
{parent=Software}
{wiki}
= Killer app
{synonym}
= Memory management
{parent=Software}
{wiki}
= Garbage collection
{disambiguate=computer science}
{parent=Software}
{wiki}
= Garbage collection
{synonym}
\Include[messaging-software]{parent=software}
= Multimedia software
{parent=Software}
= FFmpeg
{c}
{parent=Multimedia software}
{wiki}
FFmpeg is the <assembler (computing)> of audio and video.
As a result, <Ciro Santilli> who likes "lower level stuff", has had many many hours if image manipulation fun with this software, see e.g.:
* the "Media" section of <articles>.
* <image Ciro knows how to convert videos to GIFs>
As older Ciro grows, the more he notices that FFmpeg can do basically any lower level audio video task. It is just an amazing piece of software, the immediate go-to for any low level operation.
FFmpeg was created by <Fabrice Bellard>, which Ciro deeply respects.
Resize a video: https://superuser.com/questions/624563/how-to-resize-a-video-to-make-it-smaller-with-ffmpeg[]:
``
ffmpeg -i input.avi -filter:v scale=720:-1 -c:a copy output.mkv
``
Unlike every other convention under the sun, the height in `scale` is the first number.
= FFmpeg filter graph
{c}
{parent=FFmpeg}
Filter graphs are a thing of great beauty. What an amazingly obscure <domain-specific language>, but which can produce striking results with very little!!!
A quick example from https://stackoverflow.com/questions/59551013/how-to-generate-stereo-sine-wave-using-ffmpeg-with-different-frequencies-for-eac/77730492#77730492 illustrates some of the fundamentals:
``
ffplay -autoexit -nodisp -f lavfi -i '
sine=frequency=500[a];
sine=frequency=1000[b];
[a][b]amerge, atrim=end=2
'
``
which creates a graph:
``
+--------+
[sine=frequency=500]--->[a]-->| |
| amerge |-->[atrim]-->[output]
[sine=frequency=1000]-->[b]-->| |
+--------+
``
and plays 500 Hz on the left channel and 1000 Hz on the right channel for 2 seconds.
So we see the following syntax patterns:
* `sine`, `amerge` and `atrim` are filters
* `sine=frequency=500`: the first `=` says "araguments follow"
* `frequency=500` sets the `frequency` argument of the `sine` filter
* for multiple arguments the syntax is to separate arguments with colons e.g. `sine=frequency=500:duration=2`
* `;`: separates statements
* `[a]`, `[b]`: sets the name of an edge
* `,`: creates unnamed edge between filters that have one input and one output
A list of all filters can be obtained ith:
``
ffmpeg -filters
``
and parameters for a single filter can be obtained with:
``
ffmpeg --help filter=sine
``
Related question: https://stackoverflow.com/questions/69251087/in-ffmpeg-command-line-how-to-show-all-filter-settings-and-their-parameters-bef
TODO dump graph to <ASCII art>? https://trac.ffmpeg.org/wiki/FilteringGuide#Visualizingfilters mentions a `-dumpgraph` option, but haven't managed to use it yet.
Bibliography:
* https://ffmpeg.org/ffmpeg-filters.html official documentation
* https://trac.ffmpeg.org/wiki/FilteringGuide some handy tips from the FFMpeg Wiki
= ffplay
{c}
{parent=FFmpeg}
Awesome tool to view quick stuff quickly without generating files. Unfortunately it doesn't support all options that the ffmpeg CLI supports, e.g. <ffplay multiple input files>. One day, one day.
= ffplay multiple input files
{c}
{parent=ffplay}
TODO possible? https://superuser.com/questions/559768/ffplay-how-to-play-together-separate-video-and-audio-files
For synthesized streams like `sine` we can do it e.g.
``
ffplay -autoexit -nodisp -f lavfi -i '
sine=frequency=500[a];
sine=frequency=1000[b];
[a][b]amerge, atrim=end=2
'
``
but it does not seem to accept multiple `-i` for some reason. So is there a way to open a file from some filter? E.g.:
``
ffplay -i tmp.wav -i tmp.mkv -filter_complex "[0:a]atrim=end=2[a];[1:v]trim=end=2[v]" -map '[a]' -map '[v]'
``
fails with:
``
Argument 'tmp.mkv' provided as input filename, but 'tmp.wav' was already specified.
``
= FFmpeg sound synthesis
{c}
{parent=FFmpeg}
Simple <sines> and variants:
* https://unix.stackexchange.com/questions/82112/stereo-tone-generator-for-linux/536860#536860
* https://stackoverflow.com/questions/5109038/linux-sine-wave-audio-generator/57610684#57610684
* https://superuser.com/questions/724391/how-to-generate-a-sine-wave-with-ffmpeg
* https://stackoverflow.com/questions/59551013/how-to-generate-stereo-sine-wave-using-ffmpeg-with-different-frequencies-for-eac/77730492#77730492
2 second 1000 Hz:
``
ffmpeg -f lavfi -i "sine=f=1000:d=2" out.wav
``
= FFmpeg video synthesis
{c}
{parent=FFmpeg}
Video with a solid color:
* 2 second white video:
``
ffplay -autoexit -f lavfi -i 'color=white:640x480:d=3,format=rgb24,trim=end=2'
``
Also add some audio:
``
ffmpeg -lavfi "color=white:640x480:d=3,format=rgb24,trim=end=2[v];sine=f=1000:d=2[a]" -map '[a]' -map '[v]' out.mkv
``
TODO how to ffplay the video + audio directly? `-map` does not seem to work unfortunately.
* 2 second white followed by 2 second black video:
``
ffplay -autoexit -f lavfi -i 'color=white:640x480:d=3,format=rgb24,trim=end=2[a];color=black:640x480:d=3,format=rgb24,trim=end=2[b];[a][b]concat=n=2:v=1:a=0'
``
* bibliography:
* https://superuser.com/questions/1153930/how-to-generate-a-pure-white-background-video-with-ffmpeg
Display count in seconds on the video:
* black text on white background. Start from 0 and count up to 2:
``
ffplay -autoexit -f lavfi -i "
color=white:480x480:d=3,
format=rgb24,
drawtext=
fontcolor=black:
fontsize=600:
text='%{eif\:t\:d}':
x=(w-text_w)/2:
y=(h-text_h)/2
"
``
* bibliography:
* https://gist.github.com/derand/31b8312fd64156120cb8f45825a1f0f7
* https://www.reddit.com/r/ffmpeg/comments/11kxugt/make_a_video_timer_or_countdown/
= FFmpeg is the backend of YouTube
{c}
{parent=FFmpeg}
FFmpeg is likely the backend of <YouTube> through <reverse engineering>: https://streaminglearningcenter.com/blogs/youtube-uses-ffmpeg-for-encoding.html (https://web.archive.org/web/20190519135210/https://streaminglearningcenter.com/blogs/youtube-uses-ffmpeg-for-encoding.html[archive])
On <Quora>: https://www.quora.com/What-does-YouTube-use-for-encoding-video/answer/Ciro-Santilli
= Concatenate two videos with ffmpeg
{parent=FFmpeg}
https://stackoverflow.com/questions/7333232/how-to-concatenate-two-mp4-files-using-ffmpeg
= ImageMagick
{c}
{parent=Multimedia software}
{wiki}
Crop `20` pixels from the bottom of the image:
``
convert image.png -gravity East -chop 20x0 result.png
``
= ImageMagick HOWTO
{parent=ImageMagick}
= Join two images side-by-side
{parent=ImageMagick HOWTO}
https://stackoverflow.com/questions/20737061/merge-images-side-by-sidehorizontally/63575228#63575228
= Crate solid color image in ImageMagick
{parent=ImageMagick HOWTO}
``
convert -size 512x512 xc:blue blue.png
``
Bibliography:
* https://stackoverflow.com/questions/39504522/create-blank-image-in-imagemagick
= Create gradient image with ImageMagick
{parent=Crate solid color image in ImageMagick}
https://imagemagick.org/script/gradient.php
``
convert -size 256x256 gradient: out.png
convert -size 256x256 gradient:white-black out.png
convert -size 256x256 gradient:red-blue out.png
convert -size 256x256 radial-gradient: out.png
convert -size 256x256 radial-gradient:white-black out.png
``
= Crate image with given text in ImageMagick
{parent=ImageMagick HOWTO}
Digits 0 to 9, white on black background:
``
for i in `seq 0 9`; do convert -size 512x512 xc:black -pointsize 500 -gravity center -fill white -draw "text 0,0 \"$i\"" $i.png; done
``
Bibliography:
* https://stackoverflow.com/questions/67012057/how-to-generate-an-image-with-a-number-in-it
= Open source software
{parent=Software}
{wiki}
= Open source
{synonym}
= Open sourced
{synonym}
What happens when https://en.wikipedia.org/wiki/OpenCL[the underdogs] get https://stackoverflow.com/questions/1780599/what-is-the-meaning-of-posix/31865755#31865755[together] and try to factor out their efforts to beat some https://en.wikipedia.org/wiki/CUDA[evil] dominant <Microsoft Windows>[power], sometimes <Linux>[victoriously].
Or when https://github.com/facebook[startups] use the cheapest stuff available and randomly become the https://github.com/google[next big thing], and decide to keep maintaining the open stuff to get features for free from other companies, or because they are <Linux>[forced by the Holy GPL].
Open source frees employees. When you change jobs, a large part of the specific knowledge you acquired about closed source a project with your blood and tears goes to the trash. When companies get bought, projects get shut down, and closed source code goes to the trash. What sane non desperate person would sell their life energy into such closed source projects that could die at any moment? Working on open source is the single most important non money perk a company can have to attract the best employees.
Open source is worth more than the mere pragmatic financial value of not having to pay for software or the ability to freely add new features.
Its greatest value is perhaps the fact that it allows people study it, to appreciate the beauty of the code, and feel empowered by being able to add the features that they want.
That is why <Ciro Santilli> thought:
\Q[Life is too short for closed source.]
But quoting Ciro's colleague S.:
\Q[Every software is open source when you read https://en.wikipedia.org/wiki/Assembly_language[assembly code].]
And https://computergraphics.stackexchange.com/questions/7809/what-does-gpu-assembly-look-like["can <reverse engineering>[reverse engineer] the undocumented GPU hardware APIs"], <Ciro Santilli>[Ciro] would add.
While software is the most developed open source technology available in the 2010's, due to the https://www.vox.com/recode/2019/6/24/18715421/internet-free-data-ads-cost["zero cost"] of copying it over the Internet, Ciro also believes that <open knowledge>[the world would benefit enormously from open source knowledge in all areas on science and engineering], for the same reasons as open source.
= Open source advocacy entity
{parent=Open source software}
{wiki}
= GNU Project
{c}
{parent=Open source advocacy entity}
{wiki}
= GNU package
{c}
{parent=GNU Project}
= Open source hardware
{parent=Open source software}
{wiki}
= Free and open-source software
{parent=Open source software}
{wiki}
= FOSS
{c}
{synonym}
{title2}
A more precise term for those in the know: <open source software> that also has a liberal license, for some definition of liberal.
<Ciro Santilli> defines liberal as: "can be commercialized without paying anything back" (but possibly subject to other restrictions).
He therefore does not consider <Creative Commons licenses> with NC to be FOSS.
For the newbs, the term <open source software> is good enough, since most open source software is also FOSS.
But when it's not, it's crucial to know.
= Models for financing open source software
{parent=Free and open-source software}
* <sponsorware>{child}
= Software developer collective
{parent=Models for financing open source software}
This model can work well when there is a set of commonly used libraries that some developers often use together, but such that there isn't enough maintenance work for each one individually.
So what people do is to create a group that maintains all those projects, to try and get enough money to survive from the contributions done primarily for each one individually.
Examples:
* https://github.com/pmndrs
= Open source software that is not-FOSS
{parent=Free and open-source software}
{wiki}
* <GitLab>{child}
= Source code leaks
{parent=Open source software}
* <Nintendo>
* 2020-05-04 leaks via the BroadOn company
* items
* Wii <Verilog>
* coverage
* https://www.ladbible.com/technology/gaming-the-biggest-nintendo-hack-in-history-leaks-console-source-codes-20200504
* fake leak commentary
* "Sebastian" is apparently reuploaded devkit demos claiming that they are from this leak, if that is correct, <fuck> him:
* https://www.youtube.com/watch?v=6FSOFrgbi4I&lc=UgwWEWT3oyxf85J0PQB4AaABAg
* https://www.youtube.com/watch?v=teRRcouGxwc&list=PL8Yms2YDl4qOXI99jpI8oH75d3X2oxnBM&index=14
= Open knowledge
{parent=Open source software}
{wiki}
<Ciro Santilli>'s raison d'etre, one of his attempts: <OurBigBook.com>.
The outcome of closed knowledge is <reverse engineering>.
= Open educational resources
{parent=Open knowledge}
{title2=OER}
{wiki}
= Open educational content
{synonym}
Projects:
* <MIT OpenCourseWare>
* several <e-learning websites>, e.g. <OpenStax>
* https://www.oeglobal.org/
= Open Textbook Library
{c}
{parent=Open educational resources}
https://open.umn.edu/opentextbooks/
= OpenCourseWare
{c}
{parent=Open educational resources}
{wiki}
= Horrors of open source
{parent=Open source software}
Not everything is perfect.
One big problem of many big open source projects is that they are contributed to by separate selfish organizations, that have private information. Then what happens is that:
* people implement the same thing twice, or one change makes the other completely unmergeable
* you get bugs but can't share your closed source test cases, and then you can't automate tests for them, or clearly demonstrate the problem
* other contributors don't see your full semi secret important motivation, and may either nitpick too much or take too long to review your stuff
Another common difficulty is that open source maintainers may simply not care enough about their own project (maybe they did in the past but lost interest) to review external patches by people they don't know.
This is understandable: a new patch, is a new risk of things breaking.
Therefore, if you ever submit patches and they get ignore, don't be too sad. It just comes down to a question of maintenance cost, and means that you will waste some extra time on the next rebase. You just have to decide your goals and be cold about it:
* are you doing the right thing and going for a specific goal <backward design>? Then just fork, run as fast as possible towards a minimum viable product, and if you start to feel that rebase is costing you a lot, or feel you could get some open source fame for cheap, open reviews and see what upstream says. If they ignore you, politely tell yourself in your mind silently "<fuck> them", and carry on with the MVP
* otherwise, e.g. you just want to randomly help out, you have to ask them before doing anything big "how can I be of help". If I propose a patch for this issue, do you promise to review it?
Writing documentation in an open source project in which you don't have immediate push rights is another major pain due to code reviews. Code code reviews tend to be much less subjective, because if you do something wrong, stuff crashes, runs slower, or you need more lines of code to reach the same goal. There are tradeoffs, but in a limited number. Documentation code reviews on the other hand, are an open invitation to https://en.wikipedia.org/wiki/Law_of_triviality[infinite bike-shedding], since you can't "run" documentation through a standardized <brain>[brain model]. Much better is for one good documenter person to just make one cohesive <Stack Overflow> post, and ping others with more knowledge to review details or add any missing pieces :-)
= Code drop
{parent=Open source software}
<Open source> development model in which developers develop in private, and only release code to the public during releases.
Notable example project: <Android Open Source Project>.
This development model basically makes reporting bugs and sending patches a waste of time, because many of them will already have been solved, which is why this development model is <evil>{parent}.
= Closed source is less bad on online services
{parent=Open source software}
<Ciro Santilli> can accept closed source on <server> products more easily than offline, because the servers have to be paid for somehow (by stealing your private data).
= Closed source offline software used by millions
{parent=Closed source is less bad on online services}
Closed source on offline products used by millions of people is <evil>, when you could just have those for free with <open source software>! Thus Ciro's hatred for <Microsoft Windows>{child} and <MacOS>{child} (at <is the MacOS kernel open source?>[least userland, maybe]).
= Closed source software
{parent=Open source software}
= Closed source
{synonym}
The opposite of <open source software>.
= Closed standard
{parent=Closed source software}
<ISO> is the main culprit of this <bullshit>, some notable examples related to <open source software>:
* <ANSI C>
* <SQL standard>
* <Verilog>
The only low level thing that escaped this was <OpenGL> via <Khronos>, what heroes those people are.
How the hell are you supposed to develop an open source implementation of something that has a closed standard?
Not to mention open source test suites, that would be way too much to ask for, those always end up being made by some shady small companies that go bankrupt from time to time, see e.g. .
= International Organization for Standardization
{c}
{parent=Closed standard}
{wiki}
= ISO
{c}
{synonym}
= Inner source
{parent=Closed source software}
{wiki}
If you are going to do <closed source>, at least do it like this.
Basically the opposite of <need to know> for <software>.
= Personal information management
{parent=Software}
{wiki}
= Productivity software
{parent=Software}
{wiki}
= LibreOffice
{c}
{parent=Productivity software}
{wiki}
These people are heroes. There's nothing else to say.
= Program optimization
{parent=Software}
{wiki}
= Benchmark
{parent=Program optimization}
{wiki}
= Profiling
{disambiguate=computer programming}
{parent=Program optimization}
{wiki}
= Profile
{disambiguate=computer programming}
{synonym}
\Include[programming-language]{parent=software}
= Recreational programming
{parent=Software}
= Code golf
{parent=Recreational programming}
{wiki}
= Search engine
{parent=Software}
{wiki}
= Search engine optimization
{parent=Search engine}
{wiki}
= SEO
{c}
{synonym}
{title2}
= List of search engines
{parent=Search engine}
= Yandex
{c}
{parent=List of search engines}
{wiki}
= Yandexing
{synonym}
= Web crawling
{parent=Search engine}
{wiki}
= Open web crawling
{parent=Web crawling}
{wiki}
= Common Crawl
{parent=Open web crawling}
{wiki}
https://commoncrawl.org/
Amazing project, that basically makes a more searchable <Wayback Machine>.
A bit hard to use their data though, partly due to size, but also lack of free to use querrying mechanisms, and how obtuse <Amazon S3> is to use.
Notably, <aws-cli> with an account is the only reliable way, everything else is way too broken, e.g. trying the to check the an index https://index.commoncrawl.org/CC-MAIN-2023-06/ very often 500s.
But still, their projct is amazing.
The only out-of-the-box search they seem to have is: http://urlsearch.commoncrawl.org/[] for domains/URLs. It is good, but there could be so much more... notably <IPs>.
Also could should document the data shape a bit better.
Sample sizes can be found at: https://commoncrawl.org/2023/04/mar-apr-2023-crawl-archive-now-available/
To explore the data, after login:
``
aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2013-20/
``
Copy the toplevel directory only:
``
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/ . --recursive --exclude "*/*"
``
Copy some wet/wat files:
``
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz .
aws s3 sync s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz .
``
Directory structrure:
* cc-index.paths.gz (1K)
* cc-index-table.paths.gz (1K)
* segment.paths.gz (1.7K) Sample lines:
``
crawl-data/CC-MAIN-2013-20/segments/1368696381249/
crawl-data/CC-MAIN-2013-20/segments/1368696381630/
``
* index.html (2.3K)
* wat.paths.gz (98K) Sample lines:
``
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wat.gz
``
* wet.paths.gz (98K) Sample lines:
``
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wet.gz
``
* warc.paths.gz (99K)
``
crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.gz
``
* segments: directgory with actual data
* 1368696381249: one of many segments, any meaning of name?
* CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz (142M, 334M unzipped)
A tiny bit of metadata, and then plaintext content from the website, e.g. the second one:
``
WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://004eeb5.netsolhost.com/stephensilver.htm
WARC-Date: 2013-05-18T08:11:02Z
WARC-Record-ID: <urn:uuid:773b31ba-ddc6-47a5-ae24-d08141b9944d>
WARC-Refers-To: <urn:uuid:4b1bdbff-4926-4ced-86f6-072f5bb3837a>
WARC-Block-Digest: sha1:LQFSCR2LIJQYMPTXRHWU7HAPQTVSYS3A
Content-Type: text/plain
Content-Length: 12046
Stephen Silver is a journalist and editor who specializes in the areas of politics, pop culture, film and sports. He works as an editor with the North American Publishing Co. and as a film critic with The Trend, a local newspaper in the Philadelphia area.
``
No <IP> unfortunately.
* CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz (329M, 1.4G unzipped)
A lot of JSON metadata and no contents as desired. Contains IP! Some entries however are humongous with a ton of useless data, that's what bloats these so much:
``
WARC/1.0
WARC-Type: metadata
WARC-Target-URI: CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
WARC-Date: 2013-11-22T14:51:12Z
WARC-Record-ID: <urn:uuid:ec54e493-8965-41be-b344-07596cc30b3a>
WARC-Refers-To: <urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>
Content-Type: application/json
Content-Length: 1180
{"Envelope":{"Format":"WARC","WARC-Header-Length":"274","Block-Digest":"sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR","Actual-Content-Length":"372","WARC-Header-Metadata":{"WARC-Type":"warcinfo","WARC-Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz","WARC-Date":"2013-11-22T14:51:12Z","Content-Length":"372","WARC-Record-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Type":"application/warc-fields"},"Payload-Metadata":{"Trailing-Slop-Length":"0","Actual-Content-Type":"application/warc-fields","Actual-Content-Length":"372","Headers-Corrupt":true,"WARC-Info-Metadata":{"robots":"classic","software":"Nutch 1.6 (CC)/CC WarcExport 1.0","description":"Wide crawl of the web with URLs provided by Blekko for Spring 2013","hostname":"ip-10-60-113-184.ec2.internal","format":"WARC File Format 1.0","isPartOf":"CC-MAIN-2013-20","operator":"CommonCrawl Admin","publisher":"CommonCrawl"}}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"453","Header-Length":"10","Inflated-CRC":"866052549","Inflated-Length":"650"},"Offset":"0","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}
WARC/1.0
WARC-Type: metadata
WARC-Target-URI: http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions
WARC-Date: 2013-05-18T05:48:54Z
WARC-Record-ID: <urn:uuid:d519658f-7a63-46c1-849b-4cd92332ddb8>
WARC-Refers-To: <urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>
Content-Type: application/json
Content-Length: 1501
{"Envelope":{"Format":"WARC","WARC-Header-Length":"433","Block-Digest":"sha1:B2B6JDSGWCUQIIUGV54SXEE25RX4SANS","Actual-Content-Length":"302","WARC-Header-Metadata":{"WARC-Type":"request","WARC-Date":"2013-05-18T05:48:54Z","WARC-Warcinfo-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Length":"302","WARC-Record-ID":"<urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>","WARC-Target-URI":"http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions","WARC-IP-Address":"165.1.125.44","Content-Type":"application/http; msgtype=request"},"Payload-Metadata":{"Trailing-Slop-Length":"4","HTTP-Request-Metadata":{"Headers":{"Accept-Language":"en-us,en-gb,en;q=0.7,*;q=0.3","Host":"ap.org","Accept-Encoding":"x-gzip, gzip, deflate","User-Agent":"CCBot/2.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},"Headers-Length":"300","Entity-Length":"0","Entity-Trailing-Slop-Bytes":"0","Request-Message":{"Method":"GET","Version":"HTTP/1.0","Path":"/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions"},"Entity-Digest":"sha1:3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ"},"Actual-Content-Type":"application/http; msgtype=request"}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"455","Header-Length":"10","Inflated-CRC":"453539965","Inflated-Length":"739"},"Offset":"453","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}
``
Let's beautify one of them to see it better:
``
{
"Envelope": {
"Format": "WARC",
"WARC-Header-Length": "274",
"Block-Digest": "sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR",
"Actual-Content-Length": "372",
"WARC-Header-Metadata": {
"WARC-Type": "warcinfo",
"WARC-Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz",
"WARC-Date": "2013-11-22T14:51:12Z",
"Content-Length": "372",
"WARC-Record-ID": "<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>",
"Content-Type": "application/warc-fields"
},
"Payload-Metadata": {
"Trailing-Slop-Length": "0",
"Actual-Content-Type": "application/warc-fields",
"Actual-Content-Length": "372",
"Headers-Corrupt": true,
"WARC-Info-Metadata": {
"robots": "classic",
"software": "Nutch 1.6 (CC)/CC WarcExport 1.0",
"description": "Wide crawl of the web with URLs provided by Blekko for Spring 2013",
"hostname": "ip-10-60-113-184.ec2.internal",
"format": "WARC File Format 1.0",
"isPartOf": "CC-MAIN-2013-20",
"operator": "CommonCrawl Admin",
"publisher": "CommonCrawl"
}
}
},
"Container": {
"Compressed": true,
"Gzip-Metadata": {
"Footer-Length": "8",
"Deflate-Length": "453",
"Header-Length": "10",
"Inflated-CRC": "866052549",
"Inflated-Length": "650"
},
"Offset": "0",
"Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"
}
}
``
Fuck no IP addresses either. But other entries do have it, why not this one?
The reason these can be huge is the `HTML-Metadata` section which contain all outlinks! https://gist.github.com/Smerity/e750f0ef0ab9aa366558#file-bbc-pretty-wat-L34
* `CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz` ()
Obtain:
``
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz .
``
= Common Crawl Athena
{parent=Common Crawl}
TODO no <IP>? Sadface?
* https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
* https://github.com/commoncrawl/cc-index-table/blob/main/src/sql/athena/cc-index-create-table-flat.sql
* https://github.com/commoncrawl/cc-index-table/issues/30
= Reverse image search
{parent=Search engine}
{wiki}
= Scientific computing
{parent=Software}
{wiki}
= Computer simulation
{parent=Scientific computing}
{wiki}
= Scientific software
{parent=Scientific computing}
= Numerical software
{parent=Scientific software}
{wiki=Category:Numerical_software}
= Basic Linear Algebra Subprograms
{parent=Numerical software}
{wiki}
= BLAS
{synonym}
{title2}
The <original gangster>.
= Computer algebra system
{parent=Numerical software}
{wiki}
= SymPy
{parent=Computer algebra system}
{wiki}
This is the dream <exam>[cheating] software every student should know about.
It also has serious applications obviously. https://www.sympy.org/scipy-2017-codegen-tutorial/ mentions <code generation> capabilities, which sounds super cool!
The code in this section was tested on `sympy==1.8` and <Python> 3.9.5.
Let's start with some basics. <fractions>:
``
from sympy import *
sympify(2)/3 + sympify(1)/2
``
outputs:
``
7/6
``
Note that this is an exact value, it does not get converted to <floating-point numbers> where precision could be lost!
We can also do everything with symbols:
``
from sympy import *
x, y = symbols('x y')
expr = x/3 + y/2
print(expr)
``
outputs:
``
x/3 + y/2
``
We can now evaluate that expression object at any time:
``
expr.subs({x: 1, y: 2})
``
outputs:
``
4/3
``
How about a square root?
``
x = sqrt(2)
print(x)
``
outputs:
``
sqrt(2)
``
so we understand that the value was kept without simplification. And of course:
``
sqrt(2)**2
``
outputs `2`. Also:
``
sqrt(-1)
``
outputs:
``
I
``
`I` is the <imaginary unit>. We can use that symbol directly as well, e.g.:
``
I*I
``
gives:
``
-1
``
Let's do some trigonometry:
``
cos(pi)
``
gives:
``
-1
``
and:
``
cos(pi/4)
``
gives:
``
sqrt(2)/2
``
The exponential also works:
``
exp(I*pi)
``
gives;
``
-1
``
Now for some <calculus>. To find the <derivative> of the <natural logarithm>:
``
from sympy import *
x = symbols('x')
diff(ln(x), x)
``
outputs:
``
1/x
``
Just read that. One over x. Beauty.
Let's do some more. Let's solve a simple <differential equation>:
``
y''(t) - 2y'(t) + y(t) = sin(t)
``
Doing:
``
from sympy import *
x = symbols('x')
f, g = symbols('f g', cls=Function)
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x)**4)
print(dsolve(diffeq, f(x)))
``
outputs:
``
Eq(f(x), (C1 + C2*x)*exp(x) + cos(x)/2)
``
which means:
$$
f(x) = C_1 + C_2x e^x + cos(x)/2
$$
To be fair though, it can't do anything crazy, it likely just goes over known patterns that it has solvers for, e.g. if we change it to:
``
diffeq = Eq(f(x).diff(x, x)**2 + f(x), 0)
``
it just blows up:
``
NotImplementedError: solve: Cannot solve f(x) + Derivative(f(x), (x, 2))**2
``
Sad.
Let's try some <polynomial equations>:
``
from sympy import *
x, a, b, c = symbols('x a b c d e f')
eq = Eq(a*x**2 + b*x + c, 0)
sol = solveset(eq, x)
print(sol)
``
which outputs:
``
FiniteSet(-b/(2*a) - sqrt(-4*a*c + b**2)/(2*a), -b/(2*a) + sqrt(-4*a*c + b**2)/(2*a))
``
which is a not amazingly nice version of the <quadratic formula>. Let's evaluate with some specific constants after the fact:
``
sol.subs({a: 1, b: 2, c: 3})
``
which outputs
``
FiniteSet(-1 + sqrt(2)*I, -1 - sqrt(2)*I)
``
Let's see if it handles the <quartic equation>:
``
x, a, b, c, d, e, f = symbols('x a b c d e f')
eq = Eq(e*x**4 + d*x**3 + c*x**2 + b*x + a, 0)
solveset(eq, x)
``
Something comes out. It takes up the entire terminal. Naughty. And now let's try to <Abel-Ruffini theorem>[mess with it]:
``
x, a, b, c, d, e, f = symbols('x a b c d e f')
eq = Eq(f*x**5 + e*x**4 + d*x**3 + c*x**2 + b*x + a, 0)
solveset(eq, x)
``
and this time it spits out something more magic:
``
ConditionSet(x, Eq(a + b*x + c*x**2 + d*x**3 + e*x**4 + f*x**5, 0), Complexes)
``
Oh well.
Let's try some <linear algebra>.
``
m = Matrix([[1, 2], [3, 4]])
``
Let's invert it:
``
m**-1
``
outputs:
``
Matrix([
[ -2, 1],
[3/2, -1/2]])
``
= SymPy special function
{c}
{parent=SymPy}
= python/sympy_cheat/logarithm_integral.py
{file}
{parent=SymPy special function}
{title2=`li`}
= Scientific visualization
{parent=Scientific software}
= Chart
{parent=Scientific visualization}
{wiki}
= Chart type
{parent=Chart}
= Histogram
{parent=Chart type}
{wiki}
= Scientific visualization software
{parent=Scientific visualization}
https://en.wikipedia.org/wiki/Scientific_visualization
<Ciro Santilli>[Ciro's] large dataset survey: <Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli>{full}.
= Open source scientific computing consultancies
{parent=Scientific visualization software}
Huge respect to this companies.
= Enthought
{c}
{parent=Open source scientific computing consultancies}
{wiki}
= Kitware
{c}
{parent=Open source scientific computing consultancies}
{wiki}
= Plotting software
{parent=Scientific visualization software}
= Real time live plot from streaming data
{parent=Plotting software}
E.g. showing live data from a scientific instrument! TODO:
* https://superuser.com/questions/825588/what-is-the-easiest-way-of-visualizing-data-from-stdout-as-a-graph
* https://unix.stackexchange.com/questions/190337/how-can-i-make-a-graphical-plot-of-a-sequence-of-numbers-from-the-standard-input
* https://stackoverflow.com/questions/44470965/how-can-you-watch-gnuplot-realtime-data-plots-as-a-live-graph-with-automatic-up
* https://stackoverflow.com/questions/14074790/plotting-a-string-of-csv-data-in-realtime-using-linux
* https://stackoverflow.com/questions/11874767/how-do-i-plot-in-real-time-in-a-while-loop-using-matplotlib
= Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli
{c}
{parent=Plotting software}
https://stackoverflow.com/questions/5854515/large-plot-20-million-samples-gigabytes-of-data/55967461#55967461
By <Ciro Santilli>.
\Image[https://raw.githubusercontent.com/cirosantilli/media/master/VisIt_zoom_in_10_million_straight_line_plot_with_some_marked_points.png]
{source=https://stackoverflow.com/questions/5854515/large-plot-20-million-samples-gigabytes-of-data/55967461#55967461}
= Matplotlib
{c}
{parent=Plotting software}
{tag=Python library}
{wiki}
It does a huge percentage of what you want easily, and from <Python>[the language that you want to use].
Tends to be <Ciro Santilli>[Ciro]'s pick if <gnuplot> can't handle the use case, or if the project is really really serious.
Couldn't handle exploration of large datasets though: <Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli>
Examples:
* \a[matplotlib/hello.py]
* \a[matplotlib/educational2d.py]
* \a[matplotlib/axis.py]
* \a[matplotlib/label.py]
* Line style
* \a[matplotlib/line_points.py]
* \a[matplotlib/tick.py]
* \a[matplotlib/prop_cycle.py]
* Subplots
* \a[matplotlib/subplots.py]
* \a[matplotlib/subplots_add.py]
* \a[matplotlib/two_lines.py]
* \a[matplotlib/plot_matrix.py]
* \a[matplotlib/plot_matrix_x.py]
* \a[matplotlib/legend_outside.py]
* Data from files
* \a[matplotlib/plotfile.py]
* Specialized
* \a[matplotlib/bloch_sphere_walk.py]
* \a[matplotlib/bloch_sphere.py]
Tested on Python 3.10.4, <Ubuntu 22.04>.
= gnuplot
{c}
{parent=Plotting software}
{wiki=Gnuplot}
Tends to be <Ciro Santilli>'s first attempt for quick and dirty graphing: https://github.com/cirosantilli/gnuplot-cheat[].
<domain-specific language>. When it get the jobs done, it is in 3 lines and it feels great.
When it doesn't, you Google for an hours, and then you give up in frustration, and fall back to <Matplotlib>.
Couldn't handle exploration of large datasets though: <Survey of open source interactive plotting software with a 10 million point scatter plot benchmark by Ciro Santilli>
= gnuplot command line hello world
{c}
{parent=gnuplot}
https://askubuntu.com/questions/277363/gnuplot-not-showing-the-graph-window/683073#683073
CLI hello world:
``
gnuplot -p -e 'p sin(x)'
``
= Software reverse engineering
{parent=Software}
{tag=Reverse engineering}
= Software reverse engineering tool
{parent=Software reverse engineering}
= binwalk
{parent=Software reverse engineering}
https://github.com/ReFirmLabs/binwalk
= Software bug
{parent=Software}
{wiki}
= Formal verification
{parent=Software bug}
{wiki}
= Formally verified
{synonym}
= Glitch
{parent=Software bug}
{wiki}
A glitch is more precisely a <software bug> that is hard to reproduce. But it has also been used to mean a software bug that is not very serious.
= Debugging
{parent=Software bug}
{wiki}
Debugging sucks. But there's also nothing quite that "oh fuck, that's why it doesn't work" moment, which happens after you have examined and placed everything that is relevant to the problem into your brain. You just can't see it coming. It just happens. You just learn what you generally have to look at so it happens faster.
Related:
* <keep debug notes>{child}
= Ciro's call hierarchy notation
{c}
{parent=Debugging}
This is a simple hierarchical plaintext notation <Ciro Santilli> created to explain programs to himself.
It is usuall created by doing searches in an <IDE>, and then manually selecting the information of interest.
It attempts to capture intuitive information not only of the call graph itself, including callbacks, but of when things get called or not, by the addition of some context code.
For example, consider the following <pseudocode>:
``
f1() {
}
f2(i) {
if (i > 5) {
f1()
}
}
f3() {
f1()
f2_2()
}
f2_2() {
for (i = 0; i < 10; i++) {
f2(i)
}
}
main() {
f2_2()
f3()
}
``
Supose that we are interested in determining what calls `f1`.
Then a reasonable call hierarchy for `f1` would be:
``
f2(i)
if (i > 5) {
f1()
f2_2()
for (i = 0; i < 10; i++) {
f2(i)
main
f3
f3()
main()
``
Some general principles:
* start with a regular call tree
* to include context:
* remove any blank lines from the snippet of interest
* add it indented below the function
* and then follow it up with a blank line
* and then finally add any callers at the same indentation level
= Bisection
{disambiguate=software engineering}
{parent=Debugging}
One of the Holiest age old debugging techniques!
<Git> has some helpers to help you achieve bisection Nirvana: https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593
Obviously not restricted to software engineering alone, and used in all areas of engineering, e.g. <video Air-tight vs. Vacuum-tight by AlphaPhoenix (2020)> uses it in <vacuum engineering>.
The cool thing about bisection is that it is a brainless process: unlike when using a <debugger>, you don't have to understand anything about the system, and it incredibly narrows down the problem cause for you. Not having to think is great!
= Debugger
{parent=Debugging}
{wiki}
= Reverse debugging
{parent=Debugger}
= Step debugger
{synonym}
Nirvana!!!
For <compiled languages>, see: <GDB reverse debugging>{full}.
For <JavaScript>: https://stackoverflow.com/questions/17498159/how-to-go-backwards-while-debugging-javascript-in-chrome-sources-debugging/74968631#74968631
= Omniscient debugging
{parent=Reverse debugging}
What it adds on top of <reverse debugging>: not only can you go back in time, but you can do it instantaneously.
Or in other words, you can access variables from any point in execution.
TODO implementation? Apparently <Pernosco> is an attempt at it, though proprietary.
= GNU Debugger
{c}
{parent=Debugger}
{tag=GNU package}
= GDB
{c}
{synonym}
{title2}
= GDB step debug
{c}
{synonym}
Just add <GDB Dashboard>, and you're good to go.
= GDB reverse debugging
{parent=GNU Debugger}
{tag=Reverse debugging}
The best open source implementation as of 2020 seems to be: <Mozilla rr>.
= Mozilla rr
{c}
{parent=GDB reverse debugging}
https://github.com/mozilla/rr
* https://stackoverflow.com/questions/1206872/go-to-previous-line-in-gdb/46996380#46996380
* https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242
* https://stackoverflow.com/questions/3649468/setting-breakpoint-in-gdb-where-the-function-returns/46116927#46116927
* https://stackoverflow.com/questions/27770896/how-to-debug-a-rare-deadlock/50073993#50073993
* https://stackoverflow.com/questions/522619/how-to-do-bidirectional-or-reverse-debugging-of-programs/50074106#50074106 link only, marked as duplicate of go to previous line
* https://softwareengineering.stackexchange.com/questions/181527/why-is-reverse-debugging-rarely-used
= Pernosco
{c}
{parent=Mozilla rr}
{tag=Omniscient debugging}
{tag=Proprietary software}
https://pernos.co/
Proprietary extension to <Mozilla rr> by rr lead coder <Robert O'Callahan> et. al, started in 2016 after he quit Mozilla.
TODO what does it add to `rr`?
* https://youtu.be/dMroSfg9kio?t=1494
* https://robert.ocallahan.org/2018/05/update-pernosco.html
= GDB Dashboard
{c}
{parent=GNU Debugger}
https://github.com/cyrus-and/gdb-dashboard
<GDB> Nirvana?
https://stackoverflow.com/questions/10115540/gdb-split-view-with-code/51301717#51301717
\Image[https://web.archive.org/web/20200504130959if_/https://i.stack.imgur.com/mHC8f.png]
{height=600}
{title=Screenshot of terminal running GDB Dashboard}
{source=https://github.com/cyrus-and/gdb-dashboard/tree/2d31a3b391e5d0e032b791e1fb7172338b02cecb}
= Minimal working example
{parent=Software bug}
{wiki}
The <musical study> of <software engineering>.
<Ciro Santilli> is obsessed by those in order to learn any new concept, not just for bug reporting.
This includes to learn more theoretical subjects like <physics> and <mathematics>.
= Zero-based numbering
{parent=Software bug}
{wiki}
= Software company
{parent=Software}
{tag=Company}
{wiki}
\Include[microsoft]{parent=Software company}
= Adobe
{c}
{parent=Software company}
{wiki}
= Oracle Corporation
{c}
{parent=Software company}
{wiki}
= Oracle
{c}
{synonym}
<Evil> company that desecrated the beauty created by <Sun Microsystems>, and was https://en.wikipedia.org/wiki/Google_v._Oracle_America[trying to bury Java once and or all in the 2010's].
Their database is already matched by <open source> e.g. <PostgreSQL>, and https://en.wikipedia.org/wiki/Enterprise_resource_planning[ERP] and https://en.wikipedia.org/wiki/Customer_relationship_management[CRM] specific systems are boring.
Oracle basically grew out of selling one of the first <SQL> implementations in the late 70's, and notably to the <United States Government> and particularly the <CIA>. They did deliver a lot of value in those early pre-internet days, but now <open source> is and will supplant them entirely.
= Sun Microsystems
{c}
{parent=Oracle Corporation}
{tag=Computer company}
{tag=Workstation}
{wiki}
Although <Ciro Santilli> is a bit past their era, there's an aura of technical excellence about those people. It just seems that they sucked at business. Those open source hippies. Erm, wait.
Bibliography:
* https://archive.org/details/sunburstascentof00hall Sunburst: the ascent of Sun Microsystems by Mark Hall (1990)
\Video[https://www.youtube.com/watch?v=P1TsVW4P5DI]
{title=The Dawn and Dusk of <Sun Microsystems> by <Asianometry> (2022)}
{description=One of the main inspirations for the creation of their <workstations> were <CAD> applications.}
= Red Hat
{c}
{parent=Software company}
{wiki}
= Truth Happens advertisement by Red Hat
{parent=Red Hat}
{tag=The best advertisements of all time}
<video 1984 Macintosh advertisement by Apple (1984)> comes to mind.
TODO year. This was a reply to <Microsoft> anti-<Linux> propaganda it seems: https://www.ubuntubuzz.com/2012/03/truth-happens-redhats-legendary-reply.html
Trascript from: https://www.dailymotion.com/video/xw3ws
\Q[
The world is flat. Earth is the centre of the universe. Fact - until proven otherwise.\br
Despite ignorance. Despite ridicule. Despite opposition. Truth happens.
Despite ignorance. \br
The telephone has too many shortcomings to be seriously considered as a means of communication. /Western Union 1876/\br
In 1899 the US Patent Commissioner stated, everything that can be invented has been invented.
Despite ridicule.\br
The phonograph has no commercial value at all. /Thomas Edison 1880/\br
The radio craze will die out in time. /Thomas Edison 1922/\br
The automobile has practically reached the limit of its development. /Scientific American 1909/
Despite it all truth happens.\br
Man will not fly for fifty years. /Orville Wright 1901/\br
The rocket will never leave the Earth's atomosphere. /New York Times 1936/\br
There is a world market for maybe five computers. /<IBM>'s Thomas Watson 1943/\br
640K Ought to be enough for anybody. /Bill Gates 1981/
First they ignore you...\br
<Linux> is the hype du jour. /Gartner Group 1999/
Then they laugh at you...\br
We think of linux as competitor in the student and hobbyist market. But I really don't think in the commercial market we'll see it in any significant way. /Bill Gates 2001/
Then they fight you...\br
Linux isn't going away. Linux is a serious competitor. We will rise to this challenge. /Steve Ballmer 2003/
Then you win... /Mohandas Gandhi/
You are here.\br
Red Hat Linux. <IBM>.
]
\Video[https://www.youtube.com/watch?v=IE00uo3o_MU]
{title=Truth Happens <advertisement> by <Red Hat>}
= Software documentation
{parent=Software}
{wiki}
= Documentation generator
{parent=Software documentation}
{wiki}
= README
{c}
{parent=Software documentation}
{wiki}
Please, use <AsciiDoc> and <one page to rule them all>.
= Software engineering
{parent=Software}
{wiki}
= Software development
{parent=Software engineering}
{wiki}
= Software developer
{synonym}
= Software development principle
{parent=Software development}
= Don't repeat yourself
{parent=Software development principle}
{wiki}
= Yet another
{parent=Don't repeat yourself}
{wiki}
The mandatory <xkcd>: <xkcd 927: Standards>.
= DRY
{c}
{synonym}
= Software design pattern
{parent=Software engineering}
{wiki}
= Polymorphism
{disambiguate=computer science}
{parent=Software design pattern}
{wiki}
= Polymorphism
{synonym}
= Software development method
{parent=Software engineering}
{wiki}
= Pair programming
{parent=Software development method}
{wiki}
= Ciro Santilli's software engineering wisdom
{c}
{parent=Software engineering}
{tag=Essays by Ciro Santilli}
Of course, "<Ciro Santilli>" with quotes, since all of those are either taken directly from others, or had been previously formulated by others.
= Excessive encapsulation is the root of much evil
{parent=Ciro Santilli's software engineering wisdom}
{tag=You aren't gonna need it}
Some anecdotes.
<Ciro Santilli> never splits up functions unless there is more than one calling point. If you split early, the chances that the interface will be wrong are huge, and a much larger refactoring follows.
If you just want to separate variables, just use a scope e.g.:
``
int cross_block_var;
// First step.
{
int myvar;
}
// Second step.
{
int myvar;
}
``
Ciro has seen and had to deal with in his lifetime with two projects that had like 3 to 10 git separate Git repositories, all created and maintained by the same small group of developers of the same organization, even though one could not build without the other. Keeping everything in sync was Hell! Why not just have three directories inside a single repository with a single source of truth?
Another important case: <Linux> should have at least a C standard library, init system, and shell in-tree, like <BSD Operating Systems>, as mentioned at: <Linux>{full}.
= The development cycle time is your God
{parent=Ciro Santilli's software engineering wisdom}
A slow development test cycle will kill your software.
New developers won't want to learn your project, because they would rather shoot themselves.
This means that build time, and the time to run tests, must be short.
5 seconds to rebuild is the maximum upper limit.
Of course, at some point software gets large enough that things won't fit anymore in 5 seconds. But then you \i[must] have either some kind of build caching, or options to do partial builds/tests that will bring things down to that 5 second mark.
You also have to spend some time profiling execution and build from scratch times.
A slow build from scratch will mean that your <continuous integration> costs a lot, money that could be invested in a new developer!
It also means that people won't bother to reproduce bugs on given commits, or <bisection (software engineering)>[bisect stuff].
One anecdote comes to mind. <Ciro Santilli> was trying to debug something, and more experience colleague came over.
To reproduce a problem, ciro was running one command, wait 5 seconds, run a second command, wait 5 seconds, run a third command:
``
cmd1
# wait 5 seconds
cmd2
# wait 5 seconds
cmd3
``
The first thing the colleague said: join those three commands into one:
``
cmd1;cmd2;cmd3
``
And so, <the correlation between software engineers and Buddhism>[Ciro was enlightened].
\Image[https://web.archive.org/web/20220930224719im_/https://imgs.xkcd.com/comics/compiling.png]
{title=<xkcd> 303: Compiling}
{description=They should be benchmarking and fixing their shitty build system instead.}
{source=https://xkcd.com/303/}
= Everything that is not tested breaks
{parent=Ciro Santilli's software engineering wisdom}
= Everything can break everything
{parent=Ciro Santilli's software engineering wisdom}
Whenever someone asks:
\Q[I can only see this one thing different our setups, do you think it could be the cause of our different behaviour?]
you don't need to read anymore, just point them to this page immediately. <Virtualization> for the win.
= Fix it twice
{parent=Ciro Santilli's software engineering wisdom}
= You aren't gonna need it
{parent=Ciro Santilli's software engineering wisdom}
{wiki}
= YAGNI
{c}
{synonym}
{title2}
Sometimes you are really certain that something is a required substep for another thing that is coming right afterwards.
When things are this concrete, fine, just do the substep.
But you have to always beware of cases where "I'm sure this will be needed at some unspecified point in the future", because such points tends to never happen.
YAGNI is so fundamental, there are several closely related concepts to it:
* <backward design>
* <assign the hard task to the lazy person>
\Image[https://web.archive.org/web/20230130144440if_/https://imgs.xkcd.com/comics/code_lifespan.png]
{title=<xkcd> 2730: Code Lifespan}
{source=https://xkcd.com/2730/}
= KISS principle
{c}
{parent=Ciro Santilli's software engineering wisdom}
{tag=Simplicity is the ultimate sophistication}
{wiki}
= Keep it simple, stupid
{synonym}
{title2}
The software engineer phrasing of <simplicity is the ultimate sophistication>.
Like all other principles, it is not absolute.
But it is something that you should always have on the back of your mind.
<You aren't gonna need it> is closely related, as generally the extra unnecessary complications are set in place to accommodate useless features that will never be needed.
= Hofstadter's law
{c}
{parent=Ciro Santilli's software engineering wisdom}
{wiki}
= It is impossible to predict how long it will take to do something
{synonym}
The trivial takes a few hours.
The easy takes a week.
And what seemed hard takes a few hours.
As "deadlines" approach, feature sets get cut down, then there are delays, and finally a feasible feature set is delivered some time after the deadline.
The only deadlines that can be met are those of tasks which have already been done but not announced.
This is of course <Hofstadter's law>.
On the other hand, as a colleague of Ciro once mentioned, it is also known that the time it takes for a task to be done expands without limits to match the deadline. And therefore, without deadlines, tasks will take forever and never get done.
And so, in a moment, perceiving <Koan>[this paradox], <the correlation between software engineers and Buddhism>[Ciro was enlightened].
= Brooks's law
{c}
{parent=Ciro Santilli's software engineering wisdom}
{title2=Adding manpower to a late software project makes it later}
{wiki}
\Video[https://www.youtube.com/watch?v=UFFWH8N9SLk]
{title=The Misty Mountains Cold Scene from <The Hobbit (film series)>: An Unexpected Journey (2012)}
{description=\Q[I will take each and every one of these dwarves over an army from the Iron Hills. For when I called upon them they answered. Loayalty. Honour. And willing heart. I can ask no more than that.]}
= Sometimes you can debug software by staring at the code for long enough
{parent=Ciro Santilli's software engineering wisdom}
{wiki}
Once upon a time, when <Ciro Santilli> had a job, he had a programming problem.
A senior developer came over, and rather than trying to run and modify the code like an idiot, which is what <Ciro Santilli> usually does (see also experimentalism remarks at <Ciro Santilli's bad old event memory>{full}), he just stared at the code for about 10 minutes.
We knew that the problem was likely in a particular function, but it was really hard to see why things were going wrong.
After the 10 minutes of examining every line in minute detail, he said:
\Q[I think this function call has such or such weird edge case]
and truly, that was the cause.
And so, <the correlation between software engineers and Buddhism>[Ciro was enlightened].
= Office space design and remote work
{parent=Ciro Santilli's software engineering wisdom}
Working remotely is hard if you don't already highly master the software and enterprise systems used.
Also you don't feel people's love as strongly, and usefulness is built on love, see also <Steve Jobs>'s Pixar office space design philosophy.
But please, give workers a small silent office so that we can concentrate instead of a silly open space, and create an internal social network so people can see what others are doing.
Remote working is much better if the majority of the team also does it, otherwise you will get excluded. Maybe after VR...
= Keep debug notes
{parent=Ciro Santilli's software engineering wisdom}
When <debugging> complex software, make sure to keep notes of every interesting find you make in a note file, as you extract it from the <integrated development environment> or <debugger>.
Especially if your <Ciro Santilli's bad old event memory>[memory sucks like Ciro's].
This is incredibly helpful in fully understanding and then solving complex bugs.
= "Hello, World!" program
{parent=Software engineering}
{wiki}
= Hello world
{synonym}
The most important <program> ever written!!!
Other programs that can be considered "hello worlds" in different contexts:
* <web development>
* <A blog in every web framework>
* <video game>
* Doom is the hello world <shooter game>
= Time to Hello World
{parent="Hello, World!" program}
= Software engineer
{parent=Software engineering}
{wiki}
= Programmer
{synonym}
Poet warriors monkeys? Or Code peasants (https://baike.baidu.com/item/码农/10262742[码农]) according to the Chinese.
<Ciro Santilli> claims to be one of them.
Much like a pianist plays his piano, a software engineer plays his <computer>.
= Software engineer stereotype
{parent=Software engineer}
= Correlation between sofware engineers and Asian fetish
{parent=Software engineer stereotype}
{tag=Asian fetish}
https://www.quora.com/Why-do-successful-geeky-white-men-have-Asian-wives-This-seems-to-be-the-norm-in-Silicon-Valley suggests it is just an statistical inevitability.
= The correlation between software engineers and Buddhism
{parent=Software engineer stereotype}
{tag=Essays by Ciro Santilli}
= A correlation between software engineers and Buddhism
{synonym}
<Ciro Santilli> believes that there is a positive correlation between being a <software engineer>[software engineer] and liking <Buddhist>-like things.
Maybe it is linked to minimalism and <DRY>, which software engineers value so greatly.
Even Ciro had to try an unoriginal Buddhist joke intro in https://stackoverflow.com/questions/572897/how-does-javascript-prototype-work/23877420#23877420[one of this Stack Overflow answers].
Ciro also feels that his https://github.com/cirosantilli/linux-kernel-module-cheat/tree/e1d0a2fafbb35c9e65c1a8a0b6d46df3e9161461/userland["minimal reproducible example" scientific language/concept learning method obsession] of breaking things into tiny sub-problems has a strong link with <Koans>.
Some notable Buddhism/programmer examples:
* http://www.catb.org/~esr/writings/unix-koans/ "The Unix Koans of Master Foo - Rootless Root (无根的根)" by the legendary https://en.wikipedia.org/wiki/Eric_S._Raymond[Eric Steven Raymond] is notable
* http://thecodelesscode.com/ "The Codeless Code" by anonymous Qi.
* http://canonical.org/~kragen/tao-of-programming.html
* https://wiki.c2.com/?MysticalProgrammingKoans
* http://rubykoans.com/[] even <evil> programming languages adopt them!
* <The Zen of Python>
Another thing that points the correlation out is the existence of https://wattsalan.github.io/ on a `github.io` about <Alan Watts>.
= The Three Treasures of the Programmer
{c}
{parent=The correlation between software engineers and Buddhism}
= 编程三宝
{synonym}
{title2}
<Ciro Santilli>'s joke version of the Chinese <Four Treasures of the Study>!
* <web browser>
* <text editor>
* <terminal>. Though to be honest, circa 2022, Ciro learned of the ctrl + click to open file (including with file.c:123 line syntax) ability of <Visual Studio Code> (likely present in other <IDEs>), and he was starting considering dumping the terminal altogether if some implementation gets it really really right. The main thing is that it can't be a tinny little bar at the bottom, it has to be full window and super easily toggleable!
In the past, Ciro used to use <file managers>, which would be the fourth tresure. But he stopped doing so for years due to his cd alias... so it became three. He actually had exactly three windows open when he was checking if there was anything else he could not open hand of.
\Image[https://raw.githubusercontent.com/cirosantilli/media/master/Three_treasures_of_the_programmer.png]
{title=The three Treasures of the Programmer}
{description=Featuring: <Gvim>, <tmux> running in GNOME terminal, and <Chromium> browser on <Ubuntu 22.04>. The minimized windows are for demonstration purposes, <Cirism> mandates that all windows shall be maximized at all times. Splits withing a single program are permitted however.}
= List of software engineers
{parent=Software engineer}
= Aaron Swartz
{c}
{parent=List of software engineers}
{wiki}
Aaron, <Ciro Santilli> will complete <OurBigBook.com>[your quest to make eduction free]. Just <legally> this time, with the and with the <Creative Commons license> you helped to create.
Ciro likes how <The Internet's Own Boy (2014)> explains how Aaron felt like high school was <bullshit>, and that he could <autodidacticism>[learn whatever he wanted from books], <ourbigbook com/motivation>[which is one of Ciro's key feelings].
It also mentions how he was a natural teacher from a very early age.
= Guerilla Open Access Manifesto by Aaron Swartz (2008)
{c}
{parent=Aaron Swartz}
{wiki}
https://gist.github.com/briandoll/4522952
Hmmm, he does not know how to spell guerilla? <sic>? https://www.quora.com/What-is-the-correct-spelling-guerilla-or-guerrilla
Note to self: if you are going to commit a <crime>, don't publish your plans online.
<Ross Ulbricht>'s diaries come to mind.
That's how <Russian> <shadow library> maintainers do it, they know how to crime good old Russians. Maybe there is a good thing about having <dictatorships> in the world that give zero fucks about <American> copyright laws. There will always be some random <Russian> academic who will implement this and not go to jail. Maybe it's even <government>[state] sponsored.
= The Internet's Own Boy (2014)
{c}
{parent=Aaron Swartz}
{wiki}
= James Somers
{c}
{parent=List of software engineers}
{wiki}
* https://jsomers.net/
* https://www.linkedin.com/in/james-somers-4703364/
Huge interest overlap with <Ciro Santilli>, e.g. he's into
* <molecular biology> in general: <I should have loved biology by James Somers>
* <JCVI-syn3.0>: https://www.newyorker.com/magazine/2022/03/07/a-journey-to-the-center-of-our-cells
* <cryo-EM>: https://www.newyorker.com/magazine/2022/03/07/a-journey-to-the-center-of-our-cells
* <David Goodsell>: https://www.newyorker.com/magazine/2022/03/07/a-journey-to-the-center-of-our-cells
* <History of Google>: https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge
= I should have loved biology by James Somers
{c}
{parent=James Somers}
https://jsomers.net/i-should-have-loved-biology/
This resonates a lot with <Ciro Santilli>'s ideas!
* <physics and the illusion of life>
* <physics education needs more focus on understanding experiments and their history>:
* <there is value in tutorials written by early pioneers of the field>{full}
* <doing physics means calculating a number>{full}
* <education is broken>
* <molecular biology feels like systems programming>
\Q[I've never come across a subject so fractal in its complexity. It reminds me of computing that way.]
= Sandy Maguire
{c}
{parent=List of software engineers}
{wiki}
* https://sandymaguire.me
* https://github.com/isovector/
Lots of similar ideologies to <Ciro Santilli>, love it:
* https://sandymaguire.me/about/[]:
* he's an <idealist>
\Q[I might best be described somewhere between independent researcher and voluntarily-unemployed bum. At the ripe old age of 27 I decided to quit my highly-lucrative engineering job and decide to focus more on living than on grinding for the man. It's what you might call a work in progress.]
* https://sandymaguire.me/blog/reaching-climbing/[]: <don't be a pussy>
\Q[Last Friday was my final day at work. According to my facebook profile, I am now "happily retired." As of today, I don't plan to do another day of "traditional work" in my life. That's not to say that I'll be sitting idle playing tiddly winks. I want to build things, to dedicate my life to independent study, and to get really, really good with building communities. I don't have time for any of this "work" stuff that somehow pervades our entire culture, choking our inspiration and sapping our energy away from the things we'd rather be doing.]
One is also reminded of <Gwern Branwen>. Sandy is also into self-improvement stuff, so even more like Gwern. This is a point Ciro diverges on. Ciro works actively on self-worsening.
* he thinks <university> is useless:
* https://sandymaguire.me/blog/where-uni-fails/ Where University Fails (2018), mostly talking about <backward design>
* https://sandymaguire.me/blog/gatekept/ rejected from <Imperial College> <PhD> program due to <grade (exam)> being slightly too low for their stupid requirements, even though he had a referral already, and an amazing CV
* he likes <jazz>: https://sandymaguire.me/blog/too-smart/
Other interesting points:
* https://sandymaguire.me/blog/sandy-runback/ he changed his own name to Sandy because he didn't like it, he was born Alexander
* https://algebradriven.design/ <closed source> books though, ouch. At least they seem to have been made with <leanpub> though, could be worse.
He's a <Haskell> person.
= Dan Dascalescu
{c}
{parent=List of software engineers}
His website is down as of 2020, shame: https://wiki.dandascalescu.com/essays/english-universal-language
= English as a universal language by Dan Dascalescu (2008)
{c}
{parent=Dan Dascalescu}
https://wiki.dandascalescu.com/essays/english-universal-language
https://web.archive.org/web/20200317221752/https://wiki.dandascalescu.com/essays/english-universal-language
<Dan Dascalescu>'s version of <having more than one natural language is bad for the world>.
= Donald Knuth
{c}
{parent=List of software engineers}
{wiki}
= Knuth reward check
{c}
{parent=Donald Knuth}
{wiki}
* https://tex.stackexchange.com/questions/110586/327-68-knuth-reward-check
* https://www.quora.com/What-is-it-like-to-receive-a-Knuth-reward-check
* https://www-cs-faculty.stanford.edu/~knuth/boss.html
= EMBII
{c}
{parent=Software engineer}
One of the dudes from the <Cool data embedded in the Bitcoin blockchain/AtomSea & EMBII> <Bitcoin>-based file upload system.
* https://github.com/embiimob
* Real name: likely "Eric Bobby" according to:
* <LinkedIn>: https://www.linkedin.com/in/eric-bobby-0ab1b935/
* http://bitfossil.org/affbac1bfde690c1fabd60812d046c911b2882038a42b18a4d2e7cb50e989604/
According to <Cool data embedded in the Bitcoin blockchain/image Loraine.jpg> however, his mother's name was "Loraine Elizabeth White", so there's some chance his real family name is Mr. White.
However according to http://bitfossil.org/937f70bf641ccabaf623772367df64bd867ad44c53fd227d01f2662e74aeacbf/ his daughter is Maddy Bobby, so maybe he is actually Bobby.
* https://twitter.com/EMBII4U
* https://twitter.com/TheAtomSea/status/990318090738196481 sample link
\Image[https://web.archive.org/web/20230221120627im_/https://avatars.githubusercontent.com/u/6278950?v=4]
{title=<EMBII>'s usual profile image}
{source=https://github.com/embiimob}
= Gwern Branwen
{c}
{parent=Software engineer}
{tag=Ciro Santilli's e-soulmates}
Author of <gwern.net>.
Accounts:
* https://news.ycombinator.com/user?id=gwern
* https://www.lesswrong.com/users/gwern <LessWrong>
* https://twitter.com/gwern locked 2021: https://www.reddit.com/r/slatestarcodex/comments/kp2fek/does_anybody_know_what_happened_to_gwern/
* https://www.reddit.com/user/gwern/
* https://en.wikipedia.org/wiki/User:Gwern on <WikiPedia>. Self summary: https://gwern.net/wikipedia-resume[]. Also he is a critic of <deletionism on Wikipedia> like <Ciro Santilli>
He posts insanely much on these websites. It's a bit like <Ciro Santilli> on <Stack Overflow>.
<Ciro Santilli> envies this guy a bit. He <Braindumping>[dumps his brain more or less full time on his highly customized static website] partly due to early <Bitcoin> investments https://gwern.net/me[] says:
\Q[I am a freelance <American> writer & researcher. (To make ends meet, I have a <Patreon>, benefit from <Bitcoin> appreciation thanks to some old coins, and live frugally.)]
Also unsurprisingly he likes <Haskell>:
\Q[I mostly contribute to projects in Haskell, my favorite language]
<Ciro Santilli> considers Gwern <Ciro Santilli's e-soulmates> due to his interest in "dark web things" like <Bitcoin> and <Silk Road>, his immense writing output in encyclopedic book-sized articles on a <static website>, and his desire to live frugally and just research and write all day. Ah, if only Ciro had some old coins!!!
This is likely a pseudonym, his real name not being publicly unknown, e.g. at https://news.ycombinator.com/item?id=5659278[]:
\Q[
> Why do you choose relative anonymity?
For the reasons I've said in the past. To which I can add personal safety: my <Silk Road> page is a bit questionable legally, and we all know that there are ways to exploit knowledge of one's True Name and address (even if, as far as I know, I have no enemies willing to resort to, say, 'swatting' me) - one group of stalkers called up a college they thought I worked at to see if they could get me fired or otherwise ruin my day.
]
= Work by Gwern Branwen
{c}
{parent=Gwern Branwen}
= gwern.net
{c}
{parent=Gwern Branwen}
{tag=The best personal webpages of all time}
https://www.gwern.net
<Gwern Branwen>'s website.
One thing that annoys <Ciro Santilli> about that website are the footnote overload. Ciro likes linear things.
= Richard Stallman
{c}
{parent=Software engineer}
{wiki}
TODO find the best source for the amazing "I have done your mother" quote.
= Tim O'Reilly
{c}
{parent=Software engineer}
{wiki}
= Computer programming
{parent=Software engineering}
{wiki}
= Programming
{synonym}
Programming is hard. To <Ciro Santilli>, it's almost masochistic.
What makes Ciro especially mad when programming is not the hard things.
It is the things that should be easy, but aren't, and which take up a lot of your programming time.
Especially when you are already a few levels of "simple problems" down from your original goal, and another one of them shows up.
This is basically the cause of <Hofstadter's law>.
But of course, it is because it is hard that it feels amazing when you achieve your goal.
Putting a complex and useful program together is like composing a symphony, or reaching the summit of a hard <rock climbing> proble.
Programming can be an <art> form. There can be great beauty in code and what it does. It is a shame that this is hard to see from within the walls of most <companies>, where you are stuck doing a small specific task as fast as possible.
= Software portability
{parent=Software}
{wiki}
= Porting
{parent=Software portability}
{wiki}
= Port
{disambiguate=software-portability}
{synonym}
= Software quality assurance
{parent=Software}
{wiki}
= Static program analysis
{parent=Software quality assurance}
{wiki}
= Linting
{parent=Static program analysis}
{wiki=Lint (software)}
= Linter
{synonym}
= Software testing
{parent=Software quality assurance}
{wiki}
= test_executables.js
{file}
{parent=Software testing}
This script tests all executables under a selected directory.
<Ciro Santilli> has been writing scripts of that type for a long time in order to test his <Ciro Santilli's documentation superpowers>[programming self-learning setups with asserts].
The most advanced of those being the https://github.com/cirosantilli/linux-kernel-module-cheat/blob/9b6552ab6c66cb14d531eff903c4e78f3561e9ca/test[test system] of <Linux Kernel Module Cheat>.
But had too much stuff that would be specific to that project, so Ciro decided to start this new one in <Node.js>, hopefully it will also be the last he ever writes.
A sample usage of the test library can be seen at: \a[nodejs/sequelize/test].
= Assertion
{disambiguate=software development}
{parent=Software testing}
{wiki}
= Continuous integration
{parent=Software testing}
{wiki}
= CI
{c}
{synonym}
{title2}
= Test driven development
{parent=Software testing}
{wiki}
This is a good approach. The downside is that while you are developing the implementation and testing interactively you might notice that the requirements are wrong, and then the tests have to change.
One intermediate approach <Ciro Santilli> likes is to do the implementation and be happy with interactive usage, then create the test, make it pass, then remove the code that would make it pass, and see it fail. This does have a risk that you will forget to test something, but Ciro finds it is a worth it generally. Unless it really is one of those features that you are unable to develop without an automated test, generally more "logical/mathematical" stuff. This is a sort of <laziness Driven Development>.
= Laziness Driven Development
{parent=Test driven development}
= Shift-left testing
{parent=Software testing}
{wiki}
= Unit and system tests
{parent=Software testing}
= Unit test
{parent=Unit and system tests}
{wiki}
= Unit testing
{synonym}
= System test
{parent=Unit and system tests}
{wiki}
= Source code
{parent=Software}
{wiki}
= The best code is no code at all
{parent=Source code}
Some <blogs>:
* https://blog.codinghorror.com/the-best-code-is-no-code-at-all/ The Best Code is No Code At All (2007)
* http://www.skrenta.com/2007/05/code_is_our_enemy.html Code is our enemy (2007)
Also resonates with <backward design>.
= Use The Source Luke
{parent=Source code}
https://wiki.c2.com/?UseTheSourceLuke
\Include[systems-programming]{parent=software}
= Terminal emulator
{parent=Software}
{wiki}
= Terminal
{synonym}
Once upon a time young <Ciro Santilli> spent lots of time evaluating the features of different terimnals. The many windows of Terminator. The pop-uppiness of Guake/Yakuake.
But then one day he met <tmux>, and <the correlation between software engineers and Buddhism>[he was enlightened]
Terminal choice doesn't matter. Just use <tmux>.
= Yakuake
{parent=Terminal emulator}
https://github.com/KDE/yakuake
= Terminal multiplexer
{parent=Terminal emulator}
{wiki}
= Terminal multiplexers are CLI desktop environments
{parent=Terminal multiplexer}
If we didn't have <GUIs>, <terminal multiplexers> would be our <desktop environments>. E.g. they handle stuff like:
* window switching
* copy pasting across windows
* screen locking
* clock on the status bar (same one that holds tabs)
It is a thing of beauty.
= GNU screen
{c}
{parent=Terminal multiplexer}
{tag=GNU package}
{wiki}
Most important things to know:
* kill window: Ctrl + A K
= tmux
{c}
{parent=Terminal multiplexer}
{tag=Good}
{wiki}
https://github.com/tmux/tmux
If session autosave was finally mainlined, this would be Nirvana.
= Version control
{parent=Software}
{wiki}
= Monorepo
{parent=Version control}
{tag=Good}
{wiki}
= List of version control systems
{parent=Version control}
= Concurrent Versions System
{c}
{parent=List of version control systems}
{title2=CSV}
{wiki}
It is said, that once upon a time, programmers used CSV and collaborated on <SourceForge>, and that everyone was happy.
These days, are however, long gone in the mists of time as of 2020, and beyond <Ciro Santilli>'s programming birth.
Except for hardware developers of course. The are still happily using <Perforce> and <Tcl>, and shall never lose their innocence. Blessed be their souls. <Amen>.
= Git
{c}
{parent=List of version control systems}
{wiki}
= Git design rationale
{c}
{parent=Git}
The fundamental insight of Git design is: a <SHA> represents not only current state, but also the full history due to the <Merkle tree> implementation, see notably:
This makes it so that you will always notice if you are overwriting history on the remote, even if you are developing from two separate local computers (or more commonly, two people in two different local computers) and therefore will never lose any work accidentally.
It is very hard to achieve that without the <Merkle tree>.
Consider for example the most naive approach possible of marking versions with consecutive numbers:
* Local 1:
* 0: root commit
* 1: commit 1
* 2: commit 2 by local 1
* Local 2:
* 0: root commit
* 1: commit 1
* 2: commit 2 by local 2
* 3: commit 3 by local 2
* Remote
* 0: root commit
* 1: commit 1
If Local 1 were to push to Remote first, how could Local 2 notice that when it tries to push itself? The navie method of just checking: "does Remote have commit "2"" does not work, because Local 2 has a different version of commit 2 than local 1.
= Git command
{c}
{parent=Git}
= git clone
{c}
{parent=Git command}
= Download a single directory with git
{c}
{parent=git clone}
* https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository/52269934#52269934
* summaries:
* https://stackoverflow.com/questions/2425059/how-to-pull-specific-directory-with-git/54910376#54910376
* https://stackoverflow.com/questions/180052/checkout-subdirectories-in-git/52270636#52270636
* dupes:
* https://unix.stackexchange.com/questions/233327/is-it-possible-to-clone-only-part-of-a-git-project/468182#468182
* https://askubuntu.com/questions/460885/how-to-clone-only-some-directories-from-a-git-repository/1074185#1074185
* file or directory
* https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repo/56504849#56504849
* file
* https://stackoverflow.com/questions/2466735/how-to-sparsely-checkout-only-one-single-file-from-a-git-repository/52270527#52270527
* only small files:
= Git internals
{c}
{parent=Git}
= Git remote communcation
{c}
{parent=Git internals}
= Serve Git over HTTP static website
{parent=Git remote communcation}
* https://blog.thesparktree.com/git-mirror-anywhere-using-dumb-http-protocol
* https://stackoverflow.com/questions/2278888/private-git-repository-over-http
= Git object
{c}
{parent=Git internals}
= Git commit object
{c}
{parent=Git object}
https://stackoverflow.com/questions/22968856/what-is-the-file-format-of-a-git-commit-object-data-structure
= Git UI
{parent=Git}
Perfect Git integration belongs in <integrated development environments> :-)
= gitk
{parent=Git UI}
\Image[https://raw.githubusercontent.com/cirosantilli/media/master/gitk.png]
{title=<gitk> 2.34.1 running on <Ubuntu 22.04> with a simple repository.}
= tig
{disambiguate=git UI}
{parent=Git UI}
https://github.com/jonas/tig
This is good. But it misses some key operations, so much so that makes Ciro not want to learn/use it daily.
= Git web interface
{c}
{parent=Git UI}
= GitHub
{c}
{parent=Git web interface}
{wiki}
This is where <Ciro Santilli> stored his code since he started coding nonstop in 2013.
He <closed source is less bad on online services>[does not like the closed source aspect of it], but hey, there are more important things to worry about, the network effect is just too strong.
= GitHub repo
{c}
{parent=GitHub}
= GitHub book repo
{parent=GitHub repo}
Some amazing people have put book source codes on GitHub. This is a list of such repos.
= GitHub porn policy
{c}
{parent=GitHub}
https://www.quora.com/Does-github-allow-uploading-a-porn-image/answer/Ciro-Santilli
= GitHub users deleted by GitHub
{c}
{parent=GitHub}
https://stackoverflow.com/questions/50720844/can-github-delete-your-account/66105692#66105692
= GitHub Pages
{c}
{parent=GitHub}
https://pages.github.com/
The cheapest and most resilient way to publish <how to teach/text is cheaper than video>[text content] humanity has achieved so far.
Some tests:
* https://github.com/cirosantilli/jekyll-cheat[]: https://cirosantilli.com/jekyll-cheat
* https://github.com/cirosantilli/test-gh-pages[]: https://cirosantilli.com/test-gh-pages
Test with a `.nojekyll` file.
* https://github.com/cirosantilli/test-gh-pages-min[]: https://cirosantilli.com/test-gh-pages-min[]. Minimal version of the above.
= GitHub Sponsors
{c}
{parent=GitHub}
= Who are the developers that are making the most money through GitHub sponsors?
{c}
{parent=GitHub Sponsors}
https://github.com/isaacs/github/issues/1824
= Pull request
{parent=GitHub}
The heart/main innovation of GitHub!
= Octokat.js
{c}
{parent=GitHub}
https://github.com/philschatz/octokat.js
= octokit.js
{c}
{parent=Octokat.js}
https://github.com/philschatz/octokit.js
= GitLab
{c}
{parent=Git web interface}
{wiki}
GitLab was very important to Ciro because he wanted to base https://github.com/booktree/booktree[Booktree] on it.
See also: <Ciro Santilli's minor projects>.
= GitLab CI
{c}
{parent=GitLab}
{tag=Continuous integration}
https://github.com/gitlabhq/gitlab-ci
= GitLab cookbook
{c}
{parent=GitLab}
https://gitlab.com/gitlab-org/cookbook-gitlab
\Include[git-tips]{parent=git}
= SourceForge
{c}
{parent=Git web interface}
{wiki}
RIP: https://www.quora.com/Is-SourceForge-still-relevant-to-open-source-projects/answer/Ciro-Santilli
\Include[web-technology]{parent=software}
= Git implementation
{c}
{parent=Git}
= libgit2
{c}
{parent=Git implementation}
https://github.com/libgit2/libgit2
= libgit2/rugged
{c}
{parent=libgit2}
https://github.com/libgit2/rugged
= Git bibliography
{c}
{parent=Git}
= Pro Git book
{c}
{parent=Git bibliography}
https://github.com/progit/progit2
= Perforce
{c}
{parent=List of version control systems}
{wiki}
= Software bibliography
{parent=Software}
= EbookFoundation/free-programming-books
{parent=Software bibliography}
https://github.com/EbookFoundation/free-programming-books