node-js.bigb

= Node.js
{c}
{wiki}

Wellhttps://stackoverflow.com/questions/8775262/synchronous-requests-in-node-js[Sync], if you are gonna usehttps://stackoverflow.com/questions/8775262/synchronous-requests-in-node-jsp[Sync] this <JavaScript>[wonky language] thing inhttps://stackoverflow.com/questions/8775262/synchronous-requests-in-node-js[Sync] one place, you might as well usehttps://stackoverflow.com/questions/8775262/synchronous-requests-in-node-js[Sync] it everywherehttps://stackoverflow.com/questions/8775262/synchronous-requests-in-node-js[Sync] and make it more decent. See also: <how to convert async to sync in JavaScript>.

Their CLI debugger is a bit crap compared to GDB, basic functionality is either lacking or too verbose:
* https://stackoverflow.com/questions/65493221/how-to-break-at-a-specific-function-or-line-with-the-node-js-node-inspect-comman
* https://stackoverflow.com/questions/70486188/how-to-break-on-uncaught-exception-on-the-node-js-node-inspect-command-line-debu Some operations are only possible on the browser debug UI...
Documentation at: https://nodejs.org/dist/latest-v16.x/docs/api/debugger.html

= Node.js example
{parent=Node.js}

Under \a[nodejs]:
* \a[nodejs/infinite_loop.js]
* \a[nodejs/infinite_hello.js]
* \a[nodejs/performance.js]: related: https://stackoverflow.com/questions/77185767/azure-data-factory-updating-node-to-20-in-azure-devops-pipeline-fails-with-th/77718648#77718648

= nodejs/count.js
{parent=Node.js example}
{file}

This example counts to finity, sleeping 1 second between each count. Related:
* https://stackoverflow.com/questions/14249506/how-can-i-wait-in-node-js-javascript-l-need-to-pause-for-a-period-of-time

= nodejs/read_child_process_lines.js
{parent=Node.js example}
{file}

This example reads lines from a child process one by one, as soon as lines become fully available. Related:
* https://stackoverflow.com/questions/9781214/parse-output-of-spawned-node-js-child-process-line-by-line canon
  * https://stackoverflow.com/questions/14332721/node-js-spawn-child-process-and-get-terminal-output-live/77423073#77423073 answered big superset
  * https://stackoverflow.com/questions/12955154/nodejs-read-and-parse-each-line-of-stdout duplicate

= JavaScript memory usage benchmark
{parent=Node.js example}

In this section we will use the file \a[nodejs/bench_mem.js], tests are run on Node.js v16.14.2 from NVM, Ubuntu 21.10, on <ciro santilli s hardware/Lenovo ThinkPad P51 (2017)> which has 32 GB RAM.

Related answer: https://stackoverflow.com/questions/12023359/what-do-the-return-values-of-node-js-process-memoryusage-stand-for/72043884#72043884

First using `topp` from https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux/40576129#40576129 let's observe the memory usage of some baseline cases.

A C hello world with an infinite loop at the end has:
* 2.7 MB
* 770 KB

For a Node.js infinite loop \a[nodejs/infinite_loop.js]
``
topp infinite_loop.js
``
This gives approximately:
* RSS: 20 MB
* VSZ: 230 MB

Adding a single hello world to it as in \a[nodejs/infinite_hello.js] and running:
``
topp infinite_hello.js
``
leads to:
* RSS: 26 MB
* VSZ: 580 MB
We understand that Node.js preallocates VSZ wildly. No big deal, but it does mean that VSZ is a useless measure for Node.js.

Forcing garbage collection as in \a[nodejs/infinite_hello.js] brings it down to 20 MB however:
``
topp node --expose-gc infinite_hello_gc.js
``

Finally let's see a baseline for `process.memoryUsage` \a[nodejs/infinite_memoryusage.js]:
``
node --expose-gc infinite_memoryusage.js
``
which gives initially:
``
{
  rss: 23851008,
  heapTotal: 6987776,
  heapUsed: 3674696,
  external: 285296,
  arrayBuffers: 10422
}
``
but after a few seconds randomly jumps to:
``
{
  rss: 26005504,
  heapTotal: 9084928,
  heapUsed: 3761240,
  external: 285296,
  arrayBuffers: 10422
}
``
so we understand that
* `heapUsed` seems constant at 3.7 MB
* `heapTotal` is a very noisy, as it starts at 7 MB, but randomly jumps to 9 MB at one point without apparent reason

Now let's run our main test program.

First a baseline case with an array of length 1:
``
node --expose-gc bench_mem.js n 1
``
This gives the same results as `node --expose-gc infinite_memoryusage.js`. The same result is obtained by doing:
``
a = undefined
``
with:
``
node --expose-gc bench_mem.js dealloc
``

Not let's vary the size of `n` a bit with:
``
node --expose-gc bench_mem.js n N
``
which gives:
|| N
|| `heapUsed`
|| `heapTotal`
|| `rss`
|| `heapUsed` per elem
|| `rss` per elem

| 1 M
| 14 MB
| 48 MB
| 56 MB
| 10
| 30

| 10 M
| 122 MB
| 157 MB
| 176 MB
| 18
| 15

| 100 M
| 906 MB
| 940 MB
| 960 MB
| 9
| 9.3

"`rss` per elem" is calculated as: `rss` - 26 MB, where 26 MB is the baseline RSS seen on `n 1`.

Similarly "`heapUsed` per elem" deduces the 4 MB (approximation of the above 3.7 MB) seen on `n 1`.

Note that to reach https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER[MAX_SAFE_INTEGER] we would need 8 bytes per elem worst case.

Everything below 100 million (8) is therefore very memory wasteful in terms of RSS.

If we use `Int32Array` typed array buffers instead of a simple `Array`:
``
node --expose-gc bench_mem.js array-buffer n N
``
we see that the memory is now, unsurprisingly, accounted for under `arrayBuffers`, e.g. for `N` 1 million:
``
{
  rss: 31776768,
  heapTotal: 6463488,
  heapUsed: 3674520,
  external: 4285296,
  arrayBuffers: 4010422
}
``
Results for different N:
``
|| N
|| `arrayBuffers`
|| `rss`
|| `rss` per elem

| 1 M
| 4 MB
| 31 MB
| 5

| 10 M
| 40 MB
| 67 MB
| 4.6

| 100 M
| 40 MB
| 427 MB
| 4
``
We see therefore that typed arrays are much closer to what they advertise (4 bytes per element), even for smaller element counts, as expected.

Now let's try one million objects of type `{ a: 1, b: -1 }`:
``
node --expose-gc bench_mem.js obj
``
gives:
``
{
  rss: 138969088,
  heapTotal: 105246720,
  heapUsed: 70103896,
  external: 285296,
  arrayBuffers: 10422
}
``
Disaster! Memory usage is up to 70 MB! Why?? We were expecting only about 24, 4 baseline + 2 * 10 for each million int?!

And now an equivalent version using `class`:
``
node --expose-gc bench_mem.js class
``
gives the same result.

Let's try Array:
``
node --expose-gc bench_mem.js arr
``
is even worse at 78 MB!! OMG why.
``
{
  rss: 164597760,
  heapTotal: 129363968,
  heapUsed: 78117008,
  external: 285296,
  arrayBuffers: 10422
}
``

Let's change the number of fields on the object? First as a sanity check:
``
node --expose-gc bench_mem.js obj 2
``
produces as expected the smae result as:
``
node --expose-gc bench_mem.js obj
``
so adding properties one by one doesn't change anything from creating the literal all at once. Good.

Now:
``
node --expose-gc bench_mem.js obj N
``
gives `heapUsed`:
* 1: 70M
* 2: 70M
* 3: 70M
* 4: 70M
* 5: 110M
* 6: 110M
* 7: 110M
* 8: 134M
* 9: 134M
* 10: 134M
* 11: 158M

= Node.js library
{c}
{parent=Node.js}

= Node.js standard library
{c}
{parent=Node.js library}

= Node.js `worker_threads`
{c}
{parent=Node.js standard library}

= Node.js database bindings
{parent=Node.js library}

* <node.js SQLite bindings>

= Node.js ORM library
{parent=Node.js library}
{tag=Object-relational mapping}

\Include[sequelize]

= Node.js web framework
{c}
{parent=Node.js}

= Express.js
{c}
{parent=Node.js web framework}

This doesn't do a hole lot. <Ciro Santilli> wouldn't really call it a web framework. It's more like a middleware. Real web frameworks are built on top of it.

Examples under: \a[nodejs/express]:
* \a[nodejs/express/min.js]: minimal example. Visit http://localhost:3000 and it shows `hello world`. It is a bit wrong because the headers say HTML but we return plaintext.
* \a[nodejs/express/index.js]: example dump with automated tests where possible. The automated tests are run at startup after the server launches. Then the server keeps running so you can interact with it.

A live example on <Heroku> can be seen at: https://github.com/cirosantilli/heroku-node-min

= Realworld app written in Express
{c}
{parent=Express.js}

<gothinkster realworld> implementations based on <Express.js>.

= gothinkster/node-express-realworld-example-app
{parent=Realworld app written in Express}

https://github.com/gothinkster/node-express-realworld-example-app

= sigoden/node-express-realworld-example-app
{parent=Realworld app written in Express}

Appears to be a port of <gothinkster node-express-realworld-example-app> to <Sequelize>.

Seemed to just work at 68bbadfd77f679f0df0fcd0de5bceb9c37b1144a Ubuntu 20.10, was forked from parent project in 2018.

= Varun-Hegde/Conduit_NodeJS
{parent=Realworld app written in Express}

Very raw. Easy to understand. Relatively well organiezd. But also very buggy at 3ab8d9f849a1cdf2985a8d123b1893f0fd4e79ab: https://github.com/Varun-Hegde/Conduit_NodeJS/issues/3[], I just can't trust it. There must be several helper libraries that would greatly DRY up the repetitive CRUD. Ciro hates the style :-) 4 space indents, no space after commas, no semicolon. Not based on https://github.com/gothinkster/node-express-realworld-example-app[] which is essentially one of the reference implementations, so from scratch apparently, which is a bad sign.

= FeathersJS
{c}
{parent=Node.js web framework}

* https://feathersjs.com
* https://github.com/feathersjs/feathers
* https://stackoverflow.com/questions/tagged/feathersjs

Looks interesting.

It seems to abstract the part about the client messaging the backend, which focuses on being able to easily plug in a number of <Front-end web framework> to manage client state.

Has the "main web <API> is the same as the REST API" focus, which is fundamental 2020-nowadays.

Uses <Socket.IO>, which allows the client Javascript to register callbacks when data is updated to achieve <Socket.IO>, e.g. their default chat app does:
``
client.service('messages').on('created', addMessage);
``
so that message appear immediately as they are sent.

Their standard template from `feathers generate app` on `@feathersjs/cli@4.5.0` includes:
* several authentication methods, including <OAuth>
* testing
* backend <database> with one of several <object-relational mapping>! However, they don't abstract across them. E.g., the default Chat example uses NeDB, but a real app will likely use <Sequelize>, and a <port (software-portability)> is needed
which looks promising! They don't have a default template for a <Front-end web framework> however unfortunately: https://docs.feathersjs.com/guides/frameworks.html#the-feathers-chat lists a few chat app versions, which is their <hello world>:
* <Front-end web framework>: not built-in on generator, but there are some sample repos pointed from the documentation, and they did work out-of-box:
  * <feathers-chat-react>
But it is in itself a completely boring app with a single splash page, and no database interaction, so not a good showcase. The actual showcase app is <feathersjs feathers-chat>.

And there is no official example of the chat app that is immediately deployable to <Heroku>: <FeathersJS Heroku deployment>, all setups require thinking.

Global source entry point: determine on `package.json` as usual, defaults to `src/index.js`.

= FeatherJS demo apps
{c}
{parent=FeathersJS}

https://github.com/feathersjs/awesome-feathersjs#projects-using-feathers

= feathersjs/feathers-chat
{c}
{parent=FeatherJS demo apps}

https://github.com/feathersjs/feathers-chat

The main <FeathersJS> <hello world> demo. Notable missing things...
* instant <Heroku> deployability: <FeathersJS Heroku deployment>
* no <Front-end web framework> which sucks, but there are basically official demos that worked e.g. <feathers-chat-react>
* <FeathersJS signup email verification>

= feathers-chat PostgreSQL
{c}
{parent=feathersjs feathers-chat}

The default feathers-chat app runs on NeDB (local filesystem <JSON> database).

<Ciro Santilli> managed to port it to <Sequelize> for <PostgreSQL> as shown at: https://github.com/cirosantilli/feathers-chat/tree/sequelize-pg

= feathers-chat-react
{c}
{parent=feathersjs feathers-chat}

https://github.com/feathersjs-ecosystem/feathers-chat-react

Last updated 2018 as of 2021, but still just worked.

Also uses <webpack> which is fantastic.

Gotta run https://github.com/feathersjs/feathers-chat[] first: https://github.com/feathersjs-ecosystem/feathers-chat-react/issues/5[], then it worked:
``
git clone https://github.com/feathersjs/feathers-chat
cd feathers-chat
git checkout fd729a47c57f9e6170cc1fa23cee0c84a004feb5
npm install
npm start
``
and on the other terminal:
``
git clone https://github.com/feathersjs-ecosystem/feathers-chat-react
cd feathers-chat-react
git checkout 36d56cbe80bbd5596f6a108b1de9db343b33dac3
npm install
npm start
``
then visit http://localhost:3000/ and you can create an account and login, tested on Ubuntu 20.10. Data is stored on persistently.

TODO how to merge those two repos into a single repo.

If you <disable JavaScript on Chromium>, it stops working completely. There is a section on how to solve that at: https://docs.feathersjs.com/cookbook/express/view-engine.html[] but it does not cover React specifically. <Codaisseur feathersjs-react-redux-ssr> might be good to look into.

= Codaisseur/feathersjs-react-redux-ssr
{c}
{parent=FeatherJS demo apps}

https://github.com/Codaisseur/feathersjs-react-redux-ssr

Also <webpack> and <Babel (transcompiler)>, looks promising!

As of 2021, last commit from 2017.

Running:
``
git clone https://github.com/Codaisseur/feathersjs-react-redux-ssr
cd feathersjs-react-redux-ssr
npm install
``
failed on Ubuntu 20.10 <Node.js> v14.15.3 with:
``
../src/create_string.cpp:17:37: error: no matching function for call to ‘v8::String::Utf8Value::Utf8Value(v8::Local<v8::Value>&)’
   17 |   v8::String::Utf8Value string(value);
      |                                     ^
``
Likely similar <bullshit> from: https://stackoverflow.com/questions/50111688/node-sqlite-node-gyp-build-error-no-member-named-forceset-in-v8object because the Node.js version is too new.

If I try `nvm install v10`

I <Google> error messages until reaching:
``
diff --git a/gulpfile.js b/gulpfile.js
index b931e06..24d2cc8 100644
--- a/gulpfile.js
+++ b/gulpfile.js
@@ -14,34 +14,34 @@ gulp.task('css', function() {
            .pipe(gulp.dest('./dist'))
 })
 
-gulp.task('css:watch', ['css'], function() {
+gulp.task('css:watch', gulp.series('css', function() {
   gulp.watch('app/styles/**/*.sass', ['css'])
-})
+}))
 
 gulp.task('moveAssets', function() {
   return gulp.src('./app/assets/**/*')
              .pipe(gulp.dest('./dist/assets'))
 })
 
-gulp.task('build:revAssets', ['css', 'moveAssets'], function() {
+gulp.task('build:revAssets', gulp.series('css', 'moveAssets', function() {
   var rev = new $.revAll()
   return gulp.src('./dist/**/*')
              .pipe(rev.revision())
              .pipe(gulp.dest('./dist/public'))
              .pipe(rev.manifestFile())
              .pipe(gulp.dest('./dist'))
-})
+}))
 
 gulp.task('build:cpServer', function() {
   return gulp.src('./app/**/*.{js,ejs}')
              .pipe(gulp.dest('./dist/server-build'))
 })
-gulp.task('build:revServer', ['build:cpServer'], function() {
+gulp.task('build:revServer', gulp.series('build:cpServer', function() {
   var manifest = gulp.src('./dist/rev-manifest.json')
   return gulp.src('./dist/server-build/{components,containers}/**/*')
              .pipe($.revReplace({ manifest: manifest }))
              .pipe(gulp.dest('./dist/server-build'))
-})
+}))
 
 gulp.task('build', function() {
   runSequence('build:revAssets', 'build:revServer')
diff --git a/package.json b/package.json
index bcb29c3..86bd593 100644
--- a/package.json
+++ b/package.json
@@ -67,7 +67,7 @@
     "redux-thunk": "^0.1.0",
     "request": "^2.79.0",
     "rewire": "^2.3.4",
-    "run-sequence": "^1.2.2",
+    "run-sequence": "^2.2.1",
     "serve-favicon": "^2.3.2",
     "socket.io-client": "^1.7.2",
     "superagent": "^1.4.0",
@@ -86,16 +86,16 @@
     "concurrently": "^2.0.0",
     "cross-env": "^1.0.7",
     "enzyme": "^2.3.0",
-    "gulp": "^3.9.0",
+    "gulp": "^4.0.2",
     "gulp-autoprefixer": "^3.1.0",
     "gulp-load-plugins": "^1.2.0",
     "gulp-rev": "^6.0.1",
-    "gulp-sass": "^2.1.1",
+    "gulp-sass": "4.1.0",
     "gulp-sourcemaps": "^1.6.0",
     "jsdom": "^7.0.1",
     "mocha": "^2.4.5",
     "nock": "^2.17.0",
-    "node-sass": "^3.4.2",
+    "node-sass": "^5.0.0",
     "nodemon": "^1.6.0",
     "react-addons-test-utils": "^15.3.2",
     "react-transform-catch-errors": "^1.0.0",
``
and the next problem is: https://stackoverflow.com/questions/48513573/gulp-error-gulp-hastask-is-not-a-function

= randyscotsmithey/feathers-realworld-example-app
{parent=FeatherJS demo apps}

<FeathersJS> entry for <gothinkster realworld>.

<MongoDB>-based.

So once you install MongoDB, run with:
``
MONGODB_FEATHERS_REALWORLD=mongodb://localhost:27017/mydb npm start
``

Got it working on <Ubuntu> 20.10 with both <React> and <Vue.js> front-ends at https://github.com/randyscotsmithey/feathers-realworld-example-app/commit/8bc3a09242285de624c75bb8345630df499a7d07[] as mentioned at https://github.com/randyscotsmithey/feathers-realworld-example-app/issues/2 except for bad error reporting on UI.

Tests can be run with:
``
MONGODB_FEATHERS_REALWORLD=mongodb://localhost:27017/mydb npm run test
``
but there were 10 failures and 55 passes: https://github.com/randyscotsmithey/feathers-realworld-example-app/issues/3

= FeathersJS Heroku deployment
{c}
{parent=FeathersJS}

Got it working as mentioned at: https://github.com/cirosantilli/feathers-chat/tree/sequelize-pg

One major step was to <port (software-portability)> to <PostgreSQL> as shown at <feathers-chat PostgreSQL>.

Bibliography:
* https://stackoverflow.com/questions/47270219/deploy-feathersjs-app-on-heroku/66723974#66723974
* https://github.com/feathersjs/feathers/issues/1647
* https://medium.com/@mattchewone/feathersjs-deployment-gitlab-ci-cd-heroku-4ea9e34ab129

There's also a `heroku` branch at: https://github.com/feathersjs/feathers-chat/tree/heroku[], but it also seems to use NeDB? So you can have a filesystem in Heroku? Doesn't seem so: https://stackoverflow.com/questions/42775418/heroku-local-persistent-storage

= FeathersJS signup email verification
{c}
{parent=FeathersJS}

* https://github.com/feathersjs/feathers/issues/1514
* https://blog.feathersjs.com/how-to-setup-email-verification-in-feathersjs-72ce9882e744

= Meteor
{disambiguate=web framework}
{c}
{parent=Node.js web framework}

https://github.com/meteor/meteor

The idea is cool. It really unifies front-and back end.

But <Ciro Santilli> feels the approach proposed by <FeathersJS> of being a glue between bigger third-party <Front-end web frameworks> like <React> and backend (<object-relational mapping>) is more promising and flexible.

= Nest.js
{c}
{parent=Node.js web framework}

* https://github.com/lujakob/nestjs-realworld-example-app
* <object-relational mapping>: https://docs.nestjs.com/techniques/database
* <Front-end web framework>: looks weak:
  * https://github.com/yemiwebby/nest-react-project

= lujakob/nestjs-realworld-example-app
{c}
{parent=Nest.js}

<Nest.js> entry for <gothinkster realworld>.

Didn't manage to get it to work perfectly on <Ubuntu> 20.10: https://github.com/lujakob/nestjs-realworld-example-app/issues/57

= lujakob/nestjs-realworld-example-app SQLite port
{parent=lujakob nestjs-realworld-example-app}

Tried a quick port to <SQLite> to get rid of annoying local databases for development, but failed, at c1c2cc4e448b279ff083272df1ac50d20c3304fa
``
npm install sqlite3 --save-dev
``
and
``
{
  "type": "sqlite",
  "database": "db.sqlite3",
  "entities": ["src/**/**.entity{.ts,.js}"],
  "synchronize": true
}
``
then:
``
npm start
``
fails with:
``
DataTypeNotSupportedError: Data type "timestamp" in "ArticleEntity.created" is not supported by "sqlite" database.
``
Attempt to hack it:
``
--- a/src/article/article.entity.ts
+++ b/src/article/article.entity.ts
@@ -20,10 +20,10 @@ export class ArticleEntity {
   @Column({default: ''})
   body: string;

-  @Column({ type: 'timestamp', default: () => "CURRENT_TIMESTAMP"})
+  @Column({ default: () => "CURRENT_TIMESTAMP"})
   created: Date;

-  @Column({ type: 'timestamp', default: () => "CURRENT_TIMESTAMP"})
+  @Column({ default: () => "CURRENT_TIMESTAMP"})
   updated: Date;
``
and after that it seems to run.

I can signup and login, terrible error reporting as usual, make sure to use long enough usernames/passwords.

However, article creation fails with:
``
Unhandled Rejection (TypeError): Cannot read property 'slug' of undefined
``

= Sails.js
{c}
{parent=Node.js web framework}

https://github.com/balderdashy/sails

<Front-end web framework> integration: no native one:
* <React>:
  * https://github.com/markmur/sails-react-webpack
  * https://github.com/sepineda/sails-react-webpack Last updated as of 2021: 2018
  * https://stackoverflow.com/questions/28331032/sails-js-with-react-js-how-to-do-it-correctly
* <Vue.js>:
  * https://github.com/mikermcneil/ration Issue tracker disabled...
    * live at: https://ration.io/
    * selling a course at: https://courses.platzi.com/courses/sails-js/
  * https://platzi.com/cursos/javascript-pro/ non-free and in Spanish pointed to from official README...
  * <Nuxt.js>:
    * https://github.com/AngelMunoz/sails-nuxt

TODO <server-side rendering> anyone??
* https://stackoverflow.com/questions/32412590/how-to-use-react-js-to-render-server-side-template-on-sails-js
* https://stackoverflow.com/questions/54217147/ssr-for-react-redux-application-with-sails
* https://gist.github.com/duffpod/746a660bcddfd986878c92dde1a04f06
* https://www.reddit.com/r/reactjs/comments/7saoqm/sailsjs_or_adonisjs_designed_for_server_side/

= NVM
{c}
{parent=Node.js}

https://github.com/nvm-sh/nvm

The best way to install Node.js:
* https://stackoverflow.com/questions/16898001/how-to-install-a-specific-version-of-node-on-ubuntu-debian/47376491
* https://askubuntu.com/questions/49390/how-do-i-install-the-latest-version-of-node-js/425888#425888
* https://askubuntu.com/questions/594656/how-to-install-the-latest-versions-of-nodejs-and-npm/971612#971612
* https://askubuntu.com/questions/426750/how-can-i-update-my-nodejs-to-the-latest-version/1115255#1115255