Adventures in Mastodon Self-Hosting: Upgrade to 4.1.0rc1

Github Release: https://github.com/mastodon/mastodon/releases/tag/v4.1.0rc1

Note, my shell sets RAILS_ENV=production so I often skip that on a per-command bases, contra the official docs.

Be running as mastodon user: su - mastodon

Took a backup of postgres using my backup script notes : ./backup.sh

Fetch the new commits: git fetch

Checkout the release: git checkout v4.1.0rc1

Install ruby dependencies: bundle install

Install js dependencies: yarn install

DB Migrations: bundle exec rails db:migrate

1
2
3
4
Migrating to BackfillAdminActionLogsAgain (20221206114142)
== 20221206114142 BackfillAdminActionLogsAgain: migrating =====================
Scoped order is ignored, it's forced to be batch order.
== 20221206114142 BackfillAdminActionLogsAgain: migrated (0.0952s) ============

Compile assets: bundle exec rails assets:precompile

Note: very slow due to underpowered CPU and memory contention

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
mastodon@mastodon:~/live$ bundle exec rails assets:precompile
yarn install v1.22.19
[1/6] Validating package.json...
[2/6] Resolving packages...
[3/6] Fetching packages...
[4/6] Linking dependencies...
warning Workspaces can only be enabled in private projects.
[5/6] Building fresh packages...
[6/6] Cleaning modules...
Done in 28.62s.
Compiling...
Compilation failed:

<--- Last few GCs --->

[18995:0x67b8e50] 242239 ms: Mark-sweep 476.4 (494.6) -> 474.3 (494.6) MB, 6600.1 / 0.2 ms (average mu = 0.093, current mu = 0.033) allocation failure scavenge might not succeed
[18995:0x67b8e50] 245729 ms: Mark-sweep 476.4 (494.6) -> 474.5 (494.9) MB, 3299.4 / 0.0 ms (average mu = 0.078, current mu = 0.054) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb06730 node::Abort() [node]
2: 0xa1b6d0 [node]
3: 0xce1e60 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xce2207 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xe99875 [node]
6: 0xe9a356 [node]
7: 0xea887e [node]
8: 0xea92c0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xeac23e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xe6d77a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x11e64e6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x15da159 [node]

Ran out of memory. Try patching out parallelization and devtool (source-maps).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
mastodon@mastodon:~/live$ vim config/webpack/production.js
mastodon@mastodon:~/live$ git diff
diff --git a/config/webpack/production.js b/config/webpack/production.js
index 143a23b99..b22dd1661 100644
--- a/config/webpack/production.js
+++ b/config/webpack/production.js
@@ -14,7 +14,7 @@ const root = resolve(__dirname, '..', '..');

module.exports = merge(sharedConfig, {
mode: 'production',
- devtool: 'source-map',
+ devtool: 'none',
stats: 'normal',
bail: true,
optimization: {
@@ -22,8 +22,8 @@ module.exports = merge(sharedConfig, {
minimizer: [
new TerserPlugin({
cache: true,
- parallel: true,
- sourceMap: true,
+ parallel: false,
+ sourceMap: false,
}),
],
},
1
2
3
4
5
6
7
8
mastodon@mastodon:~/live$ bundle exec rails assets:precompile
yarn install v1.22.19
[1/6] Validating package.json...
[2/6] Resolving packages...
success Already up-to-date.
Done in 1.56s.
Compiling...
Compiled all packs in /home/mastodon/live/public/packs

Worked this time.

Restart all mastodon processes: (exit to root first) systemctl restart mastodon-*

Notes for next time

Probably should stop mastodon processes during the upgrade. Theres an extended period where database and code could be mismatched due to how slow the install process is on this machine. Incoming federation should be retried by the sender if we are not up to receive.

This might also give some more breathing room to the install and compile steps.

systemctl stop mastodon-* might do it