Database Borked Itself. Any ideas to repair or fix the issue?

Operating Systems & Open Source
1

Hey folks.

So my nextcloud postgres docker database desided to literally grenade itself randomly somehow without warning, nor power outtage, nor data corruption on the ZFS Raid Z2 array

The server just responds with internal error

2022-05-27 18:08:43.328 MDT [46752] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:43.328 MDT [46752] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:43.328 MDT [46752] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:43.422 MDT [46753] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:43.422 MDT [46753] CONTEXT:  automatic analyze of table "nextcloud.public.oc_appconfig"
2022-05-27 18:08:43.433 MDT [46753] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:43.433 MDT [46753] CONTEXT:  automatic analyze of table "nextcloud.public.oc_jobs"
2022-05-27 18:08:43.903 MDT [46754] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:43.903 MDT [46754] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:43.903 MDT [46754] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:43.903 MDT [46754] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.298 MDT [46755] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.298 MDT [46755] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.298 MDT [46755] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.298 MDT [46755] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.547 MDT [46756] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.547 MDT [46756] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.547 MDT [46756] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.547 MDT [46756] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.800 MDT [46757] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.800 MDT [46757] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:44.800 MDT [46757] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:44.800 MDT [46757] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:45.096 MDT [46758] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:45.096 MDT [46758] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:45.097 MDT [46758] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:45.097 MDT [46758] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:45.416 MDT [46759] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:45.416 MDT [46759] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:45.416 MDT [46759] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:45.416 MDT [46759] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:46.929 MDT [46760] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:46.929 MDT [46760] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:46.929 MDT [46760] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:46.929 MDT [46760] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.202 MDT [46761] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.202 MDT [46761] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.203 MDT [46761] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.203 MDT [46761] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.497 MDT [46762] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.497 MDT [46762] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.498 MDT [46762] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.498 MDT [46762] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.801 MDT [46763] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.801 MDT [46763] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:47.802 MDT [46763] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:47.802 MDT [46763] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.138 MDT [46764] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.138 MDT [46764] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.138 MDT [46764] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.138 MDT [46764] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.459 MDT [46765] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.459 MDT [46765] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.459 MDT [46765] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.459 MDT [46765] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.947 MDT [46774] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.947 MDT [46774] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:48.948 MDT [46774] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:48.948 MDT [46774] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:49.432 MDT [46775] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:49.432 MDT [46775] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:49.432 MDT [46775] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:49.432 MDT [46775] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.135 MDT [46776] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.135 MDT [46776] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.135 MDT [46776] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.135 MDT [46776] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.451 MDT [46777] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.451 MDT [46777] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.452 MDT [46777] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.452 MDT [46777] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.866 MDT [46778] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.866 MDT [46778] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:50.866 MDT [46778] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:50.866 MDT [46778] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.180 MDT [46779] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.180 MDT [46779] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.181 MDT [46779] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.181 MDT [46779] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.490 MDT [46780] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.490 MDT [46780] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.491 MDT [46780] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.491 MDT [46780] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.807 MDT [46781] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.807 MDT [46781] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:51.808 MDT [46781] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:51.808 MDT [46781] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.307 MDT [46782] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.307 MDT [46782] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.308 MDT [46782] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.308 MDT [46782] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.618 MDT [46783] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.618 MDT [46783] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.618 MDT [46783] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.618 MDT [46783] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.916 MDT [46784] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.916 MDT [46784] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:52.916 MDT [46784] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:52.916 MDT [46784] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.224 MDT [46785] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.224 MDT [46785] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.225 MDT [46785] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.225 MDT [46785] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.487 MDT [46786] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.487 MDT [46786] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.488 MDT [46786] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.488 MDT [46786] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.987 MDT [46787] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.987 MDT [46787] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:53.987 MDT [46787] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:53.987 MDT [46787] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:54.528 MDT [46788] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:54.528 MDT [46788] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:54.529 MDT [46788] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:54.529 MDT [46788] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:54.846 MDT [46789] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:54.846 MDT [46789] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:54.846 MDT [46789] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:54.846 MDT [46789] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.168 MDT [46790] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.168 MDT [46790] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.168 MDT [46790] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.168 MDT [46790] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.490 MDT [46791] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.490 MDT [46791] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.490 MDT [46791] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.490 MDT [46791] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.807 MDT [46792] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.807 MDT [46792] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:55.808 MDT [46792] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:55.808 MDT [46792] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:56.135 MDT [46793] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:56.135 MDT [46793] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:56.135 MDT [46793] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:56.135 MDT [46793] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:56.821 MDT [46794] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:56.821 MDT [46794] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:56.821 MDT [46794] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:56.821 MDT [46794] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.079 MDT [46795] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.079 MDT [46795] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.079 MDT [46795] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.079 MDT [46795] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.422 MDT [46796] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.422 MDT [46796] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.422 MDT [46796] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.422 MDT [46796] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.725 MDT [46797] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.725 MDT [46797] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:57.726 MDT [46797] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:57.726 MDT [46797] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:58.055 MDT [46798] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:58.055 MDT [46798] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:58.055 MDT [46798] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:58.055 MDT [46798] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:58.711 MDT [46799] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:58.711 MDT [46799] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:58.711 MDT [46799] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:58.711 MDT [46799] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:59.189 MDT [46808] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:59.189 MDT [46808] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:59.189 MDT [46808] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:59.189 MDT [46808] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:59.747 MDT [46809] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:59.747 MDT [46809] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:08:59.747 MDT [46809] ERROR:  cache lookup failed for index 3379
2022-05-27 18:08:59.747 MDT [46809] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.023 MDT [46810] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.023 MDT [46810] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.023 MDT [46810] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.023 MDT [46810] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.280 MDT [46811] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.280 MDT [46811] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.281 MDT [46811] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.281 MDT [46811] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.572 MDT [46812] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.572 MDT [46812] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.572 MDT [46812] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.572 MDT [46812] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.871 MDT [46813] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.871 MDT [46813] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:00.871 MDT [46813] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:00.871 MDT [46813] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.114 MDT [46814] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.114 MDT [46814] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.114 MDT [46814] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.114 MDT [46814] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.370 MDT [46815] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.370 MDT [46815] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.371 MDT [46815] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.371 MDT [46815] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.694 MDT [46816] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.694 MDT [46816] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:01.694 MDT [46816] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:01.694 MDT [46816] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.015 MDT [46817] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.015 MDT [46817] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.015 MDT [46817] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.015 MDT [46817] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.282 MDT [46818] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.282 MDT [46818] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.282 MDT [46818] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.282 MDT [46818] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.522 MDT [46819] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.522 MDT [46819] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.523 MDT [46819] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.523 MDT [46819] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.794 MDT [46820] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.794 MDT [46820] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:02.795 MDT [46820] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:02.795 MDT [46820] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.321 MDT [46821] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.321 MDT [46821] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.322 MDT [46821] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.322 MDT [46821] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.606 MDT [46822] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.606 MDT [46822] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.606 MDT [46822] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.606 MDT [46822] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.924 MDT [46823] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.924 MDT [46823] STATEMENT:  SELECT * FROM "oc_appconfig"
2022-05-27 18:09:03.925 MDT [46823] ERROR:  cache lookup failed for index 3379
2022-05-27 18:09:03.925 MDT [46823] STATEMENT:  SELECT * FROM "oc_appconfig"
    ~ ▓▒░

Thats the endless error. please tell me this is recoverable in someway shape or form. I have no clue how it happened. Im looking for solutions and both google and duckduckgo yield me nothing for this specific index

Any and all useful help is appreciated!

2

I’d try reindexing the database if you haven’t already. If it’s just a bad index that got corrupted somehow, then yeah you should be able to rebuild it(backup first!), or just delete it(backup first!) and still have all your data.

It’s been a while since I used postgres, but I think 3379 would be the key for the pg_class table which might help you track down the name of the index so you can drop it if needed.

4 Likes
3

Do you have any backups of your /mnt/OnePoint21GigaWatts/containers/postgresql-data/ directory? I’d probably lean towards restoring a backup of that , even if you might lose 24 hours of data.

4

I dont really have a backup of it yet because I hadnt integrated backups into that part of docker yet unfortunately.

Thanks ill try again. First time it errored out. I will be persistent.

1 Like
5

As others mentioned, this looks like an index issue. So the first thing to try is a reindex.

However the zeroth step is to take a backup, any attempt to fix corruption may cause more corruption. After you’ve taken a backup try taking a pg_dump, which will create a text-only version of the database. A pg_dump only reads the data itself and will ignore any index corruption, and it will help show any missing data in the database.

Another thing that can happen is having both the old value and new value in a table after a bad UPDATE. In theory it should never happen, but in practise I’ve seen it a few times. However this will normally only trigger strange behaviour when reindexing with UNIQUE columns.

1 Like
6

@here

this is my issue. I cannot stop a docker container and put it in single user mode and reindex… Ideas here?

Starting the container will start all nextcloud containers. They depend on each other

❯ sudo docker exec -it -u postgres nextcloud-postgres-server postgres --single --ignore_system_indexes=on --enable_indexscan=off --enable_bitmapscan=off nextcloud
2022-05-27 23:26:19.723 MDT [88756] FATAL:  lock file "postmaster.pid" already exists
2022-05-27 23:26:19.723 MDT [88756] HINT:  Is another postmaster (PID 1) running in data directory "/var/lib/postgresql/data"?
    ~ ▓▒░
7

I’d probably create a separate dockerfile just for spinning up postgres in single user mode, and then connect to that. Otherwise you’ll have to look at ways to run a shell, or some other process with postgres inside it so you can stop and start it without killing the container in the process.

2 Likes
8

thanks thats a good idea

1 Like
9

Try “docker exec … psql”, this will give you a postgres prompt for the database. A few commands:

\l - list database
\c - connect do database
\dt - show tables

Connect to the right database and you should be able to do “REINDEX DATABASE db_name”, with the appropriate db_name. This can be run in parallel with nextcloud (though current sessions may freeze due to database locks).

Otherwise it sounds like you’re stuck in container debug hell.

2 Likes
10

Im getting unbelievably frustrated with google results lately

image
image1251×937 162 KB

They are a pile of crap

Exactly all the links on the first three pages are useless

2 Likes
11

It looks like to even do this Im going to need to alter the mapping so I can bash into the proper user

Ughhh why are database containers always setup in such a shit way out of the gate

like this:

root@d86f63a59be3:/var/lib/postgresql/data# pg_ctl stop nextcloud
pg_ctl: cannot be run as root
Please log in (using, e.g., "su") as the (unprivileged) user that will
own the server process.
root@d86f63a59be3:/var/lib/postgresql/data# sudo -u postgres pg_ctl stop nextcloud
sudo: pg_ctl: command not found
root@d86f63a59be3:/var/lib/postgresql/data#

How does that work out? Gonna have to whereis and manually path. LOL

Thank goodness for stack:

image
image1065×947 157 KB

Slowly figuring out how to launch this thing in single user mode

12

I shell’d into Rootz server and did the following, we think we might have fixed it:

  1. Mount the data vol in a new container, run that container with /bin/bash
  2. pgctl start with -o "-P", which disables system indexes.
  3. psql -u nextcloud connect to the database.
  4. \l to confirm you can access the db.
  5. REINDEX DB $DBNAME
  6. ???
  7. Profit
2 Likes
13

It worked. So heres the deal. I need to write this up and write up a solid backup solution. Ill make a post on it once I figure that out so theres no more need for container debug hell like this.

Google is a pile of trash in terms of help these days

14

Temporary non backup solution

Hourly ZFS snapshots

Keep 36

15

you could also do pgdumps

16

I thought about that too but id like to systemD this and make it all seemless and easy to view in the logs. This would be my ideal solution. Then hold the last 7 days of daily dumps in an archive and delete the olders ones… this is what I would like but I havent a clue how to frame it yet.

17

cron the dumps daily, and have hourly zfs snapshots that self delete after 24 hours

this is what I do although I use mysql

18

Got that part taken care of

I have moved on from cron haha. But yeah the idea is the same. I just want to do so with systemD

Im wondering of datasets per container are smarter than one big dataset

19

what on earth are you talking about? you mean timers?

this 1000%

compartmentalization is your friend

20

Ahh yeah would just be about moving the data from one dataset to the other and remapping compose. Thats easy

Yes. System D services and timers. I prefer this over cron tasks. Also easier to view in my logs with journalctl. I have my reasons for this madness