cgroups limits no longer respected? #6882
Labels
Milestone
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Date: 2020-06-09 17:56:49 +0200
From: @swingbit
To: GDK devs <>
Version: 11.35.19 (Nov2019-SP3)
CC: @yzchang
Last updated: 2020-07-27 09:30:10 +0200
Comment 27805
Date: 2020-06-09 17:56:49 +0200
From: @swingbit
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Build Identifier:
In a system with 24GiB of RAM, MonetDB is run in Docker with memlimit=16g.
What Docker does is setting cgroups limits.
Starting from Apr2019, MonetDB reads those limits and uses them as caps for the resources it is allowed to use.
I remember testing that. mserver5 would report exactly the memory as limited by docker (thus: by cgroup). That seems to no longer be happening:
This is from inside the docker container.
cgroups limits memory to 16GiB, but MonetDB sees 24GiB:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
17179869184
/opt/monetdb/bin/mserver5 --version
MonetDB 5 server 11.35.20 (64-bit, 128-bit integers)
This is an unreleased version
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2020 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 23.5GiB available memory, 8 available cpu cores
Libraries:
libpcre: 8.43 2019-02-23
openssl: OpenSSL 1.1.1d FIPS 10 Sep 2019
libxml2: 2.9.9
Compiled by: @634c8f6244bd (x86_64-pc-linux-gnu)
Compilation: gcc -O3 -fomit-frame-pointer -pipe -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wundef -Wformat=2 -Wformat-overflow=1 -Wno-format-truncation -Wno-format-nonliteral -Wno-cast-function-type -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wno-missing-field-initializers -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wduplicated-cond -Wduplicated-branches -Wrestrict -Wnested-externs -Wno-char-subscripts -Wunreachable-code
Linking : /usr/bin/ld -m elf_x86_64 -Wl,-Bsymbolic-functions
Reproducible: Always
Comment 27806
Date: 2020-06-09 18:16:31 +0200
From: @sjoerdmullender
Are you sure mserver5 --version reported the limit before? I doubt it. The version information is printed very early on and then the server exits. It doesn't go through the code that initializes all these memory variables.
What does mserver5 say without the --version argument?
Comment 27807
Date: 2020-06-09 18:31:44 +0200
From: @swingbit
Yes, at some point is was visible:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
2147483648
/opt/monetdb/bin/mserver5 --version
MonetDB 5 server 11.33.12 (64-bit, 128-bit integers)
This is an unreleased version
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2019 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 2.0GiB available memory, 8 available cpu cores
Libraries:
libpcre: 8.43 2019-02-23
openssl: OpenSSL 1.1.1c FIPS 28 May 2019
libxml2: 2.9.9
Compiled by: @a9e9f9e384d5 (x86_64-pc-linux-gnu)
Compilation: gcc -O3 -fomit-frame-pointer -pipe -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wundef -Wformat=2 -Wformat-overflow=1 -Wno-format-truncation -Wno-format-nonliteral -Wno-cast-function-type -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wno-missing-field-initializers -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wduplicated-cond -Wduplicated-branches -Wrestrict -Wnested-externs -Wno-char-subscripts -Wunreachable-code
Linking : /usr/bin/ld -m elf_x86_64 -Wl,-Bsymbolic-functions
Another clue that the limit is not actually set is that more and more often we see MonetDB that is OOM-killed as soon as it reached the max memory allowed, with still plenty of swap and disk space. If the limits were set correctly, it should be way more difficult to reach that moment.
Comment 27808
Date: 2020-06-09 18:36:04 +0200
From: @swingbit
Also, starting mserver5 without --version (so, actually starting the default database, gives the same memory information)
Comment 27831
Date: 2020-06-12 12:14:35 +0200
From: @swingbit
I compared again a Nov2019 system with a Apr2019 system.
On a Nov2019 (11.35.20) with 24G of system RAM:
cgroups limits it to 12G
[root@9f334da3261c /] cat /sys/fs/cgroup/memory/memory.limit_in_bytes
12884901888
20G are available
sql>select * from env() where name = 'gdk_mem_maxsize';
+-----------------+-------------+
| name | value |
+=================+=============+
| gdk_mem_maxsize | 20603770839 |
+-----------------+-------------+
1 tuple
On a Apr2019 (11.33.12) with 32G of system RAM:
cgroups limits it to 2G
[root@4b5f37cbd7c9 /] cat /sys/fs/cgroup/memory/memory.limit_in_bytes
2147483648
1.6G are available:
sql>select * from env() where name = 'gdk_mem_maxsize';
+-----------------+------------+
| name | value |
+=================+============+
| gdk_mem_maxsize | 1750199173 |
+-----------------+------------+
1 tuple
Comment 27832
Date: 2020-06-12 12:18:53 +0200
From: @swingbit
I see this in the Nov2019 release notes:
Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively.
Just a guess, perhaps when these re not explicitly set their default values replace anyway what found in cgroups settings?
Comment 27833
Date: 2020-06-12 14:47:19 +0200
From: @yzchang
Those options were introduced as extra safeguard and convenience.
We try to find, e.g., CGroups limitations set on mserver5, but we don't always succeed, because the location of the CGroups information can sometimes be hard to determine.
Afaik, we take the minimum of CGroups limitations (if we can find them) and the values of "vmmaxsize" and "memmaxsize" (if set). The actual max sizes for vm and mem are printed in the mserver5 start-up msg, especially if they are less than the system's overall resources.
Comment 27834
Date: 2020-06-12 15:38:16 +0200
From: @sjoerdmullender
What might help me find the issue is if you could run mserver5 under strace. I'm interested in the output of
strace -s 1024 mserver5 --version
starting with the opening of the file /proc/self/mountinfo until the end. (You can use the strace -o option to save the output to a file.)
Comment 27837
Date: 2020-06-12 17:38:52 +0200
From: @swingbit
Created attachment 682
Output of strace
Comment 27838
Date: 2020-06-12 17:40:53 +0200
From: @swingbit
Apparently it doesn't find them:
openat(AT_FDCWD, "/sys/fs/cgroup/memory/docker/2951843bb482c7502f1c9e1ab8b5de271f6eac75ef227d5949c175e4b6687612/memory.limit_in_bytes", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/fs/cgroup/memory/docker/2951843bb482c7502f1c9e1ab8b5de271f6eac75ef227d5949c175e4b6687612/memory.soft_limit_in_bytes", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/fs/cgroup/memory/docker/2951843bb482c7502f1c9e1ab8b5de271f6eac75ef227d5949c175e4b6687612/memory.memsw.limit_in_bytes", O_RDONLY) = -1 ENOENT (No such file or directory)
Comment 27839
Date: 2020-06-12 17:43:08 +0200
From: @swingbit
What is strange is that the /docker structure should not be visible (it isn't) from inside the docker container, but only from the host.
The docker container only sees /sys/fs/cgroup/memory/memory.limit_in_bytes
Comment 27840
Date: 2020-06-12 17:51:45 +0200
From: @swingbit
Created attachment 683
Trace output of 11.33.12 (works)
Comment 27841
Date: 2020-06-12 17:52:19 +0200
From: @swingbit
11.33.12 doesn't look into the /docker structure
Comment 27845
Date: 2020-06-15 11:00:55 +0200
From: MonetDB Mercurial Repository <>
Changeset 5495166fbc3c made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.
For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=5495166fbc3c
Changeset description:
Comment 27847
Date: 2020-06-15 14:44:11 +0200
From: @swingbit
Back-ported to Nov2019 and tested, the fix works.
Comment 27849
Date: 2020-06-15 15:57:18 +0200
From: @sjoerdmullender
Great. Thanks for testing.
The text was updated successfully, but these errors were encountered: