giuseppe

giuseppe

Software engineer at Red Hat Inc.

Member Since 12 years ago

Red Hat,

Experience Points
224
follower
Lessons Completed
0
follow
Best Reply Awards
155
repos

2077 contributions in the last year

Pinned
⚡ A fast and lightweight fully featured OCI runtime and C library for running containers
⚡ Podman: A tool for managing OCI containers and pods.
⚡ FUSE implementation for overlayfs
⚡ Production-Grade Container Scheduling and Management
⚡ minimal tool for creating a new user namespace with multiple UIDs/GIDs mapped inside
⚡ a C library for accessing OCI runtime and image spec files
Activity
Jan
19
11 hours ago
Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

failed to make veth pair: operation not supported

# podman --version
podman version 3.3.1

# podman run -it --name mybusybox  docker.io/library/busybox
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 01c2cdc13739 done  
Copying config cabb9f684f done  
Writing manifest to image destination
Storing signatures
ERRO[0000] error loading cached network config: network "podman" not found in CNI cache 
WARN[0000] falling back to loading from existing plugins on disk 
Error: error configuring network namespace for container 670d4af03e7c805a9fd12b14cde25e473b89302710e316e7501ac150de4c4726: error adding pod mybusybox_mybusybox to CNI network "podman": failed to make veth pair: operation not supported


giuseppe
giuseppe

I you deem this irrelevant or "noise" feel free to remove my message.

I think it is useful for other users that end up with the same issue. Thanks for sharing it

Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

cgroup is not displayed as shared namespace in pod inspection

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman help pod create displays cgroup as one of default values of --share flag, which specifies shared namespaces in a pod. However, cgroup is not listed in SharedNameSpaces of a pod with default values. Even if cgroup is explicitly specified in --share flag, cgroup is not listed in a result of podman pod inspect and actually containers in a pod don't share a cgroup namespace. Regarding other options for --share such as ipc and net, specified values to --share are displayed as SharedNameSpaces in podman pod inspect.

In the source code, there are two similar parameters for cgroups in a pod. When cgroup is specified in --share, PodConfig.UsePodCgroup is set to true. For SharedNameSpaces in pod inspection, PodConfig.UsePodCgroupNS is referred to. There are two resources shared in a pod regarding cgroup, a cgroup parent and a cgroup namespace:

  • When PodConfig.UsePodCgroup is true, a cgroup parent is shared in a pod, so that all containers in the pod have the same cgroup parent.
  • If PodConfig.UsePodCgroupNS is true, a cgroup namespace is shared in a pod, so that all containers in the pod join the same cgroup namespace though this flag is currently never set.

There are some options for the issue:

  • Just add a note to documentation.
  • Share a cgroup parent as the current behavior and fix to display cgroup in SharedNameSpaces in podman pod inspect based on PodConfig.UsePodCgroup. Another change is required if it is necessary to share a cgroup namespace, which is currently not shared.
  • Fix to share a cgroup namespace by setting PodConfig.UsePodCgroupNS. Another fix is required for sharing a cgroup parent.

Steps to reproduce the issue:

  1. Confirm the default shared namespaces in a pod:
$ podman pod create --help

Create a new empty pod

Description:
  After creating the pod, the pod ID is printed to stdout.
<snip>
     --share string                  A comma delimited list of kernel namespaces the pod will share (default "cgroup,ipc,net,uts")
<snip>
  1. Create a pod:
$ podman pod create --name testpod
  1. Inspect shared namespaces of the created pod:
$ podman pod inspect --format '{{.SharedNamespaces}}' testpod

Describe the results you received:

"cgroup" is not included in the result:

$ podman pod inspect --format '{{.SharedNamespaces}}' testpod
[net uts ipc]

Describe the results you expected:

"cgroup" is included in the result:

$ podman pod inspect --format '{{.SharedNamespaces}}' testpod
[cgroup net uts ipc]

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Fri Nov 12 15:25:37 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "35"
  eventLogger: journald
  hostname: laptop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.7-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2182692864
  memTotal: 16295391232
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8587571200
  swapTotal: 8589930496
  uptime: 13h 58m 30.75s (Approximately 0.54 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 12
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 15:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.2-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

giuseppe
giuseppe

we already have cgroup-parent for run.

If we go the --share path, I think it should be --share=cgroup-parent.

As Matt pointed out though, I also find it confusing. Perhaps it should be podman run --cgroup-parent=pod-shared?

Activity icon
issue

giuseppe issue comment openshift/machine-config-operator

giuseppe
giuseppe

Bug 2038827: add /etc/sub{u,g}id files

to give the container managers access to user namespace ranges OOTB

Signed-off-by: Peter Hunt [email protected]

giuseppe
giuseppe

we could hardcode it in containers/storage but then there is the risk some other users on the system would end up using the same range of additional IDs.

That is probably not a problem on FCOS but it could be on systems with multiple users configured.

An alternative would be to create a containers user with no shell login and use usermod --add-subuids --add-subgids to add more IDs

open pull request

giuseppe wants to merge openshift/machine-config-operator

giuseppe
giuseppe

Bug 2038827: add /etc/sub{u,g}id files

to give the container managers access to user namespace ranges OOTB

Signed-off-by: Peter Hunt [email protected]

giuseppe
giuseppe

65536 is a very small range :-)

pull request

giuseppe merge to openshift/machine-config-operator

giuseppe
giuseppe

Bug 2038827: add /etc/sub{u,g}id files

to give the container managers access to user namespace ranges OOTB

Signed-off-by: Peter Hunt [email protected]

Activity icon
issue

giuseppe issue comment containers/buildah

giuseppe
giuseppe

rootless overlay support check fails if configured mount_program in /etc/containers/storage.conf is not in $PATH

Description Rootless buildah info fails overlay support check if the mount_program is installed at a non-default path and it's absolute path is configured in /etc/containers/storage.conf. buildah info passes if the mount_program's is found in $PATH or its absolute path is configured in $HOME/.config/containers/storage.conf or is set from the command line flag --storage-opt .

Reproducible by installing fuse-overlayfs at a non-default path - /home/ubuntu/bin/fuse-overlayfs and configure the same only in /etc/containers/storage.conf .

Steps to reproduce the issue:

  1. Install buildah from kubic repo without fuse-overlayfs
sudo buildah --debug info
{
    "debug": {
        "buildah version": "1.21.3",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.15.2"
    },
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 8771399680,
        "MemTotal": 12557430784,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "focal",
        "kernel": "5.4.0-94-generic",
        "os": "linux",
        "rootless": false,
        "uptime": "1h 41m 23.98s (Approximately 0.04 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mountopt=nodev,metacopy=on"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "true"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

[email protected]:~$ which fuse-overlayfs
[email protected]:~$

[email protected]:~$ buildah --debug info
kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted
WARN[0000] failed to shutdown storage: "kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted"
ERRO[0000] exit status 125
  1. Download fuse-overlayfs static binary, and set mount_program in /etc/containers/storage.conf
[email protected]:~$ wget https://github.com/containers/fuse-overlayfs/releases/download/v1.8/fuse-overlayfs-x86_64 -O /home/ubuntu/bin/fuse-overlayfs && chmod a+x /home/ubuntu/bin/fuse-overlayfs
[email protected]:~$ sudo sed -i 's/#mount_program.*/mount_program\ =\ "\/home\/ubuntu\/bin\/fuse-overlayfs"/' /etc/containers/storage.conf

[email protected]:~$ sudo buildah --log-level=trace info --debug
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/home/ubuntu/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
{
    "debug": {
        "buildah version": "1.21.3",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.15.2"
    },
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 8762142720,
        "MemTotal": 12557430784,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "focal",
        "kernel": "5.4.0-94-generic",
        "os": "linux",
        "rootless": false,
        "uptime": "3h 0m 8.78s (Approximately 0.12 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mount_program=/home/ubuntu/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,metacopy=on"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}
DEBU[0000] shutting down the store
  1. PASS: Rootless overlay check passes if mount_program in $PATH
[email protected]:~$ PATH=$PATH:/home/ubuntu/bin buildah --debug info
{
    "debug": {
        "buildah version": "1.21.3",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.15.2"
    },
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 8747040768,
        "MemTotal": 12557430784,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "focal",
        "kernel": "5.4.0-94-generic",
        "os": "linux",
        "rootless": true,
        "uptime": "2h 50m 46.84s (Approximately 0.08 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mount_program=/home/ubuntu/bin/fuse-overlayfs"
        ],
        "GraphRoot": "/home/ubuntu/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 1
        },
        "RunRoot": "/run/user/1000/containers"
    }
}
  1. PASS: Rootless overlay check passes if mount_program set by command line flag --storage-opt.
[email protected]:~$ which fuse-overlayfs
[email protected]:~$

[email protected]:~$ buildah --storage-driver=overlay --storage-opt="overlay.mount_program=/home/ubuntu/bin/fuse-overlayfs" info
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 8754876416,
        "MemTotal": 12557430784,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "focal",
        "kernel": "5.4.0-94-generic",
        "os": "linux",
        "rootless": true,
        "uptime": "2h 15m 44.86s (Approximately 0.08 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mount_program=/home/ubuntu/bin/fuse-overlayfs"
        ],
        "GraphRoot": "/home/ubuntu/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/user/1000/containers"
    }
}
  1. FAIL: Rootless overlay check fails if mount_program is only set in /etc/containers/storage.conf and it is not available in $PATH.
[email protected]:~$ which fuse-overlayfs
[email protected]:~$ 

[email protected]:~$ grep mount_program /etc/containers/storage.conf
mount_program = "/home/ubuntu/bin/fuse-overlayfs"

[email protected]:~$ buildah --log-level=trace info
DEBU[0000] running [buildah-in-a-user-namespace --log-level=trace info] with environment [SHELL=/bin/bash SECURITYTAGS= PWD=/home/ubuntu LOGNAME=ubuntu XDG_SESSION_TYPE=tty MOTD_SHOWN=pam HOME=/home/ubuntu LANG=C.UTF-8 
SSH_CONNECTION=172.16.91.1 62849 172.16.91.8 22 LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s USER=ubuntu SHLVL=1 XDG_SESSION_ID=11 XDG_RUNTIME_DIR=/run/user/1000 SSH_CLIENT=172.16.91.1 62849 22 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin BUILDTAGS=exclude_graphdriver_devicemapper DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus SSH_TTY=/dev/pts/0 OLDPWD=/home/ubuntu/buildah _=/usr/bin/buildah TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:1000 Size:1} {ContainerID:1 HostID:100000 Size:65536}], and GID map [{ContainerID:0 HostID:1000 Size:1} {ContainerID:1 HostID:100000 Size:65536}]
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is not supported
kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted
github.com/containers/buildah/vendor/github.com/containers/storage/drivers/overlay.checkAndRecordOverlaySupport
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/drivers/overlay/overlay.go:215
github.com/containers/buildah/vendor/github.com/containers/storage/drivers/overlay.Init
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/drivers/overlay/overlay.go:306
github.com/containers/buildah/vendor/github.com/containers/storage/drivers.GetDriver
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/drivers/driver.go:288
github.com/containers/buildah/vendor/github.com/containers/storage/drivers.New
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/drivers/driver.go:318
github.com/containers/buildah/vendor/github.com/containers/storage.(*store).getGraphDriver
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/store.go:824
github.com/containers/buildah/vendor/github.com/containers/storage.(*store).GraphDriver
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/store.go:841
github.com/containers/buildah/vendor/github.com/containers/storage.(*store).load
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/store.go:756
github.com/containers/buildah/vendor/github.com/containers/storage.GetStore
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/containers/storage/store.go:698
main.getStore
	/usr/src/packages/BUILD/src/github.com/containers/buildah/cmd/buildah/common.go:107
main.infoCmd
	/usr/src/packages/BUILD/src/github.com/containers/buildah/cmd/buildah/info.go:49
main.init.8.func1
	/usr/src/packages/BUILD/src/github.com/containers/buildah/cmd/buildah/info.go:33
github.com/containers/buildah/vendor/github.com/spf13/cobra.(*Command).execute
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/spf13/cobra/command.go:852
github.com/containers/buildah/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/spf13/cobra/command.go:960
github.com/containers/buildah/vendor/github.com/spf13/cobra.(*Command).Execute
	/usr/src/packages/BUILD/src/github.com/containers/buildah/vendor/github.com/spf13/cobra/command.go:897
main.main
	/usr/src/packages/BUILD/src/github.com/containers/buildah/cmd/buildah/main.go:220
runtime.main
	/usr/lib/go-1.15/src/runtime/proc.go:204
runtime.goexit
	/usr/lib/go-1.15/src/runtime/asm_amd64.s:1374
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is not supported
WARN[0000] failed to shutdown storage: "kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted"
ERRO[0000] exit status 125

Describe the results you received: Overlay check fails.

Describe the results you expected: Overlay check should pass if mount_program path is correctly configured in /etc/container/storage.conf.

Output of rpm -q buildah or apt list buildah:

[email protected]:~$ apt list buildah
Listing... Done
buildah/unknown,now 100:1.21.3-1 amd64 [installed]
buildah/unknown 100:1.21.3-1 arm64
buildah/unknown 100:1.21.3-1 armhf
buildah/unknown 100:1.21.3-1 s390x

Output of buildah version:

[email protected]:~$ buildah version
Version:         1.21.3
Go Version:      go1.15.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.12.0
Git Commit:
Built:           Wed Dec 31 19:00:00 1969
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

(paste your output here)

Output of cat /etc/*release:

[email protected]:~$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Output of uname -a:

[email protected]:~$ uname -a
Linux focal 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[email protected]:~$ cat /etc/containers/storage.conf
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/home/ubuntu/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"
giuseppe
giuseppe

I don't think we use /etc/containers/storage.conf for rootless, except for rootless_storage_path.

I am not sure whether we should do that, the storage configuration can be very different for root and rootless (e.g. many overlay options won't work with rootless).

Can you set the mount_path in the user storage.conf file?

Jan
18
1 day ago
Activity icon
issue

giuseppe issue comment containers/crun

giuseppe
giuseppe

pre-dump support

CRIU supports the concept of pre-copy migration. Instead of creating a complete checkpoint of a process it is possible to only write the memory of the process to disk while the process keeps on running. The first memory only checkpoint can then be transferred to the migration destination. During the transfer time it is possible take a second checkpoint which will only write the memory pages to disk that have changed since the previous checkpoint. This way it can be possible that the second checkpoint is much smaller while the process keeps on running which also means that the amount of data which needs to be transferred to the migration destination might be smaller and thus the migration downtime can be reduced. This only makes sense if the number of memory pages which are changing is rather small. There is no limit on the number pre-copy iterations.

This commit takes the interface as implemented in runc and implements it for crun. Podman already uses the pre-dump as implemented by runc.

This commit also makes sure that the underlying software stack supports the pre-dump mechanism. CRIU uses the kernel's dirty page tracking and it is not available on all architectures (aarch64 does not implement it) or might not be enabled in the kernel. If the user wants to use pre-dump on a system without dirty page tracking crun will fail early and inform the user.

This crun pre-dump implementation relies on libcriu interfaces which are not yet part of the latest release (3.16.1). So at least 3.16.2 or 3.17 is required to use pre-dump in combination with crun.

giuseppe
giuseppe

@adrianreber deps are merged. Can we merge this one as well?

Activity icon
issue

giuseppe issue comment containers/crun

giuseppe
giuseppe

crun 1.4.1 fails to build with gcc 11.1.0

crun 1.4 builds fine, but with the update to 1.4.1 compile fails with this error being printed:

python/crun_python.c: In function ‘container_kill’:
python/crun_python.c:236:20: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
  236 |   const char signal*;
      |                    ^
python/crun_python.c:236:21: error: expected expression before ‘;’ token
  236 |   const char signal*;
      |                     ^
pull request

giuseppe pull request containers/crun

giuseppe
giuseppe

python: fix build

commit 21a8dafae7f26cba1250519aaeb3e55ea445cf93 introduced the regression.

Closes: https://github.com/containers/crun/issues/854

Signed-off-by: Giuseppe Scrivano [email protected]

Activity icon
created branch

giuseppe in giuseppe/crun create branch fix-python-build

createdAt 19 hours ago
Activity icon
created tag
createdAt 22 hours ago
Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

Segfault when running from bazel

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

A gist for setting up and reproducing the error https://gist.github.com/fredr/dd0e5c3639fa109df82471292d6bc8c3

  1. Download BUILD and WORKSPACE to a folder

  2. In that folder, run:

$ bazel build //... --sandbox_writable_path=${XDG_RUNTIME_DIR} --sandbox_writable_path=${HOME}/.local/share/containers/storage --sandbox_debug

Describe the results you received:

time="2022-01-14T09:25:40+01:00" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
cannot setresgid: Invalid argument
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x557e3105594d]

goroutine 1 [running]:
github.com/containers/common/libimage.(*Runtime).Load(0x0, {0x557e31ddcaf8, 0xc000042028}, {0x7fffe5543de7, 0x50}, 0xc00058bb38)
	github.com/containers/[email protected]/libimage/load.go:27 +0xed
github.com/containers/podman/v3/pkg/domain/infra/abi.(*ImageEngine).Load(0x7fffe5543de7, {0x557e31ddcaf8, 0xc000042028}, {{0x7fffe5543de7, 0x41}, 0x0, {0x0, 0x0}})
	github.com/containers/podman/v3/pkg/domain/infra/abi/images.go:362 +0xff
github.com/containers/podman/v3/cmd/podman/images.load(0x557e329443a0, {0xc0002d9000, 0x0, 0x2})
	github.com/containers/podman/v3/cmd/podman/images/load.go:92 +0x358
github.com/spf13/cobra.(*Command).execute(0x557e329443a0, {0xc00003c0a0, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x557e32955e20)
	github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:895
main.Execute()
	github.com/containers/podman/v3/cmd/podman/root.go:91 +0xbe
main.main()
	github.com/containers/podman/v3/cmd/podman/main.go:39 +0x74

Describe the results you expected: I'm guessing something in the setup is wrong, and this should trigger an error message telling me what.

Additional information you deem important (e.g. issue happens only occasionally): Bazel executes within a sandbox, and it is when executing podman from inside that sandbox that this seems to happen. If I run the generated script that fails from my terminal, it works just fine.

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.4
Git Commit:   f6526ada1025c2e3f88745ba83b8b461ca659933
Built:        Thu Dec  9 19:30:40 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.31-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.31, commit: 7e7eb74e52abf65a6d46807eeaea75425cc8a36c'
  cpus: 16
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  hostname: runner
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.12-1-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 7517573120
  memTotal: 33400438784
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4
      commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 39h 46m 19.52s (Approximately 1.62 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/fredr/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/fredr/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/fredr/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639074640
  BuiltTime: Thu Dec  9 19:30:40 2021
  GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Qi podman
Name            : podman
Version         : 3.4.4-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  crun  fuse-overlayfs  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64  libseccomp.so=2-64  slirp4netns
Optional Deps   : apparmor: for AppArmor support [installed]
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  podman-docker: for Docker-compatible CLI [installed]
Required By     : podman-docker
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 72,79 MiB
Packager        : David Runge <[email protected]>
Build Date      : tor  9 dec 2021 19:30:40
Install Date    : tis 11 jan 2022 15:30:32
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature


Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

/usr/bin/docker is symlinked to /usr/bin/podman

giuseppe
giuseppe

when we do not have enough privileges, we re-exec and gain these privileges.

We should not get that far in the parent Podman process and re-exec from SetupRootless (pkg/domain/infra/abi/system.go).

Does the re-exec fail and Podman somehow keep going without enough privileges?

Activity icon
created tag
createdAt 1 day ago
Activity icon
issue

giuseppe issue comment containers/crun

giuseppe
giuseppe

crun 1.4.1 release artifact signatures missing

For crun 1.4.1 there are no signatures on github for the uploaded artifacts. Have they been moved?

giuseppe
giuseppe

Sorry, I forgot to upload them.

Should be fixed now

Activity icon
issue

giuseppe issue containers/crun

giuseppe
giuseppe

crun 1.4.1 release artifact signatures missing

For crun 1.4.1 there are no signatures on github for the uploaded artifacts. Have they been moved?

Activity icon
issue

giuseppe issue comment containers/crun

giuseppe
giuseppe

podman failed in executing shell script from mounted NFS path by rootless user with `keep-id`

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. environment
[email protected]:~/shared$ df
Filesystem                            1K-blocks     Used Available Use% Mounted on
0.0.0.0:/home/lsfadmin/shared    245243904 54182912 191060992  23% /home/lsfadmin/shared
[email protected]:~/shared$ pwd
/home/lsfadmin/shared
[email protected]:~/shared$ cat 1.sh 
#!/bin/bash
echo hello
[email protected]:~/shared$ /home/lsfadmin/shared/1.sh
hello
  1. it works if there is no keep-id
[email protected]:~/shared$ podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --rm ubuntu  /home/lsfadmin/shared/1.sh
hello
  1. it failed if there has keep-id
[email protected]:~/shared$  podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --userns=keep-id --rm ubuntu  /home/lsfadmin/shared/1.sh
Error: open executable: Permission denied: OCI permission denied

Describe the results you received:

Error: open executable: Permission denied: OCI permission denied

Describe the results you expected:

[email protected]:~/shared$  podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --userns=keep-id --rm ubuntu  /home/lsfadmin/shared/1.sh
hello

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[email protected]:~/shared$ podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 4
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: comports1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-91-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 4466839552
  memTotal: 8348520448
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 21473517568
  swapTotal: 21474828288
  uptime: 319h 45m 27.66s (Approximately 13.29 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/lsfadmin/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 3
    stopped: 6
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.5
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/lsfadmin/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/lsfadmin/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

pull request

giuseppe pull request containers/crun

giuseppe
giuseppe

container: attempt find_executable after setresuid

if the find_executable() call fails when running with EUID=0, ignore any error beside ENOENT and attempt again the lookup once the process switched to the final UID/GID used in the container.

This is visible on network file systems like NFS, where CAP_DAC_OVERRIDE is not honored and the EUID=0 cannot read a file if it is owned by the UID/GID in the container.

It is the same mechanism we already have in place for chdir() that might fail before the setresuid() call.

Closes: https://github.com/containers/crun/issues/852

Signed-off-by: Giuseppe Scrivano [email protected]

push

giuseppe push giuseppe/crun

giuseppe
giuseppe

container: attempt find_executable after setresuid

if the find_executable() call fails when running with EUID=0, ignore any error beside ENOENT and attempt again the lookup once the process switched to the final UID/GID used in the container.

This is visible on network file systems like NFS, where CAP_DAC_OVERRIDE is not honored and the EUID=0 cannot read a file if it is owned by the UID/GID in the container.

It is the same mechanism we already have in place for chdir() that might fail before the setresuid() call.

Closes: https://github.com/containers/crun/issues/852

Signed-off-by: Giuseppe Scrivano [email protected]

commit sha: da28cf1afe3cd4b268a2e60e0e6fc11916ba8e8b

push time in 1 day ago
Activity icon
created branch

giuseppe in giuseppe/crun create branch fix-lookup-executable-on-NFS

createdAt 1 day ago
Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

podman failed in executing shell script from mounted NFS path by rootless user with `keep-id`

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. environment
[email protected]:~/shared$ df
Filesystem                            1K-blocks     Used Available Use% Mounted on
0.0.0.0:/home/lsfadmin/shared    245243904 54182912 191060992  23% /home/lsfadmin/shared
[email protected]:~/shared$ pwd
/home/lsfadmin/shared
[email protected]:~/shared$ cat 1.sh 
#!/bin/bash
echo hello
[email protected]:~/shared$ /home/lsfadmin/shared/1.sh
hello
  1. it works if there is no keep-id
[email protected]:~/shared$ podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --rm ubuntu  /home/lsfadmin/shared/1.sh
hello
  1. it failed if there has keep-id
[email protected]:~/shared$  podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --userns=keep-id --rm ubuntu  /home/lsfadmin/shared/1.sh
Error: open executable: Permission denied: OCI permission denied

Describe the results you received:

Error: open executable: Permission denied: OCI permission denied

Describe the results you expected:

[email protected]:~/shared$  podman run -v /home/lsfadmin/shared:/home/lsfadmin/shared --userns=keep-id --rm ubuntu  /home/lsfadmin/shared/1.sh
hello

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[email protected]:~/shared$ podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 4
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: comports1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-91-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 4466839552
  memTotal: 8348520448
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 21473517568
  swapTotal: 21474828288
  uptime: 319h 45m 27.66s (Approximately 13.29 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/lsfadmin/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 3
    stopped: 6
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.5
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/lsfadmin/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/lsfadmin/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

giuseppe
giuseppe

Why? Do you think we are doing some SELinux labeling causing this issue?

my mistake, I assumed the entire storage was on NFS, not just the volume.

I think the error is caused by crun that attempt to open the file while being root before the setresuid to the expected UID/GID in the container.

That usually works, except on network file systems where CAP_DAC_OVERRIDE is not honored.

Moving to crun.

Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

Segfault when running from bazel

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

A gist for setting up and reproducing the error https://gist.github.com/fredr/dd0e5c3639fa109df82471292d6bc8c3

  1. Download BUILD and WORKSPACE to a folder

  2. In that folder, run:

$ bazel build //... --sandbox_writable_path=${XDG_RUNTIME_DIR} --sandbox_writable_path=${HOME}/.local/share/containers/storage --sandbox_debug

Describe the results you received:

time="2022-01-14T09:25:40+01:00" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
cannot setresgid: Invalid argument
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x557e3105594d]

goroutine 1 [running]:
github.com/containers/common/libimage.(*Runtime).Load(0x0, {0x557e31ddcaf8, 0xc000042028}, {0x7fffe5543de7, 0x50}, 0xc00058bb38)
	github.com/containers/[email protected]/libimage/load.go:27 +0xed
github.com/containers/podman/v3/pkg/domain/infra/abi.(*ImageEngine).Load(0x7fffe5543de7, {0x557e31ddcaf8, 0xc000042028}, {{0x7fffe5543de7, 0x41}, 0x0, {0x0, 0x0}})
	github.com/containers/podman/v3/pkg/domain/infra/abi/images.go:362 +0xff
github.com/containers/podman/v3/cmd/podman/images.load(0x557e329443a0, {0xc0002d9000, 0x0, 0x2})
	github.com/containers/podman/v3/cmd/podman/images/load.go:92 +0x358
github.com/spf13/cobra.(*Command).execute(0x557e329443a0, {0xc00003c0a0, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x557e32955e20)
	github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:895
main.Execute()
	github.com/containers/podman/v3/cmd/podman/root.go:91 +0xbe
main.main()
	github.com/containers/podman/v3/cmd/podman/main.go:39 +0x74

Describe the results you expected: I'm guessing something in the setup is wrong, and this should trigger an error message telling me what.

Additional information you deem important (e.g. issue happens only occasionally): Bazel executes within a sandbox, and it is when executing podman from inside that sandbox that this seems to happen. If I run the generated script that fails from my terminal, it works just fine.

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.4
Git Commit:   f6526ada1025c2e3f88745ba83b8b461ca659933
Built:        Thu Dec  9 19:30:40 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.31-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.31, commit: 7e7eb74e52abf65a6d46807eeaea75425cc8a36c'
  cpus: 16
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  hostname: runner
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.12-1-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 7517573120
  memTotal: 33400438784
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4
      commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 39h 46m 19.52s (Approximately 1.62 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/fredr/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/fredr/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/fredr/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639074640
  BuiltTime: Thu Dec  9 19:30:40 2021
  GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Qi podman
Name            : podman
Version         : 3.4.4-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  crun  fuse-overlayfs  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64  libseccomp.so=2-64  slirp4netns
Optional Deps   : apparmor: for AppArmor support [installed]
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  podman-docker: for Docker-compatible CLI [installed]
Required By     : podman-docker
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 72,79 MiB
Packager        : David Runge <[email protected]>
Build Date      : tor  9 dec 2021 19:30:40
Install Date    : tis 11 jan 2022 15:30:32
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature


Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

/usr/bin/docker is symlinked to /usr/bin/podman

giuseppe
giuseppe

we switched from "detect rootless" to "detect if we have CAP_SYS_ADMIN" because running with EUID=0 is not enough to perform all the operations needed to mount the storage, pull images and run containers. It is useful for example when running Podman in a container as root but without capabilities, which is somehow equivalent to run as rootless, so we need to create a user namespace to gain there the needed capabilities.

Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

failed to make veth pair: operation not supported

# podman --version
podman version 3.3.1

# podman run -it --name mybusybox  docker.io/library/busybox
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 01c2cdc13739 done  
Copying config cabb9f684f done  
Writing manifest to image destination
Storing signatures
ERRO[0000] error loading cached network config: network "podman" not found in CNI cache 
WARN[0000] falling back to loading from existing plugins on disk 
Error: error configuring network namespace for container 670d4af03e7c805a9fd12b14cde25e473b89302710e316e7501ac150de4c4726: error adding pod mybusybox_mybusybox to CNI network "podman": failed to make veth pair: operation not supported


giuseppe
giuseppe

for xt_conntrack.ko you'd probably need CONFIG_NETFILTER_XT_MATCH_CONNTRACK

pull request

giuseppe merge to cri-o/cri-o

giuseppe
giuseppe

pkg/container: fix container device GID fallback.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixes in some cases incorrectly chosen device node group ID in containers.

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

Pass host side GID, not UID, as fallback GID to getDeviceUserGroupID(). Update the corresponding unit tests to trigger/catch the original bug.

Does this PR introduce a user-facing change?

None

Activity icon
issue

giuseppe issue comment containers/podman

giuseppe
giuseppe

podman run hang with error: container creation timeout

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman run --rm -it --network=host alpine:3.14.3 hang and exit after a while with error: Error: container creation timeout: internal libpod error

From the debug log, seems container has been successfully created with crun/runc, but podman failed to connect to the container somehow.

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm -it --network=host alpine:3.14.3) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/etc/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/rockmen1/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver btrfs                     
DEBU[0000] Using graph root /home/rockmen1/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/rockmen1/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/rockmen1/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "btrfs" 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/rockmen1/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image alpine:3.14.3 (policy: missing) 
DEBU[0000] Looking up image "alpine:3.14.3" in local containers storage 
DEBU[0000] Trying "alpine:3.14.3" ...                   
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Trying "docker.io/library/alpine:3.14.3" ... 
DEBU[0000] parsed reference into "[[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage ([[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb) 
DEBU[0000] Looking up image "docker.io/library/alpine:3.14.3" in local containers storage 
DEBU[0000] Trying "docker.io/library/alpine:3.14.3" ... 
DEBU[0000] parsed reference into "[[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Found image "docker.io/library/alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage 
DEBU[0000] Found image "docker.io/library/alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage ([[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb) 
DEBU[0000] Looking up image "alpine:3.14.3" in local containers storage 
DEBU[0000] Trying "alpine:3.14.3" ...                   
DEBU[0000] Trying "docker.io/library/alpine:3.14.3" ... 
DEBU[0000] parsed reference into "[[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage ([[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb) 
DEBU[0000] Inspecting image 0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Looking up image "alpine:3.14.3" in local containers storage 
DEBU[0000] Trying "alpine:3.14.3" ...                   
DEBU[0000] Trying "docker.io/library/alpine:3.14.3" ... 
DEBU[0000] parsed reference into "[[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage 
DEBU[0000] Found image "alpine:3.14.3" as "docker.io/library/alpine:3.14.3" in local containers storage ([[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb) 
DEBU[0000] Inspecting image 0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] Inspecting image 0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb 
DEBU[0000] using systemd mode: false                    
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json" 
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host 
DEBU[0000] Allocated lock 0 for container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 
DEBU[0000] parsed reference into "[[email protected]/home/rockmen1/.local/share/containers/storage+/run/user/1000/containers]@0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] exporting opaque data as blob "sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb" 
DEBU[0000] created container "6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011" 
DEBU[0000] container "6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011" has work directory "/home/rockmen1/.local/share/containers/storage/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata" 
DEBU[0000] container "6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011" has run directory "/run/user/1000/containers/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] [graphdriver] trying provided driver "btrfs" 
DEBU[0000] mounted container "6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011" at "/home/rockmen1/.local/share/containers/storage/btrfs/subvolumes/7cc9481b3034e801010ee59ed8a6ca4431f4f60e940c411fde134cf263210a18"
DEBU[0000] Created root filesystem for container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 at /home/rockmen1/.local/share/containers/storage/btrfs/subvolumes/7cc9481b3034e801010ee59ed8a6ca4431f4f60e940c411fde134cf263210a18
DEBU[0000] network configuration does not support host.containers.internal address
DEBU[0000] skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5"
DEBU[0000] skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)"
DEBU[0000] skipping unrecognized mount in /etc/containers/mounts.conf: ""
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting CGroups for container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 to user.slice:libpod:6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/home/rockmen1/.local/share/containers/storage/btrfs/subvolumes/7cc9481b3034e801010ee59ed8a6ca4431f4f60e940c411fde134cf263210a18"
DEBU[0000] Created OCI spec for container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 at /home/rockmen1/.local/share/containers/storage/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 -u 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 -r /usr/bin/crun -b /home/rockmen1/.local/share/containers/storage/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata -p /run/user/1000/containers/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata/pidfile -n infallible_blackburn --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /run/user/1000/containers/btrfs-containers/6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/rockmen1/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg btrfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011.scope
DEBU[0240] Cleaning up container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011
DEBU[0240] Network is already cleaned up, skipping...
DEBU[0240] unmounted container "6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011"
DEBU[0240] Removing container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011
DEBU[0240] Removing all exec sessions for container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011
DEBU[0240] Cleaning up container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011
DEBU[0240] Network is already cleaned up, skipping...
DEBU[0240] Container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 storage is already unmounted, skipping...
DEBU[0240] Container 6905541d72f6bdc16f787820dec3312933bb1468f2d0e8329d27eb4432efb011 storage is already unmounted, skipping...
DEBU[0240] ExitCode msg: "container creation timeout: internal libpod error"
Error: container creation timeout: internal libpod error

Additional information you deem important (e.g. issue happens only occasionally):

On Arch Linux, kernel 5.15.2, tried both rootless and root, tried both runc and crun as well

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.17.3
Git Commit:   2ad1fd3555de12de34e20898cc2ef901f08fe5ed
Built:        Sat Nov 13 05:41:08 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.30-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: 2792c16f4436f1887a7070d9ad99d9c29742f38a'
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: luowj26-pc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 985
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.2-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 23039205376
  memTotal: 33443921920
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.2
      commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 22m 54.35s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: mirror.baidubce.com
    Prefix: docker.io
  search:
  - quay.io
  - docker.io
store:
  configFile: /home/rockmen1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/rockmen1/.local/share/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.15 '
    Library Version: "102"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/rockmen1/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636753268
  BuiltTime: Sat Nov 13 05:41:08 2021
  GitCommit: 2ad1fd3555de12de34e20898cc2ef901f08fe5ed
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Output of crun --version:

crun version 1.2
commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

Output of runc --version:

runc version 1.0.2
commit: v1.0.2-0-g52b36a2d
spec: 1.0.2-dev
go: go1.17
libseccomp: 2.5.3

Log from systemd:

conmon[2924]: conmon 6905541d72f6bdc16f78 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.UTYAD1}
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/proc/self/fd/13/attach}
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: terminal_ctrl_fd: 13
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: winsz read side: 16, winsz write side: 16
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: about to accept from console_socket_fd: 10
conmon[2925]: conmon 6905541d72f6bdc16f78 <ninfo>: about to recvfd from connfd: 12
systemd[619]: Started libcrun container.
conmon[2925]: [75B blob data]
giuseppe
giuseppe

I can't spot anything strange in the stacktrace. It is just glibc waiting for events.

Your issue though doesn't look like a race condition if you can reproduce so easily. Could you try disabling SElinux? Anything useful from --debug when you run with overlay?

Jan
17
2 days ago
Activity icon
issue

giuseppe issue comment containers/fuse-overlayfs

giuseppe
giuseppe

EIO for ioctl FS_IOC_GETFLAGS

Not sure if this is expected behavior, a fuse-overlayfs issue or a fuse issue, but I get EIO when using lsattr after it calls ioctl with FS_IOC_GETFLAGS. From the caller:

$ lsattr -d d
lsattr: Input/output error While reading flags on d

Here's the strace output from fuse-overlayfs:

openat2(3, ".", {flags=O_RDONLY|O_NONBLOCK, resolve=RESOLVE_IN_ROOT}, 24) = 8
ioctl(8, FS_IOC_GETFLAGS, 0x7ffd6d1d1bc0) = 0
writev(5, [{iov_base="(\0\0\0\0\0\0\0\204\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\27\0\0\0\0\0\0\0", iov_len=8}], 3) = -1 EINVAL (Invalid argument)
dup(2)                                  = 9
fcntl(9, F_GETFL)                       = 0x80002 (flags O_RDWR|O_CLOEXEC)
fstat(9, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(9, "fuse: writing device: Invalid argument\n", 39fuse: writing device: Invalid argument
) = 39
close(9)                                = 0
close(8)                                = 0

What I don't get is the actual ioctl call succeeds, but then writev gets EINVAL writing to /dev/fuse. This is running unprivileged from the root user namespace, but I originally hit it in a podman container. This is with fuse-overlayfs 1.7.1, fuse 3.10.3 and linux 5.13.0.

Activity icon
issue

giuseppe issue comment containers/fuse-overlayfs

giuseppe
giuseppe

Image corruption when using fuse-overlayfs instead of overlayfs inside a Docker container

When generating images for Dockerfile via buildah running in a Docker container, a file is corrupted when using fuse-overlayfs. This is not the case when using overlayfs (without fuse).

Steps to reproduce:

  • spin up a small VM (e.g. AWS/t2.small, RHOSP/g.standard.small) with FC34 or FC35 connected to the Red Hat intranet (FC35 on AWS us-east-1 is ami-08b4ee602f76bff79, from https://alt.fedoraproject.org/cloud/)

  • setup the VM via

    sudo dnf install -y moby-engine
    sudo systemctl enable docker
    sudo systemctl start docker
    
  • to verify the build is working via overlayfs on the upstream buildah/stable image, run

    sudo docker run --rm --privileged quay.io/buildah/stable:latest buildah bud --storage-opt mount_program= -f https://gitlab.com/cki-project/containers/-/raw/d3f2634e066b2eab0a4a9d84ec295b4a2543b475/Dockerfile
    
  • trying to build the Dockerfile with fuse-overlayfs instead fails with a checksum error

    sudo docker run --rm --privileged quay.io/buildah/stable:latest buildah bud -f https://gitlab.com/cki-project/containers/-/raw/d3f2634e066b2eab0a4a9d84ec295b4a2543b475/Dockerfile
    

We have only observed that with the /usr/lib64/python3.6/site-packages/libcomps/_libpycomps.so file on our ubi8/rhel8 images.

This file is present in the ubi8 image (checksum in before.sha256), and gets updated to a newer version during dnf update (checksum in after.sha256). During one of the following RUN steps, the file reverts to the original version, which rpm --verify detects. It varies at exactly what RUN step that happens. Sometimes, this reversion is even incomplete for a while, i.e. only a part of the file reverts to the original contents.

In the Docker container, /var/lib/containers is a separate volume, with either ext4 (FC34 host, and Docker using the overlayfs2 driver), or btrfs (FC35 host, and Docker using the btrfs driver).

We have two RHOSP VMs running (FC34/FC35) that can be used to reproduce the problem, let me know on RH IRC #kernelci if you want to have an SSH key added.

Activity icon
created branch

giuseppe in giuseppe/fuse-overlayfs create branch tag-1.8.1

createdAt 1 day ago
pull request

giuseppe pull request containers/storage

giuseppe
giuseppe

revert usage of github.com/valyala/gozstd

github.com/valyala/gozstd doesn't currently support building without cgo: https://github.com/valyala/gozstd/pull/41

Revert the change until it is fixed upstream.

Signed-off-by: Giuseppe Scrivano [email protected]

push

giuseppe push giuseppe/storage

giuseppe
giuseppe

Revert "chunked: use valyala/gozstd"

This reverts commit 5bb6d8e65ed440670082af4807d70088a93b945b.

Signed-off-by: Giuseppe Scrivano [email protected]

giuseppe
giuseppe

Revert "archive: use valyala/gozstd"

This reverts commit a3abf19ed46fcbed400d6909b18eb01f09751b65.

Signed-off-by: Giuseppe Scrivano [email protected]

giuseppe
giuseppe

Revert "vendor add valyala/gozstd"

This reverts commit 7ac0e7bfff3a88ce711b7744db62c1646131468b.

Signed-off-by: Giuseppe Scrivano [email protected]

commit sha: 9e9783dbe8623928fa31191977e6d2bb9c0adaa2

push time in 1 day ago
Previous