There is a hack for the second issue. When you launch the container, mount your current directory as the same directory in the container and set your working directory to the same.
“LXD is a next generation system container manager. It offers a user experience similar to virtual machines but using Linux containers instead.
It’s image based with pre-made images available for a wide number of Linux distributions and is built around a very powerful, yet pretty simple, REST API.” from https://linuxcontainers.org/lxd/
I started with this approach but needed to also cater for services that you deployed which means you really need to include systemd as part of your development environment.
Though wrapped up in a Makefile, it pretty much looks like:
Packer generates the Docker container but will also cook my production GCP, AWS, Azure, … images too.
packer.json includes a call out to a setup script that does the grunt work and installs systemd (Debian: systemd-sysv) and sets the entry point to /sbin/init; there are some other minor details (such as passwd -d root so you can do a console login and logging out is via typing halt in the container) but this is the gist of it.
To interact with the deployment you work from the host side and then just reload your service to make it live. You do need to line up your ducks in a row to get those bind mounts into the right place for your service to just pick up on but when you get it right it makes life a lot easier.
As a note I continue to use a shell script over orchestration tools, as well as other container/VM environments, so everything remains accessible to others. The above seems to work on Windows and macOS too.
At the end of the day this is about making it not just easier for myself, but for everyone else.
…those bind mounts are read-only to act as a polite reminder/guard that you should not edit files directly on your ‘server’ but instead make all changes in the project host side where they can be commited and be re-deployed (hopefully involving just a service reload).
LOL, I set this up more than 1 year ago, after tired of having to set up my vim dev env on cluster nodes for debugging: https://github.com/vicaya/vimdev/blob/master/v which doesn’t require privilege mode.
Note, the repo also integrates with docker hub build automation, which is fairly convenient as well.
I require privileged access when running the container because, as a programmer, I need to low-level access (performance counters, and so forth). Otherwise, it would not be required.
Borissays:
Have you looked at singularity?
It’s containers for HPC environments. The idea is the scientist creates his environment (e.g. his laptop) as an image and run on a cluster with this image. Should also work for programming and it’s becoming a standard in HPC.
regards,
Boris https://singularity.lbl.gov/
I am not exactly sure we need “containers for HPC”. What is the benefit over Docker?
Borissays:
I’m not a real expert, but the main ideas are,
1. security. Docker needs a daemon running as root and the containers running in a root context. Sysadmins don’t like that 😉
Singularity is a program, running under your username.
2. convenience. Singularity automatically mounts your home directory and the work directory (not sure about the later) into the container, so no extra copying back and forth.
3. portability. A singularity container is just one big file. Docker containers are build from layers. You can easily convert them to singularity. The advantage is, you can just copy this one big file to a new cluster and don’t have to rely on that there are the proper versions of your libraries/programs installed.
4. reproducability. If you publish a paper, you just have to preserve the singularity container and your dataset and can reproduce the results years later. Docker containers get updated.
Hope this explains the reasons of this development. It makes it also much easier for sysadmins not to install 10 versions of the same library for different projects. So win-win 🙂
regards,
Boris
Singularity is a program, running under your username.
So you do not get access to performance counters and the like? I need to be able to access these from time to time. So if privileged access is not feasible, that would make it impossible for me to use such an option.
Singularity automatically mounts your home directory and the work directory (not sure about the later) into the container, so no extra copying back and forth.
I do the same with Docker.
portability. A singularity container is just one big file. Docker containers are build from layers. You can easily convert them to singularity. The advantage is, you can just copy this one big file to a new cluster and don’t have to rely on that there are the proper versions of your libraries/programs installed.
I am confused about this comment. The whole point of Docker is not to have to worry about the libraries or programs installed on the host.
reproducability. If you publish a paper, you just have to preserve the singularity container and your dataset and can reproduce the results years later. Docker containers get updated.
I don’t think they do get updated. Not unless you want to get updates. That is an important feature for docker.
Hope this explains the reasons of this development. It makes it also much easier for sysadmins not to install 10 versions of the same library for different projects. So win-win 🙂
regards,
I use docker for this very purpose, so that I do not have to install different versions.
Borissays:
Ok,
I think in a self administered environment there might be not much difference. But Docker was meant to run micro services and has different goals. Sure you can tweak docker, but singularity solves some of the problems without tweaking.
I’m not sure about performance counters, but I doubt it, because of being a userland process.
The main idea behind it is, that you can test your program small scale on your local machine (maybe in docker) and once you’re happy, convert your docker image to singularity and run the production on a cluster.
The main point is not a lot of clusters will implement docker, because of the inherent security issues. And I think a lot of research needs as much computing power as possible, so you have to think how to scale out.
Anyhow, it was just a suggestion 🙂
regards,
Boris
pr0PMsays:
Use podman instead of docker most issues will be solved instantly.
For me, the point of using docker is that I can have the same workflow no matter what machine I am on (Windows, macOS, any Linux distribution).
Yes. I have VirtualBox, but it has major drawbacks for what I have to do. Running podman inside VirtualBox would be a terrible experience compared to just launching docker.
I also don’t want to mess with the systems. I don’t want to hack /etc/apt/sources.list.d under ubuntu if I don’t need to. Docker is easily to install, fully supported, pretty much everywhere.
I realize that what I write may sound unfair, but I think that using Docker, at this point in time, makes a lot of sense.
I did investigate podman. I am sure it is great for some people… but I don’t find it attractive. When I can just start podman containers with a command line under macOS, then maybe…
There is a hack for the second issue. When you launch the container, mount your current directory as the same directory in the container and set your working directory to the same.
docker run -v 'pwd':'pwd' -w 'pwd' ubuntu
Nice trick. I will use it.
As for your command, let us see what happens when we run it…
So, first, if you are using a secured linux (and many of my servers are secured linux), the volume binding won’t work. That’s easily solved.
But then, more critically, as you can see, the file permission is messed up.
You can fix the second issue with uid mapping. I haven’t needed to use that feature myself, so my advice after this point is more speculative. That said, https://seravo.fi/2019/align-user-ids-inside-and-outside-docker-with-subuser-mapping looks like another good solution.
Yes. That’s what my script does.
I use lxd because it feels more like a vm but is a container.
lxc launch ubuntu:Alias containerName
lxc exec containerName -- Command
lxc stop containerName
lxc destroy containerName
“LXD is a next generation system container manager. It offers a user experience similar to virtual machines but using Linux containers instead.
It’s image based with pre-made images available for a wide number of Linux distributions and is built around a very powerful, yet pretty simple, REST API.” from https://linuxcontainers.org/lxd/
I am not bound to Docker. It just happens to be everywhere I need to work. I must admit that my approach is a bit of a hack.
https://docs.fedoraproject.org/en-US/fedora-silverblue/toolbox/
I think in theory Fedora toolbox could be extended to support multiple Linux distros but probably LXD is your best bet for now.
Daniele, I suggest you trying this tool to develop inside a container. I use it and it is great
https://code.visualstudio.com/docs/remote/containers
It is native from vscode
I started with this approach but needed to also cater for services that you deployed which means you really need to include
systemdas part of your development environment.Though wrapped up in a Makefile, it pretty much looks like:
env TMPDIR=$(pwd) $(pwd)/packer build -on-error=ask -only docker packer.json
docker run -it --rm \
-e container=docker \
-v $(pwd)/data:/opt/VENDOR/PROJECT/data:ro \
-v $(pwd)/nginx:/opt/VENDOR/PROJECT/nginx:ro \
-v $(pwd)/lua:/opt/VENDOR/PROJECT/lua:ro \
-v $(pwd)/public:/opt/VENDOR/PROJECT/public:ro \
-v $(pwd)/src:/opt/VENDOR/PROJECT/public/gpt/src:ro \
--publish=127.0.0.1:8000:80 \
--publish=127.0.0.1:63790:6379 \
--tmpfs /run \
--tmpfs /tmp \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
--cap-add SYS_ADMIN --cap-add NET_ADMIN --cap-add SYS_PTRACE \
--stop-signal SIGPWR \
VENDOR/PROJECT:latest
Packer generates the Docker container but will also cook my production GCP, AWS, Azure, … images too.
packer.jsonincludes a call out to asetupscript that does the grunt work and installssystemd(Debian:systemd-sysv) and sets the entry point to/sbin/init; there are some other minor details (such aspasswd -d rootso you can do a console login and logging out is via typinghaltin the container) but this is the gist of it.To interact with the deployment you work from the host side and then just reload your service to make it live. You do need to line up your ducks in a row to get those bind mounts into the right place for your service to just pick up on but when you get it right it makes life a lot easier.
As a note I continue to use a shell script over orchestration tools, as well as other container/VM environments, so everything remains accessible to others. The above seems to work on Windows and macOS too.
At the end of the day this is about making it not just easier for myself, but for everyone else.
…those bind mounts are read-only to act as a polite reminder/guard that you should not edit files directly on your ‘server’ but instead make all changes in the project host side where they can be commited and be re-deployed (hopefully involving just a service reload).
Vagga creates user space containers for developers
This approach has undoubtedly been reimplemented multiple times by many people. Two examples I am aware of:
https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docker/tools/onlbuilder
https://github.com/Azure/sonic-buildimage/blob/master/Makefile.work#L244
Yes. I do not claim to be original.
LOL, I set this up more than 1 year ago, after tired of having to set up my vim dev env on cluster nodes for debugging: https://github.com/vicaya/vimdev/blob/master/v which doesn’t require privilege mode.
Note, the repo also integrates with docker hub build automation, which is fairly convenient as well.
I require privileged access when running the container because, as a programmer, I need to low-level access (performance counters, and so forth). Otherwise, it would not be required.
Have you looked at singularity?
It’s containers for HPC environments. The idea is the scientist creates his environment (e.g. his laptop) as an image and run on a cluster with this image. Should also work for programming and it’s becoming a standard in HPC.
regards,
Boris
https://singularity.lbl.gov/
I am not exactly sure we need “containers for HPC”. What is the benefit over Docker?
I’m not a real expert, but the main ideas are,
1. security. Docker needs a daemon running as root and the containers running in a root context. Sysadmins don’t like that 😉
Singularity is a program, running under your username.
2. convenience. Singularity automatically mounts your home directory and the work directory (not sure about the later) into the container, so no extra copying back and forth.
3. portability. A singularity container is just one big file. Docker containers are build from layers. You can easily convert them to singularity. The advantage is, you can just copy this one big file to a new cluster and don’t have to rely on that there are the proper versions of your libraries/programs installed.
4. reproducability. If you publish a paper, you just have to preserve the singularity container and your dataset and can reproduce the results years later. Docker containers get updated.
Hope this explains the reasons of this development. It makes it also much easier for sysadmins not to install 10 versions of the same library for different projects. So win-win 🙂
regards,
Boris
So you do not get access to performance counters and the like? I need to be able to access these from time to time. So if privileged access is not feasible, that would make it impossible for me to use such an option.
I do the same with Docker.
I am confused about this comment. The whole point of Docker is not to have to worry about the libraries or programs installed on the host.
I don’t think they do get updated. Not unless you want to get updates. That is an important feature for docker.
I use docker for this very purpose, so that I do not have to install different versions.
Ok,
I think in a self administered environment there might be not much difference. But Docker was meant to run micro services and has different goals. Sure you can tweak docker, but singularity solves some of the problems without tweaking.
I’m not sure about performance counters, but I doubt it, because of being a userland process.
The main idea behind it is, that you can test your program small scale on your local machine (maybe in docker) and once you’re happy, convert your docker image to singularity and run the production on a cluster.
The main point is not a lot of clusters will implement docker, because of the inherent security issues. And I think a lot of research needs as much computing power as possible, so you have to think how to scale out.
Anyhow, it was just a suggestion 🙂
regards,
Boris
Use podman instead of docker most issues will be solved instantly.
Podman is linux only, right?
Yes, but we have vm and wsl to take care of that. (here)
For me, the point of using docker is that I can have the same workflow no matter what machine I am on (Windows, macOS, any Linux distribution).
Yes. I have VirtualBox, but it has major drawbacks for what I have to do. Running podman inside VirtualBox would be a terrible experience compared to just launching docker.
I also don’t want to mess with the systems. I don’t want to hack /etc/apt/sources.list.d under ubuntu if I don’t need to. Docker is easily to install, fully supported, pretty much everywhere.
I realize that what I write may sound unfair, but I think that using Docker, at this point in time, makes a lot of sense.
I did investigate podman. I am sure it is great for some people… but I don’t find it attractive. When I can just start podman containers with a command line under macOS, then maybe…