Skip to content
Snippets Groups Projects

Draft: Build docker

Closed Draft: Build docker
sara.vallero/gwcelery:build-docker into main
Closed Alessio Fiori requested to merge sara.vallero/gwcelery:build-docker into main

Added a Dockerfile to build Docker containers, with base image python:3.x. The Dockerfile is in the project root directory.

Added a CI stage build-docker with one job that builds an image based on python:3.9. The job runs on the CNAF gitlab runner (tag: cnaf). The job is triggered when a new tag is created. Optionally, it can be triggered by MR to the main branch. The YAML file of the job is stored in the docker dir.

Created two scripts for building images locally (not required for the CI job). README is included.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
docker/docker-build.yml 0 → 100644
28 when: never
29 variables:
30 IMAGE_TAG: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
31 script:
32 #- docker build --build-arg from=repo --build-arg url="${CI_REPOSITORY_URL}" --build-arg tag_or_branch="${GWCELERY_TAG}" --build-arg python=${PYTHON_VERSION} -t "${CI_REGISTRY_IMAGE}/python${PYTHON_VERSION}:${IMAGE_TAG}" .
33 - docker build --build-arg from=local --build-arg python=${PYTHON_VERSION} -t "${CI_REGISTRY_IMAGE}/python${PYTHON_VERSION}:${IMAGE_TAG}" .
34 - docker push "${CI_REGISTRY_IMAGE}/python${PYTHON_VERSION}:${IMAGE_TAG}"
35
36 # Main build job.
37 # See docker/README.md for build instructions.
38 docker/build/python3.9:
39 variables:
40 PYTHON_VERSION: "3.9"
41 needs:
42 - test/poetry/python3.9
43 - test/wheel/python3.9
  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    why do we need this stage to pass. It's better to run the tests against the built image.

  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    in addition to the wheel and poetry

  • Roberto DePietri
    Roberto DePietri @roberto.depietri ·
    Owner

    @deep.chatterjee I do agree with you. Now the build is performed after the test is successful, meaning the code should we work. In this case, the docker image is built. This is the first step. Once we have that merge, we will study a way to add a final step in the build docker to test the sanity of the produced docker image. That would be easy to add once this merge request is in. I will put that on the to-do-list.

  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    The image being built does not imply the code will work. In fact right now, the code will NOT work since there is no redis server in the container. The first step to know the container is built correctly is to ensure that at least the unittests are running correctly against it. Please add that in.

  • Roberto DePietri
    Roberto DePietri @roberto.depietri ·
    Owner

    We do not want redis inside the container. The container should run against a host-installed-version instance of redis. The idea is to use one container for any worker, each one of them accessing the same instance of the redis datatabase.

  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    @roberto.depietri it is unclear to me what is being done where. I hope you agree that with me that this dockerfile, the code will not work, because celery itself will not start. If you have a dedicated config repo for the CNAF deployment of gwcelery and other services, it should live there.

    A lot of details are opaque (like I mentioned in !1072 (merged) as well). Can I suggest your and others create a different milestone, or any other gitlab feature as you see fit, and move all the containerization and kubernetes related MRs there. Since this is not a priority item, I would like to visit this all together once we have taken care of the other priority items.

  • Please register or sign in to reply
  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee ·
    Owner

    Also, installing from poetry will install the redis client as a part of celery, but the redis server is also needed. This is done as a part of the unittesting stage and can be re-used.

  • Deep Chatterjee
    Deep Chatterjee @deep.chatterjee started a thread on the diff
    • docker/docker-build.yml 0 → 100644
    18 tags:
    19 - cnaf
    20 rules:
    21 - if: $CI_COMMIT_TAG
    22 when: always
    23 variables:
    24 IMAGE_TAG: $CI_COMMIT_TAG
    25 # The following condition triggers the pipeline for MRs to the main branch
    26 # Set "when: manual" to enable it for your branch
    27 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main'
    28 when: never
    29 variables:
    30 IMAGE_TAG: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    31 script:
    32 #- docker build --build-arg from=repo --build-arg url="${CI_REPOSITORY_URL}" --build-arg tag_or_branch="${GWCELERY_TAG}" --build-arg python=${PYTHON_VERSION} -t "${CI_REGISTRY_IMAGE}/python${PYTHON_VERSION}:${IMAGE_TAG}" .
    33 - docker build --build-arg from=local --build-arg python=${PYTHON_VERSION} -t "${CI_REGISTRY_IMAGE}/python${PYTHON_VERSION}:${IMAGE_TAG}" .
  • Deep Chatterjee mentioned in merge request !1072 (merged)

    mentioned in merge request !1072 (merged)

    • Roberto DePietri
      Roberto DePietri @roberto.depietri ·
      Owner

      @deep.chatterjee The redis server is NOT needed to run gwcelery. In fact, the openmp workers are running at caltech on machines that do not have redis installed and they use the redis server running on the emfollow machine.

    • Deep Chatterjee
      Deep Chatterjee @deep.chatterjee ·
      Owner

      That is only because you have redis server running and celery can connect to. So "The redis server is NOT needed to run gwcelery" is simply wrong. Any celery application needs a result backend to run. We use redis.

      So without the server running in the container, or being a separate service or part of a compose file which should also live here, this by itself does not make much sense.

      Edited by Deep Chatterjee
    • Sara Vallero
      Sara Vallero @sara.vallero ·
      Developer

      @deep.chatterjee this build stage is the first necessary step for a complete pipeline that will deploy gwcelery, redis, gracedb and hop for acceptance/integration tests on k8s. It is not meant to replace the unittests already implemented, but just to produce a base environment with correct dependencies.

    • Please register or sign in to reply
  • Roberto DePietri
    Roberto DePietri @roberto.depietri ·
    Owner

    @deep.chatterjee @cody.messick This is the merge request we need to start working on automatic testing and CI integration. It creates a docker file (suited for the purpose) and has no side effects on gwceley functionality.

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
    • Please register or sign in to reply