Openshift: Remove Zombie machines

Openshift, Kubernetes Comments

When an Openshift MachineSet was created with a faulty configuration, Machines created from it may be undeletable by the Controller although the Machines are not existing at all. The status of the Machines will then stay in the “Deleting” state.

$ oc get machine -n openshift-machine-api
NAME                              PHASE      TYPE              REGION       ZONE   AGE
m3adow-infra-westeurope1-5asrt    Deleting                                         3d2h
m3adow-infra-westeurope1-ioq4z    Running    Standard_D2s_v3   westeurope   1      3d

To manually remove the Machine from the API, remove its finalizers via patch:

oc patch machine -n openshift-machine-api --type=merge -p '{"metadata":{"finalizers":null}}' m3adow-infra-westeurope1-5asrt

This will instantly remove it.

Blog: Codebites removal announcement

Blog Comments

I’ve been inactive for a long time and I won’t promise any more activity. I just wanted to inform all potential feed readers that I will remove Codebites as their own section soon. The plans I initially had with Codebites didn’t work well, Life and Work prevented me from writing more posts, so I don’t think having the few posts I write separated into different sections makes any sense. Therefore from now on all posts will be available in the one (and only) RSS feed (feed.xml) once again. This may lead to some old and/or already read posts reappearing in your feeds. Also, if you’re using one of the deprecated ones, you may encounter problems. While I tried redirecting the old RSS feeds to the new one (with Javascript, which surprisingly worked best for the three applications I tested with), I’m not certain all RSS readers support redirects.

Ansible: Using Python string substituition in variables

Ansible, Codebites Comments

Scenario: Utilize Pythons string substitution in Ansible variables

- hosts: localhost
  connection: local

    foobar: ">>%s<<"
    substitution: "adminswerk.de"

    - debug:
        msg: "{{ foobar % substitution }}"

Output (slightly truncated):

$ ansible-playbook string_format.yml
PLAY [localhost] **********************************************************************************************************************************************

TASK [debug] **************************************************************************************************************************************************
ok: [localhost] => {
    "msg": ">>adminswerk.de<<"

PLAY RECAP ****************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0

Ansible: Using programatically constructed variables

Ansible, Codebites Comments


Use specific variables, depending on the environment a playbook is run in. In this example either a low load or high load environment


- hosts:
    - localhost
  gather_facts: no

    - lowload_app_java_opts: "-Xms512m -Xmx2G"
    - highload_app_java_opts: "-Xms4G -Xmx8G"
    - services:
        - lowload_app
        - highload_app

    - debug:
       msg: "fixed_var: {{ lookup('vars', service_name + '_java_opts') }}"
      loop: "{{ services }}"
        loop_var: service_name

Output (truncated):

$ ansible-playbook -v programatic_variable_lookup.yml
PLAY [localhost] *******************************************************************

TASK [debug] ***********************************************************************
ok: [localhost] => (item=lowload_app) => {
    "msg": "fixed_var: -Xms512m -Xmx2G"
ok: [localhost] => (item=highload_app) => {
    "msg": "fixed_var: -Xms 4G -Xmx8G"

PLAY RECAP *************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0

Winbind: UID/GID range full

Linux, Samba Comments

We had a problem on a storage server. One user was not able to authenticate with the Samba service using his Active Directory credentials. Furthermore I couldn’t find his user via getent passwd AD\\username. After checking several LDAP/Kerberos/PAM configuration files, I had the glorious idea to also check the logs of winbind.

[2019/01/29 14:05:16.726252,  1, pid=4467, effective(0, 0), real(0, 0)]   Fatal Error: UID range full!! (max: 60000)
[2019/01/29 14:05:16.726299,  1, pid=4467, effective(0, 0), real(0, 0)]   Error allocating a new UID
[2019/01/29 14:05:16.726339,  1, pid=4467, effective(0, 0), real(0, 0)]   no backend defined for idmap config BUILTIN
[2019/01/29 14:05:16.726903,  1, pid=4467, effective(0, 0), real(0, 0)]   Fatal Error: GID range full!! (max: 60000)
[2019/01/29 14:05:16.726948,  1, pid=4467, effective(0, 0), real(0, 0)]   Error allocating a new GID

Huh, interesting. This wasn’t a heavily used server. Neither users nor groups were even in the proximity of 60000. Accordingly increasing the idmap uid/idmap gid did not help at all.
Several hours later I found the solution in the arstechnica forum:

Long story short, stop winbind, delete winbindd_cache.tdb & winbindd_idmap.tdb from /var/cache/samba, then restart winbind. Mappings now happen right. So I can log in with domain accounts and access shares.

The provided path /var/cache/samba did not fit for the Red Hat Enterprise Linux running on this server. But finding out that winbindd_cache.tdb and winbindd_idmap.tdb are located in /var/lib/samba was no big deal after nearly 60 minutes of unnecessary debugging.