Sysadmin Mega Thread

[oO.o]

- hosts: localhost
  gather_facts: false
  tasks:

    - set_fact:
        teams: "{{  teams
                    | default({})
                    | combine( { item['teamname']:  { 'members': [] }
                                                    | combine(item) } ) }}"
      vars:
        raw_string: |
          teamname:Cougars
          description:Baseball
          member:John,second
          member:Bill,first

          teamname:Rockies
          description:Baseball
          member:Phil,pitcher
          member:Peter,catcher
          member:Wilbur,centerfield

          teamname:marlins
          description:MinorLeague
      loop: "{{ raw_string
                | regex_replace(':', ': ', multiline=true)
                | regex_replace('^member:', '  -', multiline=true)
                | split('\n\n')
                | map('replace', '  -', 'members:@@@  -', 1)
                | map('regex_replace', '@@@', '\n')
                | map('from_yaml') }}"

    - debug:
        var: teams

ok: [localhost] => {
    "teams": {
        "Cougars": {
            "description": "Baseball",
            "members": [
                "John,second",
                "Bill,first"
            ],
            "teamname": "Cougars"
        },
        "Rockies": {
            "description": "Baseball",
            "members": [
                "Phil,pitcher",
                "Peter,catcher",
                "Wilbur,centerfield"
            ],
            "teamname": "Rockies"
        },
        "marlins": {
            "description": "MinorLeague",
            "members": [],
            "teamname": "marlins"
        }
    }
}

Not the worst thing I’ve ever done…

If there’s an argument you can pass to regex_replace to limit the number of replacements, that would eliminate the need to resort plain replace which can’t handle \n.

---

Edited to include empty members list and consolidate to one set_fact.

[cotton]

This is interesting! Thank you for sharing it!

[cotton]

    - set_fact:
        teams: "{{  teams
                    | default({})
                    | combine( { item['teamname']:  { 'members': [] }
                                                    | combine(item) } ) }}"

So that creates a variable called “teams” it then takes “teams” and must turn it into a dictionary. It then creates an anonymous dictionary with two items: a (list???) "teamname and a dictionary object “members” which maps to a list and combines them into an dictionary and then combines that dict with “teams”

Not sure if that’s right, but that what seems to be going on

[cotton]

@oO.o

      loop: "{{ raw_string
                | regex_replace(':', ': ', multiline=true)
                | regex_replace('^member:', '  -', multiline=true)
                | split('\n\n')
                | map('replace', '  -', 'members:@@@  -', 1)
                | map('regex_replace', '@@@', '\n')
                | map('from_yaml') }}"

It loops over the “raw_string” output and replaces all “:” with ": ", setting multiline to true
replaces any line that starts with “member:” with a ’ -’, (must be making it a list item)
creates a list and split on the double \n
no idea what

@oO.o

                | map('replace', '  -', 'members:@@@  -', 1)
                | map('regex_replace', '@@@', '\n')

Ahh I see now, you’re creating the list by replacing @@@ with a new line. Not sure what the 1 does after that.

[oO.o]

I can write a breakdown later today. This is exactly the kind of thing I’ve been doing for the past 6 months so I’ve gotten used to reading and writing relatively complex filter sequences.

[cotton]

This is something I gotta get proficient at so if you have the time it would be really helpful.

[oO.o]

@cotton

It loops over the “raw_string” output and replaces all “:” with ": ", setting multiline to true
replaces any line that starts with “member:” with a ’ -’, (must be making it a list item)
creates a list and split on the double \n

Yep, all correct.

@cotton

no idea what

@oO.o

>                 | map('replace', '  -', 'members:@@@  -', 1)
                | map('regex_replace', '@@@', '\n')
> ```

Ahh I see now, you’re creating the list by replacing @@@ with a new line. Not sure what the 1 does after that.

That is the messy part. We need to insert members: above the first - replacement we made before the split, otherwise we have an unnamed list floating in the middle of the dictionary:

description: Baseball
  - Phil,pitcher
  - Peter,catcher
  - Wilbur,centerfield
teamname: Rockies

needs to become:

description: Baseball
members:
  - Phil,pitcher
  - Peter,catcher
  - Wilbur,centerfield
teamname: Rockies

Ideally, this would be done with one filter, but afaik, replace_regex has no clean way of limiting the number of replacements. replace does (the 3rd argument for replace is an integer limiting the number of replacements), but in my testing, replace would automatically escape backslashes, so \n became \\n. So as a workaround, I used replace to insert members: once before the first dash and replace_regex to insert the newline via @@@ which is an arbitrary string that should never appear organically in the data.

replace:

description: Baseball
members:@@@  - Phil,pitcher
  - Peter,catcher
  - Wilbur,centerfield
teamname: Rockies

replace_regex:

description: Baseball
members:
  - Phil,pitcher
  - Peter,catcher
  - Wilbur,centerfield
teamname: Rockies

Then of course it’s still just a string, so from_yaml actually interprets it into an object and then you have a list of teams, but you want a dictionary of teams with the keys named after each teamname variable. And if a team has no members, the members variable is simply missing. Based on the samples you provided, if no members are present, you want members to be an empty list.

Both of those issues are rectified here:

@oO.o

        teams: "{{  teams
                    | default({})
                    | combine( { item['teamname']:  { 'members': [] }
                                                    | combine(item) } ) }}"

This takes each team object from the list, assigns it to a key named after the teamname variable and defaults members to an empty list.

[oO.o]

Also, @cotton, if you’re spending a lot of time parsing similarly structured text in Ansible, you should look into creating a custom parser:

https://docs.ansible.com/ansible/latest/network/user_guide/cli_parsing.html

This was originally part of the network suite and intended to parse output from things like Cisco switches, but recently, I think they realized this stuff is much more generally applicable and have moved it into utils.

What I did above makes sense for a one-off situation but it really scales badly if your actual use case is more complex or is central to multiple roles or playbooks.

[cotton]

This hacker-factor on that is amazing.

That’s a nice way to handle that problem. I understand what you did and that’s a good solution to a tough problem.

Thank you for sharing that.

cotton

[oO.o]

What does “access reconciliation” mean in the context of compliance requirements in a security baseline document? It’s listed under access control.

[regulareel]

@oO.o

What does “access reconciliation” mean in the context of compliance requirements in a security baseline document? It’s listed under access control.

I feel like sysadmin speak is like pretend english oftentimes - full of keyword corporate speak techy mumbo jumbo.

[cotton]

I can’t use ansible any longer for configuration management. So, I’m looking at Chef and Puppet.

Can anyone help give me advice on what I should be looking out for with these tools? I’ve used ansible pretty much daily for the past two years. So I have some familiarity with configuration management, but not really sure what to look out for when deciding on these two.

Anyone have advice on things they learned after starting using puppet or chef they wished they knew before they started using it? Or, can you summarize the tools in like a sentence each?

I understand it can be a complex topic so if I can provide specific things for clarification please let me know.

Edit: I see it says that puppet is “model” driven and chef is “code” driven. When would model driven be advantagous compared to code driven?
cotton

[Dynamic_Gravity]

Chef is Ruby based whereas all the other ones are python based.

Not sure about Puppet never used it.

May I ask why you cannot use ansible?

[cotton]

With ansible there is a one → many relationship where one system (ansible server) has to intialize connections to many clients. Chef and Puppet are many → one relationship where many clients initiate a connection to one master

[Dynamic_Gravity]

Or you can use Saltstack which allows you to configure masterless or agentless management

[cotton]

I need to check that out.

For the environment I’m in: the one → many push setup for ansible makes me jump through a lot of hoops to get it to work, which I think is way out of the ordinary deployment. Therefore, I’m looking to pivot.

BTW I’m seriously not trying to start a flame war, because this could turn into that. I’m literally just looking for options and what to look out for when deciding this.

[Dynamic_Gravity]

traditional master->minion setup (run from a master)

sudo salt 'myhost' cmd.run 'echo "Hello, World!"'

masterless minion setup (run from the minion)

sudo salt-call --local cmd.run 'echo "Hello, World!"'

agentless setup (run from a master but minion install not needed only an entry in /etc/salt/roster)

sudo salt-ssh 'myhost' cmd.run 'echo "Hello, World!"'

[Dynamic_Gravity]

@cotton

BTW I’m seriously not trying to start a flame war, because this could turn into that. I’m literally just looking for options and what to look out for when deciding this.

nah its alright fam. I love RedHat but they gotta stop with the walled garden shit with their tools.

[cotton]

How do you initiate the salt-call?

For example, let’s say I want to make sure minion1 has apache installed. How does that get called?

With ansible it’d be like on ansible server run ansible-playbook configure_http -i …

The problem that I face is action has to be driven by the client. It can’t be driven by the master…

[Dynamic_Gravity]

@cotton

How do you initiate the salt-call?

on RedHat family

sudo yum install salt salt-minion

on Debian family

sudo apt install salt-common salt-minion

One does not have the the minion.service running because the salt-call will instantiate a minion process regardless of a traditional or masterless setup. Just that the package exists.

@cotton

For example, let’s say I want to make sure minion1 has apache installed. How does that get called?

By default there needs to be a /srv/salt for your state files (yaml) and /srv/pillar for pillar data.

If doing a traditional master->minion it would go like this for the cli

sudo salt 'myhost' pkg.install apache

but if we were to use state files

/srv/salt/apache/apache.sls

httpd:
  pkg.installed: []
  service.running:
    - enable: True
    - reload: True
    - watch:
      - file: /etc/httpd/conf.d/*.conf
    - require:
      - pkg: httpd

And then apply the state

sudo salt 'myhost' state.apply apache

If it were were doing local then it would be.

sudo salt-call --local state.apply apache