Writing Ansible Libraries / Modules

I read this post by Adam the Automator and there was some talk on using win_shell for most of the tasks. I had a lot to say about the topic so instead of replying further in the twitter feed I thought I would share my thoughts in this blog . Just to address the tweet, underneath the hood of the playbooks, roles, tasks , Ansible is just executing powershell in an idempotent way. (which really is just powershell script written in a certain way. I’ll address this below). At first I had the same mindset of @Adbertram. Why use Ansible when Powershell can do it all? One of my good friends and colleague eventually got me to use it and now I can see why it is useful. The reason why I use Ansible is for organising my Powershell tasks and then orchestrating them together. For me I feel that writing Ansible playbooks, roles and modules easier than the DSC ecosubsystem. This is however is for another topic. Which brings me to when would we write ansible modules? The simple answer is when you find yourself writing blocks of tasks in shell script. The Win_Shell module is great for simple instructional tasks. So for example simple tasks Download a file from a ftp session. but when you find yourself writing task blocks of win_shell commands Red Hat’s best practice is to turn that into a module. Writing blocks of commands could be akin to using the DSC Script Resource for all tasks and creating specialised DSC resources is when you create an Ansible module. Most of the time you use DSC Script module until you can write a DSC Resource for that particular task. Writing an Ansible module for Windows can be a mystery at first but it is quite simple if you understand the core components as the Ansible team have written helper functions that make this process easy. To create an Ansible library, it is a simple matter of creating a folder called library in your role and putting your powershell scripts in there:

C:\Users\weiyentan\Documents\GitHub\AnsibleModules\roles\configure_win_failovercluster
├──defaults
│ └──main.yml
├──files
├──handlers
├──library
│ ├──win_failovercluster.ps1
│ └──win_failovercluster.py
├──meta
│ └──main.yml
├──tasks
│ └──main.yml
├──templates
└──vars

Here you can see that i have created a module called win_failovercluster.ps1. The file win_failovercluster.py is essentially the python help documentation on how to use it the module. Once I have this I can then reference this in my playbook like so:


– name: failover cluster to other nodes in Cluster
win_failovercluster:
name: ‘{{ item }}’
Clustertype: clustergroup
destination: "{{ win_sql_destination}}"
with_items:
– ‘SQL Server (MSSQL)’
– MSDTC

I won’t go into how to write Ansible playbooks the extra parameters such as Clustertype and destination are all customisable and you can define it in the ps1. Here is the anatomy of a ps1 file.

#Requires -Module Ansible.ModuleUtils.Legacy
$ErrorActionPreference = 'Stop'

$results = @{
changed = $false #mandatory key value pair
msg = '' #optional keyvalue pair
}

The first part is that you must have a hash table called results. This tells ansible whether it has changed it or not. You can control the hash table later based on your conditions. The second part is where you define your parameters:

$params = Parse-Args $args -supports_check_mode $false
$check_mode = Get-AnsibleParam -obj $params -name "_ansible_check_mode" -type "bool" -default $false
$name = Get-AnsibleParam -obj $params -name "name" -type "str" -failifempty $true
$Clustertype = Get-AnsibleParam -obj $params -name "Clustertype" -type "str" -ValidateSet "Clustergroup" , "Clusterresource" , "ClusterSharedVolume"
$destination = Get-AnsibleParam -obj $params -name "destination" -type "str"
$group = Get-AnsibleParam -obj $params -name "group" -type "str"

The Parse-Args function pulls the parameters from the yml and then puts it into the powershell script to process. Get-AnsibleParam then gets those values and then sorts them accordingly. For those that know Powershell it is very similar to powershell param block. Here is some of the examples with the powershell equivalent: -type: class. [int], [string] -failifempty: mandatory -ValidateSet The name parameter is what you call the parameter from the yml. You can create as many parameters as you like. Because of the way Ansible works, Write-Error, throw , Write-Output is not recognised by Ansible. Instead there are other ways to use these so that Ansible will behave correctly. The functions to use are: Fail-Json = Write-Error, throw Exit-Json = Write-Output Remember, when you use these functions the script will exit. I can manipulate the message that is given.

$results.msg ="The cluster group $name is already on $destination"
Exit-Json -obj $results
Fail-Json $results -message "The cluster group $name did not move successfully to $destination"

So how do we make this whole thing idempotent? The same way that DSC works but not having to know the Get, Set, Test. Consider the block of code for configuring the failover cluster. Consider the following block of script.

First we evaluate whether the cluster group exists through a helper function testclusterelement . If it doesnt exist, then we produce an error, with the error message back to Ansible. The ansible module will terminate.

Then on line 12, we do another get on a value of the current node. If the current node is the same as $destination, then we have no change , everything is set correctly, Exit the script with the message to say that the group is already on the correct destination. Once it reaches Exit-Json the script will exit.

Then it will move on, and try to move the cluster, because the assumption is in this case that we are ready to move. the cluster group. Again, we produce the message and Move-ClusterGroup executes in a try catch. We change the the results hashtable to reflect that change.
In the catch block we use Fail-Json to produce the correct errors.

The last part is where we have accounted for when the destination has -not- been specified and we just move the role to any destination.
I could have written this module better by having the exit-json output doing it only once.
For completeness I have provided the gist link below:

As you can see , Ansible doesnt replace powershell at all. It complements it. All tasks in Ansible are just Powershell scripts when it comes to Windows. It’s not a question of whether you should use Powershell or Ansible. Ansible will wrap around your existing workflows. While I do believe that I wouldnt use Ansible for day to day tasks that I interact with Windows. I would Ansible Tower/AWX to replace scheduled tasks and execution of Powershell ‘scripts’. Ever since I have started using Ansible, I have found my organisation of code to be better and my reusability of code far more proficient than I was with without.

Pester – Parameters and Hashtable Fun!

So I have been really getting into Pester over the last few months using Pester. I have been using Pester for operational validation purposes.

I could see the potential in writing operational validation tests as that is what we were essentially do when we do manual checks on server builds.

One of the things that I found in this scenario is that when we write pester test, it seemed that we have to write a test for each node that we are testing.

Then I came across an article that was written by June Blender called “How to Pass Parameters to a Pester Test Script”.

Essentially this article explained how to pass parameters into a Pester Test. I thought that was great and so I decided that I would write a function that would put the values in for me.

June explained that the syntax would follow this:


Invoke-Pester -Script @{Path='C:\Tests\MyTests.Tests.ps1';Parameters=@{Name='Pester';Version='3.4.0'}}

So in my function I could create a whole lot of parameters replacing the values and wrap it around a function…taking the above as an example I could create a pseudo function like so:

This would work but there was a problem. The problem was that it would only work on that particular Pester test. It wouldn’t adhere to the proper rules where we create a function to handle anything that you would throw at it.

So I went back and rethought how I would create a function that would take any parameter and then process it. Then I came across an idea.

The format that Pester uses to accept multiple parameters was essentially a hash table with multiple layers. ie it had a main layer that held the path parameter and then it had a parameter layer that had the values in it.

So my idea was to pass the parameters as properties in an objects to the function and it would process the results.

This would allow all the parameters needed for Pester to be in a csv spreadsheet and then I could use import-csv and foreach-object to pass through to the tool.

$test =import-csv c:\csv\test.csv

$test | ForEach-Object {$_ | Invoke-POVTest -testpath C:\test\pov pester test\povpester.test.ps1

So that would mean that I would have to find a way to get Powershell to get the properties into a hashtable. Jeff Hicks had a blog post (can’t remember where it was …will update it when I can find it) where he shows how to copy properties of an object. I follow something similar and that is to get all the noteproperty the custom object has and then copy that to the hash table.

Then I produce a hashtable that mimics the layout of what Pester wants.

$finalhash = @{ 'Path' = $testpath; 'Parameters' = $hash }
$param = @{
    'script' = $finalhash
}

On thing to note is that I create a called script. The reason why I do this is that in the final function I eventually created I wanted to add in parameters to pass through to the underlying cmdlet invoke-pester.

Below is a snippet of the heart of whats going on:

I have put up the module at GitHub under the module POV (Pester Operational Validation) here . NB . as at the 5th Jan Its in a dev branch. One I have added in inline comments I’ll merge it with the master branch.

Enjoy, I hope that helps anyone trying to understand hashtables, pester and objects

Powershell Reporting – Server Auditing – Part 1 -Getting the XML

At my current workplace every single time we install a server (yes Desired State Configuration has not caught on here yet….)there is a compliance check list that has to be filled out a to make sure that everything has been done.

I had to do the checks one time and thought it was an opportunity for something that I could automate as it was merely retrieving some information from the system itself. Besides it was taking forever to go ahead and tick some boxes.

The information that they asked for is a lot of wmi/cim information such as disk drive information, network information, operating system details and after I talked to my team leader we got it to retrieve information from group policy/SCOM/SCCM.

I was a bit new to this so after looking around powershell help, and then after looking for some inspiration   I came across Irwin Strachan post on how to create Active Directory snapshots and his other post on how to do reporting using Pscribo by Iain Brighton.

I decided to look into this further and adapt his idea of snapshots and adapt it to my problem. Thanks Irwin!

Irwin’s solution is to create variables and get the information from Active Directory and then export the information to a XML file which can then be passed through to whatever you want. . . Pester and in this case PScribo.

PScribo is a framework that allows me to the same cmdlets to produce various different types of output.

I faced a bit of challenges in regards to this. First off, the firewall ports between networks had blocked off WMI. This meant that all my Windows 2008 R2 machines were not able to be queried because the  Windows 2008 R2 machines had only Powershell V2. By default remoting was  turned off. The organisation was not keen in turning them on…it would mean changes management controls would get in the way. This didn’t really matter as there was a Windows 2012 R2 /2016  upgrade path which was in the early phases which meant that I could write my script to handle situations for future/existing versions of Windows 2012 R2.

I decided to go with the use of switches in my parameters. Due to the current limitations I thought that for every Windows 2008 R2 machine we’ll have to do the old way and copy it into the machine and then do it locally. I would have to add in the functions that I need to use for those times when I am reporting on a Windows 2008 R2 machine. I’ll use a switch -local that would tell the script to use certain methods to do things….as  some of my code did not work in a certain way. Below I’ll go over a sample of code so that you get the idea of how I am doing it and then provide the full source afterwards.


	#$filepath = Join-Path -Path $Path -ChildPath "$computer.xml"
	if ($local)
	{
		$CCMEXEC = get-process -ComputerName $computer -name CcmExec
	}
	else
	{
		$CCMEXEC = Invoke-Command -ComputerName $computer { get-process -name CcmExec }
	}
	if ($local)
	{
		$SCOM = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | where { $_.displayname -like "*operations manager agent*" } | select displayname
		
	}
	else
	{
		$SCOM = invoke-command -computer $computer { Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* } | where displayname -like "*operations manager agent*" | select displayname
	}
	
	if ($local)
	{
		#$ENDPOINT = Get-RemoteSoftware -ComputerName $computer | where name -like "*Symantec endpoint*" | select name
		$ENDPOINT = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | where { $_.displayname -like "*Symantec endpoint*" } | select displayname
	}
	if ($dcom)
	{
		$TimeService = Get-WmiObject -ComputerName $computer -classname win32_Service | where { $_.name -eq "W32time" } | select -ExpandProperty state
	}
	Else
	{
		$TimeService = Get-CimInstance -ComputerName $computer -classname win32_Service | where { $_.name -eq "W32time" } | select -ExpandProperty state
	

Thanks to Irwin, I used the variables to store the end result of values that I want to look up. I create a foreach loop and include the variables within the loop. In my original script at the end of the loop I then export that to a XML file. I am planning to change that so that it outputs that so I can pipe that to other cmdlets….and this way I could create a function out of it…and create a module.

The full code is below. Next time I will write up on how I pass this to PScribo for reporting.

A study in Powershell scripting: Part 3

This is the third part of my series of studying scripting. I will follow up with how I used the previous function I created to get the results that I wanted. (NB I have changed the name of some groups to something else to maintain privacy. )

Onto the example. I produced three groups to represent the separation of the alphabet.

Write-Verbose -Message "Collecting members from Contoso_GeneralAccess Active Directory group with surnames starting from A-H...."

$contoso_ah_extract = Export-ADGroupmember -ADGroup Contoso_GeneralAccess -ADProperty surname -regex "^[A-H]" | where {
	$_.samaccountname -notmatch "\b[A-Za-z]*admin\b" -and`
	$_.samaccountname -notmatch "\b[A-Za-z]*svc\b" -and $_.enabled -eq $TRUE} | select -ExpandProperty samaccountname

Write-Verbose -Message "Collecting members from Contoso_GeneralAccess Active Directory group with surnames starting from I-P...."
$contoso_ip_extract = Export-ADGroupmember -ADGroup Contoso_GeneralAccess -ADProperty surname -regex "^[I-P]" | where {
	$_.samaccountname -notmatch "\b[A-Za-z]*admin\b" -and`
	$_.samaccountname -notmatch "\b[A-Za-z]*svc\b" -and $_.enabled -eq $TRUE} | select -ExpandProperty samaccountname

Write-Verbose -Message "Collecting members from Contoso_GeneralAccess Active Directory group with surnames starting from R-Z...."
$contoso_qz_extract = Export-ADGroupmember -ADGroup Contoso_GeneralAccess -ADProperty surname -regex "^[Q-Z]" | where {
	$_.samaccountname -notmatch "\b[A-Za-z]*admin\b" -and`
	$_.samaccountname -notmatch "\b[A-Za-z]*svc\b" -and $_.enabled -eq $TRUE} | select -ExpandProperty samaccountname

So essentially what this snippet is saying is Find all users in the Contoso_GeneralAccess group and export all users that surname begins a-h, i-p, q-z and then dumps them as samaccountname format. It puts them into variables for the next lot of processing.

The next series of tasks is to grab the sharefile groupmembers. I want to put this in group variables so I can compare them with the extract variables above. The code is below:


Write-Verbose -Message "Collecting members from Sharefile_A-H group..."
$sharefileadgroup_ah = get-adgroup Sharefile_A-H | get-adgroupmember | Get-ADUser | select -ExpandProperty samaccountname
$sharefileadgroup_ah_obj = get-adgroup Sharefile_A-H

Write-Verbose -Message "Collecting members from Sharefile_I-P group..."
$sharefileadgroup_ip = get-adgroup Sharefile_I-P | get-adgroupmember | Get-ADUser | select -ExpandProperty samaccountname
$sharefileadgroup_ip_obj = get-adgroup Sharefile_I-P

Write-Verbose -Message "Collecting members from Sharefile_Q-Z group..."
$sharefileadgroup_qz = get-adgroup Sharefile_Q-Z | get-adgroupmember | Get-ADUser | select -ExpandProperty samaccountname
$sharefileadgroup_qz_obj = get-adgroup Sharefile_Q-Z

Then I use Compare-Object to do the comparison. My notes that I made in the script should explain the details what I am actually doing and the reasoning behind it. (NB Thats why its so important to add inline comments in my opinion so that when it comes to review in a few months time then then I can understand the logic!)


# This region compares the groups as discussed in the beginning notes. It creates variables on what to add and remove in the respective share file groups using the compare-object cmdlet. As the reference object is the  
# variable $Contoso_ range, Users which has the sideindicator => means that in this case the users exist in the differenceobject variable which is the sharefile group but not Contoso group and therefore should be removed. 
# Those that has the side indicator <= means that it exists in the Contoso group range and not the sharefile group and there should be added. #region Comparison: Write-Verbose -Message "Comparing members from Contoso_GeneralAccess group extract from A-H to decipher what to add and remove from Sharefile_A-H group..." $objects_to_remove_in_sharefile_ah_group = Compare-Object -ReferenceObject $Contoso_ah_extract -DifferenceObject $sharefileadgroup_ah | where { $_.sideindicator -eq "=>" } | Select -expandproperty inputobject
$objects_to_add_in_sharefile_ah_group = Compare-Object -ReferenceObject $Contoso_ah_extract -DifferenceObject $sharefileadgroup_ah | where { $_.sideindicator -eq "<=" } | Select -expandproperty inputobject Write-Verbose -Message "Comparing members from Contoso_GeneralAccess group extract from I-P to decipher what to add and remove from Sharefile_I-P group..." $objects_to_remove_in_sharefile_ip_group = Compare-Object -ReferenceObject $Contoso_ip_extract -DifferenceObject $sharefileadgroup_ip | where { $_.sideindicator -eq "=>" } | Select -expandproperty inputobject
$objects_to_add_in_sharefile_ip_group = Compare-Object -ReferenceObject $Contoso_ip_extract -DifferenceObject $sharefileadgroup_ip | where { $_.sideindicator -eq "<=" } | Select -expandproperty inputobject Write-Verbose -Message "Comparing members from Contoso_GeneralAccess group extract from Q-Z to decipher what to add and remove from Sharefile_Q-Z..." $objects_to_remove_in_sharefile_qz_group = Compare-Object -ReferenceObject $Contoso_qz_extract -DifferenceObject $sharefileadgroup_qz | where { $_.sideindicator -eq "=>" } | Select -expandproperty inputobject
$objects_to_add_in_sharefile_qz_group = Compare-Object -ReferenceObject $Contoso_qz_extract -DifferenceObject $sharefileadgroup_qz | where { $_.sideindicator -eq "<=" } | Select -expandproperty inputobject

#endregion

In the next post I will discuss about removing and adding users from these groups using the comparison object and how I created an email report to send out what was deleted and added.

I hope that this gives some insight to people learning Powershell.

A study in Powershell scripting: Part 2

In this post I discuss about the  function I decided to incorporate into the script which is exporting Active Directory users to three sets of groups.

I like writing my own scripts because then I have complete control as to how it will work. Again going back to my principles I look at how it can be done manually and then transform them to code.

Manually I would have to collate all the Active Directory users in the master group by their last name and filter out A-H and so forth through the alphabet. So identifying some of the core cmdlets here is Get-ADGroupMember but in order to extract out the users is all about filtering, which generally means Where-Object (a little hint is that anything to do with whittling away records through the pipeline generally requires where-object, you will find that you use that 80% of the time in Powershell for day to day tasks ;))

The second part would involve regular expressions which I am absolutely terrible.

Second step. Look at help on Regular Expressions in Powershell. It is not a cmdlet but a concept so where to look? An easy way in the help is to type in keywords. Such as examples are like Get-Help *regular* and it will list down all the helpfiles relating to this.

Another tip is that all the conceptual ideas in Powershell  generally start with the word ‘about‘ and the spaces are replaced by an underscore. Following this syntax if I type Get-Help about_Regular_Expressions would give me the required helpfile. By the way if you wanted to list through ALL helpfiles type

Get-Help about*

This will show all the help files  for conceptual ideas in Powershell.

Anyway I am digressing. Back to the matter at hand, at this stage all I needed to do was to output a set of objects from the cmdlet. So I started to play with things like:

Get-Adgroupmember -identity GroupA

Then I hit across a problem. I needed to get the last name of every user but Get-Adgroupmember produces only a unique type and has a unique set of members in it. No surname or firstname properties.

A simple pipe through to Get-Member confirmed that the only members that seemed relevant to me would be samaccountname,and name. Then I remembered something reading Don Jones’s books and other various Powershell books out there said and that was I could pipe through the value of the object through to another one –providing– that they had the same property in each cmdlet! I thought to myself what other cmdlet produces Active Directory user information? Get-Aduser is something that comes to mind. If you look at Active Directory users even in the RSAT you can see that First Name and Last Name is there in the windows environment.

So I did a Get-Member on Get-aduser to confirm their existence. Sure enough there were givenname and surname were present as members which satisfied my requirements but it also has samaccountname….which is the glue that pieces the two together…which means I could do this:

Get-ADGroupMember -Identity "Domain Users" | Select -ExpandProperty samaccountname | Get-Aduser |Where-Object {$_.surname -like "Tan"}

The sequence essentially is:

  • get all members of group domain users.
  • Select the samaccountname property and transform it to text.
  • Pass along that the Get-aduser cmdlet and filter out the surname to be like Tan.

Now I needed to rather than find a single word I needed Powershell to match users based on their surname. So I changed the operator -like to -match. I needed to paramaterize certain values. Parameterizing the value for the user was easy, but I also wanted to parameterize the comparison operator but I came up with errors.


"Unexpected token '$regex' in expression or statement"

I did some digging around this but there was nothing much definitive so I asked for help at Powershell.org and the community was very helpful. (Thanks Dave Wyatt!).

The end result of my custom cmdlet is :

[CmdletBinding()]
param
(
[Parameter(Mandatory = $true,
ValueFromPipeline = $true,
ValueFromPipelineByPropertyName = $true,
Position = 1)]
$ADGroup,
[Parameter(Mandatory = $true,
Position = 2)]
[ValidateSet("givenname", "surname")]
$ADProperty,
[Parameter(
Position = 3)]
$Operator = 'match',
[Parameter(Mandatory = $true,
Position = 4)]
$regex
)
$a = get-adgroupmember -identity $ADGroup
$splatOperator = @{ $Operator = $True }
$a | Select -ExpandProperty samaccountname | Get-Aduser |
where $ADProperty @splatOperator $regex

}

In Part 3 I will discuss how I used this function to create the rest of the script and my thought process behind that.

The three most useful commands in Powershell

Well for me anyway. 😉

If I had asked you to guess I am doubtful anyone would have guessed . These three are Get-Help, Get-Command,Get-Member. 
If I had known about about these earlier in my powershell study it would have made my life a while lot easier . 

Many people who first start learning Powershell go straight to the Internet to search for their scripting answers/help . The problem with that is that while the script may solve a generic problem no proper understanding of Powershell concepts will mean it is harder to build your own. 

Anywho I am digressing. 

Get-Help is just that. Getting help on all the cmdlets that you are available in Powershell. You can even use wild cards. 

For example Get-Help *aduser* will show all the commands (in this case all things related to active directory users if you have Active Directory modules loaded). 
Get-Command is something similar . But it just searches commands. There can be well over 2000 cmdlets in the latest version of Windows and more if you have modules.

Example is Get-Command g*aduser* -module Activedirectory. This pulls all commands from active directory that has start with the verb g and the noun aduser. Powershell is very smart in this regard. 

Get-Member is a very important cmdlet. This cmdlet will show all members of an object. All cmdlets are objects so in essence it tells what what you can refer to. 

Example: Get-Service | Get-Member should produce the members of Get-Service. You can then reference all of these properties. 

With these three commands I found I could find my way around powershell and not depend on Google. Hopefully this helps all you guys learning the Shell. :)

Hello world!

I thought it might be a good idea to segregate all my IT posts from my personal blog to this one. I am looking for a more focused view on my work in IT. For the last few months I have been learning about Powershell. The fine folks at powershell.org said that we can contribute at any level  even beginners. We can document our learning process in the hopes of helping others.

I am also learning about Alpha Five a database ISE.

Seeing that Powershell is kinda my first time scripting language, I thought I would blog about my learning experience.

I will primarily be focusing on these two things as well as a little bit of other technologies in between. See you in the next post :)