Categories

CUC (6) CUCM (28) Jabber (6) Python (4) Routing (3) Solarwinds Orion NPM (4) switching (1) Video (6) voice (3)

Monday, 28 September 2020

Solarwinds, using application templates to monitor services

A while ago, i was asked to set up alerting for a number of windows servers that would trigger when a critical component would change status, in this example when either the DNS or Active directory service is NOT RUNNING

This post will describe the set up for this kind of alerting. By no means do I claim that this is the only way to achieve this kind of alerting, because it probably is not. But at least it will provide a run down on the components that are required to achieve this. If anyone has any feedback or alternative methods, please drop me a comment below.

The primary requirement for this sort of monitoring, be it through application templates or AppInsight, solarwinds, requires WMI access to the servers in question, usually this is done through some AD service account, which password does not expire. Again, simple SNMP and ICMP access to you server, is not going to work.

This post will follow a number of steps in chronological order:


1-create your application (include you components monitors)

2-assign nodes to the application

3-set up an alert that gets triggered when the application changes status.


so lets get started


1-Create your application  (include you components monitors)

This is the first step, really what you need to remember is that by creating an application, solarwinds will start polling its components and this way you can create a trigger resulting in a certain action as soon as the component changes status.  I am using the word component here (and so does Solarwinds), because an application is built up from one or more components, depending on what you deem necessary to monitor.


For this post, I will use an existing template called, "Active Directory 2008 R2 - 2012 Services and Counters". For this, go to settings> SAM settings > manage Templates.

Here you can copy an existing, out of the box template and modify it as required.

Fig.1. Copy an existing template


Lets drill into the template and see what components make up the application (template). I will continue to call it 'application' from now on, as this truely reflects what is actually does. Although i still feel its a somewhat confusing term to use. Ok, so in the picture below you can see some of the component monitors that make up the application. In this example I disable all component monitors but the DNS service.

Fig. 2 application component monitors


so the detailed view of this component monitor is as follows (sorry i had to keep the pic so big, so you can actually read it):


Fig. 3 component monitor detail

As can be seen in Fig. 3 , the DNS service is being monitored and the component will change status to down as soon as the DNS service is not running, or if one of the Thresholds is exceeded.  if you are not interested in the threshold, just set them to 100, so they never trigger, because I do not think you can remove them individually from the component monitor.

what is interesting here is that the service gets identified by its Net service name  (DNS in this example). so for custom service you would need to add a component monitor and add a particular service to the application. To find out the actual service name, follow this  https://stackoverflow.com/questions/7172276/net-start-service-how-where-to-get-the-service-name, or double click the service, and check its name and populate that into your component monitor:




As you can see the fetching method is WMI, so again, this will only work when solarwinds has the appropriate AD access/service account to actually tap into WMI. You dont really need to filll out the creds just yet, because this will be done once assigning nodes to this application, but you can to test access is sucessfull on certain nodes, by setting Test Node and Test, as can be seen on top of the fig.3. screenshot.



2-assign nodes to the application

So now you have created your application, you will need to assign it to individual nodes, or a group of nodes. because I want to assign my application to Active Directory servers, I will add it to a group that contains all AD servers in my organisation, in my case I used a dynamic query, so that shouold new servers get added in the future the would automaticall be assign the application to be monitored (see Fig.4)

Fig. 4 assign application to nodes


Next, you will need to assign the WMI credentials that the group will use. in this case we use a single service account. See Fig. 5

Fig. 5 choose the WMNI creds


So at this point we have created our application and assigned it to a group of nodes (active directory servers in this case). the final step would be to actually alert when these service get stopped, or better, when the application changes state, when one or more of its components changes status.

3- Create an alert that triggers when the application changes status.

As always, go to settings> mange alerts and create a new alert.


Really the only interesting part of this alert is the trigger, because somehow it needs to tie back to your application created in step 1. So create an application trigger for this. As per fig. 6. below

Fig. 6 Alert Trigger


I add a double citerium for the scope, that is can only alert when the node is actually up, otherwise it will alert when the node goes down for a reboot. Because i have already applied the application to the node group, in step 2, there is no need to redifine the nodes that are in scope, its just the application name (which needs to be the same as the on used in step 1), that we need to define.  After that its the same as any alert. Maybe spend some time on the action and if you add a email action, have it contain the service that triggered the alert so your NOC people know what to do, rather than trying to solve a puzzle.

In the screeen shot below i have added the component status to be included in the email.


the actual email will then look something like this:

Its not a real good example, bacause the email above, is a result of an alert simulation, so services are up, but when a service in the aplication would be truely down, it will tell the engineer what service has the issue

namaste Suckers




Monday, 25 May 2020

Palo alto firewall upgrade and testing step by step


This post describes on how to upgrade an active-passive palo alto firewall pair. also Note that this pair does NOT have preemptive fail over turned on, so there is no automatic fail back after the passive fw recovers. fail back only occurs after ths passive firewall is put in suspsense mode.
The assumption here is is that fw001 is the active one and fw002 is the passive one.

Pre change checks/testing

-check app version to ensure you meet requirements (verify dynamic content version is same as panorama)
-capture routing protocols status (peering state, routes sent/received for protocols etc)
-check userid state:
        show user user-id-agent statistics
         show user user-id-agent state all
         show user ip-user-mapping all type UIA
-check interface states : GUI got to device > network and verify link state of all interface that  should be up  (these interfaces and aggregate interfaces shoudl recover after upgrade
-check system logs, and filter on critical, to see if no critical events have occured after upgrade
-check traffic logs on critical zone for basic health state:
    monitorlogs and filter on interface

Test HA failover Pre upgrade, to confirm the fail back to primary FW works

  • On fw001 , go to the GUI, Device / High Availability / Operational Commands / Suspend local device.
  • CLI:  verify  "show high-availability state"  and see if secondary has become active  (dashboard>widgets>high availability and check has become passive)
  • Restore fw001 to fuinctional state:  On the firewall you previously suspended, select DeviceHigh>Availability>Operational Commands and click the Make local device functional link.
  •  (when pre-empt is not on you will need to suspend fw002, to force fw001 to become active)
  •  CLI:  verify  "show high-availability state"  and see if primary has become active  (dashboard>widgets>high availability and check has become passive)


Implementation:

For this upgrade,  Palo alto's best practices are used as per

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRrCAK

pre-change implementation steps:
================

-pre download the new software (x.y.z) on the device
-create config backup:  device > setup  > operations > save named config (use local admin login)
-export config backup:  device > setup > operations > export names configuration  (use local admin)
-export device state: device > setup > operations> export device state (local admin login)
-generate tech file:  device > setup > generate tech file (local admin login)


During change
==============
1- Failover active fw001 to standy firewall  (ensure everything is working before adding a new software version in the mix).
 Active:  fw001
      On fw001 , go to the GUI, Device / High Availability / Operational Commands / Suspend local device.
           - CLI:  verify  "show high-availability state"  and see if secondary has become active

2- have officeworks TESTER (to be nominated) test all apps, now HA is running on secondary FW

3    Upgrade fw001  firewall to 8.1.14
4     Restart firewall

5   on fw001: show system info , and verify version is x.y.z

6- resume/fail back the primary fw001 to become passive:
                issue: request high-availability state functional
                issue: show high-availbility state   to verify, the primary should still be passive at this                          stage!!! and the secondary should still be active

7-verify BGP and other dynamic routing protocols, if applicable: see the routing table, a

8- verify traffic monitor/log to see if traffic passes,

9 check system logs

8 upgrade fw002 (passive firewall), in same fashion,
            -suspend machine first, now the primary should have become active again  (show high-availability status)
            -then upgrade,
            -reboot
             -request secondary as functional again,  verify high availability


Post change tests

-capture routing protocols status (peering state, routes sent/received for protocols etc)
-check userid state:
        show user user-id-agent statistics
         show user user-id-agent state all
         show user ip-user-mapping all type UIA
-check interface states : GUI got to device >  network and verify link state of all interface that  should be up  (these interfaces and aggregate interfaces shoudl recover after upgrade
-check system logs, and filter on critical, to see if no critical events have occured after upgrade
-check traffic logs on critical zone for basic health state:
    monitorlogs and filter on interface

Monday, 10 February 2020

Pycharm and git hub version control, using branches

I already created a post on how to use github repositories in Pycharm, so you can work on your code on your local client, from a central place; git hub. Granted this is an awesome feature, but it is not really doing version control, it just controls one version of a file or files and keeps them in sync, using push, pull and commit mechanisms. i suggest you read my previous post on repositories, before continuing:

https://ciscoshizzle.blogspot.com/2020/02/using-github-repositories-from-pycharm.html


Now, I am by no means an expert programmer, but one of the first thing i was looking for was a way to keep track of my code and keep the code that i was working on separate from the code that i already has an knew was working.
For instance I was working on a piece of code that, i wanted to log into a bunch of network switches, check the IOS version and write it to a file. I got this to work. The next step for me was to add multiprocessing to it, so it wouldnt take 3 hours for the script to run, on a 200 or 300 node network. I could see that i needed to add to my existing script, however I wanted to keep my already working script intact or at least make sure i wouldnt break it by mistake. and once done, merge my multiprocessing script back into my initial script without multiprocessing. I realise that there might be a dozen other motives to start using version control with brances, but this was my motivation to look into. I just could not continue to work with folders containing a multitude of version of the same code, with no easy way to revert.

in Pycharm all branc related operations are done , using the Git Branches popup, at the bottom of the Pycharm client:










Clicking on it, will take you into its menu, for instance:


Fig.1.- PyCharm Git menu

You can pretty much drive and control all your version control and branches from this menu. As can be seen in figure 1, there are 3 main components:

1-Repositories; these are the local repositories on your machine running PyCharm

2-Common local Branches - these branches exist locally on your machine, within a certain repository and are most likely in sync with Github, but dont have to be until they are pushed into github.

3-Common remote branches - these branches exist on the repositories on git hub, but dont necissarily have to exist locally, untill the branch is pulled.

When you look at the git popup you can see which branches exist, also you can see branchnames like origin/master. all revision branches are forked off the master branch.

Switching between branches

There will be times when you want to work on different branches within the same repository.  Select the branch that you want to switch to under Local Branches and choose Checkout from the list of available operations. As you can see in Fig. 2  i am working in the master branch (as it has the tag symbol in front of it and is therefore considered the current version).  So i select the "Revision_1_DFJ" branch and select check out.


Fig.2. Pycharm Git pop up switch between branches

Once finished, you should see the code under that branch in the main code window in Pycharm.


Creating a new branch locally and pushing it out to github

This comes in handy when you want to change code and put it in a separate branches and making it available for others in github. again go to git popup.

select the master branch. go to New Branch from selected


Fig.3. - create a new branch


Give the branch a name, I have called it revision_2_DFJ, hit create. This should automatically bring you in that new branch, you can check ghoing to the githgub popup, anmd check if the yellow tag icon shows up left of your brtanch, or easier: hoover over the git popup button:


Fig.4. see active branch/currernt branch

as you can see my active branch (purple) is now revision_2_DFJ.




so the next thing todo is make a change in the code, I am just gonna add a comment at the top. for this, again, select the file > git commit file and the commit changes windows pops up:


Fig. 6 - commit window
Fig.6 shows the commit window, and with version control its the same as anything else multiple people work on: only as good as its weakest link, so make sure, you put meaning full comments in your commits.  Also you can see in figure 6, that this commit applies to the revision_2_DFJ branch. 

Please note: nothing will be pushed out until your changes are comitted!

After you hit commit you can see all branch related action in the version contol popup in Pycharm, its not necassary to check this, but it provides great overview of how your branches grow:


Fig.7. Version control popup window
As you can see in Fig.7 my latest branch creation is not forked off from the root, as revision_1_DFJ is.

Now let me push it back into github, so i got to the git popup, select my revisoin_2_DFJ branch and select PUSH and verify in my event log that the push is indeed sucessful.

So now, if i go into githubv itself; I should see the newly created branch that was pushed out from Pycharm, so let me check:


Fig.8 - Github branches within the repository

As you can see above, here is a revision_2_DFJ branch in github.


Creating a new branch on github and pulling it to local machine


This will come in handy when you are an existing PyCharm user and you are invited into participating in the development on a certain piece of code that is already in github, or if you have been working on github online and you want to continue using PyCharm

By know I assume you know how to add a new branch to github

Ok let me give you a clue:


Fig.9



Because i selected the master branch first, my revision_3_DFJ will fork of from master. so master\revision_3_DFJ. there is no right or wrong here, different branches for different collaborations and code.  But remember if you create a branch out of the master branch an you alreay have existing branches, from the master branch,  that you have made alteration to, these changes will not be refelcted in your latest master\revision  branch, because the master is the original. I strongly suggest the master to stay original, unless you are 100% happy your added code works and can be merged into the master.

So you cant really work on inidvidual branches, you have to clone an existing repository from github. I have already described how to do this by going to VCS in Pycharm, so please refer to 


https://ciscoshizzle.blogspot.com/2020/02/using-github-repositories-from-pycharm.html


Ok, so I have now imported the netmiko_login_devices repositories from github into Pycharm, after this, indeed i can see the various branches in the git popup. Now i can check them out and toggle between them, as can be seen in the picture below.


Fig.10 - popup toggle between branches

Pull requestsPull requests let you tell others about changes you've pushed to a branch in a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before your changes are merged into the base branch.

for example, I am asking Jim Bob to revise a certain piece of code.


Now in github i create a pull request on a certain branch that I am collaborating on with jim bob, and i put in a comment for him to review:


Fig. 11 . Pull request in github

Dont merge the pull request yet, as you are still doing work on this code, or Jim Bob might come back with revisions.  So now back to Pycharm.


Fig. 12 - Pull request in Pycharm
Pycharm now display for revision on the code on Jim bobs Pycharm. in the Pull request part of the version control window.



reference:   https://www.jetbrains.com/help/pycharm/manage-branches.html#

Monday, 3 February 2020

Using github repositories from PyCharm

If you want to share your code with other people so they can work on it in a combined effort, dont use gmail or dropbox, like a cave man. Gmail and dropbox have no concept of version control and are therefore not suitable to collaborate on code. Github is, because it supports version control natively.  I will not go into subversions and branches in this post, pretty much because i havent figured out how to use this properly myself, so stay tuned for a post on that.

There are essentially 3 mechanisms in place to keep version control, all these are performed from your pycharm client:

commit:  this 'save' your change to the code locally on your pycharm client
(by default the \users\name\PyCharmProjects folder,  contains the repository/project data).
pull:  Pull the code off the github repository. so that is changes were made by another author, this will be reflected locally on your pycharm client (do this every time you start working on the code, so that other people's changes are refelcted)
push: use this once done with your changes, so that it gets uploaded onto your github repository, centrally. after you pushed your code,  others can see your changes.

before i continue, assure you have the following in place:

-create github account 
-create a repository on github
-install git.exe on your local machine (download from https://git-scm.com/download/win)
-configure your github creds into pycharm (go to File>setttings>version control >github and add a credential set)

At this point i am assuming you are not a coding n00b and alreayd have some meaningfull code in a repository on your github account.

Every repository in github has its own URL, this is how pycharm connects to it:

get this url from github, as per below:





Copy this URL (see picture above), into pycharm, VCS > Check out from version control and paste the URL in.  This will clone the repository locally on you machine.



You can test the connecttion, by clicking....you guessed it..., then CLONE.

Now you can work on your cloned respository, remember to save, as all your changes are local even though you use commit; changes will not automatically update the github repository, until it is manually pushed back out from your pycharm client to the github repository!!! so you will need to save your changes as per usual (CXTRL+S).




now COMMIT

righ click the file : git >  commit file

the window will also show you exactly what changes you made and you can add a commit comment in that, once pushed to the github repository will show you that comment, so your project coworkers can see what was changed, so it is important that these comments are meaningful and not just some facebook crap.



Once you are done making changes and you want to put your work back into the repsoitory, under a branch. you will need to push it back to git hub using a PUSH:

first select the directory or file that you have made changes to and select push:



This will actually show you what change you made or at least what was changed in the commit and what will be pushed out to git hub:



If for some reason you have issues with version control, you can check the Version control log in PyCharm, which you can find on the tab at the bottom of PyCharm, where you terminal and Python Console is as well:


Namaste!


Monday, 9 December 2019

Creating Solarwinds alerts based on custom pollers

Some times you have a requirement to alert on non pre-canned condition in Solarwinds, such as HA fail over, a certain change in routing table (size) or really anything that can be monitored on a devices through SNMP, or put differently: any OID value available.

For this post i am gonna use Palo alto HA fail over as a trigger for an alert. The first part is a brief explanation on how to create a costum poller for palo Alto HA state, because that is all we need to know, once you can poll the state (active/passive), you can use that value as as a condition to perform an alert action.

Below is the definition of the OID call PanSysHAState (which is A Reference to Panorama High Availability state)

Figure 1 - Custom poller definition


You can see the current value, of the OID by testing it, going into the MIB browser. as can bee seen in the screenshot below, fw002 is passive in this case, (and thus fw001 is active, in our particular scenario).

Figure 2 - Browse MIB


So now we have the correct OID, the next step is to set up a new Alert in Solarwinds.

so go to settings > all settings > manage alerts > add new alert

Figure 3 - Alert description and evaluation

Really, there is no point in evaluation the alert every 4 minutes as Fig3. depicts, if your polling frequency is more than 4 minutes (enterprise polling interval is more likely to be around 5 minutes)

Ok, now the interesting part, the trigger condition, first select . I want to alert on Custom Node Poller, as can be see in figure 4. so the scope of the alert is to only look at one particular custom poller, in our case called "panSysHAState" (see figure 1)


Figure 4 - Trigger condition

We want to trigger an action when the state on fw001 (the active on in the pair) is no longer 'active'. no need to include fw002 in the trigger condition.

Figure 5 - Trigger action
i this case I opted to send an email out as soon as the condition is met.

Namaste

Wednesday, 20 November 2019

Solarwinds SQL queries

Sometime you will just have to go into the Solarwinds Orion DB to do certain queries, that cannot be done through the solarwinds GUI. In this post i will describe a number of queries that have helped me in the past to achieve various bits and pieces. to do these queries, open the data base manager on your solarwinds box and add the default server, the select SolarwindsOrionDatabase.




Finding interfaces in UNKNOWN Status


I needed to do this query to summarise interfaces in unknown status, that were orphaned from the actual node, and that were not visible when doing a rediscovery of the particular node, simply because the interfaces (logical) had been removed and solarwinds had not cleaned up the DB for whatever reason


Select InterfaceID, NodeID, Caption, interfacename, Status, StatusLED From [dbo].[Interfaces]
Where StatusLED = 'Unknown.gif' 

Result:






To delete any of these interfaces, you need the interfaceID (above) and run:

delete from Interfaces where InterfaceID=953insert into DeletedInterfaces (InterfaceId) values (953)


Monday, 26 August 2019

VentraIP using SCP for file transfers

Okay, this turned out to be a bit of a bitch.


I have a hosting account with VentraIP, recently VentraIP did some upgrades on their front end and as a result a feauture that I always used did no longer work:

FTP to and from my own hosting partition. after some mucking around I got file transfer to work with scp. here is how:

1-log onto your VIP control panel and got to MY services > Hosting > Manage :




2-Go to Configuration > SSH access and whitelist your own IP address (use www.ipchicken to find out),  now please note this whitelist only lasts 28 days, so you have to keep redoing it.  Also note Ventra allows a non standard port of 2683 




The username, can also be found in the cpanel under Special FTP accounts.

I have used WinSCP to connect to my ventraIP hosting partition, using the details as depeicted above. 


Anyone have any new insights on how to achieve this: please drop me a line