SysAdmin Archives - The Pursuit of Knowledge

April 6, 2023bx2 No Comments

Microsoft LAPS – The LDAP server is unavailable.

If you haven’t used Microsoft LAPS, it’s a neat feature that helps enhance domain security by setting a randomized password for each local administrator account on each domain computer/server. Gone are the days of using one password for local admin access across hundreds or thousands of endpoints.

Microsoft LAPS is not new and has been around for a few years but is a must for a properly secured domain environment. The lack of randomized local admin passwords is often an item detected on internal penetration tests.

Today, I encountered an error that I haven’t seen before with a Microsoft LAPS deployment. I spent the better half of the afternoon trying to figure out why extending the domain schema wasn’t working.

I’ve been following this great Microsoft technet article on deploying LAPS and while trying to extend the schema, I was getting the following error:

I have confirmed that my management machine didn’t have windows firewall enabled, domain replication was functional and LDP.exe would connect to the schema master at the required IP:389.

Everything checked out. One thing I didn’t review yet was Event Viewer.

I remember in College, some of our professors would tell us that Event Viewer will be one of the most powerful tools that we will use, if we choose to use it.

I’ve spent a considerable amount of time in my career searching through logs, exporting and reviewing problems. Event viewer has been an extremely valuable tool and I’ve worked with other team members to educate and help them learn to use Event Viewer to assist their troubleshooting methods.

Anyways, I launched event viewer on my management station that has LAPS installed and where the Schema is attempting to be extended from. Looking under Windows Logs –> Applications, I noticed a bunch of Information events that reference PowerShell, TCP port 389 and my host IP going to a DC.

This environment is utilizing Carbon Black for its endpoint protection and it slipped my mind when I was initially installing this.

I was able to work with our security team to allow traffic from this endpoint and to the other DCs so that the schema extension can occur.

After a bit of waiting, I was given the green light and tried once more.

Voila, it worked.

I wanted to share this because after searching around, I saw references to firewalls and blocking but I can’t recall reading anything about endpoint protection/AV causing this.

Thanks for reading and hopefully this helps somebody in the future.

June 8, 2021bx2 No Comments

Convert Disk from RAID to Non-RAID – Dell PERC H730 Mini

Last week I was working on setting up two new servers at a new office about 6,000 km away. Initially, everything was going smoothly on Server #1 until I tried to configure the second server in a similar manner.

Let me explain…

We are using the following:
-Dell R730xd servers
–Bios 2.12.1
–iDRAC firmware: 2.75.100.76
-Dell PERC H730 Mini
-Seagate ST8000NM0065 SAS (6 of them)
–Revision K004
-Two volumes
–OS (RAID-1, SSDs)
–Storage (RAID-6, Seagate)

What we did on each server for the OS boot drive is combine two enterprise SSD disk into a RAID-1 configuration. This worked well for us as expected.

While investigating some options for local storage that could possibly be shared, we wanted to do some testing with Microsoft’s Storage Spaces Direct, which required us to remove the Storage Volume and convert the disks from a RAID to Non-RAID configuration.

Server #1 was completed successfully. Entering the iDRAC configuration, we expanded Overview –> Storage and then selected Virtual Disks.

We clicked on Manage and deleted the chosen volume via the drop down option under Virtual Disk Actions.

Once the volume was deleted, we needed to convert each disk from a RAID drive to Non-RAID drive.

This is done by going into the Physical Disks section under storage (within the iDRAC menu) and going to the setup section.

From there, you would just click the Setup section at the top, select each or all disks that you want to reconfigured for Non-RAID and select apply.

This worked great for the first server but not so much for the second server.

When doing so, the job would be accepted and checking the Job Queue which is under the Overview –> Server section, we noticed the following basic error message: PR21: Failed

Since the message didn’t provide enough information, we went to the Logs section under Overview –> Server and selected the Lifecycle Log section.

Here you can possibly get slightly more details but in our case, it wasn’t enough to figure out what was going wrong.

We started off by searching that error message on Dells website and found the following:

We couldn’t find out why we were not able to reformat the disks into a Non-RAID configuration. Server #1 completed this without issues. We compared both servers (exact same spec) and there was nothing out of the ordinary.

We stumbled upon an interesting Reddit post that speaks about a very similar situation. The user in this case had 520 bytes sector drives and was trying to reformat them to 512 bytes.

We compared the drives between both servers and everything was the same. We couldn’t perform the exact steps as identified on Reddit since we couldn’t get the drives detected and we didn’t have any way to hookup each SAS drive to a 3rd party adapter and check the drive details.

We decided to do a test and shut down both servers and move the drives from one unit to the other, thanks to our remote office IT employee. Doing so would identify if the issue is in fact with the drives or with the server/raid controller/configuration.

With the drives from server #2 into server #1, we were able to format them into a Non-RAID configuration with ease. We knew our issues were with the server itself.

Diving more into Dells documentation, we found one area that was not really discussed but required to reboot the server and tap F2 to enter the Controller Management window.

Here, we looked around and found what we believed to be the root cause of our issues, located in Main Menu –> Controller Management –> Advanced Controller Properties.

Look at the last selection, Non RAID Disk Mode, we had this as Disabled!

This wasn’t a setting we setup and the initial testing was done by our vendor a great distance away.

We choose the Enabled option for Non-RAID Disk Mode and applied and restarted the server

With that modified, we loaded back into iDRAC and we were finally able to select all of our disks and configure them as non-raid.

Once done, all the disks were passed through to windows and we were able to use them for our storage and to test Microsofts Storage Spaces Direct.

I wanted to take a few minutes and write this up as this was something we couldn’t pinpoint right away and took a bit of time to investigate, test and resolve.

Some resources that I came across that might help others:

http://angelawandrews.com/tag/perc-h730/

https://johannstander.com/2016/08/01/vsan-changing-dell-controller-from-raid-to-hba-mode/amp/

https://www.dell.com/support/kbdoc/en-us/000133007/how-to-convert-the-physical-disks-mode-to-non-raid-or-raid-capable

https://www.dell.com/support/manuals/en-ca/idrac7-8-lifecycle-controller-v2.40.40.40/idrac%20racadm%202.40.40.40/storage?guid=guid-9e3676cb-b71d-420b-8c48-c80add258e03

Thanks for reading!

June 5, 2021bx2 No Comments

Firmware Upgrades on Jabra Headsets (Evolve 20)

Good afternoon all,

Headsets used to be a simple ‘plug it and forget it’ kind of device but there are certain makes and models that can have firmware upgrades applicable to them.

Some headsets that I deal with on a daily basis are Plantronics HW520 with the Plantronics DA70 USB adapter and the Jabra Evolve 20 headsets.

I won’t get into specific details regarding the Plantronic headsets paired with the DA70 USB adapter but avoid that combination if you can. Compared to the Jabra headsets, I’ve had a ton of failures and issues with the Plantronics configuration listed above than I’ve had with Jabra.

Anyways, this isn’t a post to compare both but I just wanted to mention it. I might write a post about this in the future outlining my experience and the issues/failures I’ve seen.

This sunny and hot saturday afternoon, I decided to pop by work to get some quiet time and push through with some outstanding tasks on my plate.

One of the tasks is to prepare a large amount of Jabra Evolve 20 headsets to be deployed to our staff over the comming weeks.

Companies deploy most if not all staff to Work From Home (WFH) due to COVID in 2020/2021+ and while we prepare and send employees to work at home, we want to make sure we patch and reduce the amount of unnecessary calls to helpdesk.

Our staff primarily use Jabra Evolve 20 headsets and they are great, well priced and comfortable but we have had some compatability issues in the past with them.

Some of the issues we experienced was performance and stability of the headset and compatibility with platforms such as Genesys cloud dialers.

When we initially started to troubleshoot, we realized that Jabra Evolve and Plantronics headsets can have firmware upgrades applied to them via Jabra Direct or Plantronics Hub.

When comparing current software versions detected on the headset and new updates and their release notes, we found that often Performance and Stability Improvements are listed in each firmware upgrade along with software compatibility improvements.

When we updated the Jabra Evolve 20 headsets to the latest firmware version as of 2021 (version 4.3.1), we found that our issues were no longer valid. Voila!

95% of these headsets update without issues within the Jabra Direct application but this afternoon I ran into a few headsets that upon starting the firmware upgrade, would error out and no longer cooperate with the application, shown below.

The error above shows after I tried to apply the firmware upgrade, the same way I did it for the many previous headsets.

Pressing Recover Now / Recover just provides me that bland Firmware was not updated message, with the recommendation to contact the local IT Administrator (myself) or Jabra Support.

Since the Jabra Direct application refused to cooperate, I decided to check the Jabra Website to see if a manual firmware upgrade file exists. Low and behold, it does. Release date 2021/04/15, version 4.3.1. I download the file (Jabra_EVOLVE_20_30_4.3.1.zip) and look at the contents of the zip.

Inside is just a basic info.xml and a .hex file.

How do I execute this zip file or the contents of the zip file?

I do some searching online and find mention of an application called Jabra Firmware Upgrade wizard, but I wasn’t able to successfully locate it, nor unsure if it would actually work in my case.

I kept searching and eventually found an article on Jabras website that explains how to manually upgrade the firmware when a failed firmware installation occurs.

The important part of this is when you enter the Updates section of Jabra Direct, press the following keys to unlock the Update From File option.

CTRL + SHIFT + U

As you can see above, the same headset that failed the firmware and failed to recover the previous version, was successfully updated using the .zip file via the Update From File hidden option.

Thinking that the few headsets might have to be RMA’d, I was able to get them updated and ready for deployment.

As this was not an easy find, despite the instructions on Jabras site, I found many discussions and attempts to manually apply the firmware via alternative methods.

Coming across this Jabra article and the hidden menu, I knew I wanted to share it here in the event that somebody runs into the same issue as I have.

Thanks for reading.

May 13, 2021bx2 No Comments

VMware VMUG Advantage 15% off discount – 03/13/2021

Hello all,

Those of you that want to the full benefits and features of VMware for a homelab, you can register for the VMware VMUG Advantage Program get a decent discount with the following code:

ADV15OFF

The VMware Advantage is a single user, 1 year subscription for $200.00 USD but if you enter in the 15% code (ADV15OFF), you can get it for $170.00 USD.

This has its benefits as it provides you with various VMware products and the ability to have full access to ESXi and the advanced functions (ie: vSAN + more).

I have no affiliation with this code and I was able to use it today successfully, on May 13th 2021.

This code from what I tested only worked with the 1-year subscription and not the 2 or 3 year.

For those of you that want to know more about this offering from VMware, please see the following link:

https://www.vmug.com/home
https://www.vmug.com/membership/vmug-advantage-membership

It is pricey but if you are working at advancing your skills in this platform, I think it’s a small price to pay.

Sure you can just download ESXi and have the 30 day free version but this is less hassle and has a large community backing this group.

Just last week I was listening onto a session from VMware VMUG presenters about homelab configurations, costs and best practices.

I figured I’d offer this out if anybody wants to try. The code may not work by the time you check so I apologize. I only came across this code from other references on various blogs.

Good luck and stay safe!

May 13, 2021bx2 No Comments

Fantastic Article regarding Domain Controller Security

I’ve been recently doing a bit of reading regarding corporate domain security and I came across this short but detailed Microsoft Article that references some best practices.

https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/securing-domain-controllers-against-attack

One interesting thing that I came across and it might already be common knowledge is the recommendation to enable BitLocker on Domain Controllers.

I figure if anybody is looking at working on their homelab and building skills, you might as well start off with good/best practices which would follow you from the homelab to the corporate world.

The article also has links that open up further security documents such as Administering security policy settings, Avenues to Compromise and configuration of firewall for AD Domains and Trusts.

As I continue to review and read more into these, I wanted to share this link in case anybody else has an interest in this topic.

March 23, 2019bx2 No Comments

The trouble with hacks

/rant

I want to rant. I’ve been working as an IT/Sysadmin for about 2 years now and there are two things that I am haunted by.

DNS
Group Policies

Now, I am always learning and I am by no means an expert at windows systems administration. I took on more and more responsibilities that removed me from the ‘IT Support’ role and let me grow into Systems Administration and I continue to learn daily.

Now, not get into specifics, but taking over a AD infrastructure that was neglected by hacks is terrifying. I refer to hacks, as in people that neglect the network, that don’t have a proper vision for documentation and structure and that don’t understand how AD and GPOs work.

Within the IT SysAdmin community “It’s always DNS” is a common phrase and a joke at times. Well god dam, I can’t believe how accurate it is or how powerful DNS is in a network.

You know what irks me? People that use crafty stupid hostnames for critical servers or any server at that. Stupid names such as “Sugar Baby” “Super Man” “Bat Man”, etc… you get the gist.

When you take over a network and have critical servers with stupid naming conventions like that, it can get very easy to shut down the wrong server or make changes because all of the names are so irrelevant. Especially when nothing is documented and you are left to your own to research and investigate carefully.

Not that I’ve had that happen, but I have had a mishap with a DNS record that was named something ridiculous. The server wasn’t even around anymore but a critical server was using that DNS record for a link to an IP in it’s hostfile. Something I never thought to check nor look into.

The other thing that annoys me is the ignorance of not knowing how to properly setup GPO’s and push them out to AD. You do NOT need to enforce everything. Stop doing that. After spending time looking around and cleaning up GPO’s, you wonder what would drive a person to just enforce everything.

Sure, if it’s a critical policy that you want in every OU regardless if it has Inheritance blocking or not but don’t enforce everything just because you are trying to push the policy out faster or believe that it will guarantee that the policy will get to the clients.

I cannot believe that a novice admin is correcting domain wide issue that a senior IT director of many years had made.

I can spend the rest of my afternoon ranting about stuff that I’ve come across but that’s not the point of this post. I wanted to get DNS and GPO’s off my chest only.

I suppose you will find this in any job/career. People that want to take initiative, drive, pride in their work and do the best with what they can. Others will just let things fall into disarray and not bother.

/rant

November 27, 2018bx2 No Comments

Adding a vCenter 6.7 license

Hello, it’s me again.

From my recent blog post regarding setting up vCenter, I had difficulties locating the area to apply the vCenter license. From what I found on the internet, it was referenced that you should go to the Host that contains the vCenter/VCSA VM, click on the VM and click on Configure. Maybe VMware changed it in version 6.7 but I could not find the same area for license registration under the VM itself.

Under the VCSA VM –> Configure –> Settings, I should see a ‘License’ section. I could not find anything of that sort. I logged in as my admin account and my personal admin account, both that have the license role and that feature was still not available.

Frustrated, I did some looking around within the vSphere client and I found the area to do this.

You need to click on the ‘top’ FQDN vCenter identifier on the left hand side of the window, which houses your Datacenter and the nodes inside.

Once you click on it, you will see the following,

As you see, now selecting the VCSA and going to the Configure section and under Settings, we now see Licensing as an option. Now in my case, I’ve already applied the license but I’m going over where I went to do this.

You would select the Assign License button to proceed with entering your key into vCenter.

Under the Assign License window, you will have two options. To select an existing license or new license. You can import the license from your License section from the admin page or you can type in your license if you haven’t already done so. I’ve already uploaded my licenses to the Administration License section, which I will show next.

Now what I have done initially was gone into the Administration section –> Licensing –> Licenses and typed in the VMware vCenter Server 6 Essential vCenter license key. When I did this, the usage of the vCenter license was set to 0 and capacity was set to 1. This was because I never assigned the license to the vCenter itself. I did this in the Assign License window as seen above.

The last and final screenshot above shows the Administrator License window which identifies my License(s) and their state and capacity.

To note: When I was in the process of importing each host, the license for those hosts registered automatically here. I did not have to enter the VMware vSphere 6 Essentials Plus License. Those just followed with each host/node into vCenter.

November 27, 2018bx2 No Comments

My novice attempt are VMware maintenance

I’ll come out and say it, I’m not an expert or a confident user of virtualization and more specifically VMware products. Over the last bit, I’ve taken on a more senior and technical lead position at my job and that involves more to do with the infrastructure side of things and not as much ‘customer facing’. I’ve played around with VMware Workstation and Oracle VirtualBox but I haven’t done a hole lot in regards to ESXi, vCenter and the works.

I needed to ‘pull up my big boy pants’ and start learning as much as I can in the short time frame about our production ESXi cluster, trying to understand the configuration and anything that may be wrong with it.

When my department slowly withered away until it was only me, I’ve heard that our vCenter is broken and that management of the cluster is not possible. Not having VMware support, I was really concerned about this broken system and how it would negatively affect our production and highly critical cluster. I started doing some reading and came to realize that vCenter (VCSA) is only a central mangement feature. Rather than using vSphere client to manage each invidivual node/host, vCenter allows you to manage the hosts all together (in a cluster) and enabled a few features, including High Availability (HA) and vMotion (allowing to move VM’s from host to host without downtime).

Knowing this, I spent any downtime I had reading up about vCenter and VCSA. I looked at different installation methods (Windows vs Linux) the pros and cons of each. vCenter can be installed on top of a Windows installation or it can be configured on a Linux machine and often referred to VCSA (vCenter Server Appliance).

My first question was regarding what vCenter/VCSA can I use with my cluster? Luckly, I came across a page on VMware site that helps identify the version of ESXi and what version of vCenter is compatible.

With that sorted, I downloaded the most recent version of vCenter 6.7U1. I choose to download the Linux installation rather than mess with Windows and use up a license for it.

Now with the .ISO downloaded, I searched high and low to find a good step by step guide on how to complete this install. I already shut down the old vCenter VM that was previously created by our IT staff, which was having issued and filling it’s storage with logs. Rather than try to troubleshoot it, I wanted to start with a fresh install.

I came across this fantastic link that helped me tremendously for setting up and installing my VCSA. The notes and screenshots helped a novice like myself through this process.

As this was a live production setup, I was always fearful of something occurring but unfortunately I don’t have the resources to do it any other way.

Anyways, I felt that I wanted to share this quick post and the link to the site that helped me through this process. Good articles go a long way in helping others out and that is one thing I want to focus with this blog site. To provide good information that I discover or come across.

Thanks for reading!