PROBLEM DESCRIPTION
Our client is a technology company who provides solutions like multi-cloud data management, data protection, storage optimization, compliance readiness and workload portability. About the challenge centered on their Backup Appliance Services, which was a manual and cumbersome operation, for one of their major customers, a leading global retailer.
This client needed to query the level of patch applied to each of its Appliances (over 300 nodes). Being performed manually, it was time-consuming and error-prone, requiring constant attention and input from the administrator .
Our team was tasked with automating the appliance management process. Due to their restrictive IT policies, we were confronted with several key limitations:
- Could not use key authentication (ssh-keys).
- There was no CMDB with the list of nodes.
- It took between 2 and 5 minutes for each appliance to be checked manually, since different menus had to be navigated to obtain the information.
- Solutions like Ansible were not authorized due to IT restrictions and infrastructure requirements.
- The solution had to be agentless, and processes were not to be left running on the appliances.
- SNMP was not enabled to perform data extraction from the nodes.
IMPLEMENTED SOLUTION
In order to optimize the manual and repetitive tasks that do not scale well in large infrastructure deployments, 3XM proposed the development of a solution called Command Center: an operations management automation tool to manage these Appliances, and easily adaptable to manage any physical infrastructure component that supports ssh protocol.
The solution allows Infrastructure Administrators to provide a list of nodes on which to interact, credentials and scripts to use on each Appliance, through a web layer or via CLI (Command Line Interface). Command Center parallels the execution of the selected script by applying it to the selected component domain, generating a report in json format with the expected outputs, or reporting problems during execution.
It also uses templates that facilitate automation in customized menus or non-standard CLIs. We also contemplated the creation of simple templates prepared to interact with the complexity of NBU.
The client was also able to keep strict control of the published scripts using the Script Repository component that could run either as a dockerized component, or, on top of an already existing GIT repository.
To validate its operation and test the scalability of the solution, 3XM built an NBU Appliance emulator, which allowed us to test the interaction of the tool with hundreds of nodes.
OBTAINED BENEFITS
After several brief meetings with the 3XM Group's Architecture and Technical group, we were able to capture the requirements and design a solution according to the client needs.
The customized solution benefited the client in the following ways:
- It allows them to interact with more than 300 nodes in a simple way.
- The manual tasks that took over 5 minutes per node were automated and now take seconds for each of the nodes.
- The existing infrastructure required no modifications, since Command Center was built respecting IT policy restrictions.
- Operators can now use custom inventories without using a CMDB.
- The scripts created by the administrators are now stored in and managed from a centralized location: A Script Repository.
- Administrators can now choose to manage via CLI or WebUI, both friendly and intuitive.
- Script Repository can reside in the delivered Docker solution or using any GIT repository available in the market providing maximum flexibility.
- Being a Docker-based solution, this is the only dependency for deployment.
- The solution requires minimum resources and supervision to operate and we were able to reduce appliance admin time from hours to seconds eliminating error prone tasks.
- The solution can be easily and inexpensively customized and upgraded, having been built with Open Source technologies.
TECHNOLOGY STACK
- Git
- Docker
- GNU/parallel
- Python
- Ssh
- Script Server
- Alpine Linux
