February 24 began for everyone with one piece of news. For the cloud operator GigaCloud, this meant one more thing – the need to ensure the operation of client services and help reserve data for those who have not yet done so.
GigaCloud is the primary or backup site for thousands of services. Among them: Diya, ProZorro, large banks, industries, medical services and much more. For the guys from technical support, the terms “critical infrastructure” and “critical service” are commonplace. And for users – non-working bank cards, pharmacies and shops, if a critical service “falls”.
Dev.ua spoke with GigaCloud technical support specialists Andrey and Alexander, as well as with the head of the department, Sergey, about how they worked during the first attacks and sirens, and what has changed in their work over the past three months.
“If the support stops, it will be bad for everyone”
“The first day was total chaos,” recalls Sergey, GigaCloud’s head of technical support, of the day the Russian troops invaded Ukraine. – On the one hand, there was a slight panic, a lack of understanding of the situation and where all this would lead. On the other hand, several hundred calls to support. GigaCloud is a mission critical service company and we needed to keep going. If the support stops, it will be bad for everyone.”
“I soberly assessed the situation and decided to work in my apartment in Vishnevoe without going down to the bomb shelter,” Sergey continues. – I thought it was better to fall from the fourth floor than to get out from under the rubble. The first thing I did was contact my team.”
How GigaCloud technical support works
The support team of the cloud operator consists of 13 people: technical support specialists, specialized engineers, virtualization systems and hardware specialists. They process customer requests that come to the ticket system.
All requests for work converge there – allocate technical resources, issue licenses, help with reservations, and much more. All this is sorted and distributed among employees. According to Sergey, in three days, from February 24 to February 26, GigaCloud received 300 applications. So much usually comes to support in two weeks.
All the founders, management and most of the employees of the cloud operator met the beginning of the war in Kyiv. Some people were already in Lviv, because the company was building a new cluster there. Some of the employees could not be evacuated from Kyiv, because they ensure the operation of critical infrastructure in data centers. The management of the cloud operator provided them with everything they needed, and the rest offered help with the evacuation.
Engineers from the GigaCenter data center and GigaCloud cloud operator stayed in Kyiv to maintain critical infrastructure on site. In the photo: a makeshift kitchen in the data center
“In the morning, I asked Sergei in a telegram whether to go to the office,” says Alexander, “and in response I saw: “Are you not aware of the news? The war has begun.” At that time I was so frivolous about this that even this phrase did not shock me. Thought it was another panic. Understanding began to come when I heard the sounds of multiple rocket launchers and a siren. I looked out the window and saw that people were running out of houses with things.”
Alexander and his family live in Obolon. After the first explosions, he took his wife and children to the subway.
“We hurriedly gathered everything we could take with us and ran to the subway. At the same time, I noticed that we were massively flooded with customer requests. It was impossible to get into the subway – thousands of people. I left my family there and returned to the apartment to respond to customer requests. Some I decided right away, the rest I redirected to our certified specialists in virtualization systems.”
Basically, clients requested the transfer of VMware infrastructure from a cloud in Kyiv to a cloud in Lviv or Poland. At the end of the migration, the clients redirected their users and records to the new IP addresses. It sounds easy, but when there are several dozens of such clients at the same time, and hundreds of virtual machines, all this needs to be coordinated, negotiated and controlled. And all this against the background of the general panic of the first day of the war.
According to Alexander, the worst thing he experienced that day was the news that all entrances and exits to the subway were blocked. Then in social networks there was information about the battles on the dam of the Kievvodokanal, which is located near Vyshgorod.
“If the dam is destroyed, then Obolon will be completely flooded. For me it would be a real disaster. After all, it turns out that I left my family in a trap, while I myself continue to work. And when the first door to the subway was opened a few hours later, I immediately took my wife and children out of there.”
A week later, Alexander managed to take his loved ones abroad, and now they are safe. Alexander himself remained in Kyiv. To continue working from home, he had to run between two apartments.
“In the fifth microdistrict of Obolon, it was so loud both day and night that sometimes the nerves could not stand it,” recalls Alexander. – When something began to buzz and explode, I went to another apartment. My work laptop was always with me, in my backpack. In the second apartment, I could only work with mobile Internet. Came there, closed applications and returned back. It was in that apartment that I witnessed a direct hit on a neighboring residential 9-storey building at 20, Bogatyrskaya.
A rocket hit a residential building on Bogatyrskaya Street. Photo by the State Emergency Service of Ukraine“I lived for two weeks without communication, light and all the benefits of civilization”
If before the war it was possible to determine which day had the most calls, then for the first three weeks of the war every day was a peak day, whether it was a weekday or a weekend. It was the most difficult period since the beginning of the war. Often people had to be replaced, because the support staff could not work during the evacuation or active hostilities. As, for example, Andrey, lead technical support engineer.
“On the first day of the war, I did not leave Kyiv and processed customer requests until late at night,” says Andrei. – The next day I read in the news that DRGs were breaking through to Obolon. Therefore, I made, perhaps, one of the worst decisions in my life – I left Kyiv in the direction of Makarov and ended up in the cauldron of hostilities.
Andrei could only work for the first two days. Then a Russian plane was shot down over Buzovaya, it fell on power lines, and electricity went out. The Internet was a little longer, but then it was also interrupted.
“I lived for two weeks without communication, light and all the benefits of civilization,” says Andrey. – In the middle of March, we were able to return to Kyiv. Terodefense people suggested a safe way through the villages.”
“Before the war, we made mass calls, tried to help clients”
In the early days of a full-scale invasion of Russia, there could have been more client requests if GigaCloud had not been preparing for a possible war. Although no one believed that it would really begin. At the beginning of the year, the cloud operator began implementing its Business Continuity Plan (sustainability plan).
At the end of January, GigaCloud launched a technical platform in Lviv and offered customers to transfer their services there for free and place backups.
“About a month before the start of the war, our team was given the task of informing customers that it was possible to transfer infrastructure and backup copies of critical data to sites in Lvov and Poland,” Alexander recalls. — We made mass calls, tried to help clients. While talking to them, I heard a lot of skeptical comments.”
The preparation allowed the technical support team to do a huge amount of work even before the start of the war. And starting on February 24, customers who had previously been in thought began to en masse ask for help to transfer critical data to other servers. Then the main difficulties in processing applications began.
Assembled, packaged, transported – about customer migration
GigaCloud assisted private cloud clients with equipment evacuation. A private cloud is a virtual data center that is provided to the client for exclusive use, so it cannot simply be “transferred” over communication channels. Some of the clients were asked to evacuate the equipment to Lviv, and the cloud operator performed this work for free. In the conditions of active hostilities, it was a whole special operation: our employees independently assembled, packed, transported to a safe place, installed and configured part of the client’s cluster. This is how a separate service appeared in GigaCloud – remote-help.
“GigaCloud has several levels of support,” says Alexander. – If before the war we were divided and worked only on our tasks, then during the period of active hostilities we insured each other. Engineers helped technical support while someone moved and settled in a new place. And we helped the engineers move the equipment.”
Clients continue to actively migrate to GigaCloud sites in Lviv and Poland, but the peak has already subsided. “Now the situation has more or less stabilized, there are fewer applications on weekends, it’s clear that people are trying to relax,” says Sergey, head of technical support. But that doesn’t mean there’s less work to do. We didn’t have any cuts. We cannot do this, because the time for processing client requests will increase. This is now not only inappropriate, but unacceptable. Businesses have just started to launch, and I understand that we need to continue to support our customers.”
20% of customers do not make backups
The main tasks of the support remain unchanged – transferring clients to the cloud and backing up. The business continues to secure its infrastructure. But not everyone does this. “20% of customers who ordered Veeam licenses do not use them. That is, there are licenses, but they do not make backups. This can be a fatal mistake for a business in a critical situation,” Sergey said.
“We are all on our front”
“I often come across articles with headings: “What to do if you feel guilty that you don’t defend the country in the ranks of the Armed Forces of Ukraine, or didn’t join the Troops,” Sergei says. – I think that now everyone is working for the country: the Armed Forces of Ukraine protect it from enemies, and the rest support the economy, pay taxes, do everything to make it live. After all, if there is no country, then there will be nothing to protect. We are all in our place and on our front.”