Case Studies


What is the secret of successfully building an ETL environment in the cloud in-house, even for a non-engineer?


YO-HO BREWING is a craft beer manufacturer headquartered in Karuizawa, Nagano Prefecture, Japan. The company's mission is to provide a new beer culture to the Japanese beer market with the goal of "Bringing flavor to beer and happiness to life!”. With the support of Cloud Ace, YO-HO BREWING built an in-house supply/demand management system for its products in the cloud about two years ago, even though the team was made up of non-engineers. In this interview, we asked them about how they built the ETL part, which supports the core of the data infrastructure, in-house using Cloud Build.

(from left)

Cloud Ace Co., Ltd.

Technical section, system development dpt.

Leader Kohei Yoshimura


YO-HO BREWING COMPANY

System Control Tower(Information System Unit)

Mr. Kishi, Takuro

Mr. Ogino, Naoto


Cloud Ace Co., Ltd.

Technical section, system development dpt.

Odaira Ren


Decision to bring the supply and demand management system in-house

Mr. Kishi:

At first, we were using our own supply-demand management system for product manufacturing, shipping, and inventory management, which was created using Google spreadsheets and other functions and Google Apps Script. The system had many layers of sheets because we kept adding on and adding on.

Then, as staff members were replaced, it became a complete black box, with no one understanding the mechanics behind the scenes and no one able to maintain it, and we were seeing that it was not scalable when new warehouse requirements came up.

Based on these factors, we started looking into outsourcing the system construction to solve these issues in the fall of 2019. However, due to the special requirements we were seeking, we found it difficult to proceed within our budget as we received estimates from several companies.

When we were at an impasse and wondering what to do, we came up with the idea that we could micro-systemize the system while keeping costs down by utilizing the spreadsheets and reworking the database part to migrate to the Google Cloud.

When we consulted with Cloud Ace, they assured us by saying,"Yes, we can do that! In fact, you would probably be more competitive in the marketplace if we managed the core areas in-house”. On the other hand, we had little experience in building our own systems, so we asked Cloud Ace to assist us in bringing them in-house.

The project was to be carried out by three non-engineers with no experience in system development, and we had to learn as we went along. I myself had only been assigned to the systems department for about a month, so I was worried about whether we could really succeed in this project, but we were able to achieve success with the support of Cloud Ace.

The simple idea that people who know the business can make it the easiest to use

Yoshimura:

The reason I said "Yes, it can be done!" was based on the simple idea that if a product is created by the people who use it, it will be easier to use.

It may be an ideal, but if we can proceed based on this simple idea, I believe we can create a structure and system that maximizes DX (Digital Transformation) and the developer experience.

In Cloud Booster, we provide consulting services that pursue the question of "How can we create something that is easy for users to use on their own?

We provided such support to YO-HO BREWING this time as well.

Through knowledge sharing and skill transfer, we helped them develop the ability to improve and add functions to the system on their own, and bring it to the ultimate form that is easiest to use.


Decision to migrate the ETL part, which supports the foundation of the data infrastructure, in-house to the cloud

Mr. Ogino

There were many things we wanted to work on other than ETL, but the ETL part, which supports the core of the data infrastructure, was highly important and urgent, and it was an essential function to deploy company-wide, so we proceeded with it as our top priority project.

Originally, we used a third-party ETL tool that could be executed on a local PC. Since the execution environment was dependent on the client PC, if that PC broke down, the most important part of the data would stop, so we wanted to create a stable operation by somehow migrating the execution environment to the cloud.

The team had a lot of discussions, but there was no certainty that the ideas we came up with could really be built, and we were worried. In parallel, as we investigated and demonstrated various tools, we also realized that they did not match our requirements, and we wondered if we could somehow build our own system. So we consulted with Cloud Ace, who had helped us build a supply and demand management system through in-house production, and showed them the overall system design around data, and asked them what kind of ETL system would best match our company's philosophy.

Mr. Kishi

The data we handle varies from order and sales data that can be obtained on on-premise servers, to data retrieved from SaaS, and data received from business partners via e-mail, etc. We were still in the process of experimenting with the data source requirements, which were expected to increase.

On the other hand, depending on the local environment, there may be problems with maintainability and availability. This meant that it was necessary to have a switchable micro system and a tool with the required conversion function.

In addition, since it was still under development and the amount of data to be handled was not large, we wanted to start with a quick and flexible mechanism.


Shifting policy from GKE to Cloud Build in order to further reduce costs

Mr. Kishi:

Originally, we were moving toward undertaking ETL jobs in the GKE container environment, but there were still technical hurdles for the non-engineer team. Although it was expensive and a great deal of knowledge was gained, we found it difficult to adopt it to solve the issues of ETL.

To be honest, I remember finishing the first day of the workshop with Cloud Ace, thinking "this time it's difficult, but it's an investment for the future".

So, while we felt the potential of GKE to develop, we were wondering if it was the best for this project.

Cloud Ace suggested that we use Cloud Build instead of GKE to run existing ETL tools on the cloud. We felt that this would be the best direction for us at that stage, so we did exactly that.

Odaira

When the project started, we were charged even if we weren’t using GKE. This led us to switch over to Cloud Build due to cost considerations.

Since we are a non-engineer team, we were able to utilize it in a way that reduced learning costs compared to GKE.

Take the first step!

Mr. Kishi

We were able to reduce costs, as mentioned above. We were also able to reconfigure the local-PC only ETL jobs to operate on the cloud environment, so maintainability and availability also became better. We were freed from the risk of being overwhelmed by recovery work if our device breaks down.

We were also able to realize that we could operate the system even without a high level of IT skill.

On the other hand, as the requirements for ETL increase in the future, if it takes resources to create execution jobs or if there are technical hurdles, we are enthusiastic about investing in IT tools. We hope to make the right choices as our company grows.

In this sense as well, we think it was the right choice to just try it out after working to accurately understand our issues.

Since we were able to overcome the issue of implementing data into this project, we would like to use the data in the future so that it can be used for management decisions.

Have an improvement cycle for internalized systems

Mr. Kishi:

While building a supply and demand system together with Cloud Ace while using PDCA, we found many things we wanted to try, like expanding the system scale and making improvements. This led to growth in the team.

Even for employees who have never been involved in engineering, this project has created a place to take on challenges and produce results. I think it is a big growth from the perspective of the future.

Through our knowledge of programming and SQL, we were able to acquire the necessary skills for solving problems in the field, and it broadened our horizons. I feel that being able to use our abilities not only in system-related matters but also in data utilization and analysis is a great advantage for the future growth of the company.


Mr. Ogino

Since we are not an IT company, I was initially worried about incorporating new technologies and mechanisms into our business. However, with the support of Cloud Ace, we were able to support places that we could not choose alone, and by being able to accumulate successful experiences, we were able to broaden the range of choices as a team.

I've used AWS before, but since it was my first time using Google Cloud, having hands-on learning during the workshop and actually touching it was helpful because I was able to put it into practice immediately.


Yoshimura

New products and solutions are being created every day, so I think making a system on the cloud should be done by repeating the cycle of catching up on information, constantly searching for better ways, and optimizing the system.

This time, we conducted four workshops in a short period of one month, from understanding the overall mechanism to selecting and implementing tools. This is the "SI 2.0" that Cloud Ace is aiming for, and I am sure that Yo-Ho Brewing will continue to go through this cycle and grow significantly in the future.

For the future

Mr. Kishi:

We were finally able to take data that had been silo’d amongst departments and individuals and put them into BigQuery. Until now, KPIs have looked at the data that each department has, but in the future, we will have KPIs that connect to the activities of their own department from indicators based on company-wide strategies, and a platform that can multiply various data. I would like use this to create new business ideas and solve problems.


Mr. Ogino

Now that we have finally created a data base, we would like to contribute more to the business by incorporating various databases.

We would like to create new business seeds and grow existing services so that more people can enjoy beer.

We are also looking for new friends!

We would like to seek people with IT skills, but we are also looking for people who can understand the on-site needs of the company from a business perspective and discuss with the staff.

There is a culture of flat discussions regardless of position or role, so we want a team where the IT team and the field team can jointly commit to the same issue and achieve results.

In particular, the IT team only looks at the data engineering area, and the field team only looks at the business area, but I want to aim for activities that allow each other to work seamlessly with the aim of achieving results. I think that is the real thrill of a business company like ours.

Although low-code development and democratization of IT have progressed, I still feel a barrier in the technical field, so I would like to make good use of the knowledge and resources of other companies like this time to improve the internal business.

Services used in this case study

Cloud booster


* What is ETL tool?

ETL tools are a category of IT products that extracts and collects digital data scattered inside and outside an organization, transforms and processes it according to need, and loads the information where it can be used.