IACC migrates its Apache Kafka integration platform to the AWS cloud

IACC Professional Institute and their needs

IACC Professional Institute was born as an educational project in 1981 under the figure of the IACC Technical Training Center (CFT). Around 1987, it obtained official recognition from the Chilean Ministry of Education, becoming the IACC Professional Institute. As of 2008, IACC incorporates in its academic offer careers in a completely online modality for the general public, becoming the first and only Professional Institute in Chile 100% online. By 2020, the IACC Professional Institute has more than 16,000 students and more than 5,000 graduates.

To support its huge academic community, IACC has several on-premise information systems, which must communicate with each other in an integrated and reliable way. Seeking to redesign the integration of their systems, the IACC IT team conducted a proof of concept with a event-based architecture using the Apache Kafka platform, which was quite successful. Within this context, IACC turned to TruStep to migrate the on-premise environment of its Kafka proof of concept to the AWS cloud, into a robust, scalable, and highly available productive architecture.

Solution Design

The designed solution combined the new AWS Managed Streaming for Apache Kafka (MSK) service with the already traditional AWS Elastic Computing Cloud (EC2), with the support of AWS CodeCommit, AWS CodeDeploy and AWS CodePipeline services to process the versions of the applications between Development, QA and Production.

The following diagram shows the architecture adopted:

It is observed in the figure that the QA and Production environment are segregated and practically identical. Both are connected by a single flow within the AWS CodePipeline, which first deploys in QA and after an approval by the QA team, deploys the same version in Production, ensuring consistency between environments. Each environment communicates with its counterpart in the corporate data center to establish the data flow with the on-premise systems through a private VPN between AWS and the corporate datacenter.

The solution designed by TruStep deploys the Kafka Connect cluster in distributed mode through an AutoScaling Group, taking advantage of its scalability and fault tolerance. Every time a new server is created, all the necessary software components are automatically installed on that server:

• the Apache Kafka software package along with its dependencies;
• the AWS command line clients (AWS CLI);
• the CodeDeploy agent.

Next, the CodeDeploy agent retrieves the latest version of the Kafka connectors installed successfully and deploys them. As part of that automatic process, a series of Kafka Connect cluster configuration tasks are run:

• Identify and retrieve the connection strings to communicate with the Kafka cluster brokers (MSK);
• Retrieve from AWS Secrets Manager the configuration of each connector;
• Update the configuration of each connector through the Kafka Connect REST API;
• Create the needed topics within the Kafka cluster;
• Adjust the properties files of the new Kafka Connect node;
• Shutdown and starts up the Kafka Connect cluster as well as suspend and resume execution of the connectors.

In the same way, when developers create a new version of the integration software, CodePipeline takes care of the necessary steps to generate and deploy it on as many servers as necessary, in QA and in Production environments, without manual intervention from Development or Operation teams.

The Achieved Benefits

This automation process means that the IACC Operations team does not have to worry about the details of how to scale the platform: just add or remove servers from the AutoScaling Group and everything else happens automatically.

In addition, the time freed by those automations allows the team to dedicate themselves to other more important tasks for the IACC business and at the same time reduces time needed to release new versions in QA and Production, all of this within a robust, scalable and high-quality architecture that AWS delivers to its customers.

With the successful migration of the Apache Kafka integration platform running on AWS, the IACC IT team can now focus their efforts on growing the number of integrations between their systems with a robust event-driven architecture, and at same time offering management real time data and metrics to support their decision making.

Talk with TruStep

If you want to discuss your use case with us, please contact us through the form available at this link.