How IACC moved its Kafka environment to AWS with greater scalability and high availability.
IACC on AWS: Greater Scalability and High Availability with Apache Kafka
IACC is a Chilean educational institution with more than 16,000 students enrolled in its online programs. To support its growth, the institution adopted an event-driven architecture based on Apache Kafka, but required a more scalable and highly available environment.
To achieve this, it selected TruStep to migrate its on-premises Kafka environment to Amazon MSK (Managed Streaming for Apache Kafka). Here's how this migration was designed and executed.
IACC Professional Institute and Its Need
IACC Professional Institute is a higher education institution founded in 1981 in Chile. In 2008, it became a pioneer by offering 100% online education, establishing itself as a national reference.
With thousands of students and multiple integrated systems, the need emerged for a more modern, reliable, and scalable architecture.
The IT team validated an event-driven architecture using Apache Kafka and then engaged TruStep to evolve this proof of concept into a production-ready environment on AWS.
Solution Design
The solution combined Amazon MSK with AWS services such as EC2, CodeCommit, CodeDeploy, and CodePipeline, enabling full automation of the application lifecycle.
The QA and Production environments are segregated while remaining consistent with each other, with automated and controlled deployments through a pipeline.
Communication with legacy systems takes place through a secure VPN connection between AWS and the data center.
The Kafka Connect cluster was implemented in distributed mode with Auto Scaling, ensuring high availability and elasticity.
Environment Automation
Each newly provisioned server automatically receives:
- Apache Kafka and its dependencies;
- AWS CLI;
- AWS CodeDeploy Agent.
In addition, the process automates:
- Configuration of the connection to the MSK cluster;
- Retrieval of secrets through AWS Secrets Manager;
- Connector configuration through the REST API;
- Creation of Kafka topics;
- Kafka Connect cluster configuration updates and restarts.
New connector versions are automatically built and distributed through CodePipeline, with no manual intervention required.
Benefits Achieved
- Automatic scalability: dynamic infrastructure adjustment;
- High availability: resilient architecture on AWS;
- Complete automation: reduced operational effort;
- Agility: faster and more consistent deployments;
- Business focus: teams are free to innovate.
With the new architecture, IACC was able to expand integrations between systems and leverage real-time data for decision-making.
Learn more
Would you like to apply this architecture to your business?
Schedule an initial conversation with TruStep. We can review your current scenario, understand existing dependencies, and discuss paths for a gradual modernization, while keeping operations running during the transition.
