Beyond the Interview: How Brokee Helps Engineers Showcase Their True Value
By George Goodwin
Tired of Interviews That Don’t Show What You Can Really Do?
Let’s face it—technical interviews can be awkward. Whether you’re answering textbook questions, solving algorithm puzzles that never show up on the job, or fumbling through take-home tests with no feedback… it’s not always a fair shot.
That’s where Brokee comes in.
Brokee is flipping the script on traditional hiring with live, broken environments that let engineers show what they actually know. Instead of talking about their skills, engineers get to prove their value by solving real-world issues—just like they would on the job.
Let’s break down how Brokee is helping engineers shine beyond the interview.
What Is Brokee?
Brokee is a platform built by engineers—for engineers.
It was founded by Maksym Lushpenko, a frustrated Cloud Engineer who was tired of the same broken hiring process. He created Brokee to help hiring teams evaluate real skills, not just resumes or rehearsed answers.
The Concept Is Simple And Brilliant
Give engineers a broken system
Watch how they fix it
Hire the ones who can
Every challenge is designed by senior System Engineers who know what it’s like to be on-call at 2 AM trying to fix a crashing website. These tests reflect the real-world chaos engineers deal with every day.
“When a website goes down, theory won’t save it. Practical skills will.”— Brokee Team
How Brokee Helps Engineers Show Their True Value
1. No More Fluff—Just Skills
Brokee doesn’t care where you went to school or how polished your GitHub is. What matters is whether you can solve real infrastructure problems in real time.
2. Live, Broken Environments = Real-World Pressure
These aren’t simulated quizzes or pre-recorded demos. Engineers are dropped into actual broken systems and expected to troubleshoot their way out—just like in a real job.
3. Evaluations Built by Real Engineers
All Brokee assessments are created and tested by experienced engineers who’ve been in the trenches. That means every task reflects actual DevOps work.
4. No More Take-Home Tests That Waste Time
Brokee challenges are built for efficiency. Candidates can complete a practical evaluation that gives hiring teams real insight—without wasting hours on assignments that go nowhere.
5. An Empowering Experience for Candidates
For engineers, Brokee is more than a test—it’s a platform to showcase your value. You walk away knowing you were evaluated fairly, and that your skills spoke louder than a resume ever could.
Brokee’s Bigger Purpose
Not only is Brokee revolutionizing how engineers get hired—it’s also making a difference in the world. As a Ukrainian-led company, Brokee actively supports Ukraine’s tech community with education, donations, and platform access.
“We don’t just build a better way to hire—we stand for something bigger.” — Brokee Team
✨Key Takeaways
Brokee replaces outdated technical interviews with real, broken environments
Engineers are evaluated based on practical, job-ready skills
Hiring managers get faster, clearer insights into candidate capabilities
The platform empowers engineers to prove their value—beyond the interview
FAQs about Beyond the Interview: How Brokee Helps Engineers Showcase Their True Value
What is Brokee?
Brokee is a technical assessment platform that tests engineers using live, broken systems to simulate real-world scenarios.
How does Brokee help engineers?
It gives engineers a platform to showcase their hands-on skills and problem-solving ability in realistic environments—offering a better alternative to traditional interviews.
What makes Brokee better than take-home tests?
Take-home tests often don’t reflect real work and can waste time. Brokee challenges are efficient, job-focused, and designed by actual engineers.
Is Brokee only for DevOps roles?
Currently, Brokee is focused on DevOps and System Engineering roles, but its core concept can be applied to many technical areas.
Who founded Brokee?
Brokee was founded by Maksym Lushpenko, a Cloud Engineer who wanted to fix the broken hiring process.
Ready to Go Beyond the Interview?
Whether you're an engineer tired of being misjudged or a hiring manager looking for top talent, Brokee offers a smarter, more practical way to assess technical skills.
👉 Visit Brokee’s website to learn more.
👉 Get in touch with Brokee if you want high-impact content like this for your brand.
How Brokee Helps Startups & Enterprises Scale Their DevOps Teams Fast
By George Goodwin
Introduction: The DevOps Hiring Challenge
Hiring DevOps engineers is one of the biggest challenges for startups and enterprises alike. Finding the right talent with real-world problem-solving skills can be time-consuming, expensive, and frustrating. Many companies rely on traditional hiring methods like resumes, coding tests, or take-home assignments, but these approaches don’t always reveal how well a candidate can handle real system failures under pressure.
That’s where Brokee comes in. Designed to help businesses scale their DevOps teams faster and smarter, Brokee’s hands-on, real-world technical assessments take the guesswork out of hiring.
The Problem with Traditional DevOps Hiring
Why Startups Struggle to Hire DevOps Engineers
Startups often face tight deadlines and limited budgets, making it essential to hire the right DevOps engineers quickly. However, traditional hiring processes slow things down because:
Resumes don’t prove real-world skills – Just because someone has experience on paper doesn’t mean they can troubleshoot live issues.
Take-home tests are inefficient – Candidates can get outside help, and reviewing submissions takes valuable time.
Interviews don’t reflect actual job challenges – Answering theoretical questions doesn’t show how someone would respond in an actual system failure.
Why Enterprises Need a Better Hiring Process
Larger businesses may have established hiring pipelines, but they still face challenges:
High volume of applicants – Filtering out unqualified candidates can be overwhelming.
Costly hiring mistakes – A bad DevOps hire can slow down workflows, cause downtime, and hurt productivity.
Inconsistent skill assessment – Different interviewers may evaluate candidates differently, leading to inconsistent hiring decisions.
How Brokee Helps Businesses Scale DevOps Teams Fast
Brokee provides a faster, more effective way to hire DevOps engineers by using real-world testing environments that accurately measure a candidate’s ability to diagnose and fix live system issues.
1. Hands-On Testing with Live, Broken Environments
Unlike traditional hiring methods, Brokee places candidates in actual DevOps scenarios where they must troubleshoot live failures—just like they would in a real job.
✅ See real problem-solving skills in action
✅ Test practical knowledge, not just theory
✅ Ensure candidates can handle high-pressure situations
2. Faster Hiring with Automated Evaluations
Brokee’s platform automatically evaluates how candidates perform in real-world scenarios, helping businesses:
✅ Filter out unqualified candidates quickly
✅ Reduce interview time by up to 50%
✅ Hire the right engineers 3X faster
3. Scalable Solution for Startups & Enterprises
Whether you’re hiring your first DevOps engineer or scaling a global team, Brokee adapts to your needs:
For startups – Quickly identify top talent without wasting time on weak candidates.
For enterprises – Standardize hiring across multiple teams with consistent skill evaluations.
4. Built by Engineers, for Engineers
Brokee was created by Maksym Lushpenko, a frustrated Cloud Engineer who saw firsthand how broken traditional hiring methods were. That’s why Brokee’s tests are designed by senior system engineers, ensuring they truly reflect real-world job demands.
Why Businesses Trust Brokee
🚀 3X faster hiring – No more weeks of back-and-forth interviews.
💰 Save on hiring costs – Reduce bad hires that cost businesses thousands.
🔧 Accurate skill assessment – Know if a candidate can do the job before hiring them.
Key Takeaways
Traditional DevOps hiring methods, such as resumes and theoretical interviews, often fail to assess real-world skills.
Brokee provides hands-on, real-world testing to evaluate candidates’ problem-solving abilities in live environments.
The platform helps businesses hire 3X faster, reducing hiring costs and improving efficiency.
Automated evaluations ensure that only the most qualified candidates move forward in the hiring process.
Startups and enterprises alike benefit from Brokee’s scalable hiring solution that adapts to different team sizes and needs.
Created by experienced engineers, Brokee ensures accurate skill assessment, leading to better hiring decisions and stronger DevOps teams.
FAQs about How Brokee Helps Startups & Enterprises Scale Their DevOps Teams Fast
1. How does Brokee’s hiring process work?
Brokee creates real-world, broken environments where candidates must diagnose and fix DevOps issues. Their performance is evaluated automatically, helping companies hire faster and with more confidence.
2. How does Brokee save time for hiring managers?
By automating the technical evaluation process, Brokee filters out unqualified candidates before the interview stage, reducing time spent on weak applicants.
3. Can Brokee be used for remote hiring?
Yes! Brokee’s platform is 100% remote, making it perfect for companies hiring globally.
4. What types of DevOps skills does Brokee test for?
Brokee evaluates a range of skills, including system troubleshooting, cloud management, automation, CI/CD, and security best practices.
5. How does Brokee help startups hire their first DevOps engineer?
Startups often lack the technical expertise to assess candidates properly. Brokee provides an objective, skill-based assessment, ensuring they hire the right talent without needing an in-house DevOps expert.
FAQs About Brokee as a Business
1. What is Brokee?
Brokee is a technical assessment platform designed to help companies hire DevOps engineers faster by testing candidates in real-world problem-solving scenarios.
2. Who founded Brokee?
Brokee was founded by Maksym Lushpenko, a Cloud Engineer who was frustrated with traditional hiring processes.
3. What makes Brokee different from other assessment platforms?
Unlike other platforms that use multiple-choice or take-home tests, Brokee simulates real DevOps challenges, allowing companies to see how candidates perform in actual job situations.
4. What industries can use Brokee?
Any business looking to hire DevOps engineers can benefit from Brokee, including startups, enterprises, and tech companies across various industries.
5. How can I get started with Brokee?
Visit Brokee’s website to learn more and start hiring smarter today!
Conclusion: Scale Your DevOps Team the Smart Way
Hiring the right DevOps engineers shouldn’t be a gamble. With Brokee’s real-world assessments, you can build a strong, capable DevOps team quickly and efficiently—without wasting time on bad hires.
🚀 Want to hire better engineers, faster? Contact Brokee today!
Why Practical Experience Matters More Than Certifications in DevOps Hiring
Introduction
The debate between experience and certifications in DevOps hiring is ongoing. While certifications prove theoretical knowledge, hands-on experience often carries more weight in real-world problem-solving. This blog explores why practical experience is more valuable than certifications when hiring DevOps professionals and how it impacts hiring decisions.
Perception and Certifications: The "Customer's" View
Certifications provide credibility, reassuring customers and stakeholders that a DevOps engineer has met industry standards. However, they don’t always guarantee the ability to handle real-world complexities. Many organizations prioritize certified professionals for compliance reasons, but in practice, experience is what truly drives efficiency and problem-solving.
Employers must balance external perceptions with actual job performance. While certifications can be impressive on a resume, they should not be the primary factor in hiring decisions.
DevOps - Degree or Certification?
A degree provides a broad foundation, while certifications focus on specific tools like AWS, Kubernetes, and Terraform. However, the real test of a DevOps engineer is their ability to automate workflows, troubleshoot systems, and optimize infrastructure. Employers value those who can apply their knowledge rather than just pass an exam.
Instead of relying solely on a degree or certification, DevOps professionals should build a portfolio of hands-on projects showcasing their problem-solving skills.
Why I Don’t Consider Your Certifications During an Interview
Many hiring managers, including myself, don’t weigh certifications heavily during interviews. Why?
Certifications don’t always reflect real-world experience.
DevOps is constantly evolving; adaptability is more important than memorized knowledge.
Practical problem-solving skills and the ability to work in high-pressure environments are key.
Instead, I focus on candidates’ hands-on skills, their experience with troubleshooting, and their approach to solving technical challenges. A real-world example of a past project, debugging an infrastructure issue, or automating a workflow tells me far more about a candidate’s ability than a certificate ever could.
Skill and Experience Vs. Certifications
While certifications validate foundational knowledge, they rarely demonstrate proficiency. Real-world experience helps candidates:
Navigate unpredictable system failures.
Develop effective troubleshooting techniques.
Work efficiently in teams under pressure.
Adapt to new technologies on the fly.
Certifications might get candidates an interview, but experience secures the job. A candidate who has built and maintained a CI/CD pipeline in production has a significant advantage over one who has just studied DevOps concepts.
Creativity, Problem-solving, and Critical Thinking
DevOps isn’t just about following best practices—it’s about solving unique problems creatively. Experience fosters:
The ability to think outside the box.
Decision-making in high-pressure situations.
Critical thinking that certifications cannot measure.
A truly skilled DevOps engineer combines technical knowledge with problem-solving and adaptability. Hiring managers look for professionals who can handle unexpected issues and devise innovative solutions rather than those who only know theoretical best practices.
Key Takeaways
Certifications prove theoretical knowledge but don’t guarantee practical skills.
Hands-on experience is essential for troubleshooting and problem-solving in real-world DevOps environments.
Adaptability and continuous learning are more valuable than holding multiple certifications.
Hiring managers prioritize experience over certifications in interviews.
Creativity, critical thinking, and teamwork are crucial for success in DevOps.
A strong DevOps portfolio with real-world projects carries more weight than a certification.
FAQs About Why Practical Experience Matters More Than Certifications in DevOps Hiring
Is it possible to get a DevOps job without certifications?
Yes, many DevOps professionals are hired based on their hands-on experience rather than certifications. Employers value practical problem-solving skills over theoretical knowledge.
Are certifications completely useless in DevOps?
No, certifications can help demonstrate foundational knowledge and commitment to learning, but they should be supplemented with real-world experience. A certification should enhance experience, not replace it.
How can I gain practical DevOps experience?
Working on personal projects, contributing to open-source, and setting up cloud-based environments are great ways to gain hands-on experience. Internships, freelance work, and home lab setups are also effective methods.
What does a hiring manager look for in a DevOps candidate?
Problem-solving skills, real-world troubleshooting experience, adaptability, and teamwork matter more than just certifications. Demonstrating hands-on expertise in automation, cloud infrastructure, and CI/CD pipelines will make a candidate stand out.
FAQs About Innovar Marketing Agency
What services does Innovar Marketing Agency provide?
We specialize in B2B client acquisition, lead generation, and marketing strategy to help businesses grow.
How does Innovar Marketing Agency help businesses?
We use data-driven marketing strategies, automation tools, and industry expertise to generate high-quality leads and drive revenue growth.
How can I contact Innovar Marketing Agency?
Visit our website or reach out via email to learn more about how we can help your business scale.
FAQs About Brokee
What is Brokee?
Brokee is a platform designed to help businesses optimize their operations through automation and data-driven insights.
How does Brokee benefit businesses?
Brokee helps businesses streamline processes, improve efficiency, and make data-backed decisions for better performance and profitability.
Who can use Brokee?
Brokee is ideal for businesses of all sizes looking to enhance their operational workflows and improve overall productivity.
How can I get started with Brokee?
You can visit our website to learn more, sign up for a demo, or contact our team for customized solutions.
Conclusion
While certifications have their place in DevOps, they should never be a substitute for hands-on experience. Practical skills, adaptability, and problem-solving are what truly make a DevOps engineer stand out. If you’re looking to build a strong DevOps career, focus on real-world learning, continuous skill development, and building a portfolio that demonstrates your expertise.
How Brokee is Disrupting the Hiring Process for DevOps & Cloud Engineers
Introduction
Hiring DevOps and cloud engineers has become a major challenge for businesses. The demand for skilled professionals is skyrocketing, but finding the right fit is easier said than done. Traditional hiring processes are slow, expensive, and often fail to deliver the best candidates.
That’s where Brokee comes in. Brokee is shaking up the hiring industry by offering a smarter, more efficient way to connect companies with top DevOps and cloud engineers. Let’s dive into how Brokee is changing the game.
Challenges in Hiring DevOps & Cloud Engineers
1. High Demand and Talent Shortage
The DevOps field is growing fast, and companies are struggling to find experienced engineers. The competition for top talent is fierce, making it hard for businesses to attract and retain the best candidates.
2. Lengthy Hiring Processes
Traditional hiring can take months. Screening resumes, conducting multiple interviews, and negotiating offers slow down the process. By the time a company decides, the best candidates have already taken other jobs.
3. High Costs
Recruiting agencies and job boards charge high fees without guaranteeing results. Small and mid-sized companies often struggle to afford these costs while competing with tech giants for talent.
4. Skills Gap
Many engineers have certifications but lack real-world experience. Hiring managers find it difficult to assess practical DevOps skills, leading to mismatches and poor hiring decisions.
5. Retention Issues
Even after hiring, companies face challenges in keeping their DevOps engineers engaged. High turnover rates mean businesses must constantly restart the hiring cycle.
How Brokee is Changing the Hiring Process
Brokee is transforming how companies find and hire DevOps and cloud engineers. Here’s how:
1. AI-Powered Matching System
Brokee uses advanced AI to match companies with the best candidates based on skills, experience, and culture fit. This eliminates hours of resume screening and speeds up the hiring process.
2. Pre-Vetted Talent Pool
Unlike traditional job boards, Brokee curates a pool of pre-vetted DevOps and cloud engineers. Every candidate is tested on real-world tasks, ensuring they have hands-on experience, not just certifications.
3. Faster Hiring Time
With Brokee, businesses can connect with top engineers in days instead of months. The platform streamlines the hiring process, so companies don’t lose out on top talent.
4. Cost-Effective Hiring
Brokee removes the need for expensive recruiting agencies. Companies pay only for successful hires, making the process budget-friendly.
5. Flexible Hiring Options
Need a full-time DevOps engineer? Looking for a contract-based cloud expert? Brokee offers flexible hiring solutions to match different business needs.
6. Focus on Retention
Brokee doesn’t just help businesses hire engineers—it also ensures they stay. The platform offers career growth resources and mentorship programs to keep DevOps professionals engaged and satisfied.
Dos and Don’ts When Hiring Your First DevOps Engineer
Dos:
✅ Clearly define the skills and experience needed.
✅ Use platforms like Brokee to access pre-vetted candidates.
✅ Test real-world DevOps skills before hiring.
✅ Offer career growth opportunities to retain top talent.
Don’ts:
❌ Don’t rely only on resumes—test candidates with practical tasks.
❌ Don’t rush into hiring without checking culture fit.
❌ Don’t overlook retention strategies—DevOps engineers need career growth.
What You Need to Know About Hiring DevOps Engineers in 2025
The DevOps hiring landscape is constantly evolving. Here’s what businesses should expect in 2025:
Remote Work is Here to Stay – More DevOps engineers prefer remote or hybrid roles.
AI in Hiring – AI-powered hiring tools like Brokee will become the standard.
Emphasis on Automation – Companies will seek DevOps engineers skilled in CI/CD and infrastructure automation.
Increased Focus on Cloud Skills – Cloud expertise will be a must-have for DevOps professionals.
FAQs About How Brokee is Disrupting the Hiring Process for DevOps & Cloud Engineers
1. What makes Brokee different from traditional hiring platforms?
Brokee pre-vets candidates, uses AI-powered matching, and speeds up the hiring process, making it faster and more reliable than traditional job boards.
2. How does Brokee ensure candidates have real-world experience?
Brokee tests engineers on practical DevOps tasks to ensure they have hands-on experience, not just certifications.
3. Can small businesses afford Brokee?
Yes! Brokee is cost-effective and offers pay-for-success pricing, making it accessible for startups and small businesses.
4. Does Brokee help with retention?
Yes, Brokee provides career growth resources and mentorship programs to keep engineers engaged.
5. Can I hire remote DevOps engineers through Brokee?
Absolutely! Brokee connects companies with top remote and on-site DevOps engineers worldwide.
Conclusion
Brokee is revolutionizing the way companies hire DevOps and cloud engineers. By leveraging AI, pre-vetted talent, and a streamlined hiring process, Brokee makes it faster, more cost-effective, and more reliable to find top engineering talent.
If you’re struggling with DevOps hiring, it’s time to try a better way. Check out Brokee and start hiring smarter today!
A Recruiter’s Guide to Evaluating DevOps Candidates Like a Pro
George Goodwin
DevOps is one of the most in-demand skills in the tech industry today. As companies strive for faster software delivery, improved scalability, and enhanced operational efficiency, the need for skilled DevOps engineers is skyrocketing. However, finding and evaluating the right DevOps candidates can be challenging for recruiters.
This guide will walk you through everything you need to know to assess DevOps professionals effectively. We’ll cover the core tools, essential skills, hiring best practices, and market trends so you can confidently identify top talent.
What is DevOps?
DevOps is a combination of software development (Dev) and IT operations (Ops) that focuses on automation, collaboration, and continuous improvement. The goal is to streamline the software development lifecycle, ensuring rapid and reliable releases.
Key Concepts in DevOps
CI/CD (Continuous Integration/Continuous Deployment): Automates code integration and deployment, ensuring frequent and stable software updates.
Infrastructure as Code (IaC): Uses coding tools like Terraform and Ansible to manage infrastructure instead of manual configurations.
Monitoring and Logging: Tools like Prometheus and Grafana help track system performance and resolve issues proactively.
Automation: Reduces manual tasks using tools like Jenkins and GitLab CI.
Collaboration: Encourages seamless communication between developers, operations teams, and security experts.
Core DevOps Tools and Technologies
Understanding the tools a candidate works with is essential for assessing their expertise. Here are some of the most common DevOps technologies:
Version Control Systems: Git, GitHub, GitLab
CI/CD Tools: Jenkins, CircleCI, Travis CI, GitLab CI
Configuration Management: Ansible, Chef, Puppet
Containerization & Orchestration: Docker, Kubernetes
Cloud Platforms: AWS, Google Cloud, Azure
Infrastructure as Code: Terraform, CloudFormation
Monitoring & Logging: Prometheus, Grafana, ELK Stack
Scripting Languages: Python, Bash, Go
A Definitive Guide to Hiring an Excellent DevOps Engineer
Hiring a DevOps engineer requires more than just scanning resumes for buzzwords. Companies need professionals who understand automation, cloud computing, and software lifecycle management. Here’s how to find the best candidates:
1. Craft a Clear Job Description
Define the role, expectations, and required technical skills upfront. Be specific about tools, cloud experience, and automation requirements.
2. Expand Your Talent Search
Look beyond job portals like LinkedIn and Indeed. Consider sourcing talent from:
DevOps communities and meetups
GitHub and GitLab repositories
Open-source contributions
3. Use Boolean Search Strings
Widen your candidate pool by using search strings like:
("DevOps Engineer" OR "Site Reliability Engineer" OR SRE OR "Cloud Automation Engineer") AND (Jenkins OR Kubernetes OR Terraform) AND (AWS OR Azure OR Google Cloud)4. Assess Soft Skills
Since DevOps roles require collaboration, candidates should demonstrate strong communication, problem-solving, and adaptability skills.
Skills That You Must Test for a DevOps Engineer Position
Technical expertise is crucial, but evaluating a candidate’s hands-on ability is even more important. Here are the top skills to assess:
1. Scripting & Coding
Candidates should be proficient in languages like Python, Bash, or Go to automate workflows and manage infrastructure.
2. Automation & CI/CD Pipelines
Evaluate their experience with CI/CD tools and their ability to implement automation in software deployment.
3. Cloud Computing
Ensure they have hands-on experience with AWS, Google Cloud, or Azure, including deploying applications in cloud environments.
4. Containers & Orchestration
Knowledge of Docker and Kubernetes is critical for managing scalable applications.
5. Infrastructure as Code (IaC)
Test their ability to write Terraform or Ansible scripts to automate infrastructure setup.
6. Monitoring & Security
Assess their familiarity with tools like Prometheus, ELK Stack, and best security practices for cloud environments.
The DevOps Market and Adoption Statistics
The DevOps market is booming, with increasing demand for automation and cloud adoption.
The global DevOps market surpassed $7 billion in 2021 and is projected to grow by 20% from 2022 to 2028.
The U.S. DevOps market is estimated at $3.8 billion, making up 41.4% of the global market.
85% of organizations plan to adopt a cloud strategy by 2025, further driving the demand for skilled DevOps professionals.
FAQs About A Recruiter’s Guide to Evaluating DevOps Candidates Like a Pro
What are the most important skills for a DevOps candidate?
A strong DevOps candidate should have expertise in automation, cloud computing, scripting, containers, and continuous integration/deployment (CI/CD).
Where can I find top DevOps talent?
Look beyond traditional job boards. Consider GitHub, open-source contributions, DevOps meetups, and specialized DevOps communities.
How do I test a DevOps candidate’s practical skills?
Conduct live coding tests, ask them to automate a task, or have them troubleshoot a real-world DevOps scenario.
What is the average salary of a DevOps engineer?
Salaries vary, but experienced DevOps professionals can earn over $200K annually, depending on skills and location.
FAQs About Brokee
What services does Brokee offer?
Brokee specializes in B2B client acquisition, lead generation, and digital marketing strategies to help businesses grow.
How does Brokee help companies scale?
We use data-driven strategies, automation, and personalized outreach to connect businesses with high-quality leads.
Is Brokee suitable for startups?
Absolutely! We help startups accelerate growth by providing cost-effective lead generation and marketing solutions.
How can I contact Brokee?
You can reach out to us through our website or email us at info@brokee.io for more details.
Conclusion
Recruiting DevOps professionals doesn’t have to be overwhelming. By understanding key skills, tools, and hiring best practices, you can streamline your hiring process and secure top-tier DevOps talent. Use Boolean searches, test for practical skills, and leverage alternative sourcing channels to find the best fit for your company.
By following this guide, you’ll be evaluating DevOps candidates like a pro in no time!
Live Technical Assessments: The Future of Hiring System Engineers
By George Goodwin
The landscape of hiring system engineers is changing rapidly, and traditional recruitment methods are often no longer enough to identify the right talent. Technical assessments, specifically live ones, are becoming a game-changer in how companies evaluate potential hires. In this blog post, we’ll dive into the importance of technical assessments in engineering recruitment and explore how they’re shaping the future of hiring.
The Significance of Technical Assessments in Engineering Recruitment
Hiring system engineers has always been challenging, but technical assessments bring an entirely new level of precision to the process. Resumes alone can’t provide the full picture of a candidate’s capabilities. Live technical assessments offer a chance to evaluate real-world problem-solving skills, technical proficiency, and the ability to adapt to new challenges. These assessments allow companies to:
Ensure Skills Alignment: Technical assessments help ensure that candidates possess the specific skills required for the role, from coding to troubleshooting.
Validate Qualifications: With these assessments, companies can confirm the accuracy of the qualifications listed on a resume and check if the candidate truly understands the tools they claim to know.
Predict Job Performance: By assessing candidates in a real-time setting, companies can gain insight into how well a candidate might perform on the job.
Preparing for Technical Assessments in Engineering
If you’re a job seeker looking to ace your next technical assessment, preparation is key. Here’s how you can get ready:
Review Key Concepts and Projects: Brush up on the fundamentals—coding, algorithms, system design, and problem-solving. Consider revisiting past projects to discuss your process clearly.
Practice Problem-Solving: Use coding platforms like LeetCode or HackerRank to practice solving problems quickly. This will help you become familiar with time constraints and improve your critical thinking under pressure.
Mock Interviews: Simulate the real experience by practicing mock interviews with peers or mentors. The more you practice, the more comfortable and confident you’ll be in the actual assessment.
Understanding the Purpose of Technical Assessments
You might be wondering, “Why do companies conduct these live technical assessments?” The simple answer is—they provide a more accurate, hands-on evaluation of a candidate's abilities. Let’s break it down:
Objective Evaluation: Traditional interviews often rely on subjective impressions, but technical assessments use clear, measurable criteria. This helps companies avoid biases and hire based on merit alone.
Real-Time Problem Solving: During live assessments, you get to demonstrate how you tackle complex issues on the fly. Companies can see how you approach coding challenges, troubleshoot errors, and manage your time.
Better Fit: These assessments not only test technical skills but also show how well you fit within the company culture. Employers want engineers who can collaborate and adapt, and live assessments give them the opportunity to observe this in action.
Why Conduct Technical Assessments When Hiring Software Developers?
The traditional hiring process—scanning resumes and conducting interviews—just isn’t enough to fully understand a developer’s capabilities. Here’s why technical assessments have become so important in the hiring process:
Traditional Interviewing Process: The Limits
While resumes provide a good snapshot of a candidate's background, they don’t reveal how someone actually performs in a real-world setting. The standard interview focuses on theory and past experiences, but it often overlooks whether a candidate can solve the specific problems the company is facing.
Enter Live Technical Assessments
Live technical assessments focus on actual problem-solving skills. This means candidates can demonstrate how they think on their feet, work through technical challenges, and use their knowledge to deliver real solutions. By conducting live assessments, companies get to observe candidates in action, which significantly reduces the risk of hiring the wrong person.
The Future of System Engineer Hiring: Live Technical Assessments
As the demand for skilled system engineers continues to grow, traditional methods will no longer suffice. Live technical assessments are not only more efficient but also more effective at identifying the best candidates. Let’s look at how this will shape the future:
Faster Hiring Process: Live technical assessments allow recruiters to quickly see if a candidate has the necessary skills, cutting down on the time spent evaluating resumes and conducting lengthy interviews.
Improved Hiring Quality: By focusing on actual performance rather than theoretical knowledge, companies will be able to hire engineers who can truly deliver.
More Inclusive Hiring Practices: These assessments level the playing field, ensuring that candidates are judged based on their skills and abilities, not their background or how well they perform in a traditional interview setting.
FAQs About Live Technical Assessments: The Future of Hiring System Engineers
What are live technical assessments?
Live technical assessments are real-time, hands-on tests used during the hiring process to evaluate a candidate’s technical abilities. They usually involve solving coding problems, system design tasks, or troubleshooting exercises in a time-constrained environment, allowing employers to see how candidates think and solve problems.
How do live technical assessments improve hiring?
These assessments give recruiters a clearer picture of a candidate’s actual skills. Unlike traditional interviews, which may focus on theoretical knowledge, live assessments focus on practical abilities, helping employers make more informed hiring decisions.
What types of assessments are used?
Common live technical assessments include coding challenges, system design exercises, and troubleshooting tasks. These tests simulate real-world problems that system engineers will likely face on the job.
How can I prepare for live technical assessments?
Preparation involves practicing coding problems, studying key technical concepts (like algorithms and data structures), and doing mock interviews. The more you practice, the better you’ll be at solving problems under pressure.
Why is Brokee a great choice for recruitment?
At Brokee, we specialize in streamlining the recruitment process for tech companies by offering advanced technical assessments. Our platform ensures that you get the right talent, quickly, with comprehensive assessments that test real-world skills in an objective, fair, and unbiased way.
In conclusion, live technical assessments are revolutionizing the way system engineers are hired. By focusing on practical problem-solving abilities, these assessments provide both companies and candidates with a more effective, accurate, and efficient hiring experience. As the demand for skilled system engineers continues to grow, embracing live assessments will be essential to staying ahead in this competitive market.
Skills Over Resumes: How to Assess Real-World DevOps Talent
Introduction
Resumes might tell you where a candidate has worked, but they don’t always reveal how well they can actually perform in a real-world DevOps environment. In today’s fast-moving tech landscape, hiring DevOps engineers based on skills rather than just job titles or degrees is the smarter approach.
This blog post will guide you on assessing real-world DevOps talent beyond the traditional resume. We’ll explore the essential skills, practical assessments, and strategies for identifying the right DevOps engineers for your team.
Why Skills Matter More Than Resumes
A resume lists qualifications, but it doesn’t measure hands-on expertise. DevOps is a highly technical field that requires problem-solving, collaboration, and automation skills. When hiring a DevOps engineer, focusing on real-world capabilities ensures you get a candidate who can thrive in a dynamic environment.
Key Skills Every DevOps Engineer Needs
1. Automation and Scripting
A strong DevOps engineer must be proficient in automation tools like Ansible, Terraform, and Kubernetes. Scripting knowledge in languages such as Python, Bash, or PowerShell is essential for writing infrastructure-as-code and automating repetitive tasks.
2. CI/CD Pipelines
Understanding Continuous Integration/Continuous Deployment (CI/CD) pipelines is crucial for delivering reliable software quickly. DevOps professionals should be skilled in Jenkins, GitHub Actions, GitLab CI, or CircleCI to streamline software development and deployment.
3. Cloud Platforms
With most businesses moving to the cloud, experience with AWS, Azure, or Google Cloud is a must. A skilled DevOps engineer knows how to manage cloud resources efficiently and securely.
4. Containerization and Orchestration
Containers, especially Docker and Kubernetes, have revolutionized software deployment. Candidates should be able to build, deploy, and manage containerized applications effectively.
5. Infrastructure as Code (IaC)
Managing infrastructure through code ensures consistency and scalability. Familiarity with Terraform, CloudFormation, or Puppet is a major plus.
6. Monitoring and Logging
Real-time monitoring tools like Prometheus, Grafana, and ELK Stack help keep systems healthy. A DevOps engineer must be skilled in setting up and managing these tools.
7. Security Best Practices
Security should be integrated into every step of the DevOps lifecycle. Engineers must understand secrets management, secure coding practices, and compliance frameworks like SOC 2 or ISO 27001.
8. Collaboration and Communication
DevOps is as much about people as it is about technology. Engineers need strong collaboration skills to work effectively with development, operations, and security teams.
9. Problem-Solving and Adaptability
Things go wrong in tech—often. DevOps professionals need to be able to troubleshoot quickly and adapt to new technologies and workflows as they emerge.
How to Assess Real-World DevOps Skills
1. Practical Coding Challenges
Give candidates real-world problems to solve using automation scripts, CI/CD pipelines, or infrastructure as code.
2. Hands-on Labs or Take-Home Assignments
Set up a cloud environment and have them deploy a simple application, troubleshoot an issue, or set up monitoring.
3. Live Technical Interviews
Instead of just talking theory, have candidates share their screens and walk through a DevOps scenario in real-time.
4. Scenario-Based Questions
Ask how they would handle a major system failure, automate a repetitive task, or scale a cloud application under heavy load.
5. Open-Source Contributions or Past Projects
Review their GitHub repositories, DevOps blog posts, or open-source contributions to assess their real-world expertise.
FAQs about Skills Over Resumes: How to Assess Real-World DevOps Talent
1. Why is skills-based hiring important in DevOps?
Because DevOps is a hands-on field, real-world skills are more valuable than credentials. A candidate with practical experience can contribute faster and more effectively.
2. What are the best ways to assess DevOps skills?
Live coding challenges, hands-on projects, and reviewing past work (like GitHub repositories) are the best ways to gauge a candidate’s real abilities.
3. What tools should a DevOps engineer know?
Key tools include Kubernetes, Docker, Terraform, Ansible, Jenkins, AWS, Azure, Prometheus, and ELK Stack.
4. How can companies test problem-solving skills in DevOps?
Present candidates with real-world issues, such as debugging a failing deployment or optimizing a CI/CD pipeline.
5. How does DevOps hiring differ from traditional IT hiring?
Traditional hiring often relies on degrees and experience, whereas DevOps hiring focuses more on practical, hands-on skills.
FAQs about Brokee
1. What is Brokee?
Brokee is a leading platform that connects businesses with top DevOps talent through a skills-based hiring approach.
2. How does Brokee help companies hire better?
We offer hands-on skill assessments, live coding challenges, and AI-driven talent matching to ensure you hire the best DevOps engineers.
3. What makes Brokee different from traditional hiring platforms?
Unlike resume-based hiring platforms, Brokee focuses on real-world skills, providing companies with proven DevOps talent.
4. Does Brokee support remote hiring?
Yes! We connect businesses with top DevOps engineers worldwide for both remote and on-site roles.
5. How can I sign up for Brokee?
Visit our website, create a company profile, and start assessing top-tier DevOps talent today.
Conclusion
Hiring DevOps talent based on skills rather than resumes leads to stronger, more capable teams. By using real-world assessments, hands-on projects, and practical interviews, businesses can ensure they hire engineers who can perform under pressure. Whether you’re looking for your next DevOps hire or refining your own skills, focusing on hands-on expertise will set you up for success.
The Ultimate Guide to Hiring Top ReactJS Developers Using Skills Tests
Hiring the right ReactJS developer is a game-changer for businesses. But how do you ensure you’re bringing the right talent on board? Skills tests are the secret weapon that can streamline the hiring process and guarantee you’re picking the cream of the crop. In this guide, we’ll dive into everything you need to know about hiring top ReactJS developers using skills tests, from understanding React development to why testing and quality assurance (QA) are essential.
Understanding React Development
ReactJS is a powerful JavaScript library developed by Facebook for building dynamic user interfaces. Its component-based architecture and efficient virtual DOM make it a favorite for developers creating fast, scalable, and responsive web apps.
What Does a ReactJS Developer Do?
A ReactJS developer is responsible for:
Developing reusable UI components.
Managing application state using libraries like Redux or Context API.
Optimizing application performance.
Collaborating with designers and back-end developers to deliver a cohesive product.
However, a top-notch React developer goes beyond these basics. They have expertise in related tools, understand user experience, and are adept at testing and QA.
The Importance of Testing and Quality Assurance
In the fast-paced world of software development, functionality isn’t enough—reliability and user satisfaction are critical. That’s where testing and QA come in.
Why Are Testing and QA Crucial for React Development?
Prevent Bugs: Early testing catches errors before they affect users.
Enhance Stability: Quality assurance ensures your application remains stable and performs well under stress.
Improve User Experience: A bug-free, intuitive app keeps users happy.
Save Time and Money: Identifying issues early avoids costly fixes later.
By hiring a ReactJS developer with testing skills, you’re investing in a smoother development process and a better end product.
How Skills Tests Make Hiring Easier
Skills tests help you evaluate a candidate’s abilities in real-world scenarios. They’re more than just a buzzword—they’re your key to hiring success.
Benefits of Skills Tests
Objective Assessment: Evaluate candidates based on their actual work, not just resumes.
Time Efficiency: Filter out unqualified candidates quickly.
Real-World Scenarios: See how candidates perform tasks similar to those they’ll handle on the job.
Better Hiring Decisions: Focus on candidates who demonstrate the required skills and expertise.
Essential Skills to Test
When creating a skills test for ReactJS developers, consider including tasks that cover:
React Fundamentals: Building and managing components, state, and props.
JavaScript Proficiency: ES6+ syntax, promises, and async/await.
State Management: Redux, Context API, or other libraries.
Testing and QA: Writing unit and integration tests using Jest, Enzyme, or Cypress.
CSS and HTML Skills: Styling components and creating responsive designs.
Steps to Hire Top ReactJS Developers Using Skills Tests
Step 1: Write a Clear Job Description
Your job posting should specify:
Project goals.
Required skills (e.g., React, Redux, testing frameworks).
Desired experience level.
Details about the role and responsibilities.
Step 2: Screen Candidates
Look for:
Relevant project experience.
Strong portfolios showcasing React work.
Knowledge of modern development tools and practices.
Step 3: Conduct a Skills Test
Create a test that mimics real-world challenges. Example tasks include:
Debugging an existing React app.
Building a small component with specific functionality.
Writing unit tests for a given module.
Step 4: Hold a Technical Interview
Use the interview to:
Discuss their approach to the skills test.
Assess their understanding of testing and QA.
Evaluate problem-solving and communication skills.
Step 5: Make the Hiring Decision
After reviewing test results and interview performance, choose the candidate who aligns with your project needs and team culture.
FAQs about The Ultimate Guide to Hiring Top ReactJS Developers Using Skills Tests
Why Use Skills Tests to Hire ReactJS Developers?
Skills tests provide a practical way to assess a candidate’s abilities, ensuring they have the technical expertise required for your project. They’re especially useful for identifying developers with strong testing and QA skills.
What Are the Key Skills for ReactJS Developers?
Proficiency in React, JavaScript, HTML, and CSS.
Experience with state management libraries like Redux.
Familiarity with testing frameworks (Jest, Cypress).
Knowledge of agile methodologies.
How Much Does It Cost to Hire a ReactJS Developer?
Costs vary by location and experience:
North America: $80 – $150/hour.
Western Europe: $50 – $100/hour.
Eastern Europe: $30 – $60/hour.
Asia: $20 – $40/hour.
Can I Save Money by Hiring Offshore Developers?
Yes! Offshore teams often offer competitive rates without compromising quality. Just ensure they have strong testing and QA processes in place.
How Does Brokee Help with Hiring ReactJS Developers?
At Brokee, we specialize in connecting businesses with top-tier developers. Our curated talent pool ensures you’ll find professionals with the exact skills you need, including ReactJS and testing expertise. We make hiring stress-free by handling the screening and vetting for you.
What Makes Brokee Different?
Customized Talent Matching: We match developers to your specific needs.
Pre-Vetted Experts: All candidates are rigorously screened.
Support Throughout the Hiring Process: From job posting to onboarding, we’ve got you covered.
Conclusion
Hiring a top ReactJS developer doesn’t have to be daunting. By leveraging skills tests, you can objectively assess candidates’ abilities and find the perfect match for your project. Whether you’re building a dynamic web app or scaling an existing product, a skilled React developer with strong testing and QA expertise is your ticket to success.
Ready to hire your next ReactJS superstar? Let Brokee help you find the right fit today!
How to Build an Effective DevOps Hiring Strategy with Skills Assessments
By George Goodwin
Recruiting the right DevOps engineer is a critical step for companies aiming to streamline their software delivery and improve operational efficiency. But how can you make sure your hiring strategy is effective? The secret lies in combining targeted recruitment strategies with real-world skills assessments. Let’s break it down.
Understanding DevOps and Its Importance
What is DevOps?
DevOps is a culture and set of practices that bridge the gap between development and operations teams. It focuses on collaboration, automation, and continuous delivery. DevOps engineers play a key role in ensuring smooth workflows, reduced development cycles, and high-quality software delivery.
Why is DevOps Recruitment Challenging?
The demand for skilled DevOps professionals is skyrocketing, with roles growing by 25% annually. This makes recruitment competitive and requires companies to stand out with a well-defined hiring strategy and attractive employer branding.
Strategy to Hire a Skilled DevOps Engineer
1. Define Your Needs
Before starting your search, get clear about the role requirements. Ask yourself:
What are the technologies and tools the candidate must know (e.g., Docker, Kubernetes, CI/CD systems)?
What kind of projects will they work on?
Are there specific challenges they need to address?
2. Create a Compelling Job Description
A good job description can attract the right candidates. Highlight:
Responsibilities and expectations.
Skills and experience required.
Your company culture and values.
Opportunities for growth and innovation.
3. Leverage Multiple Recruitment Channels
Don’t rely solely on traditional job boards. Use:
Online forums and communities for DevOps professionals.
LinkedIn and other networking platforms.
Employee referrals to tap into trusted talent pools.
Top Skills and Qualities for DevOps Engineer Professionals
Technical Skills to Look For
Cloud Expertise: AWS, Azure, or GCP knowledge.
Containerization: Mastery of Docker and Kubernetes.
CI/CD Pipelines: Experience with Jenkins, GitLab CI/CD, or similar tools.
Infrastructure as Code (IaC): Skills in Terraform or Ansible.
Scripting Languages: Python, Bash, or PowerShell proficiency.
Soft Skills to Prioritize
Strong problem-solving abilities.
Excellent communication and collaboration.
Adaptability and a learning mindset.
Resilience under pressure.
How to Conduct Online Assessments for DevOps Hiring
Online assessments are essential for evaluating candidates’ skills in real-world scenarios. Here’s how to do it effectively:
1. Create Scenario-Based Tests
Present candidates with challenges they might face in the role, such as optimizing a CI/CD pipeline or resolving an outage.
2. Evaluate Technical Skills
Use coding exercises and system simulations to test programming and systems management knowledge.
3. Assess Collaboration and Communication
Include tasks that require teamwork or explain decision-making processes. This helps gauge how well they’ll fit into your team.
Why Do Companies Use Real-World DevOps Tasks to Judge Engineers?
Practical Insight
Real-world tasks reveal how candidates think, troubleshoot, and apply their knowledge in practical situations. It’s a better indicator of their abilities than theoretical questions.
Cultural Fit
Seeing how candidates handle collaborative scenarios can show how well they align with your company’s culture and workflow.
Problem-Solving Skills
DevOps often involves solving unexpected problems. Scenario-based assessments help you understand how candidates approach challenges.
Future Aspects of DevOps Engineers
The role of DevOps engineers is evolving rapidly. Future trends include:
Greater emphasis on automation and AI-driven DevOps tools.
Increased demand for security expertise in DevOps pipelines.
Growth in hybrid cloud and multi-cloud environments, requiring broader skillsets.
Investing in ongoing training and professional development for your DevOps team will ensure they stay ahead in this fast-changing field.
FAQs About How to Build an Effective DevOps Hiring Strategy with Skills Assessments
Why are skills assessments important in DevOps hiring?
Skills assessments provide practical insights into candidates’ abilities, ensuring they have the technical expertise and problem-solving skills required for the role.
How do I attract top DevOps talent?
Focus on employer branding, offer competitive compensation, and showcase opportunities for growth and innovation in your company.
What tools can I use for online skills assessments?
Platforms like Codility, HackerRank, or custom tools tailored to DevOps challenges can help design effective assessments.
How can I ensure a cultural fit?
Use behavioral interview questions to explore candidates’ teamwork, adaptability, and past experiences in collaborative environments.
About Brokee
What is Brokee?
Brokee is a cutting-edge platform designed to simplify hiring for technical roles, including DevOps engineers. We offer:
Real-world skills assessments tailored to your hiring needs.
Advanced analytics to help you identify the best candidates.
Seamless integration with your existing hiring process.
How does Brokee help with DevOps hiring?
Brokee provides scenario-based tests and technical assessments to evaluate DevOps candidates effectively. Our tools save time and ensure you hire the right talent for your team.
Building an effective DevOps hiring strategy requires a mix of clear planning, targeted recruitment, and practical assessments. By focusing on the right skills, leveraging real-world tasks, and prioritizing cultural fit, you can build a team that drives innovation and success. Ready to transform your DevOps hiring? Contact us today!
Integrating NATS and JetStream: Modernizing Our Internal Communication
Discover how Brokee transformed its microservice architecture from a chaotic spaghetti model to a streamlined, reliable system by integrating NATS.io. Leveraging NATS request-reply, JetStream, queue groups for high availability, and NATS cluster mode on Kubernetes, we achieved clear communication, scalability, and fault-tolerant operations. Learn how NATS.io empowered us to build a robust event-driven architecture tailored for modern DevOps and cloud engineering needs.
High-level NATS architecture
Introduction
Brokee was built using microservice architecture from day one as the initial focus for skills assessment was Kubernetes, and later we expanded to other technologies. At the same time, as new services were added, we sometimes took shortcuts with design decisions. Over the years, it resulted in a spaghetti architecture where many services were interconnected with each other and it became harder and harder to reason about dependencies and figure out which functionality should go to which service.
Discover how we improved our system's communication by integrating NATS messaging system and their JetStream functionality. We delve into the challenges we faced, the lessons we learned, and how we simplified our setup to make it more efficient. This integration has laid the foundation for a more scalable and resilient infrastructure, enabling us to adapt and innovate as our platform grows.
Why Change?
Our previous architecture relied heavily on a synchronous request-response model. While this served us well initially, it began to show limitations as our platform grew:
Scalability issues: Increasing traffic caused bottlenecks in our services.
Lack of flexibility: Adding new features required significant changes to the existing communication flow.
Reduced reliability: Single points of failure in the system led to occasional downtime.
Even though we use backoff and retry strategies in our APIs, requests can still fail if the server is unreachable, unable to handle them, or overwhelmed by too many requests. We needed a more robust, asynchronous system that could scale effortlessly. That’s when we turned to NATS and JetStream, which offered persistence.
Old architecture: tightly coupled services using synchronous request-response communication.
What is NATS and JetStream?
NATS is a lightweight, high-performance messaging system that supports pub/sub communication. JetStream extends NATS by adding durable message storage and stream processing capabilities, making it ideal for modern, distributed systems. For developers using the SDK, NATS offers support for a variety of programming languages, making it a flexible solution for integrating messaging capabilities.
With NATS and JetStream, we could:
Decouple services: Allow services to communicate without direct dependencies.
Enable persistence: Use JetStream’s durable subscriptions to ensure no messages are lost.
Simplify scaling: Seamlessly handle spikes in traffic without major architectural changes.
New architecture: decoupled services with asynchronous pub/sub communication via NATS.
The Integration Process
Here’s how we integrated NATS into our platform:
1. Setting Up NATS
We deployed NATS using Helm. Helm made the installation and configuration straightforward, allowing us to define resources and dependencies in a consistent, repeatable way.
To ensure reliability and scalability, we set up 3 running server instances of NATS, leveraging its clustering capabilities and the Raft consensus algorithm to handle increased traffic and provide fault tolerance.
For storage, we used persistent volumes, ensuring durability. NATS also offers the option to use memory-based storage. However, to optimize memory usage and prevent overload on our nodes, we decided to switch to persistent volume storage.
Additionally, we made the deployment more resilient by ensuring NATS instances were safely scheduled on separate nodes to avoid single points of failure and ensure high availability. We opted for the NATS headless service type as NATS clients need to be able to talk to server instances directly without load balancing.
config:
jetstream:
enabled: true
fileStore:
enabled: true
pvc:
enabled: true
size: 10Gi
storageClassName: premium-rwo-retain
cluster:
enabled: true
replicas: 3
statefulSet:
merge:
spec:
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
podTemplate:
topologySpreadConstraints:
kubernetes.io/hostname:
maxSkew: 1
whenUnsatisfiable: "DoNotSchedule"
2. Migrating to Pub/Sub
Our first step was replacing direct request-response calls with pub/sub communication. For example:
Before: Common Service would send an HTTP request directly to Auth Service and await a response.
After: Common Service publishes a message to the subject
auth.users.roles.assign, which is then processed asynchronously by the Auth Service that subscribes to this subject.
We incorporated the Request-Reply pattern, which NATS makes simple and efficient using its core pub/sub mechanism. In this pattern, a request is published on a subject with a unique "inbox" reply subject. Responders send their replies to the inbox, enabling real-time responses. This approach is particularly useful for scenarios requiring immediate feedback.
To distribute the workload randomly across multiple instances, the Auth Service subscribes as part of a queue group, ensuring messages are distributed to different instances. NATS automatically manages to scale responders through these groups and ensures reliability with features like "drain before exiting" to process pending messages.
In the next Golang example, we prepare a payload and publish it as a request to a mentioned subject from the Common service using NATS. This demonstrates how the Request-Reply pattern enables sending data to a subject and awaiting a response.
func NATSRequestAssignCompanyRoleForUser(
nc *nats.Conn,
userID string,
roleID string,
timeout int,
) error {
// subject -> 'auth.users.roles.assign'
subject := models.Nats.Subjects.UsersRoleAssign
payload := models.RoleAssignmentPayload{
UserID: userID,
RoleIDs: []string{roleID},
}
payloadBytes, err := json.Marshal(payload)
if err != nil {
return fmt.Errorf("failed to marshal payload: %w", err)
}
msg, err := nc.Request(subject, payloadBytes, time.Duration(timeout)*time.Second)
if err != nil {
return fmt.Errorf("failed to send NATS request: %w", err)
}
var response map[string]interface{}
if err := json.Unmarshal(msg.Data, &response); err != nil {
return fmt.Errorf("failed to unmarshal response: %w", err)
}
success, ok := response["success"].(bool)
if !ok || !success {
return fmt.Errorf("role assignment failed, response: %v", response)
}
return nil
}
In this example, we set up a subscriber with a queue group that listens to the same subject in the Auth service. The queue group ensures load balancing among subscribers, while the handler processes the requests with the relevant business logic, sending responses back to the requester.
func SubscribeToRoleAssignQueue(
nc *nats.Conn, handler func(msg *nats.Msg),
) error {
_, err := nc.QueueSubscribe(
models.Nats.Subjects.UsersRoleAssign,
models.Nats.Queues.UserRolesAssign,
func(msg *nats.Msg) {
handler(msg)
})
if err != nil {
return err
}
return nil
}
In a typical pub/sub setup, if a service fails or is unavailable, there’s no automatic way to repeat the message, and it can fail silently. To address this, we turned to JetStream, which provides message persistence and reliable delivery. With JetStream, even if a service goes down, messages can be reprocessed once the service is back online, ensuring no data is lost and improving overall system reliability.
3. Implementing JetStream
JetStream added persistence to our messaging:
Streams: We defined streams to capture messages, grouping related data for efficient processing. For example, an
stack.deletecould store all stacks destroying messages, ensuring messages are retained and available for subscribers even during downtime.In the example below, we defined a JetStream stream named
STACKSfor managing testing stack operations. It subscribes to a single subject,stack.deletebut multiple subjects can be specified. The stream has a 1GB storage limit (maxBytes) and uses file storage with three replicas for fault tolerance. The retention policy is set toworkqueue, ensuring messages are retained until processed, and once a message is acknowledged, it will be deleted from the stream. It connects to the specified NATS server instances for message handling.
apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
name: stacks
spec:
name: STACKS
description: "Manage stack operations"
subjects: ["stack.delete"]
maxBytes: 1073741824
storage: file
replicas: 3
retention: workqueue
servers:
- "nats://nats-headless.nats-system:4222"
Durable Subscriptions: Services could subscribe to streams and resume from where they left off, ensuring no data loss.
To provide flexibility and control over JetStream and consumer (a component that subscribes to a stream and processes the messages stored in that stream), we manage configurations through a manifest chart using JetStream Kubernetes controller called NACK, minimizing the need for code editing and rebuilding.
In the code, only minimal edits are required for specifying the subject, consumer, and queue group names. This approach ensures the configuration of streams and consumers is easily adjustable.
Additionally, we use push mode for streams, where messages are handled when placed in the queue. For durable queue consumers, the consumer and delivery group names must be the same to maintain consistency and work as expected.
Backoff and Acknowledgments: We use backoff in consumer configuration to control the number of retry attempts for message redelivery. Additionally, we set
ackWaitandmaxDeliverto define how long to wait before knowing if a message is acknowledged and after will be delivered.In some places, we use backoff, while in others, we use
ackWaitwith maxDeliver. You can use either backoff orackWait, but not both together: for multiple retries, backoff is preferred; for fewer retries,ackWaitis set to the execution time of your handler plus an additional 20-30% buffer, ensuring sufficient time to prevent premature exits and unacknowledged message.We also manually acknowledge messages after executing code, particularly in cases where validation fails due to invalid data, as there’s no need to redeliver the message. This helps to avoid unnecessary retries.
The next configuration sets up a JetStream consumer named
stack-deletefor the deletion of infrastructure stacks. It subscribes to thestack.deletesubject same as in stream subjects(viafilterSubject) and uses a durable nameSTACK_DELETE, ensuring message delivery resumes from where it left off.
apiVersion: jetstream.nats.io/v1beta2
kind: Consumer
metadata:
name: stack-delete
spec:
ackPolicy: explicit
ackWait: 20m
deliverGroup: STACK_DELETE
deliverSubject: deliver.stack.delete
deliverPolicy: all
description: Delete stack resources
durableName: STACK_DELETE
filterSubject: stack.delete
maxAckPending: 1000
maxDeliver: 5
replayPolicy: instant
servers:
- "nats://nats-headless.nats-system:4222"
streamName: STACKS
An example of using backoff instead of ackWait: By setting the desired retry interval instead of using ackWait, we ensure the total backoff interval is less than the maxDeliver value, or it will fail during creation/update. If there’s free interval capacity, it will reattempt with the last backoff interval.
...
spec:
ackPolicy: explicit
backoff:
- 1m
- 5m
- 10m
Key settings include:
ackPolicy: Explicit acknowledgment ensures messages are redelivered if not acknowledged.
ackWait: Set to 20 minutes to accommodate infrastructure destruction that can take up to 10-15 minutes in some cases.
deliverGroup & deliverSubject: Enables queue group-based delivery to
STACK_DELETE, ensuring load balancing among subscribers.maxAckPending: Limits unacknowledged messages to 1,000.
maxDeliver: Allows up to 5 delivery attempts per message, retrying every 20 minutes. If the message is not acknowledged after 5 attempts, it will remain in the stream.
replayPolicy: Instant replay delivers messages as quickly as possible.
servers: The consumer connects to the
STACKSstream on specified NATS servers for processing messages.
Next, we send a message to the stack.delete subject to request the deletion of a stack (the following example is written in Python). The process is straightforward: we create a message with the necessary information (userhash and test_id), and then publish it to the NATS server. Once the message is sent, we close the connection and return a response indicating whether the operation was successful or not.
async def delete_infra_stack(
userhash: str,
test_id: str,
) -> Dict[str, str]:
try:
nc = NATS()
await nc.connect(servers=[NATSConfig.server_url])
message = {"candidateId": userhash, "testId": test_id}
await nc.publish(
subject=NATSConfig.sub_stack_delete,
payload=json.dumps(message).encode("utf-8"),
)
await nc.close()
response = {
"success": True,
"message": f"Published {NATSConfig.sub_stack_delete} for {userhash}-{test_id}",
}
except Exception as e:
response = {
"success": False,
"message": str(e),
}
return response
In the next code snippet written in Golang (we use multiple languages for our backend code), the consumer subscribes to the stack.delete subject using the STACK_DELETE durable name. This allows the consumer to handle stack deletion requests while maintaining message persistence and retry logic as configured in JetStream. As you may notice subscribing is pretty straightforward as we manage the consumer configuration through the chart, which simplifies setup and allows easy adjustments without complex code changes.
func SubscribeToJSDestroyStack(js nats.JetStreamContext, svc Service) error {
subject := Nats.Subjects.StackDelete
durableName := Nats.DurableName.StackDelete
_, err := js.QueueSubscribe(subject, durableName, func(msg *nats.Msg) {
handleDeleteStack(msg, svc)
}, nats.Durable(durableName), nats.ManualAck())
if err != nil {
return fmt.Errorf("Error subscribing to %s: %v", subject, err)
}
return nil
}
func handleDeleteStack(msg *nats.Msg, svc Service) {
var req deleteStackRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
// ack on bad request data
msg.Ack()
return
}
if _, err := svc.DeleteStack(context.Background(), req.TestId, req.CandidateId, msg); err == nil {
// ack on success
msg.Ack()
}
}
4. Testing and Optimisation
We rigorously tested the system under load to ensure reliability and fine-tuned the configurations for optimal performance. Through this process, we identified the ideal settings for our message flow, ensuring efficient redelivery and minimal retries.
Challenges and Lessons Learned
Integrating NATS into our system posed several challenges, each of which provided valuable lessons in how to leverage NATS' features more effectively:
Request/Reply and Durable Subscriptions:
Initially, we thought the request/reply pattern would work well for durable subscriptions, as it seemed like a good way to ensure that every request would be retried in case of failure. However, we quickly realized that request/reply is more suited for real-time, immediate communication rather than long-term durability.
For durability, JetStream turned out to be the better option, as it ensures messages are stored persistently and retried until successfully processed. However, JetStream only delivers each message to a single designated consumer (the one configured to handle it), rather than broadcasting it to all subscribers.
Consumer and Queue Group Names:
We learned that for durable consumers to function properly, the consumer name and the queue group must be the same. If they don't match, the consumer won't subscribe to the stream, leading to issues in message delivery and distribution.This realization came after some trial and error. We tried subscribing to durable subscriptions but encountered errors. To understand what went wrong, we dug into the source code of the SDK and discovered the importance of matching the consumer name and queue group. Surprisingly, we didn’t find this mentioned clearly in the documentation, or perhaps we missed it.
Backoff vs. AckWait:
At first, we experimented with using both backoff and ackWait together, thinking it would allow us to fine-tune the retry behavior. We expected ackWait to control the waiting period for message acknowledgment, and then back off would manage retries with delays.
We first applied changes to the settings through Helm, and there were no errors, so we thought the changes were successfully applied. However, during testing, we noticed that the behavior wasn't as expected. When we checked the settings using NATS-Box Kubernetes pod, we found that the changes hadn’t taken effect. We then tried to edit the configurations directly in NATS-Box but encountered an error stating that the settings were not editable. This led to further investigation, as we realized that only one of either ackWait or backoff should be used to make it work.
Manual Acknowledgment:
One of the key lessons was the importance of manual acknowledgment. During our tests, we encountered situations where, even though the handler failed for some subscriptions, the message was still automatically acknowledged.
For instance, when an internal server error occurred, the message was considered acknowledged even though it wasn’t fully processed. We initially assumed that the acknowledgment would happen automatically if the message was successfully handled, similar to how HTTP requests typically behave.
However, when we moved to manual acknowledgment and controlled the timing ourselves, it worked perfectly. This change prevented false positives and ensured that messages weren’t prematurely acknowledged, even when an error or timeout occurred.
Testing with NATS-Box:
NATS-Box(available as part of NATS deployment) became an invaluable tool for us in testing and creating configurations. It allowed us to experiment and understand the impact of different settings on system behavior, helping us refine our approach to ensure optimal performance in real-world scenarios.
As we mentioned earlier, it helped us uncover small misunderstandings and nuances that weren't immediately obvious, giving us a deeper insight into how our configurations were being applied.
Conclusion
In conclusion, integrating NATS into our system proved to be a fast and efficient solution for our messaging needs. It wasn't without its challenges, but through testing and exploration, we were able to fine-tune the configurations to fit our needs. While we started with a simple setup, we may expand the use of NATS beyond internal communication to incorporate more features like monitoring and dead-letter queues. Additionally, we are considering replacing more of our internal architecture communication with NATS' pub/sub, and even potentially using NATS for external communication, replacing some of our REST APIs.
Based on our experience, using NATS with JetStream for durable messaging has proven to be a solid solution for ensuring reliable communication in our system. If you're looking to improve your system’s communication and explore event-driven architecture, we recommend considering NATS as a scalable and dependable choice, particularly for internal communication needs.