I started on a friday morning and had to fix the production database in the afternoon

Leboncoin.fr is the biggest classifieds site in France with 26M unique visitors per month. Jean-Louis Bergamo is the Director of Infrastructure and is an associate of the CTO Partners fund.

Company size at time of writing: 600
Tech team composition: 120 to 130 tech people
Infrastructure team: 15 people split in 3 teams — Site infrastructure, tools & processes and IT helpdesk

What’s on your pizza ?

Regina at La Pizza qui fume

Let’s talk about you

I started on a Friday morning. In the afternoon somebody told me the production database blew up and I had to help fix it.

What’s your background pre-leboncoin?

I’ve worked in infrastructure management for 20 years. I started in 98 in a IT services company, and didn’t stay long but met really good people. I then worked for an open-source search engine, and met even more good people.

My first big job was at Easynet — an Internet Service Provider — for about 7 years, then at Skyblog for 3 years, which was a very enriching from all points of view, with an incredible growth rate. (note: skyblog was the main teenagers’ blogging platform in France in the 2000’s, with millions of users). Then Facebook showed up and the audience declined, that’s about when I jumped ships.

I then worked for Winamax, an online poker company in 2010 when France finally opened its economy to online gambling. After that, I move to Figaro Classifieds (note: a competitor of leboncoin holding sites like keljob, viadeo or cadremploi).

I finally joined leboncoin in March 2013!

Just so we know what it means to manage leboncoin’s infrastructure, can you give us a few stats about the traffic?

26M unique visitors / month
About 12/13 GBPS of traffic during peak hours, with 100k requests per second
800k to 1M new ads per day

How did you join leboncoin ? You must be one of the old guys now.

I was the third sysadmin in the team. We’re about 15 now!

They contacted me and I first liked the company but it seemed like something that was running smoothly. The site was fast and not moving too much, so I first thought the job would be to maintain a bunch of running servers, which wasn’t too exciting. But I went to the interviews and realized there that there was a lot of work to be done !

I started on a Friday morning, thinking I’d had an easy weekend after having met a bunch of people and drunk coffee. But in the afternoon somebody told me the production database blew up and I had to help fix it. The team was lacking core database competences so everybody was on deck to help troubleshoot fast. After that we called a consulting firm to help us clean up and make things way more resilient.

In a few words, what’s your job today?

Mostly management. I have 3 teams to overview, with each one manager. My job is to make sure the teams work well, and make the voice of the infrastructure team heard elsewhere in the tech team in general and the company.

I also handle pretty much alone any modification on leboncoin’s network infrastructure. There are other people that know about the network and can handle it if problems arise, but I’m the only one actually making changes to the infrastructure when necessary.

Has you job changed since you started?

Greatly! I started as totally hands on and operational, and now my job is 90% management. It was a soft transition, going along with the growth of the company and therefore the team.

We have a real problem regarding the size of the team. We’re always planning to be about 8 to 9% of the total tech team, but as developers get hired faster, we have a hard time sticking to that number.

We are also constantly reorganizing company and the team. We went from having a team that really needed to work on the core of the infrastructure to a quite stable architecture, which made us split the team and have the sysadmins work directly in the development teams they are involved with.

Let’s talk high traffic tech

We’re completely killing the old platform to build a microservices infrastructure but we need to make sure it won’t impact performance

What’s your stack, and why?

For now we’re using the platform from the original swedish site — https://www.blocket.se/. It’s written in C and is very, very efficient. For the same traffic, we’re using 10 servers handling direct traffic versus 150 when I was at skyblog.

It runs a custom templating language as well as a completely custom index engine for the classifieds. The Blocket platform is very monolithic and relies on one big central database, running on 4 huge servers (160 cores, 2Tb of RAM and SSD disks). We need vertical scalability on this database.

You’re hinting that this stack will change?

We’re completely killing the old platform to build a microservices infrastructure. The front will be in ReactJS. Any feature that we now need to modify generates a new microservice, so we already started the change.

The custom templating language of the current platform is efficient but makes things harder to change, and make it harder to hire people. It took us almost a year to make the site responsive! So we’re throwing everything to the trash and go for state of the art, sexy technologies.

We’ll have some of the ReactJS pages in production at the end of the summer, we just need to make sure that we won’t impact performance in production — we’re currently deploying a server farm that will handle ReactJS prerendering.

Have you ever had to solve a crisis?

I’d say about once a year. This year was on July 5th, so not long ago! We basically lost a datacenter. The entire team jumped on the problem and worked hard to fix everything. Nobody blamed anybody, it was real, beautiful teamwork.

When the datacenter went down, some things worked well and some things didn’t. We definitely planned the time when this would happen, but we did not test at scale, and some of the untested redundancy features didn’t work because of the heavy traffic. Timeouts started popping up, and cascaded into loss of packets, services that didn’t respond, etc… Domino effect.

Now we learnt, and we will do all of our testing with high scale traffic in the future.

Every day life in the infrastructure team

What’s your hardest challenge right now?

Humans: making sure everybody has everything they need to work to their full potential, and that everybody works well together. I know it sounds cheesy, but it’s true !

Then you’ve got all the tech challenges, but the rest of the team handles that.

If you had to change something you did since the beginning, what would it be?

It’s hard to answer this question because technologies evolve too fast. Things we do now were just not doable 5 years ago, so I can’t say that I’d prefer for us to have done it earlier!

At the very beginning though, we really worked on the infrastructure itself. Our customers were ourselves. When we started interacting a lot more with the developers, we realized they just wanted “infrastructure as a service”, something that would just work.

So now we’re rebuilding a few things as a service, and that would not have been necessary if we had though that through beforehand.

The people at leboncoin

Can you describe the infrastructure team? The people, the spirit?

We still have a team spirit, even if we’re all split within different teams of developers. We all sit together at least once a week, and we also work in duos on some projects.

There’s a lot of stuff to do so we mostly hire experienced profiles that can become autonomous quickly, and can handle several projects. Some of our guys are devops and can definitely look in the code when necessary, or build tools for the infrastructure.

We’re always search for the “sheep with 5 legs”. But right now we’re also hiring a database administrator, as we need some specialization.

The main thing you are looking for when hiring?

Personality. We’re also searching for multi-disciplinary people, even if this is not always a blocker. But what we want to see is somebody that has the energy to search if they don’t know how to solve a problem.

What I’m trying to see in an interview is if you’re willing to understand and learn about a challenge, or if you’re locked down in what you know. So the interviews are mostly informal discussions, I rarely give tech challenges to solve.

We’re also heavily based on cooptation, we hire a lot of people that other people in the company already know.

Any hiring tips for startups?

Well.. good luck!

A startup needs to find that 5-legged sheep, without a lot of money to offer, which makes it even harder. It’s a great experience for the employees, but when you’ve got Google or Facebook offering twice the salary, a lot of people will take the money.

So I’d say: let your network do the magic, don’t rely on classifieds for that!

Future of leboncoin

Leboncoin in 5 years, what is it?

That’s way too far ahead, I have no idea! It will likely be a group in France, several companies. We started buying companies in specific niche markets.

On the tech side, I have no idea. In 2 years we’ll have finished the transition to microservices and will have killed the monolith. We might even have problems because we’ll have too many microservices, who knows?

The biggest problems you are going to face in the next few years?

Like any big company, we’re not as fast as startups. We’re working against that, we want to be as agile as possible, and that’s why we keep reorganizing the teams and the company in general.

I started on a friday morning and had to fix the production database in the afternoon — Jean-Louis…