The ipernity webmaster - word of the sunday

I'm going to finish my investigations with ipernity now.
I have a guess how it can be solved, but the MySQL master and slave situation with ipernity
is very confusing. Sometimes a server is at the same time master for another server, but is also at the same time slave for yet another server.
slave for yet another server. The server concept of ipernity is really confusing.

I am now writing down all the information I have obtained so far, which should help the people at Source Code not to have to
to start all over again with the investigations. Even though they certainly have more experience with MySQL replication than I do.
As I said, I had my first experience with this on Friday morning - and without really wanting to...
I had planned other things for this weekend. Photography, for example. The night-time, corona pandemic-related curfew was lifted in my city last week. I would have loved to be out photographing at night again. Instead, I sat at my computer from morning till night. Well, at least the weather was lousy too.

I know where the failure originated, I know what caused it, and I know what could possibly be done to solve the problem.
could be done to fix the problem. But I'm not one hundred percent sure about it.
And if I'm not one hundred percent sure about something like that, I don't start it. In this case, the risk that things will subsequently get worse rather than better is too great. In this case, you only have one attempt to fix the problem.
only one attempt...

Above all, I don't start repairs on the ipernity-MySQL database "on a trial basis", because I can't run through these things locally on my (private) server in advance.
Because I only run the latest programme versions on my server and on all my computers. Which
are not necessarily compatible with the old server programme packages used by ipernity. Otherwise one would
would not continue to use such old versions at ipernity. It is even very likely that ipernity would no longer run on a server with a current software environment. In any case, it is much more likely than that everything would continue to work as it did at ipernity - until this momentous error in one of the MySQL databases on Friday morning or Thursday evening.
While I slept peacefully.

And now the people from source code are supposed to take care of it, who presumably had a quiet weekend so far (they should, let them have it!).

Unfortunately, this will cost ipernity money again. But I tried my best to get as familiar with MySQL replication as I could from Friday morning to Sunday evening. And you can believe me - apart from short lunch breaks, I really didn't do anything else from Friday morning to Sunday evening. It's now after 7pm again and I've only had a short breakfast as a break so far. It hasn't been any different the last few days. But it's not too bad, but that's enough now.

Nevertheless, a weekend is simply not enough time to read up on such a complex subject and then to be really sure.
to be really sure.
So sure that you know exactly what you are doing when this and that problem occurs and how to fix it.

Yes, there are manuals on the subject. Lots and lots of manuals. I've seen countless of them on sites like stackexchange.com,
stackoverflow.com and other sites like that. Some of them I know almost by heart.
And many of them even write something similar for troubleshooting, which is not always the case - but still
a good sign that it might actually work that way.

However, most instructions also assume an installation in which there is only one MySQL master server and one or two MySQL slave servers copying over from there.
ipernity has a lot more.

If I have a good overview, ipernity has just 3 MySQL master servers and 6 MySQL slave servers affected by the problem.
And some of them even communicate with each other like an octopus.

And above all, such a (probably relatively trivial) problem cannot be solved if one is not even clear at the beginning how ipernity is set up in terms of server structure. Which server is responsible for what exactly. And how exactly they communicate with each other. Even apart from the topic of MySQL and databases.

It takes months to understand and comprehend such a system. If you have worked with such strongly interconnected systems before.

I have exactly one web server. With a MySQL server. There are many different websites and projects running on it, but they don't communicate with each other. But they don't communicate with each other. ipernity's technical structure feels more like several websites that exchange data with each other. It's like merging Google and Wikipedia together with YouTube, Facebook and Instagram and then exchanging data. Oops. That does happen in part - but that's a whole other story ;-)

I have been the webmaster responsible for you and the ipernity website for 6 weeks now. It took me the first three to four weeks to get used to the system enough to understand what was happening. In the meantime, I'm able to change things on the site if it's just a matter of programming. Partly more badly than rightly, but it works increasingly better.
I have never been a database administrator before. I use Linux as my actual operating system.

That's why it wasn't a big problem a fortnight ago to fix the site's outage within about an hour. The only reason it took longer was because I was out during the day. Forgive me. ipernity is just one of my hobbies, and occasionally I do other things. Maybe even photography, if I ever get around to it.

Yesterday lunchtime - during a short lunch break - I had to read elsewhere that one would like to see a different management. One that was staffed with professional people. They would like more information about the progress of the ipernity repair.

Yes, I say to myself - first of all, I am not a member of any management. I'm a photographer who manages ipernity's website on the side. Who tries to keep it running as well as possible. I do this in my spare time and for free. Who pays his membership fees to the association like everyone else. And then I seriously read into this some kind of sense of entitlement on the part of others who long for a professional board of directors who would do everything so much faster and better than these incompetent hobbyists? As professional as ipernity used to be? Where if there was a system error on a Friday before the next Monday, they wouldn't even have started troubleshooting? And this despite the fact that they surely retained a reasonable salary from the customers' remittances? I would like to refer to http://www.ipernity.com/blog/team/807898 as an example in this context. Read the comments below. The customers from back then - many of whom are thankfully still club members today - complain about uploads not working for weeks or months.
Funnily enough, you can also read in the comments that the management at the time even blocked the account of the company founder. That's how well it worked at ipernity, back when the management was still staffed with professional employees ;-)

Back then you were paying customers. Back then you were allowed to complain if you couldn't upload photos at all or only very slowly for weeks or months. I did that back then, too.
Of course, you can still complain about it today. But at least wait a weekend before you do that. And complain to the right people. The people who are now working for ipernity in their spare time are not responsible for many of the problems that are only appearing today due to software updates outside ipernity.

Quite apart from that: before the end of the weekend, nothing would have happened in the direction of solving the problem, even with the professional people at source code. At least I hope so, because I don't begrudge anyone who does this professionally their well-deserved weekend.

And then you, as an honorary, paying member of the association, do nothing else all weekend but try to solve the problem and the first thing you read from others on the second day is a complaint that ipernity has not given any new information about what is being worked on today for another day? Are you serious?!

Would it have helped you if I had written to you on Friday afternoon that I was looking through the system log with the name zxy on the server with the name xyz-1 to find out why the page has been in read-only mode since Friday morning?
Or would it have done you any good if I had told you that I spent this Sunday morning accessing the server named xyz-2 through an ssh pipe? A pipe to get around the problem that I have read and write access to a user named ab3-user-something because I have his login data encrypted in an RSA private key, but I can't access the MySQL database with it? At least not via a pipe from my system? This requires rights for the user root or mysql. But I don't know any of their passwords. Yes, I know the command sudo su, as I said, I use a linux-based system myself. But it didn't help me in this case either.

But what kind of information am I supposed to give you while I continue to try to read up on and familiarise myself with the topic in question?
Why is a "we are aware of the problem and are working on the solution" not enough for you? Should I have written it Saturday then "We are also working on a solution to the problem this Saturday" just to add something topical?

It's not very motivating to continue to get involved in a new, very complex issue if the first thing you then read from others is "nagging" and a cry for "professional management".
Pay me a professional salary and I can afford professional training in MySQL database management!

Until then, you'll have to make do with your hobby webmaster, who actually prefers taking photos. He does use his weekends and works around the clock to make the site writable again as quickly as possible - but I can't guarantee anything and have never done so. But even professional database specialists can't do that. However, if a problem occurs on a Friday, they will only start troubleshooting on Monday morning at the earliest. To familiarise yourself with a new topic on a Friday - as a professional? Forget it. On Fridays, as a professional, you think about which after-work club you want to grace this time. Sorry for my sarcasm.

So as a conclusion of the past weekend:
I really enjoy the stuff here. And I will continue to do it. As well as I can. Help would have been very welcome, for example, before the last general meeting. But unfortunately there were no other candidates. And someone had to do the "job". A site like ipernity is not a dream job.

So please have a little more understanding that I am more a photographer than a database specialist. That would be very pleasant. Especially in times like these, which put a lot of pressure on me. Which in turn has an effect on the quality of my sleep. Another topic, yes. "It's my own fault" too. I didn't have to make myself available. But then the website would have been dead two weeks ago. And nobody knows for how long. Today we still have a readable website, unlike a fortnight ago. And at that time, none of the people involved could do anything about the fact that the site was completely offline.

And I'm sure our site will become writable again. But that sometimes needs patience. And understanding. What it doesn't need are unqualified "grumblings", they help neither in solving the problem nor in motivation.

I wish you a pleasant weekend. Well, it's over now, even if I didn't feel I had one. Then I'll wish you a pleasant week and hand you over to the people at Quellcode. As I said, they cost money, but maybe they'll be able to fix what is probably a minor problem. They are professionals after all. :)

I'm out of there for now. I have to take pictures. I trust that I will soon be able to show you the photos again on ipernity!