Suse downtime conversation: #suse

<xaphir> CygniX, you wouldn't last long where I work at
* lurchi__ has quit (Ping timeout: 248 seconds)
* GPenguin has quit (Quit: Konversation terminated!)
* mart has quit (Ping timeout: 255 seconds)
* CosmeFulanito (~Fruno__@190.191.73.29) has joined
<wingsorc> so are there mirrors or can i just not use zypper all weekend?
* Vojtaeus has quit (Remote host closed the connection)
* Vojtaeus (~vojta@opensuse/member/vojtaeus) has joined
<Aristar> could sed all your repos to a mirror in the meantime if your zypp is not working
<Aristar> but it is dumb that their main thing isn't properly locating mirrors
<jcsl> Maintenance finish today, so tomorrow you'll be able to use zypper again. Anyway, only OBS seems to be affected because I'm updating a VM with only the standard repos enabled.
<wingsorc> are obs repos not being mirrored?
* moozaad has quit (Read error: Connection reset by peer)
* moozaad (~moozaad@cpc92094-nrte30-2-0-cust102.8-4.cable.virginm.net) has joined
* moozaad has quit (Changing host)
* moozaad (~moozaad@opensuse/member/moozaad) has joined
* CyBerNetX (~jbmasurel@108.242.88.92.rev.sfr.net) has joined
<jcsl> It seems so. In the mailing lists they listed the affected services.
<wingsorc> im a bit baffled that there is no solution in place when they need to do maintenance. They just take nearly everything down.
* Vojtaeus has quit (Remote host closed the connection)
<jcsl> It was an electrical maintenance IIRC.
<Aristar> upgrading the hamster wheels
* Vojtaeus (~vojta@opensuse/member/vojtaeus) has joined
<jcsl> and the hamsters themselves
<CygniX> xaphir: I thought my answer as very informative ;)
* dddh_ has quit (Quit: Konversation terminated!)
 CygniX cb400f ced117 Chotaire ChrisBr Churr0z clintar clopez cmurphy comrad Condoulo Conny Cool_Fire cornfeedhobo CosmeFulanito Crendgrim CubicleNate Cust0sLimen CustosLimen CyBerNetX cyboman33 cybrian cyphar czechton[m]
<paulcarroty> https://mirrors.opensuse.org/
* bugfinder (Thunderbir@nat/novell/x-fstxdjfioeweiptw) has joined
<CygniX> python repo including mirrors for tumbleweed
<paulcarroty> I'm wonder - it's too hard to add the mirror?
<CygniX> is down
<paulcarroty> http://pkg.adfinis-sygroup.ch/opensuse/tumbleweed/repo/oss/suse/x86_64/
<paulcarroty> minimum the  mirror in Switzerland is up
<xaphir> paulcarroty, you can get to the mirror sources like gwdg.de, you can see the directories there if you do an ftp browse, but problem is we can't search for package locations.
<paulcarroty> zypper ar -f URL eu_mirror
<tigerfoot> paulcarroty: it's not up to date mirror, so you've got more chance to break your TW.
<paulcarroty> tigerfoot, if package is old the manager just ignore it
<CygniX> I would definitely advice to wait for the maintenance to be completed before doing any updates/upgrades
* kpfc has quit (Ping timeout: 260 seconds)
<paulcarroty> http://mirror.de.leaseweb.net/ 6hours ago the last sync
<tigerfoot> wingsorc: why didn't you propose your help and knowledge to support the community effort to minimize the effect. start to learn how much data, servers, services are behing the infrastructure, which is most of the time freely available to you.
<CygniX> like tigerfoot said, you could end up breaking your system
<paulcarroty> please help me to understand how I can break my system with OLD packages
<tigerfoot> The outage was annonced more than one month ago, better with a no cloud sky, to take a break, and drink and enjoy outside until monday.
<Aristar> and meanwhile all development grinds to a half? doesn't seem very productive
<tigerfoot> paulcarroty: if the mirror is not in sync and you already have TW, especially with the last 3 snapshots, you will break everything cause the right way to update is dup, dup will replace newer by older
<CygniX> Aristar: which development?
<Aristar> tumbleweed repos haven't seen updates all weekend for example, so i assume testing and development on TW now has to catch up after maintenance
<paulcarroty> tigerfoot, I use only zypper up without any downgrades. You should to force every downgrade manually.
<CygniX> if using tumbleweed, you are doing it wrong if using zypper up
<Miuku> wingsorc: Money money money, must be funny.. in a rich mans world.
<tigerfoot> paulcarroty: you have to take care, cause some of the downgrade are normal, others not ... Before the --no-allow-vendor-change by default zypper up was a solution I've used, but no more
<wingsorc> tigerfoot: you are not being helpful either. Why should I not voice my concerns?
<Miuku> wingsorc: Because you're not paying a dime for the service?
<wingsorc> Miuku: my ubuntu machine doesnt have uptime issues
<CygniX> you should probably be using ubuntu then :)
<Miuku> Indeed.
<tigerfoot> wingsorc: I don't it would work without any electricity :-)
<Miuku> This isn't a religion, you know, you can freely move from one to the other and not have your head chopped off.
<wingsorc> Huh, Im just voicing my displeasure whats wrong with it?
<wingsorc> no need to get so defensive
<CygniX> nothing
<Miuku> wingsorc: Because we've heard the same thing a hundred times and there's nothing we here can do about it.
<CygniX> anyway, the downtown was scheduled, didn't happen out of the blue
<CygniX> downtime
<wingsorc> Yeah but obs package search breaks every other weak, those aren't planned
<CygniX> hmmm
* tigerfoot is offering a led light to all Nurnberg Geeko plushes, they stay in the dark during all the week-end. I cross my fingers they don't turn red or browny :-))
* }ls{ has quit (Ping timeout: 240 seconds)
* bugfinder has quit (Quit: bugfinder)
* frispete_ (~frispete@p54A908FD.dip0.t-ipconnect.de) has joined
<Aristar> i understand OBS is a free and generously offered service so i understand if that is too large to be mirrored but the main repos should not have malfunctioned. people couldn't even download iso's due to mirrorbrain misconfiguration and such
<tigerfoot> wingsorc: if you have a cheap and efficient way to move 800 builders, including S390 and a bunch of PowerPC and arm in less than 4 hours back and forth, I encourage you to send an emal to admin@o.o so they would have a chance to do it better.
<Miuku> Aristar: I agree, the d.o.o mirrorbrain issue wasn't a good one but let's face it - it's over the weekend and you can still access the mirror page. Not the end of the world by any means.
<wingsorc> tigerfoot: yeah let me drive from zürich to nürnberg to help those o.o plebs
<tigerfoot> Aristar: you mean download.o.o is not working for you ? cause it works here ????
<Aristar> yes but perhaps a good time to prioritize and move more critical services, for example what if a major security bug hit us over the weekend?
* frispete has quit (Ping timeout: 255 seconds)
<tigerfoot> wingsorc: take a flash light then, there's no electricity in the whole bloc in Nü
* GPengOFF has quit (Quit: leaving)
<Aristar> i haven't checked my SLE servers but i would hope they use working repos
<Miuku> Aristar: SMT syncing has been working fine over the weekend.
<Aristar> but it seems too much of the main repos even rely on OBS to build their packages
* }ls{ (~kalle@unaffiliated/ls/x-8089558) has joined
<wingsorc> tigerfoot: i thought this is a planned maintencance how come there is no power in nürnberg
<tigerfoot> Aristar: the whole download.o.o is mirrored and working actually.
<CygniX> hmmmm
<Aristar> ah okay, well yesterday it wasn't but perhaps temporary issue
<tigerfoot> wingsorc: because that the reason of the outage.
<CygniX> not sure about that, tigerfoot
* Dense (~Dense@95.151.51.210) has joined
* BoggGod (~cesar@197.246.4.113) has joined
<Miuku> Aristar: I was able to sync (I use rsync to fetch the entire distribution trees, updates and such) fine this morning so at least d.o.o worked then.
<wingsorc> germans just can't handle trains and electricity, don't ever remember there being a power outage in zürich :)
<Aristar> still though, i had to manually compile 4.13.6 kernel due to a bug that hit me in 4.13.5 and i'm unsure if development/testing is able to continue on kernel:/stable during such outages
<Krennic> hmm opensuse repository for irssi test still down
<Aristar> (on my TW workstation anyhow)
<tigerfoot> Aristar: and if there's a high security issue, there's still engeener in prague, tawain, australia, US and so on. Able to publish that fix on main (during the week-end) provo
<Aristar> so we are to solely rely on SUSE for security and user developers cannot contribute?
* Dense has quit (Client Quit)
<BoggGod> Hey, I've got Zypper shooting errors when retrieving "Publishing Tools and Libraries" repo. "Valid metadata not found at specified URL".
<CygniX> !contribute
<wingsorc> suse server are down
<wingsorc> opensuse*
<Krennic> yup
<shodan45> BoggGod: see the topic
<tigerfoot> Aristar: if you compile it yesterday for example locally, you can still compile it locally with osc build --clean --no-service --offline
<BoggGod> ah thanks, thought it was my btrfs fs
<wingsorc> BoggGod: Don't try to poke tigerfoot with a stick he is really tense
<Aristar> ah, interesting
<tigerfoot> wingsorc: nope, I don't care that much, The outage was annonced, so I take actions to minimize the impact on my workflow :-) Just that people that complain are only complaining usually, and when it's time for actions or contributions, there's nobody :-]]
<Aristar> thanks, i guess I should have done that. maybe in the future useful links to alternate developer/contribution resources should be posted in the news so some aren't confused. or maybe it's just me since I haven't read _ALL_ of the documentation
<tigerfoot> so yeap perhaps I'm too rude to things like this, and you're right about that.
<CygniX> factoid needs to be updated with 'contribute' how to
<BoggGod> wingsorc, tigerfoot, I'm all good, I just missed the announcement, I'm not here to for a bruisin :)
<jcsl> Aristar: you only have to read the mailing lists
<tigerfoot> Aristar: man osc can be your bible for the week-end then :-)
* SiFuh_ (~SiFuh@115.187.139.142) has joined
<wingsorc> tigerfoot: maybe im just contributing to different FOSS software than opensuse stuff.
* iranen has quit (Quit: Leaving)
* gmo (503e7460@gateway/web/cgi-irc/kiwiirc.com/ip.80.62.116.96) has joined
<tigerfoot> wingsorc: like me, what else ? ;-)
<Aristar> jcsl: mailing lists were down too yesterday from what i saw
* tigerfoot btw time for a drink.
<Aristar> but i guess i should subscribe ahead of time
<jcsl> Aristar: maybe, but it was announces a few days ago
* cisc (~cisc@unaffiliated/cisc) has joined
<Aristar> true, my fault then
<CygniX> https://status.opensuse.org/
<Aristar> and yeah the status page said some things were up but weren't working still (yesterday anyways, not sure today)
<tigerfoot> Aristar: news.o.o has the annonce October 4th :-) https://news.opensuse.org/2017/10/04/planned-downtime-to-affect-opensuse-services/
<malcolmlewis> Aristar, you cna browse the archives on gmane, I nomail subscribe to the ML's and get/send via nntp
<Aristar> but such confusion to end users is not good in general, especially if trying to promote openSUSE in magazines (which i saw recently)
<Aristar> i'm not a new user but SUSE/openSUSE isn't the only platform i use, so I don't follow as closely as I should
<BoggGod> Aristar, it would be nice of course if it showed in a banner in everything opensuse, forums, main page etc.
<malcolmlewis> Aristar, but how, we even had a forum notice, but users still keep blinkers on and post ;)
<Aristar> yeah i understand that must be frustrating
<malcolmlewis> even here the topic says it all....
<Aristar> but also frustrating to the uneducated or those who don't know where all of the resources/documentation is ahead of time in order to prepare
<Aristar> the topic should have been updated a few days ago IMO
<CygniX> perhaps
* margeas (~marco@93.115.85.148) has joined
<Aristar> but downtime like this is one of the reasons i don't rely solely on SUSE/openSUSE on my workstations and servers (though no distro is perfect), and perhaps may reflect poorly on the distribution in general
<wingsorc> I'd even argue downtime is one of the major drawback of opensuse. But the rest works pretty nicely.
<malcolmlewis> some things like arm build hosts won't come up until perhaps mid week
<jcsl> c'mon, it doesn't happen all the time
<Aristar> there was an unexpected one a month or two ago as well
<Aristar> with data recovery and such
<malcolmlewis> wingsorc, donate some 12 core 128GB servers ;)
<jcsl> Yeah, twice in I don't remember how much time
<wingsorc> malcolmlewis, and lets place them in nürnberg where the is no power ;)
* gerhard1111 (~g2@h081217074044.dyn.cm.kabsi.at) has joined
<malcolmlewis> wingsorc, they can be anywhere, just need good bandwidth
<Aristar> are end users even allowed to contribute build services from their own servers? or does it all rely on SUSE datacenter?
<jcsl> There is no power because of the electrical maintenance. Do you want fried operators?
<jcsl> Nope, you can't even donate cpu cycles
<malcolmlewis> Aristar, nope security/integrity etc
<jcsl> ↑ yep
<Aristar> couldn't they take the datacenter down in sections at a time without affecting services?
<Aristar> hmm ic... then perhaps some security/integrity checking improvements should be evaluated in the future. many other distros allow this
<wingsorc> electrical maintenance something something and fried operators
<Aristar> so are we to expect downtime anytime a UPS battery dies?
<Aristar> sorry, that was somewhat rude of me
<wingsorc> What happens if there is a fire in the nürnberg datacenter, are there any backups of the obs stuff somewhere else?
<Aristar> that's also an important question too
<Aristar> we don't want another amazon s3
<plinnell> yes, we do have off site backup
<malcolmlewis> which are all questions that can be asked by contributing to the Hero's team AFAIK, there are only four people looking after it all for openSUSE
* diogenes_ has quit (Quit: vergissmeinnicht)
* dedeckeh (~dedeckeh@ptr-fvl7vteb6vjd542m2s0.18120a2.ip6.access.telenet.be) has joined
<wingsorc> you mean, four guys are handling the nürnberg stuff?
<Aristar> but offsite backups take time to restore, i don't see why home:/ repos couldn't be mirrored even by users if cost or mirror size is an issue (as long as verification is enforced
<Aristar> and some other OBS stuff
* Ada_Lovelace has quit (Ping timeout: 246 seconds)
<jcsl> Everythink is possible as long as someone do the job
<jcsl> *Everything
* Guest3269 is now known as lex
<malcolmlewis> wingsorc, for openSUSE infrastructure, yes, see Lars's comment here https://news.opensuse.org/2017/10/04/planned-downtime-to-affect-opensuse-services/
<wingsorc> doesn't it also require political will? I thought there are some guys from SUSE and the community making decisions
<Aristar> maybe not as "open"suse as it ought to be if manpower is needed
<wingsorc> the incapacitated chameleon is pretty cute
<malcolmlewis> wingsorc, the board has a SUSE rep, rest are voted in by the membership
* striker (striker@nat/redhat/x-kbaftloaxnzehbfz) has joined
* rriemann (~rriemann@2a02:a03f:3c83:d100:b198:1e08:f610:28a3) has jo