Dark Crag and Iron Rock. Historic testing sites of live server patching. It's always a good sign when one of these two fine communities gets shafted with a few hours of downtime, combined with a likely following of extra special bugs, BUT hopefully getting the target bugs fixed in the first place. Now, granted, these players didn't exactly sign up for this sort of service and abuse, but it's good for the rest of us. What I find the most curious is that we don't have a Live PublicTestServer.

Yes, we do have a PTS before patches and it does get some pretty good populations for focused testing and feedback. Regardless of what Mythic decides to do with the feedback and bug reports, it doesn't seem to affect these patches before they hit Live servers, let alone all the chaos they cause on Live servers to begin with. Admittedly the PTS is different from Live servers. From what I understand the PTS is a pretty clean and sterile environment that the Devs work in to do all their coding OR it's the sterile environment they base most of their coding on, and it all gets assembled at the same time. Not real clear there, but it'll do.

The issue is how patch cycles work. Generally, by the time one patch goes live, there's already another patch being worked on in a future cycle. This means code is being built upon a patch a full cycle before it's released. Example, since that confuses me a little. Devs work on patch 1.2, it gets approved for release and begins testing, Devs begin work on 1.2.1 using the 1.2 base, 1.2 finishes PTS and is launched with subsequent hotfixes, (here's the tricky part) amid hotfixes released to 1.2 Devs must adjust 1.2.1 to comply with said hotfixes at the same schedule as prepping 1.2.1 and getting it approved. Things are missed, 1.2.1 is approved and pushed to PTS, old bugs live again. Devs begin work on 1.3 using the 1.2.1 base as 1.2.1 is PTS'd and launched... hotfixes...rinse wash repeat.

Most of this is standard procedure, but why do old bugs live and die with hotfixes and official patching over and over again? Here we need to classify different Dev's for what they work on. Thusly, in my mind, we have the Build Devs and Hotfix Devs. I can imagine the mentality between the two already.

Build Devs need to have absolute solid stability in whatever they're working at the time, they're on a deadline so sometimes corners need to be cut, and shortcuts taken. If their code is bad, or doesn't work, under sever scrutiny, it gets rejected, and they just wasted the past 2 months doing nothing of consequence. I see these guys getting buried in Hotfix code that they have to scrape through to find out if anything is going to affect their particular code.

Enter Hotfix Devs. These guys are the rock stars of the coding cubicles. They can whip out code like no ones business, and constantly have new fixes to apply. Unfortunately, they have to work with the Live servers and their field of vision thusly becomes very small. Affecting too much at one time can cause people to report bugs currently being worked on. Regardless, these guys tear through code, and spit out results constantly.

For every Hotfix hastily torn asunder to appease the fans, imagine one little thing breaking that's entirely unrelated. Fixes made in the past become unraveled to fix a bug in the present. The narrow scope of vision allows Hotfix coders only so much rope as they're focused on their precious Tracker, and little else. Oh sure, there's likely to be SOME things they have to check for, but in the reality of working against a clock, and seething hoard of pissed off players holding your paycheck against the wall, some things get slid back the the Build Devs.

Viciously nasty circle of life ain't it?

This is mostly speculation on my part, though I've worked in situations similar to this that ultimately fell apart because of it. It's said that most development happens in this manner, and all is good. Well maybe, but in your case Mythic, it's just not good enough. Make a Live PTS, send your Build Devs AND Hotfix Devs there. Hell, you wouldn't even need to distinguish between the two. Give players a reason to be there, even if it's just to tear ass through the end game with sweet decked out toons insta-leveled to 40 with templated gear. It's only going to hold their attention for so long, but the simple act of calling it a Test Server will keep the vast public off of it as they go along playing as intended.

Back to the point. It's not really fair to treat Dark Crag and Iron Rock as your test dummies. God knows you guys have enough extra servers sitting around. Copy over a live gold version to one of them, throw on some character creation templates, and start converting Devs over to it sooner than later. It's FINE. We can handle going without some new content for a while, at least we can hope you guys will work out the silly amount of "anniversary" bugs that got released with 1.3.1




-most of this article was derived from speculation on the few tidbits of facts that i've absorbed over the past months about how the development process works. I may be wrong, but this is the best representation I can give, and I feel that it's pretty damn close to accurate.-
A look back at patching...

1.2 - 3/3
Client Stability 3/3
Server Maint 3/4
Hot Fix 3/4
Server Maint - Transfers 3/5
Server Maint 3/5
Hot Fix 3/5
Server Maint - Transfers 3/10
Hot Fix 3/10
Login Fix 3/10
Server Maint - Transfers 3/11
Hot Fix 3/11
Server Maint - Transfers 3/12
Patcher Issues 3/12
Server Maint - Limited 3/13
Server Maint - Oceanic 3/13
Hot Fix 3/13

10 days, 16 hotfixes and server maintenance. 12 not counting Transfer issues.

1.2.1 - 4/16
Server Stability 4/16
Server Downtime 4/16
Sever Maint 4/17
Hot Fix 4/17
Server Maint 4/18
Hot Fix 4/18
Server Maint 4/20
Server Maint (DC only) 4/21
Server Maint 4/22
Hot Fix 4/23
Hot Fix 4/23
Hot Fix 4/24
Server Maint (Iron Rock) 4/28
Server Maint 4/29
Server Maint 4/30
Hot Fix 4/30

15 days, 17 hotfixes and server maintenance. No transfer issues with this batch...

1.3 6/16
Patcher Issues 6/16
Hot Fix 6/16
Server Maint 6/17
Hot Fix 6/17
Server Maint 6/18
Hot Fix 6/18
Server Maint 6/19
Hot Fix 6/19
Server Maint (DC only) 6/22
Hot Fix 6/22
Hot Fix 6/23
Server Maint 6/24
Server Maint (Iron Claw only) 6/25
Hot Fix 6/25
Server Maint 6/26
Hot Fix 6/27

11 days, 16 hotfixes and server maintenance.

1.3.0b 7/22
Server Maint (Iron Rock) 7/23
Server Maint 7/24
Server Maint 7/27
Server Maint 7/30
Server Maint 7/31
Hot Fix 7/31
Server Maint 7/31

9 days, only one Hot Fix... I guess that's OK, since it wasn't a huge patch anyway.
Just the blanket AoE nerf and Shaman/Archmage overhaul.

1.3.1 8/18
Server Maint 8/19
Hot Fix 8/20
Stability Issues 8/21
Server Maint 8/21

Here's the point of all this. Unless there's a hotfix to download before the weekend, my trending shows we won't have one until Monday. Technically, we should have an update every other day next week considering how bad things are, depending on how fast they get fixed. What's particularly upsetting about the 1.3.1 patch, is the amount of OLD bugs that have been reintroduced into the game, and the number of rollbacks that happened. There's bugs from LAUNCH that popped back up with this patch. Maybe it's part of their 1 year anniversary?

If I weren't such a WAR addict, I would say I'll be staying away from the game over the weekend. I know this isn't true though. It's like some sick masochistic obsession for me. Once we lose a couple thousand more players, I think we'll be down to the baseline of players like myself.

If Mythic is smart, they'll realize what kind of problem they opened up with this patch, and give everyone on their staff a bit of overtime for the weekend. Waiting until Monday is going to hurt them even more.


Happy (belated) Fathers day to all you proud daddies out there. Me and my daughter went camping over the weekend, and she has got to be the most hardcore 2 1/2 year old I know. Took our 4-wheeler into the woods and she was begging for speeds in excess of 50mph! God I love my child. Just need to train her in the arts of WAR so she can level me toons while I grind away at work.

In the WAR, the weekend has been quiet, while the past week was hotfix after patch after server maintenance with a scrambling of activity to plug the holes and right the wrongs. I honestly had a lot of fun over the weekend while Keep doors were broken open. The Live Event and Reikwald Factory have been excellent, and it's all coming to an end next week so get your fill! I still have 3 toons to run through the LE awards, even if it is only for a stupid pair of Titles. Praag was awfully competitive during the week in expedition resources, but over the weekend we went from a 30k lead, to a 200k deficit. Looks like Destruction on Praag will have access to LotD first after all. Ah well, it just means we get the first chance to run em out of town! Ha ha!

Jeff and Paul are back in a special double-wide, extra frame, fatty fat fat goodness comic. Mostly because I felt like it! Don't try and judge my rhyme or reason, lest it may tempt me into doing more (horrible) MSPaint sketching. Hehe. Games Day Toronto is coming up, and I have a feeling I'll be making the trek from Detroit to Canadia for the event. It's the closest one to brush by so far, and I really can't blame ANYONE for avoiding Detroit like the plague. How many major international cities get their mayor thrown in prison? Think you might show up? Leave a comment and we can rally down there. There's a post at the official forums for Praag server here.

Got a WCPI ready for tomorrow, and maybe a post later tonight if my guild members get me all rant-y again.