Arfman schreef op woensdag 29 januari 2014 @ 19:16:
Net even met de Radical op Mosport gereden, supergaaf karretje is het geworden! Goed te vangen als de banden eenmaal op temperatuur zijn.
Helaas is de site nu alweer down lijkt het, of heel traag in ieder geval. Ik wil nog zoveel auto's proberen ...
Ik heb net met de Radical op Sonoma getest (brandstof bepalen tbv Monday Tournament). En ja je hebt gelijk. Echt super autootje geworden
Hier wat uitleg van Tony over de problemen:
quote: Tony Gardner
As I said in another post, again sorry for the member site down time over the last day+. Just about nothing is a higher priority to us than having a fast, stable site that is working well. However, unfortunately, it can be difficult even with all the money and resources we spend to achieve that goal 24/7/365. We definitely do not skimp on anything when it comes to site stability. Cost is not a concern compared to uptime and a stable site/product. That is a fundamental philosophy which is probably not surprising to most. If there is something we can buy or get to make iRacing as stable as possible we do it or buy it without hesitation but that is almost always not the issue in the rare times we might go-down unplanned. We always have plenty of extra capacity in place. It can be a challenge to identically recreate the exact member site load with a site, product, and infrastructure that is always dynamically changing. There is a lot of moving parts with our software and beyond our software in our infrastructure and we try and anticipate and test everything but on occasion we fail when the actual full load tests every single thing. It usually is something very small and hard to find that causes a major problem. We also do have plans in place to consider all types of failures, everything we can think of that could go wrong in case we do, so we can recover as quickly as possible. We have learned a lot, have a lot of experience and have tremendously bright, professional and experienced people managing our infrastructure and software deployments but are still always learning, sometimes the hard way. There is a team that will not sleep (team could be different depending on the issue) until we are stable so in this case thanks to mainly Chris, Dean and Scott who resolved the problem and can now catch some sleep. No excuses, again our apologies.
Here is what went wrong and what we did:
What caused it?
Over the last 3 months we have had some rare occurrences where our servers would spike out of the blue and take up all of the CPU on a given machine. It would happen anywhere from once every two weeks to twice a week. We have been scouring the code, logs, online, trying to find the root cause. It was such a rarity that we could never track it down.
With this new season release we changed the method of how the site determines who is online and what they are doing. This change was made so that in the future, this information could be shared with servers that exist in different cells at different service providers. This change worked as designed in our development environment, our alpha environment, our beta environment and our staging environment. It also worked on the production site, to a degree.
As we continued to look at logs yesterday, trying to find the issue, we noticed on one of the servers a reference to the online status. We thought it was worth a shot so we reverted the code back to how it was pre-rollout. When we opened the site, things looked good, it seemed to be running fine. After an hour, the site spiked and crashed across the entire cluster of servers. We checked logs again and saw on one server, and only one, a reference about the online status again. We completely turned off the online status tracking. Since that time the load has been absolutely fantastic across all of the servers, we have not had any spikes and believe this to be the root of the problem.
What did we do about it?
Yesterday, we took numerous steps to try and fix the issue.
1/28
1. We upgraded all of our servers to the newest fixpacks – in hope that the issue was a webserver software issue.
2. We disabled the race guide (because we some references in the logs that we thought could be pointing to the issue)
3. We disable the spectator sessions page (as we have seen this page cause a load issue in the distant past)
4. We reverted the Online Status code back to the pre-season rollout version
5. We disabled the Online Status tracking
1/29
1. We re-enabled the race guide
2. We re-enabled spectator sessions
What are we doing now?
We are leaving the online status tracking off while we investigate a new implementation. The new implementations will allow us to easily turn it off and on so that we can test and get the site running again quickly if there are any issues. We are considering writing a few different implementations that we can switch between “on-the-fly” so that we can find the best solution with the least downtime.
Good news from our perspective is this unfortunate incident has actually helped us to find an elusive root cause, and we are sorry for the downtime. We will continue to do our best to provide a quality service and experience for our members. Again our apologies
Good Racing
Tony
quote: Chris Page
Here is some more technical info for those with an interest:
We use java virtual machines to power the site and make extensive use of web containers and services. This solution is a threaded architecture. Our management of who is online was implemented with a service that updates a cache that is replicated across the cluster. It was extremely simple and efficient and could stand up to unit load testing well. It was integrated into our log in process, and there were no issues during testing.
Things became a problem when made part of a wider cluster, and reached a tipping point, when just about every thread, was a log in thread. That code is called not only when you explicitly log in, but also when we log you in behind the scenes when you are connected to a new back end server. This new simple service couldn't keep up. This caused these threads to be queued, and delay the processing that was of actual interest. Eventually, all threads, on all machines would be in a busy wait state, and there would be no CPU cycles or available threads to service requests.
Determining the root cause of issue was elusive as it was not tied to any one call. There were no discernible patterns. Everything was seemingly hanging up at random. When we see this, it is often a shared resource, and most often a database lock. However, there were no issues no matter how hard we tried to find one there!
We even tried introducing more hardware, and as you might imagine, that just increased the load on that one service, and we crashed even sooner. There were reports of many threads being hung up. And they all seemed to be doing different things. It could have been anywhere. In the end, it turned out, it was this one little unassuming services. It hadn't stood out as being the issue, because it, like others, seemed to be hung up.
We'd like to thank the numerous messages of support we have received from the community. They really helped keep us going. We would have kept going anyhow, but it made it easier for sure. Thanks for that.
En dan nog een reactie over het terugschakel probleem van de RUF (en McLaren):
quote: Eric Hudec
It's a bug. Turns out we didn't have enough throttle blip with downshift attempts. The same problem could happen on the McLaren but was much more rare because of the characteristics of the driveline. Therefore there wasn't enough force to get the gear to change down. Fix will be rolled out asap.
[
Voor 89% gewijzigd door
JoepW op 29-01-2014 20:21
]
:strip_exif()/u/1839/avatar_drome.gif?f=community){kind=avatar}
) en dat wil nog wel, maar lager moet je niet komen want dan ben je meer in je spiegels aan het kijken dan vooruit.
/u/285775/crop60657c7849d6a_cropped.png?f=community)
:strip_icc():strip_exif()/u/202645/Untitled-4.jpg?f=community)
:strip_icc():strip_exif()/u/356256/iracing%2520avatar.jpg?f=community){kind=avatar}
/u/51800/Porsche.png?f=community)
:strip_icc():strip_exif()/u/50286/iRacing_Koen.jpg?f=community)
:strip_icc():strip_exif()/u/387936/crop678dc91459451_cropped.jpg?f=community)
:strip_exif()/u/242341/crop55d72c2ca29b1.gif?f=community)
). 
/u/227108/crop6027919ad38b8.png?f=community)
:strip_icc():strip_exif()/u/485686/crop585156a6a93aa_cropped.jpeg?f=community)
een server opzetten.
/u/495912/cool-tux_big.JPG?f=community)
Ik wou vanavond 2 races doen, maar nu ga ik het er niet op wagen...
:strip_icc():strip_exif()/u/60639/60x60.jpg?f=community)
:strip_icc():strip_exif()/u/507184/crop5a53686e3de75_cropped.jpeg?f=community)
:strip_exif()/u/34748/ferrari_avatar.gif?f=community){kind=avatar}
:strip_icc():strip_exif()/u/12401/crop62ebeb663c6e7_cropped.jpg?f=community)
/u/149597/Belgium-Flag-64.png?f=community)
:strip_icc():strip_exif()/u/329010/avatar%2520helmet%25202014.jpg?f=community){kind=avatar}
:strip_icc():strip_exif()/u/18407/got_thumb.jpg?f=community)
:strip_icc():strip_exif()/u/382285/nieuws2003_seat_kl.jpg?f=community)
/u/375241/213.png?f=community)
Dit topic is gesloten.
