Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Businesses Novell Open Source SuSE The Almighty Buck United Kingdom Linux News

London Stock Exchange Price Errors 'Emerged At Linux Launch' 168

Posted by timothy from the hiccough-hiccough-hiccough dept.
DMandPenfold writes "Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources. Thomson Reuters, Interactive Data and Netbuilder are among the largest data vendors, providing share prices to traders, that have been displaying pricing problems on some stocks throughout the week. Even the LSE's own data vendor, ProQuote, experienced problems. Concerns are being raised that there could be mistakenly setup connections or incorrect software interfaces at some of the large data vendors. Alternatively, there may be a data caching issue at the LSE that means data going out is not properly synchronised between different systems."
This discussion has been archived. No new comments can be posted.

London Stock Exchange Price Errors 'Emerged At Linux Launch' More

London Stock Exchange Price Errors 'Emerged At Linux Launch'

Comments Filter:

  • Testing? (Score:3)

    by Tubal-Cain ( 1289912 ) on Saturday February 19, 2011 @03:08AM (#35252370) Journal

    Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources.

    Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.

    • Re:Testing? (Score:5, Informative)

      by PNutts ( 199112 ) on Saturday February 19, 2011 @03:15AM (#35252394)

      Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.

      From TFA: "Meanwhile, the private view at the exchange is that the 15 month testing window should have been plenty of time for the vendors to interface perfectly with it."

      • Re:Testing? (Score:5, Informative)

        by cronius ( 813431 ) on Saturday February 19, 2011 @04:00AM (#35252528)

        From this article a couple of days earlier http://www.computerworlduk.com/news/it-business/3261625/frustrations-mount-over-london-stock-exchange-data-interface-problems/ [computerworlduk.com] :

        Publicly, the LSE and the data vendors say they are working together as engineers scramble to fix the price data problems, even though some statements to clients suggest elements of blame.

        But behind the scenes, sources close to the several of the parties, including the exchange, told Computerworld UK they were immensely frustrated at the reputational impact of the problems.

        The LSE is taking the position that its data feeds are working correctly. Industry sources said the exchange had placed a great deal of emphasis on the launch, which was largely providing successful high-speed trading, and that it had allocated sufficient time - 15 months - for the vendors to be fully prepared for the new system.

        The LSE statement appeared to place some blame with the vendors. "Unfortunately a couple of market data vendors have experienced some specific issues aligning to the new Millennium Exchange platform and we are actively working with them to help resolve their issues," it said.

        "All other trading customers and vendors are successfully trading on the new platform, benefiting from Millennium Exchange's superior functionality and speed."

        Emphasize mine. Looks to me like the exchange is getting the flack for a couple of amateurish (or saboteurish?) vendors.

        • Re: (Score:1)

          by Yvanhoe ( 564877 )

          Emphasize mine. Looks to me like the exchange is getting the flack for a couple of amateurish (or saboteurish?) vendors.

          So you mean they asked their vendors to change a communication protocol and never tested before day 1 ? Honestly, that not how you test such a development. Sure, one can blame vendors for not correctly implementing a spec, but this is regular, expectable error in the software world. The real mistake is the lack of a test with the various sources.

          • Emphasize mine. Looks to me like the exchange is getting the flack for a couple of amateurish (or saboteurish?) vendors.

            So you mean they asked their vendors to change a communication protocol and never tested before day 1 ? Honestly, that not how you test such a development. Sure, one can blame vendors for not correctly implementing a spec, but this is regular, expectable error in the software world. The real mistake is the lack of a test with the various sources.

            Huh? Did you skip the earlier sentence?

            The LSE is taking the position that its data feeds are working correctly. Industry sources said the exchange had placed a great deal of emphasis on the launch, which was largely providing successful high-speed trading, and that it had allocated sufficient time - 15 months - for the vendors to be fully prepared for the new system.

            I think he means it was available for test for 15 months. I don't think the LSE can force the vendors to use the new protocol, but giving them 15 months to do so seems fairly reasonable. On the other hand, it could be that the quantity of testing was fine, but the quality (as in how accurate with respect to important things like scaling) was poor. You can test for years, but if your testbed doesn't look anything like the real environment, what are you really testing?

        • Re:Testing? (Score:5, Insightful)

          by CaymanIslandCarpedie ( 868408 ) on Saturday February 19, 2011 @07:07AM (#35253000) Journal

          I think you can take off the tinfoil hat on the sabotage thoughts. For one LSE's own quote providing solution is experiencing the same issue. So unless they have some diabolical plan to sabotage themselves, I think that fact should rule that out. Also, this is effecting large players. These are billion dollar companies whose sole propose is to provide accurate data and provide it fast. Without that they simply don't exist. You'd have to live in a very conspiracy theory driven existence to think they'd all really all throw their own companies and livelihoods away in some dark plan to get the LSE or Linux.

          Now would all these large players all have made the exact same errors in their interfaces so they are seeing these same issues while the smaller players seem to have got it right? Sure. But the simplest explanation, of course is they are all experiencing the same issue because of some upstream problem with scaling to their large volumes. Those actually involved seem to point to some caching issue. Is that true? No clue, but they are certainly in a better position to offer analysis of the issue than you or I. That doesn't mean Linux is bad for christ's sake (so you don't have to go on some crazy conspiracy hunt to explain it away). If it is true (and we certainly don't know that it is, though the information currently available may point that way), you know what? There is software which just happens to be running on Linux, which may have an issue. No big deal. Happens every day on every platform. Not some big black eye on Linux if some software that happens to run on Linux has issues and no need to start jumping to conspiracies to explain it away.

          • Also, this is effecting large players

            Really? I hadn't heard about that aspect of the story. So previously there were only small players, but the effect of this bug is causing large players to be created. Gosh, I hope they are not also affected by the problem.

        • Re: (Score:2)

          by spydum ( 828400 )

          This is a common problem when you operate infrastructure that interfaces with multiple vendors. Sometimes you just have to forge ahead and fix the problems as they come up. I tend to agree with your assumption: lazy vendors not getting on the ball and doing their share of the legwork to stay compatible with the upgrade, who then spun stories about how the LSE's new system is "broken".

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        First off, you can have 15 YEARS To test, if you are understaffed and underskilled you will discover jack squat.

        Second, THIS news comes out three days after Slashdot wrote what amounts to a slam piece on .Net's failures to handle this business. Hmm, what happened here? These problems are noted in the title of the article itself as emerging after Linux/C++ solutions were brought to bear.

        Lets face facts, the core of EVERY issue (the failures in the .Net system that led to its replacement, the failures we're w

    • Re: (Score:3)

      by Davorama ( 11731 )

      Indeed, but please stop calling me Shirly.

    • If only they had thought of that (Score:5, Funny)

      by raftpeople ( 844215 ) on Saturday February 19, 2011 @03:46AM (#35252488)

      Surely they would have run test data before letting it go live. Maybe even feed it the actual data and simply not publish its results.

      Quote from the project manager:
      "Dang. If only we had read Tubal-Cain's post before going live. Who would have thought to run test data through this darn thing?"

    • Re: (Score:3, Funny)

      by Anonymous Coward

      When Apple released a phone they hadn't tested and it turned out "holding it a certain way" prevented calls, it suddenly changed to "holding it wrong", thus blaming the user. No scope for that here?

    • How did it work before? Did the old system run with zero errors?

    • Unless they had bosses that were under the gun and willing to take that risk, and no matter how many time the architects told them not to do it, they said cut the corners and let's get it out there....happens to me all the time, and if it fails, guess who gets thrown under the bus.

  • Off (Score:3, Funny)

    by Anonymous Coward on Saturday February 19, 2011 @03:18AM (#35252402)

    Turn it off and on again.

  • either sympathy or accusation (Score:5, Insightful)

    by DreadPirateShawn ( 1246208 ) on Saturday February 19, 2011 @03:22AM (#35252414)
    My heart goes out to the devs "working long hours and night shifts" to suss this out.

    That being said, the line that catches my eye most is: "The fact the majority of smaller vendors were fine demonstrated that those having trouble had made mistakes." In my experience, that means one of two things:

    1) The devs configuring the system didn't properly account for the sheer scale of the stress on their systems.
    2) The smaller vendors took the change more seriously, and being smaller and more flexible, successfully updated their systems to interact properly with the new systems.

    Or, of course, both.

    • Re:either sympathy or accusation (Score:5, Interesting)

      by micheas ( 231635 ) on Saturday February 19, 2011 @03:39AM (#35252464) Homepage Journal

      Or the big vendors were doing excessive cost cutting to maintain profits during the recession.

      As long as this is just a one weekend nightmare, the execs that ordered the QA not done are looking like great business monkeys. they took a risk that cut costs and managed to defect the blame to the exchange when it blew up, and it looks like nothing bad will happen to the big vendors, unless one of the small vendors gets a really unethical sales monkey that steals a large chunk of business from a couple of the big vendors.

      • Re: (Score:1)

        by Anonymous Coward

        Unethical? Why couldn't they steal lots of business from the big guys by showing their customers that they handled the transition properly while the big guy botched it? Why would unethical sales be needed to steal business from an apparently incompetent incumbent?

    • Re: (Score:1)

      by fatp ( 1171151 )
      Maybe the smaller vendors have fewer customer so there is smaller chance to detect the problem.

      Also, it said "the majority of smaller vendors", not all smaller vendors were fine.

    • Re:either sympathy or accusation (Score:5, Interesting)

      by Anonymous Coward on Saturday February 19, 2011 @09:35AM (#35253334)

      As I work for one of the vendors cited in the article, I can assure you that the reason is freaking simple: the systems that consume the feeds to move them into the trader's desktop are absolute mess of epic proportions riddled with mind boggling wtfs. Missing data, blank values, wrong numbers are common on those systems (root cause are mainly race conditions everywhere, and assumptions about missing data), assembled from random left and right acquisitions. There is no way to really know how they will behave under stress, and corporate culture will not allow managers to actually fix the issues (extremely costly, and corporate bonuses are not aligned to quality).

      In fact, it is much cheaper for everyone to just pretend this is a one time occurrence, to fire a couple of middle managers and continue frantically fixing bugs in the mess until it sorta works.

      • They typically have decent to excellent central systems, and it feels like they're treating it seriously and spending lots of money to maintain and upgrade them.

        Now at the periphery ... the interconnections, for electronic payment and so on, they're a disgusting mess. It doesn't help that some protocols are truly horrible and should have died 40 years ago, but even when they've moved into the IP era, they keep on using outdated shit and/or idiotic settings.

      • I think we'd all agree that a crappy app atop a good OS can lead to a crappy experience. Ergo, a crappy experience does not imply a crappy OS.

    • Those poor devs, who are earning 500-800 pounds a day. My heart goes out to them too.

      • Re: (Score:2)

        by Svartalf ( 2997 )

        Really? I thought it more along the lines of 270 pounds or so. That was about what I was getting paid when I worked for one of those companies doing this. Of course, that's converting it from the $440 a day I was making there in the States.

        • Re: (Score:2)

          by Jayson ( 2343 )

          800 pounds seems about right. $440/day is only $88,000/yr. I made $480,000 last year doing HFT core and strategy coding. Of course my salary is technically only $100,000/yr and the rest is in bonus. I'm hoping to crack $650,000 this year since my strategies are producing better PNL and my Sharpe ratio is stratospheric.

          I will never understand the Slashdot and geek dislike of HFT. It pays coders more than any other industry, and you get exactly what you deserve. But I guess a lot of people here are lazy tool

    • Let me add another (unfortunately realistic) possibility to your list:

      1) The devs configuring the system didn't properly account for the sheer scale of the stress on their systems.
      2) The smaller vendors took the change more seriously, and being smaller and more flexible, successfully updated their systems to interact properly with the new systems.

      3) Documentation provided to vendors is either incorrect or sufficiently vague that some software shops interpret the spec incorrectly and end up depending on t

  • Quote Services (Score:5, Informative)

    by Anonymous Coward on Saturday February 19, 2011 @04:04AM (#35252536)

    Article is a a bit misleading....
    Reuters, Netbuilder et al are quote services that run externally and fetch quotes (price/volume) from the LSE and are *not* part of what would be considered the main trading platform. The problem isnt the actual OS (Linux) or tradiing platform, which has been heralded as a big step forward from the MS solution, but rather how the resulting trade data that is pushed from the exchange to the 3rd party sites and then ultimately to the end users.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Article is a a bit misleading....
      Reuters, Netbuilder et al are quote services that run externally and fetch quotes (price/volume) from the LSE and are *not* part of what would be considered the main trading platform. The problem isnt the actual OS (Linux) or tradiing platform, which has been heralded as a big step forward from the MS solution, but rather how the resulting trade data that is pushed from the exchange to the 3rd party sites and then ultimately to the end users.

      BTW... these protocols while well documented are complicted. To get an idea, check out either the NYSE ArcaEX Direct API [nyse.com] or the NYSE Arca FIX specs [nyse.com]

      If someone could point out the LSE equivalent, I'd be interested in seeing the differences

      • Re:Quote Services (Score:5, Interesting)

        by Anonymous Coward on Saturday February 19, 2011 @05:11AM (#35252740)

        Hi,

        The NYSE Arca FIX docs are not quite relevant to this - as what we're talking about here is the market data API.

        The FIX protocol (although it does support market data transactions), is typically not the protocol of choice for market data. Typically it'll be non standard provider specific protocols which are optimised for delivering partial updates of information in a very bandwidth efficient manner.

        In the parent post someone mentions that Reuters, Netbuilder etc fetch quotes - which is also not entirely correct either. They accept push feeds from the exchange, which contain tick updates to the information about the instruments traded on that exchange. The generation of these ticks is triggered when a match occurs on the order book - which may (or may not) involve a change in price data. They then typically augment this information with other data - such as other identifiers, or derived information such as previous days close value / percentage etc.

        It is with these derived fields that the ongoing issues seem to lie, although for a certain period of time, the exchange was actually pushing out zero values for the bid/offer fields. As you can imagine, these plays merry hell with systems that attempt to drive automatic execution based on current price.

        I am informed by Netbuilder that they will have pushed a final fix to live within a subset of their servers - but it will probably be Tuesday before this is available across their whole system. This is over a week since the original identification of the issue, not good!

  • I thought I'd stuffed up big time when I hooked all of our users (around 1000 users at the time) up to test market prices rather than live data (Australian Stock Exchange) over market opening one morning (our product is aimed at day traders too). My stuff up seems minor in comparison to this.

    • I of course don't make mistakes, but what do you think would happen if for example you wiped 8.5 billion dollars off of the value of your company?

      http://finance.yahoo.com/q/bc?s=NOK+Basic+Chart&t=3m [yahoo.com]

      Do we have a world record here Mr. Elop?
       

      • The drop in stock price after the announcement of the Nokia Microsoft venture was sharp for a single day. It's also one of the highest volume days in the last 5 years.

        However, the drop is insignificant compared to the drop from late 2007 to now. Market Cap at Nokia is 1/4 what it was. Stock prices have dropped from mid 40's / share to ~ 10/share.

        Besides, the 8.5B drop in market cap isn't nearly as important to the bottom line as the billions in cash coming in from Microsoft as part of this deal.

    • What happened to you after you did that? Reminds me of one time when our project manager (he was a decent programmer, too) was digging around our Oracle database install and found some useless 'backup' files and just deleted them. Somehow immediately after that, all customer data was gone. The ironic thing is we were an automatic backup company, and we didn't keep any backups.
  • Data is always seems to be the least respected item in the IT stack (TBV). Yet its the lifeblood. Referential integrity, blah. At your peril.

  • The offending code was traced back to :

            $stockprice = power( 2, rand()*10 + 0.0001 );

    Data vendors quickly addressed the issue by cutting the LSE out entirely, and rolling their own rand() function.

    Fucking stock markets... .what a joke!

    • You got that wrong... they use SuperRand()...

      http://thedailywtf.com/Articles/SuperRand.aspx [thedailywtf.com]

    • $stockprice = power( 2, rand()*10 + 0.0001 );

      I considered arguing with your example on the basis of the presence of floating-point (prices are discrete - any sane market uses fixed-point integer arithmetic). However, since you did not provide a complete running program I have to admit that it's possible that your code is C++ and rand() is a function that returns an object, and the object's member function operator+(float) simply returns *this as a red-herring to confuse the interns.

  • On the other hand, a moon mission interfaced a host of complex systems including the lives of the astronauts. Difference is the level of integration of QA into the project.

  • Effect on stockbrokers (Score:5, Informative)

    by Anonymous Coward on Saturday February 19, 2011 @05:02AM (#35252716)

    As someone on the receiving end of this problem (as head of Development for one of the largest UK stockbrokers), it has been immensly frustrating trying to get an admission of blame from anyone involved in this issue. Our clients are complaining to use because elements of the data we're displaying on our site are just so out of whack they are laughable. In the end we've just had to remove those elements of data from our website, and they remain removed until the fix is confirmed working on Monday. This is after several attempts to fix the issues during the week.

    It's also causing serious issues for things like our black box trading engine which is used to drive automatic execution of client orders when the price moves into range for execution. The data being bad means these systems just have to be turned off, and clients are being advised to simply delete their orders.

    This is, frankly, a pretty appalling situation as the service to our clients is being impacted pretty severely.

    For an exchange that is attempting to position itself as the true 'global' exchange, this is doing an immense amount of damage to it's reputation.

    • As with most major issues there's bound to be a big ol' postmortem on this. As head of Dev you've probably got a unique insight into this, I'm curious as to your perspective on this, what you think the cause of failure might be? More strategic or more technical? Poor interface specification? Inability to handle queries under full load? From TFA there was supposedly 15 months of testing. So I'm curious as to why it failed, whether the testing simply wasn't realistic and/or thorough enough, or it was so

    • Re: (Score:2)

      by kievit ( 303920 )

      Did you check your data during the 15-month testing period, and were the errors also apparent during that time? If yes, did you report the issues or take other actions to get the problems fixed during the testing period?

      If you did not check your data during the testing period: why not?

  • is the concept of "high speed trading",
    This horsecrap has to stop, the stock market has to be dismantled

    • the stock market has to be dismantled

      Pull your money out of the stock market and convince all of your friends to do the same. Find a way to invest your money in a way that you feel is proper. If enough people reject the market in its current incarnation, change will be forced. If not, at least you know that you're doing what you can.

      Just don't get upset when many people disagree with you. We all choose our own poison.

  • It's only a stock exchange. Why the fuck should they bother testing it first, right? This just goes to underline that QA does not exist in the software world.

    • They did. They had 15 months of/to test. It's right in the article

      • Re: (Score:2)

        by Dunbal ( 464142 ) *
        We have a different definition of testing. Testing != compile it and see if it runs. Testing != hiring some minimum wage loser to tell you exactly what you want to hear.

    • Re: (Score:2)

      by Zoxed ( 676559 )

      > This just goes to underline that QA does not exist in the software world.

      My 24 years in the software industry tell me that QA is available: the customer just has to be prepared to pay for it !

    • Re: (Score:2)

      by jo_ham ( 604554 )

      It's only a news discussion site. Why the fuck should they bother reading the article before commenting on it? This just goes to underline that reading comprehension is non existent in the slashdot world, yet still they believe they have all the facts and can pass judgement.

  • what does that have to do with Linux? (Score:4, Insightful)

    by t2t10 ( 1909766 ) on Saturday February 19, 2011 @06:44AM (#35252940)

    If it didn't crash and didn't drop its network connections, Linux was doing its job.

    If the application software had bugs, then the application software developers are to blame.

    • Re: (Score:3, Insightful)

      by $pace6host ( 865145 )

      If it didn't crash and didn't drop its network connections, Linux was doing its job.

      If the application software had bugs, then the application software developers are to blame.

      Even if the OS itself isn't at fault, don't think that someone with an axe to grind won't blame it. You'll just hear something like "It's Too Hard to develop on Linux. A platform with more / better tools would have made it easier to develop and comprehensively test." I don't buy that either (regardless of what price you quote me! ;) ), but I'm sure someone will be saying it. Especially someone in marketing.

    • Re: (Score:3, Insightful)

      by Blakey Rat ( 99501 )

      But when they had problems with the Windows-based system, obviously it was the fault of the operating system! Right? I seem to remember that being the case.

      Because Slashdot ain't anything if not hypocritical!

  • Nothing to do with Linux (Score:3)

    by Wowsers ( 1151731 ) on Saturday February 19, 2011 @06:52AM (#35252970) Journal

    An attempt to rubbish Linux.

    After 20 seconds they should have realised that they should have tested more, and fired the programmers for allowing such mistakes. This is nothing to do with Linux.

    You'd think the LSE would have learnt from their last computer system rollout which also had massive problems. No quality control = management problem.

    • This is nothing to do with Linux.

      Yes, but can you imagine if the new system had a part of the MS stack running it? Slashdot would be full of people casually blaming that, instead of the app/dev side of things. You know it's true.

      • Re: (Score:2)

        by cynyr ( 703126 )

        Well, is it was caused by bugs in .Net's XML tools? sure MS's fault. Remember that linux is just a kernel, and the rest of the GNU software may or may not be running on this beast and most of the important software(the push server thingy) is likely to be custom in house stuff.

    • The lack of testing, in a corporate setting, is not the programmers fault, it is a management failure and as such the managers should be fired.

      • Indeed. If the coders are anything like myself, they are doing one of two things ; cringing uncontrollably because a piece of code that they wrote is causing problems. Or genuinely not giving a rats arse because they were ordered not to do an appropriate level of testing by the immediate management who feared that they would miss a deadline.

  • That quoted phrase 'emerged at linux launch' is NOWHERE in the article. The article says "London Stock Exchange price data failures ‘emerged immediately at Millennium launch’". This is obviously an application issue, why is the summary sort of blaming the operating system?

  • LSE is a buerocratic mess (Score:3)

    by alexmin ( 938677 ) on Saturday February 19, 2011 @12:00PM (#35253912)

    No surprise here for anyone working with them. Their marked data specs have hundreds of pages while Chi-X EU have about 20. Chi-X EU quoting and trading volume is far greater. Coincidence? I say no - people voting with their feet and going from LSE to Chi-X EU (biggest trading platform in EU now), BATS EU, and other MTFs. Maybe the only thing keeping LSE still afloat is a fact that FTSE 100 index is computed using LSE prices. Once that gone, potentially as soon as MIFID II comes out, LSE is going to circle in the crapper.

  • This is strictly an application server implementation issue, not a Linux one. They had more (though probably different) problems with their previous MS Server infrastructure, which is, as I understand it, the reason behind the migration to Linux. So, don't tie this fiasco to the use of Linux, but rather to inadequate testing of the new systems in a real-world load scenario. This is really difficult for large scale distributed systems such as this is. I spent a LOT of time and engineering effort to build a t

  • Microsoft has has had a lot of bad press lately as linux and java seem to be eating its lunch in the new world (high-performance, web base computing, small form factor and mobile computing). I suspect these issues are true but exaggerated.

    Remember that the LSE had serious problems using the MS-based system before. Remember the curious wording of the recent problems with the .NET-based system. Somehow, they tried to link this to the fact that the Linux-based system was being tested (of course it had nothing

Slashdot Top Deals

"If it ain't broke, don't fix it." - Bert Lantz

Close