Really simple batch job scheduling / workqueue for Linux via xargs

I just discovered a really simple way to create a workqueue on Linux that someone else might find useful.  If you have a bunch of jobs to run that all require different parameters and that potentially take different amounts of time to complete, it’s difficult to schedule them in a way that makes maximum use of the available cores short of using some sort of batch scheduling system, which is overly complicated for a lot of prototyping purposes.  It turns out that the xargs command has builtin workqueue scheduling that is really easy to use.

basic syntax (assuming you want to run the program ‘command’ with one parameter, and that you want to have four processes running at any one time):

echo param1 param2 param3 param4 param5 param6 | xargs -n1 -P4 command

will run

command param1 &      # & => background
command param2 &
command param3 &
command param4 &

then when the first command of those four completes, it will run

command param5 &

then command param6 &, etc.

If your command requires two parameters, do:

echo param1a param1b param2a param2b [etc.] | xargs -n2 -P4 command

If you have a quad-core processor with hyperthreading, you could do -P8, etc.

You can also obviously store the params in a file and do cat file | xargs … .

The nice thing about this approach over batch scheduling for prototyping is that if you hit Ctrl-C, it kills all the child processes.

I haven’t experimented yet to find the optimal way to generate a separate logfile for each child process, but I also just discovered PPSS which is a more powerful system for achieving the same thing as xargs, and supports separate logfiles: http://code.google.com/p/ppss/

I hope this is as useful to someone else as it is going to be to me!!

Balloon Boy

So I can’t stop laughing about Balloon Boy… some of the dialog between TV commentators while this was going on is just hilarious.

Be the first to buy my Balloon Boy design on Zazzle:

This design by somebody else is great too

Connecting the Dots: Verizon and Google — *much* bigger than a couple of phones

The Web is buzzing today with the news that Google and Verizon are entering a partnership to release Android-powered cellphones on the Verizon network, with the first two phones being released this year.  I suspect there is a lot more going on to this deal than just another carrier signing on to carry Android handsets.

Eric Schmidt, left, CEO of Google, holding the HTC Hero, and Lowell McAdam, CEO of Verizon Wireless, holding the Motorola Sholes (both unreleased)

Anyone remember the airwave auctions?  A recap: with Analog TV expiring in the US due to federal mandate, the 700MHz frequency band was about to be vacated.  This frequency band is particularly valuable because it can travel through thick concrete walls with comparative ease, bounce off interfaces between atmospheric layers to travel over mountains, and travel long distances without significant atmospheric attenuation.  Google threw down the gauntlet by offering to place an initial bid of $4.64B for a block of the spectrum if the FCC would enforce four specific openness provisions upon that block of spectrum: open applications, open devices, open services and open networks.  Verizon bid more than $9M and won most of the the coveted “C block” and parts of the A and B blocks, and AT&T mopped up most of the rest of the available spectrum.  Google seemed to have “lost” the auction by being vastly outbid by Verizon — but really emerged as one of the biggest winners, because they never intended to buy the spectrum, just to have the openness provisions enforced.  Google announced their intentions and the openness provision on 20 Jul 2007, they unveiled the Android platform on 5 Nov 2007, and the auction closed on 28 Jan 2008.  Verizon (after initially fighting Google over the openness provisions, at least publicly) started displaying a commitment to openness right around the time they won the license block, and that openness has continued with the current press release:

On a conference call with analysts and journalists Tuesday, Lowell McAdam, Verizon Wireless’s chief executive, said the first two Android-powered phones would be available this year. He also said that they will include Google Voice, a calling application that generated controversy when it was rejected for Apple’s device.

“You either have an open device or not, and this will be open,” Mr. McAdam said.

Eric Schmidt, Google’s CEO, hailed Verizon’s data network and scale and said that the carrier’s openness “was, frankly, enormously surprising, given the history and the old-line nature of telcos.”

He added: “In Verizon, somehow, the leadership has decided to embrace a very different philosophy, which works very, very well with the Internet.”

Fast forward 9 months to 25 Sept 2008, and another intriguing piece of the puzzle emerges: a patent, originally filed by Google in March 2007, is published publicly, and claims

A method of initiating a telecommunication session for a communication device include submitting to one or more telecommunication carriers a proposal for a telecommunication session, receiving from at least one of the one or more of telecommunication carriers a bid to carry the telecommunications session, and automatically selecting one of the telecommunications carriers from the carriers submitting a bid, and initiating the telecommunication session through the selected telecommunication carrier.

This is simply one of the most brilliant, industry-disrupting ideas to emerge in a long time, and it perfectly dovetails with Google’s planned openness provisions.  The patent applies what Google has learned through AdWords to the economics of mobile communications.  Most importantly, this patent indicates Google’s desire to innovate in the wireless space, starting with upending current business models.

One more important point to take into account: rumors have abounded for a long time that Google is building, or planning to build, a nationwide wireless network of some kind.  These rumors are pretty unsubstantiated, and don’t fit with Google’s usual patterns of behavior (giving away for free potential income streams just to promote greater use of the Web and therefore greater income from ads: giving away Gmail, PicasaWeb, the Android platform etc., not taking a cut on Android Market purchases, and not even forcing Gmail-over-IMAP users to view ads).  The rumors are also overly short-sighted for Google’s usual pattern of world domination: Google is not likely to build just a nationwide wifi network, that would be far too short-sighted in several different ways…

And now we have the announcement between Verizon (which, incidentally, is still not a member of the Open Handset Alliance, perhaps just a technicality) and Google:

Integral to this agreement is a commitment by the companies to devote substantial resources to accelerate delivery of leading-edge innovation that will put unique applications in the hands of consumers quickly. The two industry leaders will create, market and distribute products and services, with Verizon Wireless also contributing the breadth of its nationwide distribution channels. Consumers will be able to purchase products resulting from the collaboration in Verizon Wireless retail and online stores.

So, in brief, we have:

1. Google raises the stakes on an auction of the perfect frequency-band for next-gen wireless communications, and gets the FCC to enforce openness provisions.
2. Google unveils the Android open handset platform.
3. Verizon wins the auction and expresses its commitment to openness.
4. Google applies for a patent on open wireless communications (specifically for playing off wireless providers against each other), indicating a move to start innovating in the wireless space.
5. Google has “probably” been interested in building a wireless network of their own, of some kind, for a long time.
6. Google and Verizon announce that they are entering into a partnership together to develop next-generation mobile devices, products and services.

All that’s left is to connect the dots: does anyone other than me think this all points much more going on than just the release of Android phones on Verizon’s network?

Hint: when somebody thinks that Google is doing something, they’re almost inevitably right about the fact that Google is doing it, but they’re completely wrong about the scope of it — Google always thinks much bigger than the rumor-mongers can imagine.  (Case in point: compare all the gPhone rumors that were out there for years with what Google actually had in mind with Android…)

 

 

 

Please donate to support continued development of awesome features for Android (like multitouch zoom)!

SU article

A fairly decent short piece on Singularity University on CNET.

Singularity University: “South Park meets Harvard Business School”

South Park meets Harvard Buliness School, article in FT.

Moore’s Law — law of nature or artifact of belief?

“Secondly, even a cursory glance reveals the astounding regularity of Moore’s line. From the earliest points its progress has been eerily mechanical.  Without interruption for 50 years, chips improve exponentially at the same speed of acceleration, neither more nor less. It could not be more straight if it had been engineered by a technological tyrant. Yet, we are to believe that this strict nonwavering trajectory came about via the chaos of the global marketplace and uncoordinated ruthless scientific competition. The line is so straight and unambiguous that it seems curious anyone would need convincing by Moore and Mead to “believe” in it. The question of faith lies in whether one believes the force of this “law” lies within the technology itself, or in a self-fulfilling social prophecy. Is Moore’s law inevitable, a direction pushed forward by the nature of matter and computation, and independent of the society it was born into, or is it an artifact of self-organized scientific and economic ambition?” — The Technium

Parachutes and exponential curves

“You have to learn to make the parachute on the way down” — Peter Diamandis at Singularity University describing the need to extrapolate the current trends and start building something that requires a piece of technology that won’t exist for 5 years, but that you absolutely count on existing at that point in time.

(My new favorite quote.)

“They shall laugh and reach out their hands amidst the stars”

Great Singularity University-themed quote: “It is possible to believe that all the past is but the beginning of a beginning, and that all that is and has been is but the twilight of the dawn. It is possible to believe that all that the human mind has accomplished is but the dream before the awakening; out of our lineage, minds will spring that will reach back to us in our littleness to know us better than we know ourselves. A day will come, one day in the unending succession of days, when beings, beings who are now latent in our thoughts and hidden in our loins, shall stand upon this earth as one stands upon a footstool, and shall laugh and reach out their hands amidst the stars.”  –H.G. Wells

(Thanks, Michael!)

Singularity University students named

Singularity University is finally moving out of stealth mode — the list of students of the Graduate Studies Program Inaugural Class (GSP-09) has just been announced.  The media will be invited in to come meet the students over the next few weeks.

Singularity University: the first three weeks in pictures

Keith Kleiner posted some photos and description of the first three weeks of SU.