The power of text

After spending 4 or 5 years working on servers with decent amounts of data going through them, I've had to do a fair bit of forensic work with saved databases, giant log files, etc. AWK, sed, Groovy, EMACS, and their kin have been my friends. You can use them to distill unstructured text into a structured form for summaries, trending, etc.

I am starting to turn Ober into a console for this type of forensic work. I can see how farsighted Niklaus Wirth was when he made Oberon. This doesn't look to me like a temporary phase in computing. If anything, unstructured text is even more prevalent today than it has been in the past. Google has shown people what you can do with it. We'll see where Ober goes from here.

For the future, I'm planning on adding some more Wily-like features to Ober, like preceding commands with |, <, and >. Here's an update on Ober...

Ober is becoming usable now. Here are the items I checked off of the list from the last post:

• execute HOME/.oberrc on startup (if it exists)
  
• Scala expressions embedded in {...}
• ctrl-click opens a file. Actually, it tells the namespaces to "surf", so surfing depends on the namespaces your viewer is using
  
• fixed redraw bug

I also added some cosmetic commands, like TrackPos, Width, ViewerPos, and Height.

Using the context-sensitive surfing, I made a "data browser" to examine some object databases I use. It was pretty simple, since I could already dump objects and display the guids of the objects they reference, so when you ctrl-click on a guid, it goes to the viewer for that object (or creates it) where it displays the dump of that object. If you type things or mess up the text and you want to redisplay, you just ctrl-click on the viewer name and it refreshes.

Here is the Ober's current source code and here is Ober's current help page.

A tribute to Niklaus Wirth

If you haven't heard of Niklaus Wirth's Oberon or you haven't seen it, you should check it out. Messing with it reminded me of when I first learned Smalltalk in 1988. There's an air of playfulness and potential to it, just like in Smalltalk-80. The current version of Smalltalk is called Squeak and it's well worth checking out as well.

Oberon is sort of a Pascal take on Smalltalk. Like Smalltalk, it's both a language and an environment. Like Smalltalk, it has built-in programming tools (compiler, editor, etc), and it's interactive; you're programming "in" the language, so the edit-compile-run cycle is VERY tight (like when you're using the scala command and you can evaluate expressions). Oberon's environment was similar to Smalltalk's, but had some major differences and innovations, the most valuable one being (in my opinion) that you can click on a word anywhere in the environment and it will attempt to execute it as a command. There are other innovations too, but that's my favorite.

Harnessing this capability, Rob Pike at AT&T based a text editor/shell on Oberon as part of Plan 9, called Acme. Then, Gary Capell made Wily for Unix. Then, in 2002, I made one based on Wily for Ruby, called Meep. And THEN, I rewrote it in Java and renamed it "Ober," to point back to Oberon (and because ober means "waiter" in German). Somewhere along the line, I don't remember whether it was in Meep or just Ober, I added a "namespace" twist, to allow scoped commands, like Oberon has, but identified in the "tag" of the viewer (the text field at the top of the viewer which contains handy commands).

So. Time for another rewrite. Here's Ober-scala now.



It's two eclipse projects, one Java, and one Scala 2.8 (I'm having trouble with Java and Scala in the same project). To run it, just use the "Scala Ober" launcher that's in the scala project. There's a LOT of room for improvement, of course, but it functions, at least. On Linux. I haven't tested it on Windows. Sorry Windows :(. It's not that I think Windows sucks, it's just that I don't use it and it's pretty inconvenient for me to mess with it on my current setup.

Why did I do this? I wanted a "programmer's workbench" that was a combination program runner, simple editor, and shell. You know, like EMACS, but by Scala and for Scala. And the Oberon environment is very cool. When I read that Martin Odersky studied under Niklaus Wirth, I figured, hey, maybe some Scala people would be interested in this thing too. We'll see.

What can it do? If you righth-click a word, it will try to execute it as a command. At this point, there are 3 types of namespaces:

1. closure namespaces, which look up named closures
2. class namespaces, which look up classes
3. system namespaces, which look up shell commands

So, you can test it by right-clicking on some of the words that are already there when you start it (like New, Newcol, Del, Quit, ...). You can type in the name of a class in your class path and right-click on that (like tc.ober.Test, for instance); it will split the rest of the line by spaces and send the strings to the main method of the class. You can type in a shell command and right-click that; it will execute the word plus the rest of the line as a system command (using Runtime's exec(String) method).

What's left?

• execute .oberrc on startup
• Scala commands embedded in {...}
• file handling (crtl-click on a word and it opens as a file). See Wily for this behavior.
• meta commands for your .oberrc (commands to define commands, etc.)
• classpath manipulation
• fix bugs (like when you delete the last viewer in a track, it doesn't redraw)
  

Happy hacking!

Safe navigaion in Scala, the aftermath

Wow, I sure stirred up a hornet's nest with my safe navigation experiment. Some people posted some very useful information and good alternative approaches!

As for comments about "ugliness", being able to read this in 6 months, etc.; I'm not making a policy decision here, just experimenting. The fact that people are so put off by experimentation makes me wonder... Anyway, we'll see if it comes in handy or not. Things like this can easily be "compiled" by the mind as "programming idioms".

But as for unreadable, it's only a cascade of lambdas, after all. Personally, I've been very comfortable with functional style, since I first learned LISP, freshman year in college. It's amazing to me how emotional some programmers get when they see code using functional style. Does this mean "imperative -> good, functional -> bad?"

I guess there's always been a divide between functional and imperative approaches. I'm glad Church and Turing could see that their different approaches were accomplishing the same thing. I really think more universities should teach functional languages in their freshman level courses, before people have a chance to ingrain imperative as the one true way. My university (Purdue) did not; I just happened to learn it from a friend, because I was interested in what LISP was and how to use it. At the time, I was amazed (and a bit envious) to find out that some universities taught LISP or Scheme to freshmen.

I think LISP and Scheme are excellent first languages, because they abstract computation away from the machine; they don't encourage you to simulate a computer in your head when you write code. Simulating a stack machine can be a great barrier to actually understanding recursion and higher order functions. Freshman and sophomore year, I helped quite a few students learn recursion, but I had to "unteach" the technique of "stack simulation" first, so they actually had a chance of understanding something like A-B recursion or composite recursion.

It's interesting, because I just started teaching a guy to program last week, and I chose to start with Scheme (although eventually he probably needs to learn Java). I found good site with some basic tutorials and a free book on programming that actually teaches recursion the way I taught it freshman year, way back in 1984. It looks like I'm not the only one with those ideas about how to teach recursion.

For those interested, here's the PLaneT Scheme Site and here's the book (linked from their site as well).

Safe navigaion in Scala, take 2

OK, revisiting this topic after a very short time, I have a new implementation, again using that interesting anonymous class technique from James Iry's comment:

implicit def richOption[T](x : T) = new {
 def ??[B](f : T => B):B = if (x != null) f(x) else null.asInstanceOf[B]
}

I changed the operator from "?!" to "??" because that's a lot easier to type and I chose "??" instead of "?" so people wouldn't confuse it with the questi-colon operator from Java and C. It's stand-alone (doesn't depend on Option or anything) and I had to use a cast to get the types to work out properly, but the test case from my last post does work

println(p3??(_.bud)??(_.bud)??(_.bud)??(_.name.drop(1)))

prints null and

println(p3??(_.bud)??(_.bud)??(_.name.drop(1)))

prints ubba.

For those new to Scala, the expressions in parentheses are anonymous functions because they use "_" as identifiers. (_.bud) expands to (x => x.bud), where Scala chooses an x that is unique in the scope of the expression. Of course, you can put whatever you want in the final block there, or the other blocks. You could say this instead

p3??(_.bud)??(_.bud)??(x => println(x.name.drop(1)))

You might want to do this if you only want to print provided that the full path exists for instance, which demonstrates that this construct is more powerful than Groovy's, because each step can be more than just another increment along the path.

This has an advantage over the previous solutions of not requiring the ! operator at the end to extract the value out of the option because the ?? operator returns a raw value instead of an Option. Not having used Scala for more than a week (I haven't even used LISP since the late 80s), I can't speak as an authority on when it's appropriate to use Options, but this seems to me like an alternative to Option for at least some cases.

Safe navigation operators for Scala

Welcome to my first blog post.

I searched the net for a Scala analog to Groovy's safe navigation, but I didn't find anything, so I made my own. It's like a monad but I haven't thought much about the implications of really making it into a real monad so I haven't monadized it at this point.

Here is some example data to demonstrate how you use these safe navigation operators in Scala:

case class Person(var name: String, var bud: Person = null)

val p1 = new Person("bubba")
val p2 = new Person("fred", p1)
val p3 = new Person("mary", p2)

println(opt(p3)?!(_.bud)?!(_.bud)!(_.name.drop(1)))

This code prints "ubba". It uses ?! like Groovy's ?. operator (?. isn't a legal operator in Scala because of the .), but you need the extra parentheses or it doesn't parse like you would want (I think Scala needs the parentheses to disambiguate the scope for the _). The final ! operator returns the closure value instead of wrapping it with an OptionalRef.

If you go one more step, you'll get () as the return value:

println(opt(p3)?!(_.bud)?!(_.bud)?!(_.bud)!(_.name.drop(1)))

That code prints "()". If you don't like the first opt(Any) and you want to do implicit conversions, you can import OptionalRef.Implicit._

So, it's noisier than Groovy, but a lot less noisy than nested "if" expressions.


Here's the code, modified from James Iry's comment:

def notNull[T](x : T) = if (x == null) None else Some(x)
implicit def richOption[T](x : Option[T]) = new {
  def ?![B](f : T => B) = notNull(f(x get))
  def ![B](f : T => B) = if (x isDefined) f(x get)
}

I like that much better than my old, slightly longer, code :)

object OptionalRef {
 object Implicit {
  implicit def obj2OptionalRef[T <: Object](input: T): OptionalRef[T] = {
    if (input == null) NoRef else Ref(input)
  }
 }
 def opt[T](obj: T) = Ref[T](obj)
 abstract class OptionalRef[+T] {
  def isEmpty: Boolean
  def ?![U](st: (T) => U): OptionalRef[U] = {
   (if (isEmpty) NoRef else st(get)) match {
    case r: OptionalRef[U] => r
    case null => NoRef
    case o: U => Ref(o)
   }
  }
  def !(block: (T) => Any): Any = if (!isEmpty) block(get)
  def get: T
 }
 class Ref[+T](x: T) extends OptionalRef[T] {
  def isEmpty = false
  def get = x
  override def toString = "Ref(" + get + ")"
 }
 object Ref {
  def apply[T](obj: T) = new Ref(obj)
 }
 object NoRef extends OptionalRef[Nothing] {
  def isEmpty = true
  def get = throw new NoSuchElementException("NoRef.value")
  override def toString = "NoRef"
 }
}