A disruptive technology for code generation

OK, now I have to support my claim. Here we go.

I've been (ab)using StringBuffer.append() for code generation (like everyone else) while looking for safer alternatives (safer meaning, compile-time type-checked). I've seen all the model-to-text languages. But the rest of this post is not about them.

One way to ensure syntactic validity when generating source code relies on instantiating EMF classes that stand for Java source elements. Verbosity can be reduced using fluent interfaces instead of directly invoking EMF factories, as reported in http://bugs.eclipse.org/234003

But problems remain. Another (perceived?) problem of AST-to-AST is the laborious coding of visitors, pattern matching and rewriting, multi-stage AST transformations, yes, model transformations. As usual, DSLs have been proposed to solve that, too. They are DSLs, i.e., another language to learn. You want to learn them? This way please.

But LINQ is on the march, and it's being used for code generation. It's being used to query the input (which can be even C# API elements, using LINQ to Reflection) and it's being used to piece together the resulting code:

• http://themechanicalbride.blogspot.com/2007/02/do-you-know-linq-fu.html
  
• http://www.interact-sw.co.uk/iangblog/2007/04/03/linqcodegen
  
• http://msdn.microsoft.com/en-us/library/bb546150.aspx
  

As the ultimate test of suitability for AST-to-AST, LINQ is being used to refactor (refactor!) C# ASTs, no less. Compare with the article Unleashing the Power of Refactoring (and that article does a great job of explaining the JDT API for refactoring, so the comparison is *not* biased toward LINQ)

LINQ is not a DSL, its declarative constructs are based on logical commonalities in data access, better factored than those of model transformation DSLs. To my (informed) taste at least. And you're always free to extend LINQ with operators of your own, in a type-safe way that is. Or perhaps what finally convinced me were all the optimizations performed out-of-the-box when evaluating LINQ:

• indexes on in-memory collections, or
• taking advantage for free from multicores, or
• resultsets automatically refreshed on the wake of updates to their base data.

It's a pity we have no Java-based library to execute LINQ queries, queries expressed as ASTs (remember http://bugs.eclipse.org/234003 ?) although dedicated textual syntax would be nice, too. All right, all right, even if you'll settle only on textual syntax there's an interim solution till Java supports lambda expressions.

There's also talk about "LINQ in Scala". What I say is: it's not about syntax! (that's the easy part in fact) What makes a BIG difference are the optimizations supported by the engine when evaluating LINQ queries, input syntax notwithstanding.

If the above strikes a chord with you, let's join forces! I want to get rid of StringBuffer.append()! One of the grassroots movements to implement LINQ in Java is http://groups.google.de/group/jlinq and with some luck we won't need to wait for Java 7+ to enjoy the benefits of language-integrated queries.

P.S. If you need to convince your university department that the above qualifies for a master thesis, just flash the following papers and they'll accept immediately:

• Phil Wadler and Simon Peyton Jones.
  Comprehensive comprehensions: comprehensions with "Order by" and "Group by"
  


• Darren Willis, David J. Pearce and James Noble.
  Caching and Incrementalization for the Java Query Language.
  

(you could also slip in one of my papers ;-)

• Garcia M., Möller R.
  Incremental evaluation of OCL invariants in the Essential MOF object model
  

Of course there are other opinions on best practices around code generation. But I hope you'll be reminded of LINQ when tracking down some obscure bug due to StringBuffer.append() :-)

Miguel
http://www.sts.tu-harburg.de/~mi.garcia

Big revamp of Chapter 1 ("Frameworks for text editors: JFace Text and Gymnast Runtime") of the project report. With that, a user of the IDE generator is also in a position to *understand* the generated code. Features are classified, code snippets and references to further literature given.

----------------------------------------------------------------------

The features per category are:

a) The Gymnast Runtime extension to the JFace Text framework

• background parsing and notification of changes to the CST of the document
• context outline (with nodes implementing navigation back to the text region, and property source)
• custom problem markers
• parse-tree view

b) Customizations requiring CST or AST

• hovers over text fragments
• hovers over annotations (annotations on the vertical ruler)
• hyperlinks
• mark occurrences
• select in outline
• show in
• context menu on the editor surface, for example Go to Declaration
• views, for example EMF Type Hierarchy
• content formatting
• content assist

c) Customizations directly working on the document text

• document partitioning
• show range indicator
• bracket matching (brace, parenthesis, etc.)
• AutoEdits (for example, SmartBrace: after typing { in the JDT editor, an indented newline and a closing } are added below it). Another example is AutoIndent.
• Double-click strategy (in JDT, double-clicking an opening brace selects all the text up to the matching closing brace)
• Automatic Indentation

d) Document-independent customizations

• templates, including associated Preferences page
• actions on the toolbar and menu bar, in particular retargetable actions (i.e., those actions whose UI activators are shared between different editors, such as e.g. Content Assist)
• new file wizard, so that the document does not start empty but in syntactically valid state (thus avoiding spurious parse errors the moment background parsing kicks in)
• Preferences
  

----------------------------------------------------------------------

It's incredible the amount of detail hiding in each feature ... an Eclipse-based text editor can really be a machine with lots of moving parts ...

http://www.sts.tu-harburg.de/~mi.garcia/SoC2007/draftreport.pdf

cheatsheet

1) Bugzilla for SoC projects

https://bugs.eclipse.org/bugs/show_bug.cgi?id=186907



2) Eclipse Legal Resources

http://www.eclipse.org/legal/



3) Artifacts

1. a feature that builds and exports fine
2. update a common site project with its feature,
3. making sure the site exports fine with its and all other included features

4) cvs at soc.eclipse.org or cvs/svn at sf.net


5) guideline for structuring cvs repo

/cvsroot/soc/org.eclipse.soc.ewi/
plugins/
org.eclipse.soc.ewi
features/
org.eclipse.soc.ewi-feature
org.eclipse.soc.ewi.sdk-feature
tests/
org.eclipse.soc.ewi.test
org.eclipse.soc.ewi.test-feature
doc/
org.eclipse.soc.ewi.doc
org.eclipse.soc.ewi.doc-feature
examples/
org.eclipse.soc.ewi.example
org.eclipse.soc.ewi.example-feature

I've completed this week the "grammar to Ecore-model" converter plugin, and had even time to write a chapter about it in the (draft) progress report for my Google SoC project :)

A small experiment in diagram-layout-from-AST

If the previous post sounded too vaporware to you, this time you'll see code snippets for a view that auto-layouts the EPackage contents of a selected EClass, for example in the Ecore Sample Editor. OK, right away I want to mention that this code is my HelloWorld 0.1 attempt at layouting. If you're already into programming diagram editors, stop reading this post!!!



Most of the setup code for the view in question comes from a plugin to visualize OCL Abstract Syntax Trees, an article about which you can find here.

Instead of using a TreeViewer, this time I'm using a GraphViewer, courtesy of Zest, the Eclipse Visualization Toolkit. Again, I'm using Zest in a most basic manner, you'll see the moment you visit Zest's homepage that it's way more powerful than what my example conveys.

In a lucid moment I realized that in order to customize the figure to display (a class figure) an IStylingGraphModelFactory should be provided, like this one:

public class EcoreGraphModelFactory extends GraphModelFactory {   public EcoreGraphModelFactory(AbstractStructuredGraphViewer viewer) {     super(viewer);     // TODO Auto-generated constructor stub   }   @Override   public GraphNode createNode(Graph graph, Object element) {     GraphNode gn = super.createNode(graph, element);     if (element instanceof ENamedElement) {       IFigure cf = createFigure(element);       Dimension s = cf.getSize();       gn.setCustomFigure(cf);       gn.setSize(s.width, s.height);     }     return gn;   }   private IFigure createFigure(Object element) {     IFigure res = null;     if (element instanceof EClass) {       res = createEClassFigure((EClass)element);       // res = UMLExample.createClassFigure1();     }     return res;   }      private IFigure createEClassFigure(EClass eC) {     Font classFont = new Font(null, "Arial", 12, SWT.BOLD);     Image eCImage = Activator.getImage(eC);     Label classLabel1 = new Label(eC.getName(), eCImage);     classLabel1.setFont(classFont);     UMLClassFigure classFigure = new UMLClassFigure(classLabel1);          for (EAttribute eA : eC.getEAttributes()) {       String str = eA.getName() + " : " + GenericsUtil.getText(eA.getEGenericType());       Label attributeL = new Label(str, Activator.getImage(eA));       classFigure.getAttributesCompartment().add(attributeL);     }          for (EReference eR : eC.getEReferences()) {       String str = eR.getName() + " : " + GenericsUtil.getText(eR.getEGenericType());       Label attributeL = new Label(str, Activator.getImage(eR));       classFigure.getMethodsCompartment().add(attributeL);     }          classFigure.setSize(-1, -1);     return classFigure;   } }

All in all, the plugin structure is as follows:


You can find (at your own risk, no support, thanks) the Eclipse project in question in this zip file.

I don't even dare call it a "proof of concept", but anyway I felt some code should be thrown in to support my argument.

The IDE generator I'm working on should provide an extension point to let querying the current AST, to support this and other use cases. Yes. I think so.

diagram layout out of textual representation

How to cook your own DSL language. That’s what most DSL tool suppliers want you to do right away, don't they? I say: no problem, but please notice there’s already a plethora of languages (and their notations) that you could leverage, gain experience from, and then move on to design your own brainchild. Examples follow.

Consider statecharts. Diagrams are nice to browse, but hard to edit. Let’s say we have an IDE for a textual notation of statecharts. I like this comparison [1]:


Which is faster?

textual Environment:
1 move cursor to position
2 type “|| await C”

graphical Environment:
1 make room: shift neighbor states, enlarge parent state
2 click on “add state”
3 move mouse to location and place new state
4 click on “add state”
5 move mouse to location and place new state
6 double click on new state, toggle terminal field
7 click on “initial state”
8 move mouse to location and place new initial state
9 click on “transition”
10 move mouse to location of initial state
11 press left mouse button and keep pressed until reaching state
12 click on “transition”
13 move mouse to location of state
14 press left mouse button and keep pressed until reaching terminal
state
15 double click on transition
16 write “C” in trigger field
17 press “OK”
18 click on “delimiter line”
19 move mouse to location and place delimiter line


The underlying argument is that having a true DSL infrastructure, including access to the Abstract Syntax Tree (AST) of the doc being edited, gives you a lot of mileage, e.g. enabling the suggested use (auto-layout).

There's more related work, of course (the idea cannot be new), but I don't see it being mentioned often enough in DSL tooling circles. Diomidis Spinellis has a nice prototype to layout UML class diagrams from Java source, to achieve things like:

given the following input:

// Author:  Vadim Nasardinov
// Version: $Id: ceg-mv.xml,v 1.1 2005/11/23 22:21:22 dds Exp $

import java.util.List;
import java.util.Map;

/**

 * @assoc "1..1" - "0..n" Adapter
 * @assoc "" - "0..n" ObjectType

 * @assoc "" - "0..n" ObjectMap
 * @assoc "" - "0..n" Table
 * @assoc "" - "0..n" DataOperation

 **/
class Root {
  private Map m_adapters;
  private List m_types;
  private List m_maps;
  private List m_tables;
  private List m_ops;

  public Adapter getAdapter(Class klass) {}
}


class Adapter {
  public Root getRoot();
}

abstract class Element {
  Root getRoot() {}
}

class ObjectType extends Element {}


/**
 * @has "1..1" - "1..1" ObjectType
 **/
class ObjectMap extends Element {
  private ObjectType m_type;
}


class Table extends Element {}

class DataOperation extends Element {}

Notice that it's possible to generate different diagram views from the same source description.

Actually, all this detour into diagrams actually aims at reinforcing the view that language engineering (by means of metamodeling, with well-formedness and type-checking conditions specified in OCL) is a good thing, that also comes to benefit in case you're all for diagrams. But I won't peruse any further this visual path, for that you can check two generators of diagram editors:

1. the Eclipse Graphical Modeling Framework
   
2. the Tiger Project at Tech Univ Berlin (Germany)
   

Have fun (and please come back to this textual DSLs blog soon :)

References:

[1] The Kiel Statechart Layouter

existing prototypes

I'm quoting from a presentation I gave on this topic:

• xText by the OpenArchitectureWare team
  
• TCS, by Frédéric Jouault (INRIA)
  
• TEF, by Markus Scheidgen (Uni Humboldt)
  
• Gymnast, by Chris J Daly (IBM)
  
• SAFARI (just about to become an Eclipse project)
  

Actually each of them has its unique approach towads jumpstarting a toolset for a custom DSL. Comparison would be a lengthy, lengthy, affair. I can sum it all up by saying that, provided your requirements fit the design criteria of your tool of choice, they'll save you a lot of work.

A hard test for any IDE generator for a custom DSL is whether the input specification encodes more constraints than just grammar (I'm talking about well-formedness rules such as declarations-before-usages and type-checking rules. Same observations go for other tools not on this list). More details on this criteria can be found in the aforementioned presentation.